From 33354132e2d7d75f081f5707750b21b4a8d2120e Mon Sep 17 00:00:00 2001 From: Karoly Gerner <30629375+gerner-spryker@users.noreply.github.com> Date: Fri, 25 Oct 2024 19:44:47 +0200 Subject: [PATCH 01/26] Create chunk-size-calculation.md --- .../data-manipulation/queue/chunk-size-calculation.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.md diff --git a/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.md b/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.md new file mode 100644 index 00000000000..8b137891791 --- /dev/null +++ b/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.md @@ -0,0 +1 @@ + From 456bbd0dc73f8e9a36c07eaedfb89b3d494d03fe Mon Sep 17 00:00:00 2001 From: Karoly Gerner <30629375+gerner-spryker@users.noreply.github.com> Date: Fri, 25 Oct 2024 19:51:04 +0200 Subject: [PATCH 02/26] Update chunk-size-calculation.md --- .../queue/chunk-size-calculation.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.md b/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.md index 8b137891791..290f228254a 100644 --- a/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.md +++ b/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.md @@ -1 +1,20 @@ +--- +title: Chunk Size Calculation +description: Describes the challenges and solutions of selecting proper chunk sizes for project requirements +last_updated: Oct 25, 2024 +template: concept-topic-template +redirect_from: + - /docs/dg/dev/backend-development/data-manipulation/queue/queue.html#concepts +related: + - title: Basic Chunk Size Calculation + link: docs/dg/dev/backend-development/data-manipulation/queue/basic-chunk-size-calculation.html + - title: Advanced Chunk Size Calculation + link: docs/dg/dev/backend-development/data-manipulation/queue/advanced-chunk-size-calculation.html + - title: Expert Chunk Size Calculation + link: docs/dg/dev/backend-development/data-manipulation/queue/expert-chunk-size-calculation.html + +--- + +## Chunk Size Calculation + From d2881150872c11c90225f39f8b3fa48e3dec1892 Mon Sep 17 00:00:00 2001 From: Karoly Gerner <30629375+gerner-spryker@users.noreply.github.com> Date: Fri, 25 Oct 2024 20:02:10 +0200 Subject: [PATCH 03/26] Update chunk-size-calculation.md --- .../queue/chunk-size-calculation.md | 60 +++++++++++++++++++ 1 file changed, 60 insertions(+) diff --git a/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.md b/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.md index 290f228254a..a1480d86661 100644 --- a/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.md +++ b/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.md @@ -17,4 +17,64 @@ related: ## Chunk Size Calculation +### Problem Statement +In an e-commerce framework, selecting the correct chunk size for processing data is critical for ensuring optimal performance and efficient resource usage. The challenge arises from the diversity of business entities, stores, and locales, each with different memory and CPU requirements for denormalization. Furthermore, the frequency of updates, data sizes, and the specific configurations of various system services (e.g., Redis, Elasticsearch, RabbitMQ) make it difficult to determine the appropriate chunk size for each queue. + +Without proper chunk size configuration, systems can either overconsume resources, leading to crashes or lag, or underutilize resources, resulting in slow performance. This is where the **Chunk Size Calculator** comes in—offering a solution that helps developers fine-tune their chunk sizes based on the specific characteristics of their system. + +### What is the Chunk Size Calculator? + +The **Chunk Size Calculator** is a tool that helps developers determine the appropriate chunk sizes for processing data across different queues in a resource-efficient way. It ensures that memory and CPU usage are optimized and prevents the system from being overwhelmed during data processing tasks. This tool is designed to handle the variability in entity sizes, stores, locales, and update frequencies, giving developers confidence that their system will run smoothly in production environments. + +### The Three Levels of the Chunk Size Calculator + +The **Chunk Size Calculator** is divided into three levels: **Basic**, **Advanced**, and **Expert**. Each level is designed to accommodate different degrees of system complexity and customization. Below is an overview of each: + +#### 1. Basic Chunk Size Calculator + +The **Basic Chunk Size Calculator** is designed for small to medium B2C webshops or systems with minimal customization. It assumes that the default configuration of business entities, stores, and locales is sufficient, and that the resource consumption patterns are predictable. + +With the Basic calculator, developers only need to provide a minimal set of inputs, such as high level store configuration and high traffic entity counts. The calculator uses these inputs to recommend chunk sizes for each queue. This is ideal for developers who are working with out-of-the-box setups and need a simple, reliable way to configure their system. + +**When to use**: This is the default starting point for any project. If your system has not been heavily customized, this calculator will give you the necessary chunk sizes with minimal effort. + +Find more details on the [Basic Chunk Size Calculation](#docs/dg/dev/backend-development/data-manipulation/queue/basic-chunk-size-calculation.html) page. + +#### 2. Advanced Chunk Size Calculator + +The **Advanced Chunk Size Calculator** builds upon the basic level, requiring developers to have a deeper understanding of the services that make up the system. In addition to understanding the basic chunk size concepts, developers will need to account for service elements like: + +- **Persistence** (Relational Database Service) +- **Storage** (Redis) +- **Search** (Elasticsearch) +- **Message Broker** (RabbitMQ) +- **Scheduler** + +The Advanced calculator allows developers to select from AWS container types and configure memory sizes for each service. It also exposes additional performance metrics, such as the average number of triggered events per second and message size limits in RabbitMQ. While the calculator provides recommendations, developers will need to input detailed configuration data to fine-tune the system. + +**When to use**: The Advanced calculator is suited for systems that have been moderately customized or when developers need more precise control over performance and resource usage. + +Find more details on the [Advanced Chunk Size Calculation](#docs/dg/dev/backend-development/data-manipulation/queue/advanced-chunk-size-calculation.html) page. + +#### 3. Expert Chunk Size Calculator + +The **Expert Chunk Size Calculator** is designed for highly customized systems where one or more entities deviate significantly from the norm. This includes scenarios where entities are unusually large or where the system handles massive data volumes that require frequent updates. + +In this case, developers need a deep understanding of how the system’s components interact, including the scheduler’s worker, and the memory distribution across workers, and tasks. The Expert calculator gives full visibility into all performance metrics, allowing developers to tweak each queue individually. + +Despite the complexity, the calculator still ensures that the entire system remains balanced, preventing one queue from consuming too many resources at the expense of others. + +**When to use**: The Expert calculator is reserved for production environments with complex, heavily customized setups that require fine-tuning of every performance metric. + +Find more details on the [Expert Chunk Size Calculation](#docs/dg/dev/backend-development/data-manipulation/queue/expert-chunk-size-calculation.html) page. + +### Summary + +The **Chunk Size Calculator** provides developers with a powerful tool for optimizing their system’s resource consumption. Each level of the calculator is designed to address different use cases, from simple out-of-the-box configurations to highly customized, complex environments. Developers are encouraged to start with the **Basic Chunk Size Calculator** and, if necessary, progress to the **Advanced** or **Expert** calculators as the complexity of their system grows. + +#### Access the Calculators + +- [Basic Chunk Size Calculator](#) +- [Advanced Chunk Size Calculator](#) +- [Expert Chunk Size Calculator](#) From d514e15c0bb5dd7c98e475539b5e6d1e081b690f Mon Sep 17 00:00:00 2001 From: Karoly Gerner <30629375+gerner-spryker@users.noreply.github.com> Date: Fri, 25 Oct 2024 20:02:39 +0200 Subject: [PATCH 04/26] Update chunk-size-calculation.md --- .../data-manipulation/queue/chunk-size-calculation.md | 6 ------ 1 file changed, 6 deletions(-) diff --git a/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.md b/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.md index a1480d86661..70eb73d3e3b 100644 --- a/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.md +++ b/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.md @@ -72,9 +72,3 @@ Find more details on the [Expert Chunk Size Calculation](#docs/dg/dev/backend-de ### Summary The **Chunk Size Calculator** provides developers with a powerful tool for optimizing their system’s resource consumption. Each level of the calculator is designed to address different use cases, from simple out-of-the-box configurations to highly customized, complex environments. Developers are encouraged to start with the **Basic Chunk Size Calculator** and, if necessary, progress to the **Advanced** or **Expert** calculators as the complexity of their system grows. - -#### Access the Calculators - -- [Basic Chunk Size Calculator](#) -- [Advanced Chunk Size Calculator](#) -- [Expert Chunk Size Calculator](#) From 626e70d2b623192f983b0ac208896d93c95881a5 Mon Sep 17 00:00:00 2001 From: Karoly Gerner <30629375+gerner-spryker@users.noreply.github.com> Date: Fri, 25 Oct 2024 20:06:06 +0200 Subject: [PATCH 05/26] Update queue.md --- .../dg/dev/backend-development/data-manipulation/queue/queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/dg/dev/backend-development/data-manipulation/queue/queue.md b/docs/dg/dev/backend-development/data-manipulation/queue/queue.md index e02fd139688..a26dbd4435d 100644 --- a/docs/dg/dev/backend-development/data-manipulation/queue/queue.md +++ b/docs/dg/dev/backend-development/data-manipulation/queue/queue.md @@ -149,7 +149,7 @@ $config[QueueConstants::QUEUE_MESSAGE_CHUNK_SIZE_MAP] = [ ]; ?> ``` - +For more detailed information on the Chunk Size Calculator and how it optimzes system performance, [see further details here](#docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.html). 2. Register the plugin in `QueueDependencyProvider::getProcessorMessagePlugins()`: From fdb0daef90c19d6fc8662bec2294a54bb07b1bdf Mon Sep 17 00:00:00 2001 From: Karoly Gerner <30629375+gerner-spryker@users.noreply.github.com> Date: Fri, 25 Oct 2024 20:07:08 +0200 Subject: [PATCH 06/26] Update queue.md --- .../dg/dev/backend-development/data-manipulation/queue/queue.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/dg/dev/backend-development/data-manipulation/queue/queue.md b/docs/dg/dev/backend-development/data-manipulation/queue/queue.md index a26dbd4435d..5670da889b9 100644 --- a/docs/dg/dev/backend-development/data-manipulation/queue/queue.md +++ b/docs/dg/dev/backend-development/data-manipulation/queue/queue.md @@ -12,6 +12,8 @@ related: link: docs/scos/dev/back-end-development/data-manipulation/queue/queue-pool.html - title: Upgrade the RabbitMQ module link: docs/scos/dev/module-migration-guides/migration-guide-rabbitmq.html + - title: Chunk Size Calculation + link: docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.html --- ## Concepts From 27497a7ffde51de334197d8afd7fcd0c391fc18e Mon Sep 17 00:00:00 2001 From: Karoly Gerner <30629375+gerner-spryker@users.noreply.github.com> Date: Fri, 25 Oct 2024 20:54:21 +0200 Subject: [PATCH 07/26] Create basic-chunk-size-calculation.md --- .../queue/basic-chunk-size-calculation.md | 74 +++++++++++++++++++ 1 file changed, 74 insertions(+) create mode 100644 docs/dg/dev/backend-development/data-manipulation/queue/basic-chunk-size-calculation.md diff --git a/docs/dg/dev/backend-development/data-manipulation/queue/basic-chunk-size-calculation.md b/docs/dg/dev/backend-development/data-manipulation/queue/basic-chunk-size-calculation.md new file mode 100644 index 00000000000..396728904ed --- /dev/null +++ b/docs/dg/dev/backend-development/data-manipulation/queue/basic-chunk-size-calculation.md @@ -0,0 +1,74 @@ +--- +title: Basic Chunk Size Calculation +description: Gives an overview over the basic chunk size calculation +last_updated: Oct 25, 2024 +template: concept-topic-template +redirect_from: + - /docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.html +related: + - title: Advanced Chunk Size Calculation + link: docs/dg/dev/backend-development/data-manipulation/queue/advanced-chunk-size-calculation.html + - title: Expert Chunk Size Calculation + link: docs/dg/dev/backend-development/data-manipulation/queue/expert-chunk-size-calculation.html + - title: Queue + link: docs/dg/dev/backend-development/data-manipulation/queue/queue.html + +--- + +## Basic Chunk Size Calculation + +The **Basic Chunk Size Calculator** is designed to help developers configure the correct chunk sizes for their Spryker Commerce Operating System based on the traffic and data patterns in their system. This tool simplifies the setup process for out-of-the-box webshops that use the **publish and synchronize**, ensuring that the system can handle high-traffic entities efficiently without over-consuming resources. + +### Problem Overview + +In an e-commerce environment, certain business entities generate a large volume of update events due to frequent refreshes and high data volume. These **high traffic entities** account for the majority of the traffic within the **publish and synchronize**. Misconfiguring chunk sizes for these entities can lead to inefficient resource consumption, system lags, or overloads. The **Basic Chunk Size Calculator** offers a straightforward way to address this by determining the appropriate chunk size for each queue based on the production environment’s data profile. + +### Input Parameters + +To calculate the correct chunk sizes, developers must provide the following information based on their specific production environment: + +#### 1. High Traffic Entities + +The **high traffic entities** in an out-of-the-box e-commerce environment are those that generate significant traffic due to their volume and update frequency. These include entities such as products, prices, merchant offers, among others. + +Developers need to provide: +- **Total number of each high traffic entity across all stores** in the production environment. +- **Estimated daily refresh rate**: The approximate percentage or number of entities that are expected to be updated on a daily basis for the remainder of the system’s lifetime. + +#### 2. Stores and Locales + +Each production environment can have multiple **stores** (e.g., a French store, a German store), each supporting multiple **locales**. Since chunk size calculation depends on how data is distributed across stores and locales, developers need to provide: +- **Number of stores**: The total number of storefronts in the system. +- **Maximum number of locales**: The highest number of locales supported across all stores. + +> For more information on stores and locales in our system, [click here](https://docs.spryker.com/docs/pbc/all/dynamic-multistore/202410.0/base-shop/dynamic-multistore-feature-overview.html). + +#### 3. Publish and Synchronize Setup + +The **publish and synchronize** middleware processes entity data updates, and the worker setup plays a crucial role in determining how this is managed. Developers need to specify how their workers are set up in relation to stores. + +> For more information on workers, tasks, and how they are related to stores, [click here](https://docs.spryker.com/docs/pbc/all/dynamic-multistore/202410.0/base-shop/dynamic-multistore-feature-overview.html). + +#### 4. Number of Tasks Per Worker + +Developers are required to provide the **number of tasks per worker**. This value is essential to calculating how resources are distributed among tasks. Note that there is no additional help or explanation for determining this number, as it is specific to each setup. + +> For more information on workers, tasks, and how they are related to stores, [click here](https://docs.spryker.com/docs/pbc/all/dynamic-multistore/202410.0/base-shop/dynamic-multistore-feature-overview.html). + +### Output + +Once the required data is entered into the **Basic Chunk Size Calculator**, it will compute the optimal chunk sizes for each queue used by the system. These queues handle different business entities, and setting the right chunk size ensures efficient processing and resource allocation. + +The result will be a **chunk size for each queue** that is provided out-of-the-box with the Spryker Commerce Operating System. Developers will need to configure these chunk sizes in their system to align with the calculated values. + +> For instructions on how to set up chunk sizes for the queues, [click here](https://docs.spryker.com/docs/dg/dev/backend-development/data-manipulation/queue/queue.html#configuration-for-chunk-size). + +### Important Notes + +- The **Basic Chunk Size Calculator** is designed for systems that follow a standard, out-of-the-box configuration. If your system is more customized, consider using the **Advanced** or **Expert Chunk Size Calculator** for fine-tuning. +- This calculator only requires a basic understanding of the system's entity data and store structure. For more complex metrics like memory usage or container performance, the advanced calculators may be necessary. +- Always ensure that the chunk sizes provided by the calculator are properly configured to avoid system performance issues. + +--- + +For more detailed information about the different levels of the **Chunk Size Calculator**, see the [overview here](https://docs.spryker.com/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.html). From e550f462776e4ac6e3e18c5ac2ee98fd777a314c Mon Sep 17 00:00:00 2001 From: Karoly Gerner <30629375+gerner-spryker@users.noreply.github.com> Date: Fri, 25 Oct 2024 20:55:09 +0200 Subject: [PATCH 08/26] Update queue.md --- .../dg/dev/backend-development/data-manipulation/queue/queue.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/dg/dev/backend-development/data-manipulation/queue/queue.md b/docs/dg/dev/backend-development/data-manipulation/queue/queue.md index 5670da889b9..7e87d262498 100644 --- a/docs/dg/dev/backend-development/data-manipulation/queue/queue.md +++ b/docs/dg/dev/backend-development/data-manipulation/queue/queue.md @@ -151,7 +151,7 @@ $config[QueueConstants::QUEUE_MESSAGE_CHUNK_SIZE_MAP] = [ ]; ?> ``` -For more detailed information on the Chunk Size Calculator and how it optimzes system performance, [see further details here](#docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.html). +For more detailed information on the Chunk Size Calculator and how it optimzes system performance, [see further details here](https://docs.spryker.com/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.html). 2. Register the plugin in `QueueDependencyProvider::getProcessorMessagePlugins()`: From b33dbc99b2eacd5aa58e17446ab1bd47053c67d0 Mon Sep 17 00:00:00 2001 From: Karoly Gerner <30629375+gerner-spryker@users.noreply.github.com> Date: Fri, 25 Oct 2024 20:56:03 +0200 Subject: [PATCH 09/26] Update chunk-size-calculation.md --- .../data-manipulation/queue/chunk-size-calculation.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.md b/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.md index 70eb73d3e3b..5e214a2b0c2 100644 --- a/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.md +++ b/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.md @@ -39,7 +39,7 @@ With the Basic calculator, developers only need to provide a minimal set of inpu **When to use**: This is the default starting point for any project. If your system has not been heavily customized, this calculator will give you the necessary chunk sizes with minimal effort. -Find more details on the [Basic Chunk Size Calculation](#docs/dg/dev/backend-development/data-manipulation/queue/basic-chunk-size-calculation.html) page. +Find more details on the [Basic Chunk Size Calculation](https://docs.spryker.com/docs/dg/dev/backend-development/data-manipulation/queue/basic-chunk-size-calculation.html) page. #### 2. Advanced Chunk Size Calculator @@ -55,7 +55,7 @@ The Advanced calculator allows developers to select from AWS container types and **When to use**: The Advanced calculator is suited for systems that have been moderately customized or when developers need more precise control over performance and resource usage. -Find more details on the [Advanced Chunk Size Calculation](#docs/dg/dev/backend-development/data-manipulation/queue/advanced-chunk-size-calculation.html) page. +Find more details on the [Advanced Chunk Size Calculation](https://docs.spryker.com/docs/dg/dev/backend-development/data-manipulation/queue/advanced-chunk-size-calculation.html) page. #### 3. Expert Chunk Size Calculator @@ -67,7 +67,7 @@ Despite the complexity, the calculator still ensures that the entire system rema **When to use**: The Expert calculator is reserved for production environments with complex, heavily customized setups that require fine-tuning of every performance metric. -Find more details on the [Expert Chunk Size Calculation](#docs/dg/dev/backend-development/data-manipulation/queue/expert-chunk-size-calculation.html) page. +Find more details on the [Expert Chunk Size Calculation](https://docs.spryker.com/docs/dg/dev/backend-development/data-manipulation/queue/expert-chunk-size-calculation.html) page. ### Summary From 6eaa48d9f29efcc31b0bf620a93b39c342c263f6 Mon Sep 17 00:00:00 2001 From: Karoly Gerner <30629375+gerner-spryker@users.noreply.github.com> Date: Fri, 25 Oct 2024 20:57:08 +0200 Subject: [PATCH 10/26] Update basic-chunk-size-calculation.md --- .../data-manipulation/queue/basic-chunk-size-calculation.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/dg/dev/backend-development/data-manipulation/queue/basic-chunk-size-calculation.md b/docs/dg/dev/backend-development/data-manipulation/queue/basic-chunk-size-calculation.md index 396728904ed..71d20333059 100644 --- a/docs/dg/dev/backend-development/data-manipulation/queue/basic-chunk-size-calculation.md +++ b/docs/dg/dev/backend-development/data-manipulation/queue/basic-chunk-size-calculation.md @@ -19,6 +19,9 @@ related: The **Basic Chunk Size Calculator** is designed to help developers configure the correct chunk sizes for their Spryker Commerce Operating System based on the traffic and data patterns in their system. This tool simplifies the setup process for out-of-the-box webshops that use the **publish and synchronize**, ensuring that the system can handle high-traffic entities efficiently without over-consuming resources. +### The Basic Chunk Size Calculator +The **Basic Chunk Size Calculator** is available [here](link to google spreadsheet). + ### Problem Overview In an e-commerce environment, certain business entities generate a large volume of update events due to frequent refreshes and high data volume. These **high traffic entities** account for the majority of the traffic within the **publish and synchronize**. Misconfiguring chunk sizes for these entities can lead to inefficient resource consumption, system lags, or overloads. The **Basic Chunk Size Calculator** offers a straightforward way to address this by determining the appropriate chunk size for each queue based on the production environment’s data profile. From e70e8178d4e519d7fd208599b279e73e461dfe6e Mon Sep 17 00:00:00 2001 From: Karoly Gerner <30629375+gerner-spryker@users.noreply.github.com> Date: Mon, 28 Oct 2024 14:37:49 +0100 Subject: [PATCH 11/26] Create advanced-chunk-size-calculation.md --- .../queue/advanced-chunk-size-calculation.md | 37 +++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100644 docs/dg/dev/backend-development/data-manipulation/queue/advanced-chunk-size-calculation.md diff --git a/docs/dg/dev/backend-development/data-manipulation/queue/advanced-chunk-size-calculation.md b/docs/dg/dev/backend-development/data-manipulation/queue/advanced-chunk-size-calculation.md new file mode 100644 index 00000000000..98183211fb5 --- /dev/null +++ b/docs/dg/dev/backend-development/data-manipulation/queue/advanced-chunk-size-calculation.md @@ -0,0 +1,37 @@ +--- +title: Advanced Chunk Size Calculation +description: Gives an overview over the advanced chunk size calculation +last_updated: Oct 25, 2024 +template: concept-topic-template +redirect_from: + - /docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.html +related: + - title: Basic Chunk Size Calculation + link: docs/dg/dev/backend-development/data-manipulation/queue/basic-chunk-size-calculation.html + - title: Expert Chunk Size Calculation + link: docs/dg/dev/backend-development/data-manipulation/queue/expert-chunk-size-calculation.html + - title: Queue + link: docs/dg/dev/backend-development/data-manipulation/queue/queue.html + +--- + +## Advanced Chunk Size Calculation + +The **Advanced Chunk Size Calculator** builds upon the Basic level by allowing to fine-tune chunk sizes based on custom hardware limitations and custom performance metrics. + +### Additional Fine Tuning Configuration + +- This tool is ideal for scenarios where your system's resource consumption and performance vary from the standard configurations provided by the e-commerce framework. For example, if your **scheduler setup** differs from the default—such as running multiple containers or distributing workers differently within containers. +- Additionally, the **Advanced Calculator** is essential when you are working with a customized hardware environment. If you're using different instance types or have specific memory and CPU limits for your services (Persistence, Storage, Search, or Message Broker) the Advanced Calculator enables you to fine-tune the chunk sizes based on the actual hardware configuration. +- Beyond hardware, this calculator is also useful if your system processes events at a rate that deviates from the default settings. For example, if you need to adjust the rate at which messages are triggered into queues, or if your application has a custom warm-up time, you can modify these parameters to better reflect your system’s real-world performance. +- This tool is also ideal for fine-tuning memory and CPU allocation for tasks, especially in environments where specific performance thresholds are important. For instance, you can control how much of the scheduler’s resources are dedicated to processing tasks versus how much is reserved for the scheduler’s core operations. If your tasks are more resource-intensive than the default configuration assumes, this calculator allows you to ensure that the chunk size is set to avoid overwhelming system resources while still maximizing throughput. + +### Important Notes + +- The **Advanced Chunk Size Calculator** allows to further configure chunk sizes based on custom hardware limitations and custom performance metrics. +- For systems that require individual configuration of queues and detailed customisation of message setups, consider using the **Expert Chunk Size Calculator**. +- Always ensure that the chunk sizes provided by the calculator are properly configured to avoid system performance issues. + +--- + +For more detailed information about the different levels of the **Chunk Size Calculator**, see the [overview here](https://docs.spryker.com/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.html). From 5eb3f5dde338327dbddba7a026f3676a01a025b1 Mon Sep 17 00:00:00 2001 From: Karoly Gerner <30629375+gerner-spryker@users.noreply.github.com> Date: Mon, 28 Oct 2024 17:28:12 +0100 Subject: [PATCH 12/26] Update basic-chunk-size-calculation.md --- .../queue/basic-chunk-size-calculation.md | 38 +++++-------------- 1 file changed, 9 insertions(+), 29 deletions(-) diff --git a/docs/dg/dev/backend-development/data-manipulation/queue/basic-chunk-size-calculation.md b/docs/dg/dev/backend-development/data-manipulation/queue/basic-chunk-size-calculation.md index 71d20333059..82dbc4df084 100644 --- a/docs/dg/dev/backend-development/data-manipulation/queue/basic-chunk-size-calculation.md +++ b/docs/dg/dev/backend-development/data-manipulation/queue/basic-chunk-size-calculation.md @@ -17,7 +17,7 @@ related: ## Basic Chunk Size Calculation -The **Basic Chunk Size Calculator** is designed to help developers configure the correct chunk sizes for their Spryker Commerce Operating System based on the traffic and data patterns in their system. This tool simplifies the setup process for out-of-the-box webshops that use the **publish and synchronize**, ensuring that the system can handle high-traffic entities efficiently without over-consuming resources. +The **Basic Chunk Size Calculator** is designed to help developers configure the correct chunk sizes for their Spryker Commerce Operating System based on the traffic and data patterns in their system. This tool simplifies the setup process for out-of-the-box and low-customised webshops, ensuring that the system can handle high-traffic entities efficiently without over-consuming resources. ### The Basic Chunk Size Calculator The **Basic Chunk Size Calculator** is available [here](link to google spreadsheet). @@ -28,35 +28,15 @@ In an e-commerce environment, certain business entities generate a large volume ### Input Parameters -To calculate the correct chunk sizes, developers must provide the following information based on their specific production environment: +To calculate the correct queue chunk sizes, developers must provide the following information based on their specific production environment: -#### 1. High Traffic Entities - -The **high traffic entities** in an out-of-the-box e-commerce environment are those that generate significant traffic due to their volume and update frequency. These include entities such as products, prices, merchant offers, among others. - -Developers need to provide: -- **Total number of each high traffic entity across all stores** in the production environment. -- **Estimated daily refresh rate**: The approximate percentage or number of entities that are expected to be updated on a daily basis for the remainder of the system’s lifetime. - -#### 2. Stores and Locales - -Each production environment can have multiple **stores** (e.g., a French store, a German store), each supporting multiple **locales**. Since chunk size calculation depends on how data is distributed across stores and locales, developers need to provide: -- **Number of stores**: The total number of storefronts in the system. -- **Maximum number of locales**: The highest number of locales supported across all stores. - -> For more information on stores and locales in our system, [click here](https://docs.spryker.com/docs/pbc/all/dynamic-multistore/202410.0/base-shop/dynamic-multistore-feature-overview.html). - -#### 3. Publish and Synchronize Setup - -The **publish and synchronize** middleware processes entity data updates, and the worker setup plays a crucial role in determining how this is managed. Developers need to specify how their workers are set up in relation to stores. - -> For more information on workers, tasks, and how they are related to stores, [click here](https://docs.spryker.com/docs/pbc/all/dynamic-multistore/202410.0/base-shop/dynamic-multistore-feature-overview.html). - -#### 4. Number of Tasks Per Worker - -Developers are required to provide the **number of tasks per worker**. This value is essential to calculating how resources are distributed among tasks. Note that there is no additional help or explanation for determining this number, as it is specific to each setup. - -> For more information on workers, tasks, and how they are related to stores, [click here](https://docs.spryker.com/docs/pbc/all/dynamic-multistore/202410.0/base-shop/dynamic-multistore-feature-overview.html). +- **High Traffic Entities**: Provide the total count of each high-traffic entity (e.g., products, prices, offers) across all stores, and estimate the daily refresh rate (percentage or count of entities updated daily) for ongoing system operations. +- **Stores and Locales**: Provide the total number of stores in the system and the maximum number of supported locales across all stores, as these factors impact chunk size calculation for data distribution. + > For more information on stores and locales in our system, [click here](https://docs.spryker.com/docs/pbc/all/dynamic-multistore/202410.0/base-shop/dynamic-multistore-feature-overview.html). +- **Publish and Synchronize Setup**: The **publish and synchronize** processes entity data updates, and the worker setup plays a crucial role in determining how this is managed. Developers need to specify how their workers are set up in relation to stores. + > For more information on workers, tasks, and how they are related to stores, [click here](https://docs.spryker.com/docs/pbc/all/dynamic-multistore/202410.0/base-shop/dynamic-multistore-feature-overview.html). +- **Number of Tasks Per Worker**: Developers are required to provide the **number of tasks per worker**. This value is essential to calculating how resources are distributed among tasks. Note that there is no additional help or explanation for determining this number, as it is specific to each setup. + > For more information on workers, tasks, and how they are related to stores, [click here](https://docs.spryker.com/docs/pbc/all/dynamic-multistore/202410.0/base-shop/dynamic-multistore-feature-overview.html). ### Output From 70263e71e940979db779bc389408abc5359dbab7 Mon Sep 17 00:00:00 2001 From: Karoly Gerner <30629375+gerner-spryker@users.noreply.github.com> Date: Mon, 28 Oct 2024 17:37:30 +0100 Subject: [PATCH 13/26] Update basic-chunk-size-calculation.md --- .../data-manipulation/queue/basic-chunk-size-calculation.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/dg/dev/backend-development/data-manipulation/queue/basic-chunk-size-calculation.md b/docs/dg/dev/backend-development/data-manipulation/queue/basic-chunk-size-calculation.md index 82dbc4df084..a7d68c8dd1f 100644 --- a/docs/dg/dev/backend-development/data-manipulation/queue/basic-chunk-size-calculation.md +++ b/docs/dg/dev/backend-development/data-manipulation/queue/basic-chunk-size-calculation.md @@ -40,9 +40,7 @@ To calculate the correct queue chunk sizes, developers must provide the followin ### Output -Once the required data is entered into the **Basic Chunk Size Calculator**, it will compute the optimal chunk sizes for each queue used by the system. These queues handle different business entities, and setting the right chunk size ensures efficient processing and resource allocation. - -The result will be a **chunk size for each queue** that is provided out-of-the-box with the Spryker Commerce Operating System. Developers will need to configure these chunk sizes in their system to align with the calculated values. +Once the required data is entered into the **Basic Chunk Size Calculator**, it will compute the optimal chunk sizes for each queue used by the system. These queues handle different business entities, and setting the right queue chunk size ensures efficient processing and resource allocation. Developers will need to configure these queue chunk sizes. > For instructions on how to set up chunk sizes for the queues, [click here](https://docs.spryker.com/docs/dg/dev/backend-development/data-manipulation/queue/queue.html#configuration-for-chunk-size). From 382e7becd04db780daa3d875a814ba4b9f8ecb8c Mon Sep 17 00:00:00 2001 From: Karoly Gerner <30629375+gerner-spryker@users.noreply.github.com> Date: Mon, 28 Oct 2024 17:49:07 +0100 Subject: [PATCH 14/26] Update advanced-chunk-size-calculation.md --- .../queue/advanced-chunk-size-calculation.md | 25 +++++++++++++++---- 1 file changed, 20 insertions(+), 5 deletions(-) diff --git a/docs/dg/dev/backend-development/data-manipulation/queue/advanced-chunk-size-calculation.md b/docs/dg/dev/backend-development/data-manipulation/queue/advanced-chunk-size-calculation.md index 98183211fb5..56b90a4892d 100644 --- a/docs/dg/dev/backend-development/data-manipulation/queue/advanced-chunk-size-calculation.md +++ b/docs/dg/dev/backend-development/data-manipulation/queue/advanced-chunk-size-calculation.md @@ -19,12 +19,27 @@ related: The **Advanced Chunk Size Calculator** builds upon the Basic level by allowing to fine-tune chunk sizes based on custom hardware limitations and custom performance metrics. -### Additional Fine Tuning Configuration +### The Advanced Chunk Size Calculator +The **Advanced Chunk Size Calculator** is available [here](link to google spreadsheet). + +### Problem Overview + +While the **Basic Chunk Size Calculator** offers a starting point, it doesn’t account for the nuances of resource allocations (like container sizes, CPU, and memory limits) or performance-sensitive variables such as application warm-up time and event message size. The **Advanced Chunk Size Calculator** addresses these issues by incorporating these additional metrics, enabling a more precise configuration that enhances stability and performance in production environments. + +### Input Parameters + +To calculate the correct queue chunk sizes, developers must provide the following information based on their specific production environment: +- **Scheduler and Worker Setup**: Provide details on any non-standard configurations, such as environments with multiple containers or distinct worker distributions within containers, if your scheduler setup differs from the boilerplate defaults. +- **Resource Configuration**: Provide information on your hardware setup, including instance types, CPU, and memory limits for services like Persistence, Storage, Search, or Message Broker, to allow the **Advanced Calculator** to optimize chunk sizes based on actual resource availability. +- **Detailed Product Configuration**: Provide specific metrics related to products, the highest-traffic entity. This supports more precise chunk sizing for products without requiring an in-depth understanding of Publish & Synchronize. +- **Event and Message Processing Metrics**: Provide expected event processing metrics, including deviations from default settings such as message trigger rates, custom application warm-up times, event size limits, and data division rate multipliers, to enable configuration adjustments that align with real-world performance. + +### Output + +Once the required data is entered into the **Basic Chunk Size Calculator** and **Advanced Chunk Size Calculator**, it will compute the optimal queue chunk sizes for each queue. Developers will need to configure these queue chunk sizes in their project to align with the calculated values. + +> For instructions on how to set up chunk sizes for the queues, [click here](https://docs.spryker.com/docs/dg/dev/backend-development/data-manipulation/queue/queue.html#configuration-for-chunk-size). -- This tool is ideal for scenarios where your system's resource consumption and performance vary from the standard configurations provided by the e-commerce framework. For example, if your **scheduler setup** differs from the default—such as running multiple containers or distributing workers differently within containers. -- Additionally, the **Advanced Calculator** is essential when you are working with a customized hardware environment. If you're using different instance types or have specific memory and CPU limits for your services (Persistence, Storage, Search, or Message Broker) the Advanced Calculator enables you to fine-tune the chunk sizes based on the actual hardware configuration. -- Beyond hardware, this calculator is also useful if your system processes events at a rate that deviates from the default settings. For example, if you need to adjust the rate at which messages are triggered into queues, or if your application has a custom warm-up time, you can modify these parameters to better reflect your system’s real-world performance. -- This tool is also ideal for fine-tuning memory and CPU allocation for tasks, especially in environments where specific performance thresholds are important. For instance, you can control how much of the scheduler’s resources are dedicated to processing tasks versus how much is reserved for the scheduler’s core operations. If your tasks are more resource-intensive than the default configuration assumes, this calculator allows you to ensure that the chunk size is set to avoid overwhelming system resources while still maximizing throughput. ### Important Notes From 3873b5c1e7d693345ade089a271be87843796642 Mon Sep 17 00:00:00 2001 From: Karoly Gerner <30629375+gerner-spryker@users.noreply.github.com> Date: Mon, 28 Oct 2024 17:58:38 +0100 Subject: [PATCH 15/26] Update basic-chunk-size-calculation.md --- .../data-manipulation/queue/basic-chunk-size-calculation.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/docs/dg/dev/backend-development/data-manipulation/queue/basic-chunk-size-calculation.md b/docs/dg/dev/backend-development/data-manipulation/queue/basic-chunk-size-calculation.md index a7d68c8dd1f..0bd909c62d1 100644 --- a/docs/dg/dev/backend-development/data-manipulation/queue/basic-chunk-size-calculation.md +++ b/docs/dg/dev/backend-development/data-manipulation/queue/basic-chunk-size-calculation.md @@ -19,7 +19,6 @@ related: The **Basic Chunk Size Calculator** is designed to help developers configure the correct chunk sizes for their Spryker Commerce Operating System based on the traffic and data patterns in their system. This tool simplifies the setup process for out-of-the-box and low-customised webshops, ensuring that the system can handle high-traffic entities efficiently without over-consuming resources. -### The Basic Chunk Size Calculator The **Basic Chunk Size Calculator** is available [here](link to google spreadsheet). ### Problem Overview @@ -33,9 +32,9 @@ To calculate the correct queue chunk sizes, developers must provide the followin - **High Traffic Entities**: Provide the total count of each high-traffic entity (e.g., products, prices, offers) across all stores, and estimate the daily refresh rate (percentage or count of entities updated daily) for ongoing system operations. - **Stores and Locales**: Provide the total number of stores in the system and the maximum number of supported locales across all stores, as these factors impact chunk size calculation for data distribution. > For more information on stores and locales in our system, [click here](https://docs.spryker.com/docs/pbc/all/dynamic-multistore/202410.0/base-shop/dynamic-multistore-feature-overview.html). -- **Publish and Synchronize Setup**: The **publish and synchronize** processes entity data updates, and the worker setup plays a crucial role in determining how this is managed. Developers need to specify how their workers are set up in relation to stores. +- **Publish and Synchronize Setup**: The **publish and synchronize** processes entity data updates, and the worker setup plays a crucial role in determining how this is managed. Developers need to specify how project workers are set up in relation to stores. > For more information on workers, tasks, and how they are related to stores, [click here](https://docs.spryker.com/docs/pbc/all/dynamic-multistore/202410.0/base-shop/dynamic-multistore-feature-overview.html). -- **Number of Tasks Per Worker**: Developers are required to provide the **number of tasks per worker**. This value is essential to calculating how resources are distributed among tasks. Note that there is no additional help or explanation for determining this number, as it is specific to each setup. +- **Number of Tasks Per Worker**: Provide the **number of tasks per worker**. This value is essential to calculating how resources are distributed among tasks. Note that there is no additional help or explanation for determining this number, as it is specific to each setup. > For more information on workers, tasks, and how they are related to stores, [click here](https://docs.spryker.com/docs/pbc/all/dynamic-multistore/202410.0/base-shop/dynamic-multistore-feature-overview.html). ### Output From b37ebe8d398a921436f8673a7de67fb2465c01c0 Mon Sep 17 00:00:00 2001 From: Karoly Gerner <30629375+gerner-spryker@users.noreply.github.com> Date: Mon, 28 Oct 2024 18:00:29 +0100 Subject: [PATCH 16/26] Update advanced-chunk-size-calculation.md --- .../data-manipulation/queue/advanced-chunk-size-calculation.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/dg/dev/backend-development/data-manipulation/queue/advanced-chunk-size-calculation.md b/docs/dg/dev/backend-development/data-manipulation/queue/advanced-chunk-size-calculation.md index 56b90a4892d..d9c0cd17906 100644 --- a/docs/dg/dev/backend-development/data-manipulation/queue/advanced-chunk-size-calculation.md +++ b/docs/dg/dev/backend-development/data-manipulation/queue/advanced-chunk-size-calculation.md @@ -19,7 +19,6 @@ related: The **Advanced Chunk Size Calculator** builds upon the Basic level by allowing to fine-tune chunk sizes based on custom hardware limitations and custom performance metrics. -### The Advanced Chunk Size Calculator The **Advanced Chunk Size Calculator** is available [here](link to google spreadsheet). ### Problem Overview @@ -36,7 +35,7 @@ To calculate the correct queue chunk sizes, developers must provide the followin ### Output -Once the required data is entered into the **Basic Chunk Size Calculator** and **Advanced Chunk Size Calculator**, it will compute the optimal queue chunk sizes for each queue. Developers will need to configure these queue chunk sizes in their project to align with the calculated values. +Once the required data is entered into the **Basic Chunk Size Calculator** and **Advanced Chunk Size Calculator**, it will compute the optimal queue chunk sizes for each queue. Developers will need to configure these queue chunk sizes in the project to align with the calculated values. > For instructions on how to set up chunk sizes for the queues, [click here](https://docs.spryker.com/docs/dg/dev/backend-development/data-manipulation/queue/queue.html#configuration-for-chunk-size). From 67996e81f6e549a7b397174f832899151138fd88 Mon Sep 17 00:00:00 2001 From: Karoly Gerner <30629375+gerner-spryker@users.noreply.github.com> Date: Mon, 28 Oct 2024 18:29:51 +0100 Subject: [PATCH 17/26] Update chunk-size-calculation.md --- .../queue/chunk-size-calculation.md | 34 +++++++------------ 1 file changed, 13 insertions(+), 21 deletions(-) diff --git a/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.md b/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.md index 5e214a2b0c2..f62a91cf62f 100644 --- a/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.md +++ b/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.md @@ -17,51 +17,43 @@ related: ## Chunk Size Calculation -### Problem Statement +In an e-commerce framework, selecting the correct queue chunk size for processing data is critical for ensuring optimal performance and efficient resource usage. The challenge arises from the diversity of business entities, stores, and locales, each with different memory and CPU requirements for denormalization. Furthermore, the frequency of updates, data sizes, and the specific configurations of various system services (e.g., Redis, Elasticsearch, RabbitMQ) make it difficult to determine the appropriate chunk size for each queue. -In an e-commerce framework, selecting the correct chunk size for processing data is critical for ensuring optimal performance and efficient resource usage. The challenge arises from the diversity of business entities, stores, and locales, each with different memory and CPU requirements for denormalization. Furthermore, the frequency of updates, data sizes, and the specific configurations of various system services (e.g., Redis, Elasticsearch, RabbitMQ) make it difficult to determine the appropriate chunk size for each queue. - -Without proper chunk size configuration, systems can either overconsume resources, leading to crashes or lag, or underutilize resources, resulting in slow performance. This is where the **Chunk Size Calculator** comes in—offering a solution that helps developers fine-tune their chunk sizes based on the specific characteristics of their system. +Without proper queue chunk size configuration, projects can either overconsume resources, leading to crashes or lag, or underutilize resources, resulting in slow performance. This is where the **Chunk Size Calculator** comes in—offering a solution that helps developers fine-tune queue chunk sizes based on the specific characteristics of the project. ### What is the Chunk Size Calculator? -The **Chunk Size Calculator** is a tool that helps developers determine the appropriate chunk sizes for processing data across different queues in a resource-efficient way. It ensures that memory and CPU usage are optimized and prevents the system from being overwhelmed during data processing tasks. This tool is designed to handle the variability in entity sizes, stores, locales, and update frequencies, giving developers confidence that their system will run smoothly in production environments. +The **Chunk Size Calculator** is a tool that helps developers determine the appropriate queue chunk sizes for processing data across different queues in a resource-efficient way. It ensures that memory and CPU usage are optimized and prevents the system from being overwhelmed during data processing tasks. This tool is designed to handle the variability in entity sizes, stores, locales, and update frequencies, giving developers confidence that project will run smoothly in production environments. ### The Three Levels of the Chunk Size Calculator -The **Chunk Size Calculator** is divided into three levels: **Basic**, **Advanced**, and **Expert**. Each level is designed to accommodate different degrees of system complexity and customization. Below is an overview of each: +The **Chunk Size Calculator** is divided into three levels: **Basic**, **Advanced**, and **Expert**. Each level is designed to accommodate different degrees of project complexity and customization. Below is an overview of each: #### 1. Basic Chunk Size Calculator -The **Basic Chunk Size Calculator** is designed for small to medium B2C webshops or systems with minimal customization. It assumes that the default configuration of business entities, stores, and locales is sufficient, and that the resource consumption patterns are predictable. +The **Basic Chunk Size Calculator** is designed for small to medium B2C projects with minimal customization. It assumes that the default configuration of business entities, stores, and locales is sufficient, and that the resource consumption patterns are predictable. -With the Basic calculator, developers only need to provide a minimal set of inputs, such as high level store configuration and high traffic entity counts. The calculator uses these inputs to recommend chunk sizes for each queue. This is ideal for developers who are working with out-of-the-box setups and need a simple, reliable way to configure their system. +With the Basic calculator, developers only need to provide a minimal set of inputs, such as store configuration and high traffic entity counts. The calculator uses these inputs to recommend chunk sizes for each queue. This is ideal for developers who are working with out-of-the-box setups and need a simple, reliable way to configure project. -**When to use**: This is the default starting point for any project. If your system has not been heavily customized, this calculator will give you the necessary chunk sizes with minimal effort. +**When to use**: This is the default starting point for any project. If your project has not been heavily customized, this calculator will give you the necessary queue chunk sizes with minimal effort. Find more details on the [Basic Chunk Size Calculation](https://docs.spryker.com/docs/dg/dev/backend-development/data-manipulation/queue/basic-chunk-size-calculation.html) page. #### 2. Advanced Chunk Size Calculator -The **Advanced Chunk Size Calculator** builds upon the basic level, requiring developers to have a deeper understanding of the services that make up the system. In addition to understanding the basic chunk size concepts, developers will need to account for service elements like: - -- **Persistence** (Relational Database Service) -- **Storage** (Redis) -- **Search** (Elasticsearch) -- **Message Broker** (RabbitMQ) -- **Scheduler** +The **Advanced Chunk Size Calculator** builds upon the basic level, requiring developers to have a deeper understanding of the services that make up the project. In addition to understanding the basic chunk size concepts, developers will need to account for service elements like: Persistence, Storage, Search, Message Broker and Scheduler. -The Advanced calculator allows developers to select from AWS container types and configure memory sizes for each service. It also exposes additional performance metrics, such as the average number of triggered events per second and message size limits in RabbitMQ. While the calculator provides recommendations, developers will need to input detailed configuration data to fine-tune the system. +The Advanced calculator addresses resource allocations or performance-sensitive variables, enabling a more precise configuration that enhances stability and performance in production environments. While the calculator provides recommendations, developers will need to input detailed configuration data to fine-tune the project. -**When to use**: The Advanced calculator is suited for systems that have been moderately customized or when developers need more precise control over performance and resource usage. +**When to use**: The Advanced calculator is suited for projects that have been moderately customized or when developers need more precise control over performance and resource usage. Find more details on the [Advanced Chunk Size Calculation](https://docs.spryker.com/docs/dg/dev/backend-development/data-manipulation/queue/advanced-chunk-size-calculation.html) page. #### 3. Expert Chunk Size Calculator -The **Expert Chunk Size Calculator** is designed for highly customized systems where one or more entities deviate significantly from the norm. This includes scenarios where entities are unusually large or where the system handles massive data volumes that require frequent updates. +The **Expert Chunk Size Calculator** is designed for highly customized projects where one or more entities deviate significantly from the norm. This includes scenarios where entities are unusually large or where the project handles massive data volumes that require frequent updates. -In this case, developers need a deep understanding of how the system’s components interact, including the scheduler’s worker, and the memory distribution across workers, and tasks. The Expert calculator gives full visibility into all performance metrics, allowing developers to tweak each queue individually. +In this case, developers need a deep understanding of how the project’s components interact, including the scheduler’s worker, and the memory distribution across workers, and tasks. The Expert calculator gives full visibility into all performance metrics, allowing developers to tweak each queue individually. Despite the complexity, the calculator still ensures that the entire system remains balanced, preventing one queue from consuming too many resources at the expense of others. @@ -71,4 +63,4 @@ Find more details on the [Expert Chunk Size Calculation](https://docs.spryker.co ### Summary -The **Chunk Size Calculator** provides developers with a powerful tool for optimizing their system’s resource consumption. Each level of the calculator is designed to address different use cases, from simple out-of-the-box configurations to highly customized, complex environments. Developers are encouraged to start with the **Basic Chunk Size Calculator** and, if necessary, progress to the **Advanced** or **Expert** calculators as the complexity of their system grows. +The **Chunk Size Calculator** provides developers with a powerful tool for optimizing their system’s resource consumption. Each level of the calculator is designed to address different use cases, from simple out-of-the-box configurations to highly customized, complex environments. Developers are encouraged to start with the **Basic Chunk Size Calculator** and, if necessary, progress to the **Advanced** or **Expert** calculators as the complexity of their project grows. From fd41ee1bd726c4d1a699109f4b3e822c237a1d59 Mon Sep 17 00:00:00 2001 From: Karoly Gerner <30629375+gerner-spryker@users.noreply.github.com> Date: Mon, 28 Oct 2024 18:33:52 +0100 Subject: [PATCH 18/26] Update basic-chunk-size-calculation.md --- .../data-manipulation/queue/basic-chunk-size-calculation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/dg/dev/backend-development/data-manipulation/queue/basic-chunk-size-calculation.md b/docs/dg/dev/backend-development/data-manipulation/queue/basic-chunk-size-calculation.md index 0bd909c62d1..a29528a041e 100644 --- a/docs/dg/dev/backend-development/data-manipulation/queue/basic-chunk-size-calculation.md +++ b/docs/dg/dev/backend-development/data-manipulation/queue/basic-chunk-size-calculation.md @@ -17,7 +17,7 @@ related: ## Basic Chunk Size Calculation -The **Basic Chunk Size Calculator** is designed to help developers configure the correct chunk sizes for their Spryker Commerce Operating System based on the traffic and data patterns in their system. This tool simplifies the setup process for out-of-the-box and low-customised webshops, ensuring that the system can handle high-traffic entities efficiently without over-consuming resources. +The **Basic Chunk Size Calculator** is designed to help developers configure the correct chunk sizes for the project based on the traffic and data patterns in the system. This tool simplifies the setup process for out-of-the-box and low-customised webshops, ensuring that the system can handle high-traffic entities efficiently without over-consuming resources. The **Basic Chunk Size Calculator** is available [here](link to google spreadsheet). From 935bc8b0831deb1137ccc759adff708232f6bfb7 Mon Sep 17 00:00:00 2001 From: Karoly Gerner <30629375+gerner-spryker@users.noreply.github.com> Date: Mon, 28 Oct 2024 18:35:55 +0100 Subject: [PATCH 19/26] Update advanced-chunk-size-calculation.md --- .../data-manipulation/queue/advanced-chunk-size-calculation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/dg/dev/backend-development/data-manipulation/queue/advanced-chunk-size-calculation.md b/docs/dg/dev/backend-development/data-manipulation/queue/advanced-chunk-size-calculation.md index d9c0cd17906..c852ef676cf 100644 --- a/docs/dg/dev/backend-development/data-manipulation/queue/advanced-chunk-size-calculation.md +++ b/docs/dg/dev/backend-development/data-manipulation/queue/advanced-chunk-size-calculation.md @@ -27,7 +27,7 @@ While the **Basic Chunk Size Calculator** offers a starting point, it doesn’t ### Input Parameters -To calculate the correct queue chunk sizes, developers must provide the following information based on their specific production environment: +To calculate the correct queue chunk sizes, developers must provide the following information based on the specific production environment: - **Scheduler and Worker Setup**: Provide details on any non-standard configurations, such as environments with multiple containers or distinct worker distributions within containers, if your scheduler setup differs from the boilerplate defaults. - **Resource Configuration**: Provide information on your hardware setup, including instance types, CPU, and memory limits for services like Persistence, Storage, Search, or Message Broker, to allow the **Advanced Calculator** to optimize chunk sizes based on actual resource availability. - **Detailed Product Configuration**: Provide specific metrics related to products, the highest-traffic entity. This supports more precise chunk sizing for products without requiring an in-depth understanding of Publish & Synchronize. From 365c0da0d23ea0f0745a8b5e90ac38895f182c85 Mon Sep 17 00:00:00 2001 From: Karoly Gerner <30629375+gerner-spryker@users.noreply.github.com> Date: Mon, 28 Oct 2024 19:01:41 +0100 Subject: [PATCH 20/26] Create expert-chunk-size-calculation.md --- .../queue/expert-chunk-size-calculation.md | 63 +++++++++++++++++++ 1 file changed, 63 insertions(+) create mode 100644 docs/dg/dev/backend-development/data-manipulation/queue/expert-chunk-size-calculation.md diff --git a/docs/dg/dev/backend-development/data-manipulation/queue/expert-chunk-size-calculation.md b/docs/dg/dev/backend-development/data-manipulation/queue/expert-chunk-size-calculation.md new file mode 100644 index 00000000000..b876a8fbe2d --- /dev/null +++ b/docs/dg/dev/backend-development/data-manipulation/queue/expert-chunk-size-calculation.md @@ -0,0 +1,63 @@ +--- +title: Expert Chunk Size Calculation +description: Gives an overview over the expert chunk size calculation +last_updated: Oct 25, 2024 +template: concept-topic-template +redirect_from: + - /docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.html +related: + - title: Basic Chunk Size Calculation + link: docs/dg/dev/backend-development/data-manipulation/queue/basic-chunk-size-calculation.html + - title: Advanced Chunk Size Calculation + link: docs/dg/dev/backend-development/data-manipulation/queue/advanced-chunk-size-calculation.html + - title: Queue + link: docs/dg/dev/backend-development/data-manipulation/queue/queue.html + +--- + +## Expert Chunk Size Calculation + +### Overview + +The **Expert Chunk Size Calculator** is designed for developers working with heavily customized entities within the project. Whether it's a single entity or multiple entities, when these entities are significantly customized in terms of size, relationships, or data complexity, this calculator provides the granular control needed to fine-tune each queue's performance. + +As the complexity of an entity increases, so does the denormalization time, which can slow down the entire system. Developers using the **Expert Chunk Size Calculator** must have a solid understanding of how containerization works in the project, how resources like memory and CPU are distributed among containers, workers, and tasks, as well as the limitations of the receiving side services (such as search and storage) and the provider systems (like the database). The message broker, which delivers messages and imposes throughput limits, is also a critical component of the overall system architecture that needs to be considered. + +The **Expert Chunk Size Calculator** is available [here](link to google spreadsheet). + +### Problem Statement + +In highly customized systems, Basic or Advanced queue chunk size configurations may not suffice. Complex entities with large data sets and relationships demand more fine-tuned control over how tasks are processed, how resources are allocated, and how messages are handled. The **Expert Chunk Size Calculator** is needed to provide detailed, queue-by-queue configuration for developers who need to optimize the project's performance under these conditions. + +### Input Parameters + +The **Expert Chunk Size Calculator** requires a wide range of detailed inputs to properly configure chunk sizes. Developers need to provide in-depth information about the production environment, including: + +- **Entity Customization**: The size and cardinality of the entities, which affects how much memory and CPU is consumed during the denormalization process. +- **Message Handling**: Specific configuration data regarding the size of messages that will be processed by the system and the limits imposed by the message broker and receiving systems. + +The expert calculator offers the ability to set individual performance and resource consumption metrics for each queue, making it possible to precisely optimize the entire **publish and synchronize** process. + +### Output + +The result of the **Expert Chunk Size Calculator** is a set of optimized queue chunk sizes for each individual queue in the project. + +> For instructions on how to set up chunk sizes for the queues, [click here](https://docs.spryker.com/docs/dg/dev/backend-development/data-manipulation/queue/queue.html#configuration-for-chunk-size). + +### Important Notes + +- The **Expert Chunk Size Calculator** is intended for projects that have significant customizations at the entity level. If your system follows a more standard setup, consider using the **Basic** or **Advanced Chunk Size Calculators**. +- This calculator requires an in-depth understanding of how system components interact, including containerization, message brokers, search and storage, and resource distribution across workers and tasks. +- For systems that require individual configuration of queues and detailed customization of message handling, consider using the **Expert Chunk Size Calculator**. + +### Additional Knowledge Required + +To effectively use the **Expert Chunk Size Calculator**, developers must have a strong grasp of several key concepts related to resource management and system architecture. + +#### 1. Container-Worker-Task Resource Relationship + +This section will cover how resources (memory, CPU, etc.) are allocated between containers, workers, and tasks. It will explain how container boundaries are defined and the importance of understanding how these resources are distributed across the system to maintain healthy processing. + +#### 2. Publish and Synchronize Queues + +This section will explain how queues work in the publish and synchronize middleware, how they process multiple entities, and how factors like entity size and denormalization times impact CPU and memory consumption. Understanding these relationships is key to optimizing each queue’s performance through the expert calculator. From ae480d32f4d17586616d5724fc40a5e4aa53bf3b Mon Sep 17 00:00:00 2001 From: Karoly Gerner <30629375+gerner-spryker@users.noreply.github.com> Date: Tue, 29 Oct 2024 09:10:38 +0100 Subject: [PATCH 21/26] Update expert-chunk-size-calculation.md --- .../data-manipulation/queue/expert-chunk-size-calculation.md | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/docs/dg/dev/backend-development/data-manipulation/queue/expert-chunk-size-calculation.md b/docs/dg/dev/backend-development/data-manipulation/queue/expert-chunk-size-calculation.md index b876a8fbe2d..20d67be5362 100644 --- a/docs/dg/dev/backend-development/data-manipulation/queue/expert-chunk-size-calculation.md +++ b/docs/dg/dev/backend-development/data-manipulation/queue/expert-chunk-size-calculation.md @@ -54,10 +54,7 @@ The result of the **Expert Chunk Size Calculator** is a set of optimized queue c To effectively use the **Expert Chunk Size Calculator**, developers must have a strong grasp of several key concepts related to resource management and system architecture. -#### 1. Container-Worker-Task Resource Relationship +#### Container-Worker-Task Resource Relationship -This section will cover how resources (memory, CPU, etc.) are allocated between containers, workers, and tasks. It will explain how container boundaries are defined and the importance of understanding how these resources are distributed across the system to maintain healthy processing. -#### 2. Publish and Synchronize Queues -This section will explain how queues work in the publish and synchronize middleware, how they process multiple entities, and how factors like entity size and denormalization times impact CPU and memory consumption. Understanding these relationships is key to optimizing each queue’s performance through the expert calculator. From 1fb17d356a149c6acda9ac46c6d3ed5acf70a1f0 Mon Sep 17 00:00:00 2001 From: Spryker Release Manager Date: Tue, 29 Oct 2024 14:52:32 +0100 Subject: [PATCH 22/26] Merging chunk-size-calculator documents --- .../queue/advanced-chunk-size-calculation.md | 51 --------- .../queue/basic-chunk-size-calculation.md | 54 --------- .../queue/chunk-size-calculation.md | 104 ++++++++++++++---- .../queue/expert-chunk-size-calculation.md | 60 ---------- 4 files changed, 84 insertions(+), 185 deletions(-) delete mode 100644 docs/dg/dev/backend-development/data-manipulation/queue/advanced-chunk-size-calculation.md delete mode 100644 docs/dg/dev/backend-development/data-manipulation/queue/basic-chunk-size-calculation.md delete mode 100644 docs/dg/dev/backend-development/data-manipulation/queue/expert-chunk-size-calculation.md diff --git a/docs/dg/dev/backend-development/data-manipulation/queue/advanced-chunk-size-calculation.md b/docs/dg/dev/backend-development/data-manipulation/queue/advanced-chunk-size-calculation.md deleted file mode 100644 index c852ef676cf..00000000000 --- a/docs/dg/dev/backend-development/data-manipulation/queue/advanced-chunk-size-calculation.md +++ /dev/null @@ -1,51 +0,0 @@ ---- -title: Advanced Chunk Size Calculation -description: Gives an overview over the advanced chunk size calculation -last_updated: Oct 25, 2024 -template: concept-topic-template -redirect_from: - - /docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.html -related: - - title: Basic Chunk Size Calculation - link: docs/dg/dev/backend-development/data-manipulation/queue/basic-chunk-size-calculation.html - - title: Expert Chunk Size Calculation - link: docs/dg/dev/backend-development/data-manipulation/queue/expert-chunk-size-calculation.html - - title: Queue - link: docs/dg/dev/backend-development/data-manipulation/queue/queue.html - ---- - -## Advanced Chunk Size Calculation - -The **Advanced Chunk Size Calculator** builds upon the Basic level by allowing to fine-tune chunk sizes based on custom hardware limitations and custom performance metrics. - -The **Advanced Chunk Size Calculator** is available [here](link to google spreadsheet). - -### Problem Overview - -While the **Basic Chunk Size Calculator** offers a starting point, it doesn’t account for the nuances of resource allocations (like container sizes, CPU, and memory limits) or performance-sensitive variables such as application warm-up time and event message size. The **Advanced Chunk Size Calculator** addresses these issues by incorporating these additional metrics, enabling a more precise configuration that enhances stability and performance in production environments. - -### Input Parameters - -To calculate the correct queue chunk sizes, developers must provide the following information based on the specific production environment: -- **Scheduler and Worker Setup**: Provide details on any non-standard configurations, such as environments with multiple containers or distinct worker distributions within containers, if your scheduler setup differs from the boilerplate defaults. -- **Resource Configuration**: Provide information on your hardware setup, including instance types, CPU, and memory limits for services like Persistence, Storage, Search, or Message Broker, to allow the **Advanced Calculator** to optimize chunk sizes based on actual resource availability. -- **Detailed Product Configuration**: Provide specific metrics related to products, the highest-traffic entity. This supports more precise chunk sizing for products without requiring an in-depth understanding of Publish & Synchronize. -- **Event and Message Processing Metrics**: Provide expected event processing metrics, including deviations from default settings such as message trigger rates, custom application warm-up times, event size limits, and data division rate multipliers, to enable configuration adjustments that align with real-world performance. - -### Output - -Once the required data is entered into the **Basic Chunk Size Calculator** and **Advanced Chunk Size Calculator**, it will compute the optimal queue chunk sizes for each queue. Developers will need to configure these queue chunk sizes in the project to align with the calculated values. - -> For instructions on how to set up chunk sizes for the queues, [click here](https://docs.spryker.com/docs/dg/dev/backend-development/data-manipulation/queue/queue.html#configuration-for-chunk-size). - - -### Important Notes - -- The **Advanced Chunk Size Calculator** allows to further configure chunk sizes based on custom hardware limitations and custom performance metrics. -- For systems that require individual configuration of queues and detailed customisation of message setups, consider using the **Expert Chunk Size Calculator**. -- Always ensure that the chunk sizes provided by the calculator are properly configured to avoid system performance issues. - ---- - -For more detailed information about the different levels of the **Chunk Size Calculator**, see the [overview here](https://docs.spryker.com/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.html). diff --git a/docs/dg/dev/backend-development/data-manipulation/queue/basic-chunk-size-calculation.md b/docs/dg/dev/backend-development/data-manipulation/queue/basic-chunk-size-calculation.md deleted file mode 100644 index a29528a041e..00000000000 --- a/docs/dg/dev/backend-development/data-manipulation/queue/basic-chunk-size-calculation.md +++ /dev/null @@ -1,54 +0,0 @@ ---- -title: Basic Chunk Size Calculation -description: Gives an overview over the basic chunk size calculation -last_updated: Oct 25, 2024 -template: concept-topic-template -redirect_from: - - /docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.html -related: - - title: Advanced Chunk Size Calculation - link: docs/dg/dev/backend-development/data-manipulation/queue/advanced-chunk-size-calculation.html - - title: Expert Chunk Size Calculation - link: docs/dg/dev/backend-development/data-manipulation/queue/expert-chunk-size-calculation.html - - title: Queue - link: docs/dg/dev/backend-development/data-manipulation/queue/queue.html - ---- - -## Basic Chunk Size Calculation - -The **Basic Chunk Size Calculator** is designed to help developers configure the correct chunk sizes for the project based on the traffic and data patterns in the system. This tool simplifies the setup process for out-of-the-box and low-customised webshops, ensuring that the system can handle high-traffic entities efficiently without over-consuming resources. - -The **Basic Chunk Size Calculator** is available [here](link to google spreadsheet). - -### Problem Overview - -In an e-commerce environment, certain business entities generate a large volume of update events due to frequent refreshes and high data volume. These **high traffic entities** account for the majority of the traffic within the **publish and synchronize**. Misconfiguring chunk sizes for these entities can lead to inefficient resource consumption, system lags, or overloads. The **Basic Chunk Size Calculator** offers a straightforward way to address this by determining the appropriate chunk size for each queue based on the production environment’s data profile. - -### Input Parameters - -To calculate the correct queue chunk sizes, developers must provide the following information based on their specific production environment: - -- **High Traffic Entities**: Provide the total count of each high-traffic entity (e.g., products, prices, offers) across all stores, and estimate the daily refresh rate (percentage or count of entities updated daily) for ongoing system operations. -- **Stores and Locales**: Provide the total number of stores in the system and the maximum number of supported locales across all stores, as these factors impact chunk size calculation for data distribution. - > For more information on stores and locales in our system, [click here](https://docs.spryker.com/docs/pbc/all/dynamic-multistore/202410.0/base-shop/dynamic-multistore-feature-overview.html). -- **Publish and Synchronize Setup**: The **publish and synchronize** processes entity data updates, and the worker setup plays a crucial role in determining how this is managed. Developers need to specify how project workers are set up in relation to stores. - > For more information on workers, tasks, and how they are related to stores, [click here](https://docs.spryker.com/docs/pbc/all/dynamic-multistore/202410.0/base-shop/dynamic-multistore-feature-overview.html). -- **Number of Tasks Per Worker**: Provide the **number of tasks per worker**. This value is essential to calculating how resources are distributed among tasks. Note that there is no additional help or explanation for determining this number, as it is specific to each setup. - > For more information on workers, tasks, and how they are related to stores, [click here](https://docs.spryker.com/docs/pbc/all/dynamic-multistore/202410.0/base-shop/dynamic-multistore-feature-overview.html). - -### Output - -Once the required data is entered into the **Basic Chunk Size Calculator**, it will compute the optimal chunk sizes for each queue used by the system. These queues handle different business entities, and setting the right queue chunk size ensures efficient processing and resource allocation. Developers will need to configure these queue chunk sizes. - -> For instructions on how to set up chunk sizes for the queues, [click here](https://docs.spryker.com/docs/dg/dev/backend-development/data-manipulation/queue/queue.html#configuration-for-chunk-size). - -### Important Notes - -- The **Basic Chunk Size Calculator** is designed for systems that follow a standard, out-of-the-box configuration. If your system is more customized, consider using the **Advanced** or **Expert Chunk Size Calculator** for fine-tuning. -- This calculator only requires a basic understanding of the system's entity data and store structure. For more complex metrics like memory usage or container performance, the advanced calculators may be necessary. -- Always ensure that the chunk sizes provided by the calculator are properly configured to avoid system performance issues. - ---- - -For more detailed information about the different levels of the **Chunk Size Calculator**, see the [overview here](https://docs.spryker.com/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.html). diff --git a/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.md b/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.md index f62a91cf62f..5682e24317e 100644 --- a/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.md +++ b/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.md @@ -6,10 +6,6 @@ template: concept-topic-template redirect_from: - /docs/dg/dev/backend-development/data-manipulation/queue/queue.html#concepts related: - - title: Basic Chunk Size Calculation - link: docs/dg/dev/backend-development/data-manipulation/queue/basic-chunk-size-calculation.html - - title: Advanced Chunk Size Calculation - link: docs/dg/dev/backend-development/data-manipulation/queue/advanced-chunk-size-calculation.html - title: Expert Chunk Size Calculation link: docs/dg/dev/backend-development/data-manipulation/queue/expert-chunk-size-calculation.html @@ -27,40 +23,108 @@ The **Chunk Size Calculator** is a tool that helps developers determine the appr ### The Three Levels of the Chunk Size Calculator -The **Chunk Size Calculator** is divided into three levels: **Basic**, **Advanced**, and **Expert**. Each level is designed to accommodate different degrees of project complexity and customization. Below is an overview of each: +The **Chunk Size Calculator** is a powerful tool for optimizing Publish and Sync’s resource consumption. It consists of three levels—**Basic**, **Advanced**, and **Expert**—each designed to cater to varying use cases, from simple out-of-the-box configurations to highly customized, complex environments. Developers are encouraged to start with the **Basic Chunk Size Calculator** and progress to the **Advanced** or **Expert** calculators as the complexity of their projects increases. + +- **Basic**: Ideal for standard configurations with minimal customization. +- **Advanced**: Suited for moderately customized projects requiring more precise control over performance and resource usage. +- **Expert**: Designed for highly customized entities that demand detailed configurations and fine-tuning. #### 1. Basic Chunk Size Calculator -The **Basic Chunk Size Calculator** is designed for small to medium B2C projects with minimal customization. It assumes that the default configuration of business entities, stores, and locales is sufficient, and that the resource consumption patterns are predictable. +The **Basic Chunk Size Calculator** is a tool designed to help developers configure optimal queue chunk sizes for small to medium B2C projects with low customization with minimal configuration effort. It ensures efficient processing by aligning queue chunk sizes with traffic and data distribution. Ideal for out-of-the-box or minimally customized setups, it is available for use [here](link to Google spreadsheet). + +##### Purpose and Use Case + +In e-commerce environments, some business entities generate frequent updates due to high transaction volumes, commonly referred to as **high traffic entities**. These entities—such as products, prices, and offers—drive most of the traffic within the **publish and synchronize** operations. Misconfiguring queue chunk sizes can result in inefficient resource consumption, performance bottlenecks, or system overloads. The **Basic Chunk Size Calculator** streamlines this process by computing chunk sizes based on your project’s data profile, making it an ideal starting point for projects with standard configurations and predictable traffic patterns. + +##### Key Input Parameters + +To calculate the correct queue chunk sizes, the following data should be provided: + +- **High Traffic Entities**: Input the total count of each high-traffic entity type (e.g., products, prices, offers) across all stores, along with an estimate of the daily refresh rate (either as a percentage or a daily update count). +- **Stores and Locales**: Specify the total number of stores and the maximum number of supported locales across these stores. This information impacts data distribution and chunk sizing. + > For detailed information on configuring stores and locales, [click here](https://docs.spryker.com/docs/pbc/all/dynamic-multistore/202410.0/base-shop/dynamic-multistore-feature-overview.html). +- **Publish and Synchronize Setup**: Since entity data updates are managed through **publish and synchronize** operations, describe how workers are set up in relation to stores in the system. + > For additional details on workers and tasks, refer to [this link](https://docs.spryker.com/docs/pbc/all/dynamic-multistore/202410.0/base-shop/dynamic-multistore-feature-overview.html). +- **Number of Tasks Per Worker**: Define the number of tasks each worker will handle, as this affects resource allocation and task distribution. This parameter should align with your project’s specific setup. + > Learn more about configuring tasks per worker [here](https://docs.spryker.com/docs/pbc/all/dynamic-multistore/202410.0/base-shop/dynamic-multistore-feature-overview.html). + +##### Output and Configuration -With the Basic calculator, developers only need to provide a minimal set of inputs, such as store configuration and high traffic entity counts. The calculator uses these inputs to recommend chunk sizes for each queue. This is ideal for developers who are working with out-of-the-box setups and need a simple, reliable way to configure project. +After entering the required data, the **Basic Chunk Size Calculator** will determine optimal queue chunk sizes for each queue within your system. These chunk sizes help to balance load across queues, facilitating smooth operations and resource efficiency. Developers should configure these queue chunk sizes based on the calculator’s recommendations. -**When to use**: This is the default starting point for any project. If your project has not been heavily customized, this calculator will give you the necessary queue chunk sizes with minimal effort. +> For instructions on setting up chunk sizes for queues, [click here](https://docs.spryker.com/docs/dg/dev/backend-development/data-manipulation/queue/queue.html#configuration-for-chunk-size). -Find more details on the [Basic Chunk Size Calculation](https://docs.spryker.com/docs/dg/dev/backend-development/data-manipulation/queue/basic-chunk-size-calculation.html) page. +##### Important Considerations + +- The **Basic Chunk Size Calculator** is optimized for systems with standard, out-of-the-box configurations. If your setup includes significant customization, consider using the **Advanced** or **Expert Chunk Size Calculator** for fine-tuned configuration. +- This calculator requires only a basic understanding of your system’s entity data and store structure. More complex metrics, like memory usage and container performance, might necessitate advanced calculators. +- Ensure the recommended queue chunk sizes are correctly configured to maintain optimal system performance. #### 2. Advanced Chunk Size Calculator -The **Advanced Chunk Size Calculator** builds upon the basic level, requiring developers to have a deeper understanding of the services that make up the project. In addition to understanding the basic chunk size concepts, developers will need to account for service elements like: Persistence, Storage, Search, Message Broker and Scheduler. +The **Advanced Chunk Size Calculator** builds upon the **Basic Chunk Size Calculator**, requiring developers to have a deeper understanding of the services that make up the project. It is available for use [here](link to Google spreadsheet). + +##### Purpose and Use Case + +The **Advanced Chunk Size Calculator** is ideal for moderately customized projects or when developers need greater control over performance and resource usage. It goes beyond the **Basic Chunk Size Calculator** by taking into account resource allocations, such as container sizes, CPU, and memory limits, allowing for a more tailored configuration that aligns with specific project needs. + +##### Key Input Parameters + +To calculate the correct queue chunk sizes, the following data should be provided: + +- **Scheduler and Worker Setup**: Provide details on non-standard configurations, such as multiple containers or distinct worker distributions within containers if your scheduler setup differs from the default. +- **Resource Configuration**: Specify the hardware setup, including instance types, CPU, and memory limits for services like **Persistence**, **Storage**, **Search**, or **Message Broker** to optimize chunk sizes based on actual resource availability. +- **Detailed Product Configuration**: Provide specific metrics for products, the highest-traffic entity. This supports more precise chunk sizing without requiring an in-depth understanding of **Publish & Synchronize**. +- **Event and Message Processing Metrics**: Provide expected processing metrics, including deviations from default settings such as message trigger rates, custom application warm-up times, event size limits, and data division rate multipliers, allowing configuration adjustments that align with real-world performance. -The Advanced calculator addresses resource allocations or performance-sensitive variables, enabling a more precise configuration that enhances stability and performance in production environments. While the calculator provides recommendations, developers will need to input detailed configuration data to fine-tune the project. +##### Output and Configuration -**When to use**: The Advanced calculator is suited for projects that have been moderately customized or when developers need more precise control over performance and resource usage. +Once the required data is entered into the **Advanced Chunk Size Calculator**, it will compute the optimal queue chunk sizes for each queue. Developers should configure these queue chunk sizes in the project to align with the calculated values. -Find more details on the [Advanced Chunk Size Calculation](https://docs.spryker.com/docs/dg/dev/backend-development/data-manipulation/queue/advanced-chunk-size-calculation.html) page. +> For instructions on setting up chunk sizes for queues, [click here](https://docs.spryker.com/docs/dg/dev/backend-development/data-manipulation/queue/queue.html#configuration-for-chunk-size). + +##### Important Considerations + +- The **Advanced Chunk Size Calculator** allows for further configuration of chunk sizes based on custom hardware limitations and performance metrics. +- For systems requiring individual queue configurations and detailed customization of message setups, consider using the **Expert Chunk Size Calculator**. +- Ensure the recommended chunk sizes are correctly configured to maintain optimal system performance. #### 3. Expert Chunk Size Calculator -The **Expert Chunk Size Calculator** is designed for highly customized projects where one or more entities deviate significantly from the norm. This includes scenarios where entities are unusually large or where the project handles massive data volumes that require frequent updates. +The **Expert Chunk Size Calculator** is designed for developers working with heavily customized entities within their projects. It provides the granular control needed to fine-tune each queue's performance, especially when entities are significantly customized in terms of size, relationships, or data complexity. + +As the complexity of an entity increases, so does the denormalization time, which can slow down the entire system. Developers using the **Expert Chunk Size Calculator** must have a solid understanding of how containerization works in the project, including the distribution of resources like memory and CPU among containers, workers, and tasks. Additionally, understanding the limitations of the receiving side services (such as search and storage) and provider systems (like the database) is crucial. The message broker, which delivers messages and imposes throughput limits, is also a critical component of the overall system architecture that needs to be considered. + +The **Expert Chunk Size Calculator** is available [here](link to google spreadsheet). + +##### Purpose and Use Case + +The **Expert Chunk Size Calculator** is essential for projects that involve complex entities with large data sets and intricate relationships. Basic or Advanced queue chunk size configurations may not suffice in such scenarios. This calculator provides detailed, queue-by-queue configuration options that enable developers to optimize performance under these conditions. + +##### Input Parameters + +To properly configure chunk sizes, the **Expert Chunk Size Calculator** requires a wide range of detailed inputs. Developers need to provide in-depth information about the production environment, including: + +- **Entity Customization**: Details about the size and cardinality of the entities, which affect how much memory and CPU is consumed during the denormalization process. +- **Message Handling**: Specific configuration data regarding the size of messages processed by the system and the limits imposed by the message broker and receiving systems. + +The expert calculator offers the ability to set individual performance and resource consumption metrics for each queue, allowing for precise optimization of the entire **publish and synchronize** process. + +##### Output + +The result of the **Expert Chunk Size Calculator** is a set of optimized queue chunk sizes for each individual queue in the project. -In this case, developers need a deep understanding of how the project’s components interact, including the scheduler’s worker, and the memory distribution across workers, and tasks. The Expert calculator gives full visibility into all performance metrics, allowing developers to tweak each queue individually. +> For instructions on how to set up chunk sizes for the queues, [click here](https://docs.spryker.com/docs/dg/dev/backend-development/data-manipulation/queue/queue.html#configuration-for-chunk-size). -Despite the complexity, the calculator still ensures that the entire system remains balanced, preventing one queue from consuming too many resources at the expense of others. +##### Important Notes -**When to use**: The Expert calculator is reserved for production environments with complex, heavily customized setups that require fine-tuning of every performance metric. +- The **Expert Chunk Size Calculator** is intended for projects with significant customizations at the entity level. For more standard setups, consider using the **Basic** or **Advanced Chunk Size Calculators**. +- This calculator requires an in-depth understanding of how system components interact, including containerization, message brokers, search and storage, and resource distribution across workers and tasks. -Find more details on the [Expert Chunk Size Calculation](https://docs.spryker.com/docs/dg/dev/backend-development/data-manipulation/queue/expert-chunk-size-calculation.html) page. +##### Additional Knowledge Required -### Summary +To effectively use the **Expert Chunk Size Calculator**, developers must have a strong grasp of key concepts related to resource management and system architecture, including: -The **Chunk Size Calculator** provides developers with a powerful tool for optimizing their system’s resource consumption. Each level of the calculator is designed to address different use cases, from simple out-of-the-box configurations to highly customized, complex environments. Developers are encouraged to start with the **Basic Chunk Size Calculator** and, if necessary, progress to the **Advanced** or **Expert** calculators as the complexity of their project grows. +###### Container-Worker-Task Resource Relationship +TBD diff --git a/docs/dg/dev/backend-development/data-manipulation/queue/expert-chunk-size-calculation.md b/docs/dg/dev/backend-development/data-manipulation/queue/expert-chunk-size-calculation.md deleted file mode 100644 index 20d67be5362..00000000000 --- a/docs/dg/dev/backend-development/data-manipulation/queue/expert-chunk-size-calculation.md +++ /dev/null @@ -1,60 +0,0 @@ ---- -title: Expert Chunk Size Calculation -description: Gives an overview over the expert chunk size calculation -last_updated: Oct 25, 2024 -template: concept-topic-template -redirect_from: - - /docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.html -related: - - title: Basic Chunk Size Calculation - link: docs/dg/dev/backend-development/data-manipulation/queue/basic-chunk-size-calculation.html - - title: Advanced Chunk Size Calculation - link: docs/dg/dev/backend-development/data-manipulation/queue/advanced-chunk-size-calculation.html - - title: Queue - link: docs/dg/dev/backend-development/data-manipulation/queue/queue.html - ---- - -## Expert Chunk Size Calculation - -### Overview - -The **Expert Chunk Size Calculator** is designed for developers working with heavily customized entities within the project. Whether it's a single entity or multiple entities, when these entities are significantly customized in terms of size, relationships, or data complexity, this calculator provides the granular control needed to fine-tune each queue's performance. - -As the complexity of an entity increases, so does the denormalization time, which can slow down the entire system. Developers using the **Expert Chunk Size Calculator** must have a solid understanding of how containerization works in the project, how resources like memory and CPU are distributed among containers, workers, and tasks, as well as the limitations of the receiving side services (such as search and storage) and the provider systems (like the database). The message broker, which delivers messages and imposes throughput limits, is also a critical component of the overall system architecture that needs to be considered. - -The **Expert Chunk Size Calculator** is available [here](link to google spreadsheet). - -### Problem Statement - -In highly customized systems, Basic or Advanced queue chunk size configurations may not suffice. Complex entities with large data sets and relationships demand more fine-tuned control over how tasks are processed, how resources are allocated, and how messages are handled. The **Expert Chunk Size Calculator** is needed to provide detailed, queue-by-queue configuration for developers who need to optimize the project's performance under these conditions. - -### Input Parameters - -The **Expert Chunk Size Calculator** requires a wide range of detailed inputs to properly configure chunk sizes. Developers need to provide in-depth information about the production environment, including: - -- **Entity Customization**: The size and cardinality of the entities, which affects how much memory and CPU is consumed during the denormalization process. -- **Message Handling**: Specific configuration data regarding the size of messages that will be processed by the system and the limits imposed by the message broker and receiving systems. - -The expert calculator offers the ability to set individual performance and resource consumption metrics for each queue, making it possible to precisely optimize the entire **publish and synchronize** process. - -### Output - -The result of the **Expert Chunk Size Calculator** is a set of optimized queue chunk sizes for each individual queue in the project. - -> For instructions on how to set up chunk sizes for the queues, [click here](https://docs.spryker.com/docs/dg/dev/backend-development/data-manipulation/queue/queue.html#configuration-for-chunk-size). - -### Important Notes - -- The **Expert Chunk Size Calculator** is intended for projects that have significant customizations at the entity level. If your system follows a more standard setup, consider using the **Basic** or **Advanced Chunk Size Calculators**. -- This calculator requires an in-depth understanding of how system components interact, including containerization, message brokers, search and storage, and resource distribution across workers and tasks. -- For systems that require individual configuration of queues and detailed customization of message handling, consider using the **Expert Chunk Size Calculator**. - -### Additional Knowledge Required - -To effectively use the **Expert Chunk Size Calculator**, developers must have a strong grasp of several key concepts related to resource management and system architecture. - -#### Container-Worker-Task Resource Relationship - - - From a5bdb53940b7f4281fd85eee08bf0010bd651ca8 Mon Sep 17 00:00:00 2001 From: Karoly Gerner <30629375+gerner-spryker@users.noreply.github.com> Date: Tue, 29 Oct 2024 14:57:59 +0100 Subject: [PATCH 23/26] Update chunk-size-calculation.md --- .../data-manipulation/queue/chunk-size-calculation.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.md b/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.md index 5682e24317e..cb863ce0391 100644 --- a/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.md +++ b/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.md @@ -6,8 +6,6 @@ template: concept-topic-template redirect_from: - /docs/dg/dev/backend-development/data-manipulation/queue/queue.html#concepts related: - - title: Expert Chunk Size Calculation - link: docs/dg/dev/backend-development/data-manipulation/queue/expert-chunk-size-calculation.html --- From 651067937987e88af250140e7791026e63a7f36c Mon Sep 17 00:00:00 2001 From: Karoly Gerner <30629375+gerner-spryker@users.noreply.github.com> Date: Tue, 29 Oct 2024 15:46:33 +0100 Subject: [PATCH 24/26] Update chunk-size-calculation.md --- .../data-manipulation/queue/chunk-size-calculation.md | 7 ------- 1 file changed, 7 deletions(-) diff --git a/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.md b/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.md index cb863ce0391..4270edc221d 100644 --- a/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.md +++ b/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.md @@ -119,10 +119,3 @@ The result of the **Expert Chunk Size Calculator** is a set of optimized queue c - The **Expert Chunk Size Calculator** is intended for projects with significant customizations at the entity level. For more standard setups, consider using the **Basic** or **Advanced Chunk Size Calculators**. - This calculator requires an in-depth understanding of how system components interact, including containerization, message brokers, search and storage, and resource distribution across workers and tasks. - -##### Additional Knowledge Required - -To effectively use the **Expert Chunk Size Calculator**, developers must have a strong grasp of key concepts related to resource management and system architecture, including: - -###### Container-Worker-Task Resource Relationship -TBD From b42e6be1a606e8c0e82afc450da026a65a352cb3 Mon Sep 17 00:00:00 2001 From: Karoly Gerner <30629375+gerner-spryker@users.noreply.github.com> Date: Wed, 30 Oct 2024 09:36:55 +0100 Subject: [PATCH 25/26] Update chunk-size-calculation.md --- .../data-manipulation/queue/chunk-size-calculation.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.md b/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.md index 4270edc221d..5ea6dd929f1 100644 --- a/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.md +++ b/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.md @@ -29,7 +29,7 @@ The **Chunk Size Calculator** is a powerful tool for optimizing Publish and Sync #### 1. Basic Chunk Size Calculator -The **Basic Chunk Size Calculator** is a tool designed to help developers configure optimal queue chunk sizes for small to medium B2C projects with low customization with minimal configuration effort. It ensures efficient processing by aligning queue chunk sizes with traffic and data distribution. Ideal for out-of-the-box or minimally customized setups, it is available for use [here](link to Google spreadsheet). +The **Basic Chunk Size Calculator** is a tool designed to help developers configure optimal queue chunk sizes for small to medium B2C projects with low customization with minimal configuration effort. It ensures efficient processing by aligning queue chunk sizes with traffic and data distribution. Ideal for out-of-the-box or minimally customized setups, it is available for use [here](link to Google spreadsheet); please make a copy of the spreadsheet to experiment with and adjust the settings as needed for your project. ##### Purpose and Use Case @@ -61,7 +61,7 @@ After entering the required data, the **Basic Chunk Size Calculator** will deter #### 2. Advanced Chunk Size Calculator -The **Advanced Chunk Size Calculator** builds upon the **Basic Chunk Size Calculator**, requiring developers to have a deeper understanding of the services that make up the project. It is available for use [here](link to Google spreadsheet). +The **Advanced Chunk Size Calculator** builds upon the **Basic Chunk Size Calculator**, requiring developers to have a deeper understanding of the services that make up the project. It is available for use [here](link to Google spreadsheet); please make a copy of the spreadsheet to experiment with and adjust the settings as needed for your project. ##### Purpose and Use Case @@ -94,7 +94,7 @@ The **Expert Chunk Size Calculator** is designed for developers working with hea As the complexity of an entity increases, so does the denormalization time, which can slow down the entire system. Developers using the **Expert Chunk Size Calculator** must have a solid understanding of how containerization works in the project, including the distribution of resources like memory and CPU among containers, workers, and tasks. Additionally, understanding the limitations of the receiving side services (such as search and storage) and provider systems (like the database) is crucial. The message broker, which delivers messages and imposes throughput limits, is also a critical component of the overall system architecture that needs to be considered. -The **Expert Chunk Size Calculator** is available [here](link to google spreadsheet). +The **Expert Chunk Size Calculator** is available [here](link to google spreadsheet); please make a copy of the spreadsheet to experiment with and adjust the settings as needed for your project. ##### Purpose and Use Case From 15bd7f239570f77ce7b6f2b9c545baa391cce06d Mon Sep 17 00:00:00 2001 From: Karoly Gerner <30629375+gerner-spryker@users.noreply.github.com> Date: Wed, 30 Oct 2024 09:37:52 +0100 Subject: [PATCH 26/26] Update chunk-size-calculation.md --- .../data-manipulation/queue/chunk-size-calculation.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.md b/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.md index 5ea6dd929f1..befc140f122 100644 --- a/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.md +++ b/docs/dg/dev/backend-development/data-manipulation/queue/chunk-size-calculation.md @@ -29,7 +29,9 @@ The **Chunk Size Calculator** is a powerful tool for optimizing Publish and Sync #### 1. Basic Chunk Size Calculator -The **Basic Chunk Size Calculator** is a tool designed to help developers configure optimal queue chunk sizes for small to medium B2C projects with low customization with minimal configuration effort. It ensures efficient processing by aligning queue chunk sizes with traffic and data distribution. Ideal for out-of-the-box or minimally customized setups, it is available for use [here](link to Google spreadsheet); please make a copy of the spreadsheet to experiment with and adjust the settings as needed for your project. +The **Basic Chunk Size Calculator** is a tool designed to help developers configure optimal queue chunk sizes for small to medium B2C projects with low customization with minimal configuration effort. It ensures efficient processing by aligning queue chunk sizes with traffic and data distribution. Ideal for out-of-the-box or minimally customized setups. + +The **Basic Chunk Size Calculator** is available for use [here](link to Google spreadsheet); please make a copy of the spreadsheet to experiment with and adjust the settings as needed for your project. ##### Purpose and Use Case @@ -61,7 +63,9 @@ After entering the required data, the **Basic Chunk Size Calculator** will deter #### 2. Advanced Chunk Size Calculator -The **Advanced Chunk Size Calculator** builds upon the **Basic Chunk Size Calculator**, requiring developers to have a deeper understanding of the services that make up the project. It is available for use [here](link to Google spreadsheet); please make a copy of the spreadsheet to experiment with and adjust the settings as needed for your project. +The **Advanced Chunk Size Calculator** builds upon the **Basic Chunk Size Calculator**, requiring developers to have a deeper understanding of the services that make up the project. + +The **Advanced Chunk Size Calculator** is available for use [here](link to Google spreadsheet); please make a copy of the spreadsheet to experiment with and adjust the settings as needed for your project. ##### Purpose and Use Case