Skip to content
This repository was archived by the owner on Apr 29, 2019. It is now read-only.

Commit c6aabe3

Browse files
committed
MAGETWO-83347: Benchmark updates
1 parent 0483421 commit c6aabe3

File tree

2 files changed

+33785
-23938
lines changed

2 files changed

+33785
-23938
lines changed

setup/performance-toolkit/README.md

Lines changed: 58 additions & 67 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,8 @@ The Performance Toolkit enables you to test the performance of your Magento inst
1919

2020
## Quick Start
2121

22-
Before running the JMeter tests for the first time, you will need to first use the `php bin/magento setup:performance:generate-fixtures {profile path}` command to generate the test data. You can find the configuration files of available profiles in the folders `setup/performance-toolkit/profiles/ce` and `setup/performance-toolkit/profiles/ee`.
22+
Before running the JMeter tests for the first time, you will need to first use the `php bin/magento setup:performance:generate-fixtures {profile path}` command to generate the test data.
23+
You can find the configuration files of available B2C profiles in the folders `setup/performance-toolkit/profiles/ce` and `setup/performance-toolkit/profiles/ee`.
2324

2425
It can take a significant amount of time to generate a profile. For example, generating the large profile can take up to 4 hours. So we recommend using the `-s` option to skip indexation. Then you can start indexation manually.
2526

@@ -55,9 +56,9 @@ The following parameters can be passed to the `benchmark.jmx` scenario:
5556

5657
| Parameter Name | Default Value | Description |
5758
| --------------------------------------------- | ------------------- | ---------------------------------------------------------------------------------------- |
58-
| host | | URL component 'host' of application being tested (URL or IP). |
59-
| base_path | | Base path for tested site. |
60-
| admin_path | backend | Admin backend path. |
59+
| host | localhost | URL component 'host' of application being tested (URL or IP). |
60+
| base_path | / | Base path for tested site. |
61+
| admin_path | admin | Admin backend path. |
6162
| admin_user | admin | Admin backend user. |
6263
| admin_password | 123123q | Admin backend password. |
6364
| customer_password | 123123q | Storefront customer password. |
@@ -67,39 +68,37 @@ The following parameters can be passed to the `benchmark.jmx` scenario:
6768
| frontendPoolUsers | 1 | Total number of Frontend threads. |
6869
| adminPoolUsers | 1 | Total number of Admin threads. |
6970
| browseCatalogByGuestPercentage | 30 | Percentage of threads in Frontend Pool that emulate catalog browsing activities. |
70-
| searchQuickPercentage | 18 | Percentage of threads in Frontend Pool that emulate catalog search activities. |
71-
| searchQuickFilterPercentage | 9 | Percentage of threads in Frontend Pool that emulate catalog search activities. |
72-
| searchAdvancedPercentage | 3 | Percentage of threads in Frontend Pool that emulate catalog search activities. |
71+
| browseCatalogByCustomerPercentage | 0 | Percentage of threads in Frontend Pool that emulate catalog browsing activities. |
72+
| siteSearchPercentage | 30 | Percentage of threads in Frontend Pool that emulate catalog search activities. |
73+
| searchQuickPercentage | 60 | Percentage of threads in Frontend Pool that emulate catalog search activities. |
74+
| searchQuickFilterPercentage | 30 | Percentage of threads in Frontend Pool that emulate catalog search activities. |
75+
| searchAdvancedPercentage | 10 | Percentage of threads in Frontend Pool that emulate catalog search activities. |
7376
| checkoutByGuestPercentage | 4 | Percentage of threads in Frontend Pool that emulate checkout by guest. |
7477
| checkoutByCustomerPercentage | 4 | Percentage of threads in Frontend Pool that emulate checkout by customer. |
75-
| addToCartPercentage | 28 | Percentage of threads in Frontend Pool that emulate abandoned cart activities. |
78+
| addToCartByGuestPercentage | 28 | Percentage of threads in Frontend Pool that emulate abandoned cart activities. |
7679
| addToWishlistPercentage | 2 | Percentage of threads in Frontend Pool that emulate adding products to Wishlist. |
7780
| compareProductsPercentage | 2 | Percentage of threads in Frontend Pool that emulate products comparison. |
7881
| productCompareDelay | 0 | Delay (s) between iterations of product comparison. |
7982
| promotionRulesPercentage | 10 | Percentage of threads in Admin Pool that emulate creation of promotion rules. |
8083
| adminPromotionsManagementDelay | 0 | Delay (s) between creation of promotion rules. |
81-
| adminCategoryManagementPercentage | 5 | Percentage of threads in Merchandising Pool that emulate category management activities. |
82-
| adminProductEditingPercentage | 27 | Percentage of threads in Merchandising Pool that emulate product editing. |
83-
| adminProductCreationPercentage | 18 | Percentage of threads in Merchandising Pool that emulate creation of products. |
84-
| adminPromotionRulesPercentage | 10 | Percentage of threads in Admin Pool that emulate admin rules creating activities. |
84+
| adminCategoryManagementPercentage | 10 | Percentage of threads in Merchandising Pool that emulate category management activities. |
85+
| adminProductEditingPercentage | 35 | Percentage of threads in Merchandising Pool that emulate product editing. |
86+
| adminProductCreationPercentage | 25 | Percentage of threads in Merchandising Pool that emulate creation of products. |
87+
| adminPromotionRulesPercentage | 15 | Percentage of threads in Admin Pool that emulate admin rules creating activities. |
8588
| adminCategoryManagementDelay | 0 | Delay (s) between iterations of category management activities. |
86-
| apiProcessOrdersPercentage | 30 | Percentage of threads in Admin Pool that emulate orders processing activities. |
8789
| apiProcessOrders | 5 | Number of orders for process in Admin API - Process Orders. |
88-
| adminEditOrderPercentage | 10 | Percentage of threads in Admin Pool that emulate order edit. |
89-
| csrPoolUsers | 0 | Users of CSR Pool. |
90-
| csrBrowseCustomersOddPercentage | 2 | Percentage of threads in CSR Pool that emulate customers browsing activities. |
91-
| csrBrowseCustomersEvenPercentage | 2 | Percentage of threads in CSR Pool that emulate customers browsing activities. |
92-
| csrBrowseCustomersFilteringPercentage | 2 | Percentage of threads in CSR Pool that emulate customers browsing activities. |
93-
| csrBrowseCustomersSortingPercentage | 2 | Percentage of threads in CSR Pool that emulate customers browsing activities. |
94-
| csrBrowseCustomersFilteringSortingPercentage | 2 | Percentage of threads in CSR Pool that emulate customers browsing activities. |
95-
| csrCreateOrderPercentage | 70 | Percentage of threads in CSR Pool that emulate creation of orders. |
96-
| csrCreateProcessReturnsPercentage | 20 | Percentage of threads in CSR Pool that emulate creation/processing of returns. |
97-
| csrCreateProcessReturnsDelay | 0 | Delay (s) between creation of returns. |
90+
| adminEditOrderPercentage | 15 | Percentage of threads in Admin Pool that emulate order edit. |
91+
| csrPoolUsers | 0 | Users of Customer Support Request (CSR) Pool. |
92+
| othersPoolUsers | 0 | Users of Others Pool. |
93+
| browseCustomerGridPercentage | 10 | Percentage of threads in CSR Pool that emulate customers browsing activities. |
94+
| adminCreateOrderPercentage | 70 | Percentage of threads in CSR Pool that emulate creation of orders. |
95+
| adminReturnsManagementPercentage | 20 | Percentage of threads in CSR Pool that emulate creation/processing of returns. |
96+
| adminCreateProcessReturnsDelay | 0 | Delay (s) between creation of returns. |
9897
| wishlistDelay | 0 | Delay (s) between adding products to Wishlist. |
99-
| categories_count | 200 | Total number of categories that are be used in scenario. |
98+
| categories_count | 100 | Total number of categories that are be used in scenario. |
10099
| simple_products_count | 30 | Total number of simple products that are be used in scenario. |
101100

102-
Parameters must be passed to command line with the `J` prefix:
101+
Parameters must be passed to the command line with the `J` prefix:
103102

104103
`-J{parameter_name}={parameter_value}`
105104

@@ -114,12 +113,12 @@ There are some options that you should pass to JMeter in the console mode:
114113

115114
To get more details about available JMeter options, read [Non-GUI Mode](http://jmeter.apache.org/usermanual/get-started.html#non_gui).
116115

117-
For example, you can run a scenario via console with 100 threads for 5 minutes as follows:
116+
For example, you can run the B2C scenario via console with 90 threads for the Frontend Pool and 10 threads for the Admin Pool:
118117

119118
cd {JMeter path}/bin/
120-
jmeter -n -t {path to peformance toolkit}/benchmark.jmx -j ./jmeter.log -l ./jmeter-results.jtl -Jhost=magento2.dev -Jbase_path=/ -Jadmin_path=admin -Jfiles_folder = /opt/mpaf/tool/fragments/files/
119+
jmeter -n -t {path to peformance toolkit}/benchmark.jmx -j ./jmeter.log -l ./jmeter-results.jtl -Jhost=magento2.dev -Jbase_path=/ -Jadmin_path=admin -JfrontendPoolUsers=90 -JadminPoolUsers=10
121120

122-
As a result, you will get `jmeter.log` and `jmeter-results.jtl`. The`jmeter.log` contains information about the test run and can be helpful in determining the cause of an error. The JTL file is a text file containing the results of a test run. It can be opened in GUI mode to perform analysis of the results (see the *Output* section below).
121+
As a result, you will get `jmeter.log` and `jmeter-results.jtl`. The`jmeter.log` contains information about the test run and can be helpful in determining the cause of an error. The JTL file is a text file containing the results of a test run. It can be opened in the GUI mode to perform analysis of the results (see the *Output* section below).
123122

124123

125124
The following parameters can be passed to the `benchmark_2015.jmx` scenario:
@@ -145,7 +144,7 @@ The following parameters can be passed to the `benchmark_2015.jmx` scenario:
145144

146145
### Run JMeter scenario via GUI
147146

148-
**Note:** Use the GUI mode only for scenario debugging and viewing reports. Use the console mode for real-life load testing, because it requires significantly fewer resources.
147+
**Note:** Use the GUI mode only for scenario debugging and viewing reports. Use console mode for real-life load testing, because it requires significantly fewer resources.
149148

150149
- Change directories to `{JMeter path}/bin/` and run `jmeter.bat`.
151150
- Click *File -> Open (Ctrl+O)* and select `benchmark.jmx` file or drag and drop the `benchmark.jmx` file in the opened GUI.
@@ -189,57 +188,49 @@ For more details, read [Summary Report](http://jmeter.apache.org/usermanual/comp
189188

190189
## Additional Information
191190

192-
### Threads
191+
### Scenarios
193192

194-
`benchmark.jmx` scenario has the following thread groups and default percentage breakdown:
193+
`benchmark.jmx` scenario has the following pools and default percentage breakdown for each scenario:
195194

196-
- **Frontend Pool** (frontendPoolUsers)
195+
**Frontend Pool** (frontendPoolUsers)
197196

198-
| Thread Group Name | Label Suffix | % of Pool |
199-
| ------------------------- | ------------------------------------------------------ | --------- |
200-
| Catalog Browsing By Guest | Catalog Browsing By Guest | 30 |
201-
| Site Search | SearchQuick, SearchQuickWithFilter and SearchAdvanced | 30 |
202-
| Add To Cart (Guest) | Add To Cart By Guest | 28 |
203-
| Add to Wishlist | WishList | 2 |
204-
| Compare Products | Product Compare By Guest | 2 |
205-
| Checkout By Guest | Checkout By Guest | 4 |
206-
| Checkout By Customer | Checkout By Customer | 4 |
197+
| Scenario Name | % of Pool |
198+
| ------------------------- | --------- |
199+
| Catalog Browsing By Guest | 30 |
200+
| Site Search | 30 |
201+
| Add To Cart By Guest | 28 |
202+
| Add to Wishlist | 2 |
203+
| Compare Products | 2 |
204+
| Checkout By Guest | 4 |
205+
| Checkout By Customer | 4 |
207206

208207
Site Search thread group contains 3 variations:
209208
- Quick Search (60%)
210-
- Quick Search With Filtering (30%)
209+
- Quick Search With Filtration (30%)
211210
- Advanced Search (10%)
212211

213-
- **Admin Pool** (adminPoolUsers)
212+
**Admin Pool** (adminPoolUsers)
214213

215-
| Thread Group Name | Label Suffix | % of Pool |
216-
| ----------------------------------------- | ------------ | --------- |
217-
| Admin Promotion Rules | - | 10 |
218-
| Admin API - Process Orders | - | 30 |
219-
| Admin Edit Order | - | 10 |
220-
| Admin Category Management (Merchandising) | - | 5 |
221-
| Admin Edit Product (Merchandising) | - | 27 |
222-
| Admin Create Product (Merchandising) | - | 18 |
214+
| Scenario Name |% of Pool |
215+
| ----------------------------| --------- |
216+
| Admin Promotion Rules | 15 |
217+
| Admin Edit Order | 15 |
218+
| Admin Category Management | 10 |
219+
| Admin Edit Product | 35 |
220+
| Admin Create Product | 25 |
223221

224-
- **CSR Pool** (csrPoolUsers)
222+
**CSR Pool** (csrPoolUsers)
225223

226-
| Thread Group Name | Label Suffix | % of Pool |
227-
| -------------------------- | ------------ | --------- |
228-
| CSR Browse Customers | - | 10 |
229-
| CSR Create Order | - | 70 |
230-
| CSR Create/Process Returns | - | 20 |
231-
232-
CSR Browse Customers thread group contains 5 variations:
233-
- View Odd Grid Pages (20%)
234-
- View Even Grid pages (20%)
235-
- Filtering (20%)
236-
- Sorting (20%)
237-
- Filtering and Sorting (20%)
224+
| Scenario Name | % of Pool |
225+
| -------------------------- | --------- |
226+
| Browse Customer Grid | 10 |
227+
| Admin Create Order | 70 |
228+
| Admin Returns Management | 20 |
238229

239230
**Legacy Threads**
240231

241232
The `benchmark_2015.jmx` script consists of five thread groups: the setup thread and four user threads.
242-
By default, the percentage ratio between thread groups is as follows:
233+
By default, the percentage ratio between the thread groups is as follows:
243234
- Browsing, adding items to the cart and abandon cart (BrowsAddToCart suffix in reports) - 62%
244235
- Just browsing (CatProdBrows suffix in reports) - 30%
245236
- Browsing, adding items to cart and checkout as guest (GuestChkt suffix in reports) - 4%
@@ -249,7 +240,7 @@ By default, the percentage ratio between thread groups is as follows:
249240

250241
In order to build an aggregate report from the results of the `benchmark.kmx` scenario run, use the script `generate-b2c.php` in the folder `setup/performance-toolkit/aggregate-report`.
251242

252-
The script parses the JTL file and generates an aggregate report in CSV format. The report consists of the 4 sections separated by two empty lines:
243+
The script parses the JTL file and generates an aggregate report in CSV format. The report consists of the following four sections separated by two empty lines:
253244

254245
1. Summary information: Checkouts Per Hour, Page Views Per Hour and Test Duration (in seconds)
255246
2. Aggregated information about all requests within each thread group (median time, average time, min/max, amount of hits per hour, etc.)
@@ -296,4 +287,4 @@ Find all rows with the label *Simple Product # View* and calculate the summary t
296287

297288
If you need information about the summary throughput of the *Checkout* actions, find the rows with labels *Checkout success* and make the same calculation.
298289

299-
For the total number of page views, sum up all actions, minus the setup thread.
290+
For the total number of page views, you will want to sum up all actions minus the setup thread.

0 commit comments

Comments
 (0)