Skip to content

Commit fa86a58

Browse files
update READMEs with new information and refs to java stress test doc (Azure#47323)
* update READMEs with new information and refs to java stress test doc * remove section on logging, fix spelling errors * add comment about internal links for stress test framekwork
1 parent 8629acc commit fa86a58

File tree

3 files changed

+102
-220
lines changed

3 files changed

+102
-220
lines changed

sdk/storage/azure-storage-blob-stress/README.md

Lines changed: 33 additions & 73 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,10 @@
11
# Azure Blob Storage Stress Tests
22

3-
Represents stress tests for Azure Storage Blob client library.
3+
Represents stress tests for Azure Storage Blob client library.
4+
5+
<blockquote style="padding:10px; border-left:2px dashed #ccc;">
6+
The instructions are for internal Azure SDK developers to run stress tests in the stress test infrastructure.
7+
</blockquote>
48

59
## Running tests in stress infra
610

@@ -18,12 +22,18 @@ Check out [Azure SDK Stress Test Wiki][azure_sdk_stress_test] for general inform
1822

1923
### Deploy Stress Test
2024

21-
cd into `azure-sdk-for-java` root folder and run command to deploy the package to cluster:
25+
cd into `azure-sdk-for-java` root folder and run command to deploy all packages to the cluster:
2226

2327
```shell
2428
.\eng\common\scripts\stress-testing\deploy-stress-tests.ps1 -SearchDirectory .\sdk\storage
2529
```
2630

31+
Run the following command, if you only want to deploy this package to the cluster:
32+
33+
```shell
34+
.\eng\common\scripts\stress-testing\deploy-stress-tests.ps1 -SearchDirectory .\sdk\storage\azure-storage-blob-stress
35+
```
36+
2737
### Check Status
2838

2939
Only the most frequently used commands are listed below. See [Deploying A Stress Test][deploy_stress_test] for more details.
@@ -34,7 +44,7 @@ List deployed packages:
3444
helm list -n <stress test namespace>
3545
```
3646

37-
the namespace usually matches your username.
47+
The namespace usually matches your username and `java` for auto-deployed tests.
3848

3949
Get stress test pods and status:
4050

@@ -70,28 +80,13 @@ helm uninstall <stress test name> -n <stress test namespace>
7080

7181
### Fault injection
7282

83+
For details on using the HTTP fault injector and how it is wired into the Java stress tests, see the [Java Stress Testing documentation](https://msazure.visualstudio.com/One/_wiki/wikis/One.wiki/697419/Java-Stress-Testing-Documentation) and the [HTTP fault-injector documentation][http-fault-injector].
84+
7385
## Running tests locally
7486

7587
You can also run stress tests locally with or without fault-injection.
76-
To run test locally:
77-
1. Build `azure-storage-blob-stress` jar file. For example, you can do it with:
78-
```powershell
79-
cd .\sdk\storage
80-
mvn clean install -pl .\azure-storage-stress\,.\azure-storage-blob-stress
81-
```
82-
As a result, you should have fat jar in `azure-storage-blob-stress\target\azure-storage-blob-stress-1.0.0-beta.1-jar-with-dependencies.jar`
83-
2. Configure environment:
84-
- Set `STORAGE_CONNECTION_STRING`environment variable (connection string to storage account). You may also provide it with `--cs` command line option
85-
- Configure Application Insights:
86-
- Set `APPLICATIONINSIGHTS_CONNECTION_STRING` (connection string to application insights). You may also provide it in any other
87-
way mentioned in the [Enable Application Insights steps][enable_application_insights]
88-
- Set `APPLICATIONINSIGHTS_ROLE_NAME` to `storage-{ random string }` (for example, `storage-foobar`).
89-
3. Run the test, for example, with:
90-
```powershell
91-
java -javaagent:path\to\applicationinsights-agent.jar `
92-
-jar .\azure-storage-blob-stress\target\azure-storage-blob-stress-1.0.0-beta.1-jar-with-dependencies.jar `
93-
downloadtofile --duration 10 --size 1024 --parallel 2 --warmup 0
94-
```
88+
To run test locally from the command line, refer to the Java stress testing documentation:
89+
- [Java Stress Testing Documentation](https://msazure.visualstudio.com/One/_wiki/wikis/One.wiki/697419/Java-Stress-Testing-Documentation)
9590

9691
### Running locally with fault-injection
9792

@@ -127,36 +122,24 @@ To get a super-quick idea about test results, look for a log record that looks l
127122
"succeeded":468,"failed":0}
128123
```
129124

130-
After the stress test is deployed on the cluster, we can monitor the progress on the Application Insights resource inside the stress test resource group.
125+
After the stress test is deployed on the cluster, we can monitor the progress in the Log Analytics Workspace resource inside the stress test resource group.
131126
There are several dashboards within the stress test resource group that we can use to monitor the AKS pod and stress test status.
132127

133128
#### Stress Test Dashboard
134129

135-
General-purpose stress test dashboard is available at https://aka.ms/azsdk/stress/dashboard. It shows:
136-
- Pod status events
137-
- CPU and memory utilization of the stress test pods
138-
- Container logs and events
139-
140-
Stress test dashboard does not know about local stress test runs.
141-
142-
#### Application Insights
143-
144-
Application Insights agent brings rich monitoring experience including:
145-
- resource utilization metrics (CPU, memory, GC, threads, etc.)
146-
- live metrics, performance overview, etc
147-
- distributed tracing and dependency calls (HTTP, Azure SDK calls)
148-
- exceptions and logs
149-
- profiling in production
130+
General-purpose stress test dashboard is available at [Stress Test Dashboard][stress-test-dashboard]. It shows:
150131

151-
Application Insights is useful to:
152-
- monitor and compare throughput and latency across runs
153-
- investigate issues and find bottlenecks
132+
- Stress test pods per namespace
133+
- Successful runs
134+
- Duration
135+
- Container status
136+
- The ability to query details on each of the pods
154137

155-
Application Insights is available for local runs (as long as you provide `-javaagent` option and make sure connection string is configured).
138+
Stress test dashboard does not know about local stress test runs.
156139

157-
##### Stress test workbook
140+
##### Azure Monitor
158141

159-
Storage stress test workbook is available [here][storage-workbook] and allows to pick a specific run and see it's summary:
142+
Storage stress test workbook is available [here][azure-monitor] and allows to pick a specific run and see its summary:
160143
1. Key test parameters
161144
2. Throughput and latency charts
162145
3. Failed operations including their status and fault injected (if any)
@@ -167,9 +150,7 @@ The workbook relies on tests to emit:
167150
- certain logs in a specific format produced by [TelemetryHelper][telemetry-helper] class
168151
- certain spans produced in [BlobScenarioBase][blob-scenario-base] class.
169152
- report cloud role name that follows `storage-{runId}` pattern.
170-
Make sure to set `APPLICATIONINSIGHTS_ROLE_NAME` environment variable accordingly to make sure run appears on the dashboard.
171-
172-
_Note: some failures are expected and there is no clear 'success' criteria for the stress test_
153+
Make sure to set `APPLICATIONINSIGHTS_ROLE_NAME` environment variable accordingly to make sure run appears on the dashboard.
173154

174155
Here are a few things that clearly indicate an issue:
175156
- Content mismatch - they can be detected in following ways:
@@ -178,35 +159,13 @@ Here are a few things that clearly indicate an issue:
178159
- Throughput is noticeably lower than for previous runs with the similar parameters. This would normally correlate with latency being higher than usual.
179160
- CPU/memory consumption is noticeably higher than for previous runs with the similar parameters
180161

181-
### Logging
182-
183-
We use [logback.xml][logback_xml] to configure the logging. By default, the stress test run on cluster will output
184-
`INFO` level log which you may adjust based on your needs.
185-
186-
The [storage workbook](#stress-test-workbook) needs `com.azure.storage.blob.stress` and `com.azure.storage.stress` to stay at the `INFO` level.
187-
188-
You may also control the verbosity of logs that go to Application Insights - see [Application Insights logging configuration][application-insights-logging] for more details.
189-
190-
See also [Logging in Azure SDK][logging-azure-sdk] for more details.
191-
192162
## Key concepts
193163

194164
### Project Structure
195165

196-
See [Layout][stress_test_layout] section for details.
166+
See [Layout][stress_test_layout] section for details as described by the Azure SDK team.
197167

198-
Below is the current structure of project:
199-
```
200-
.
201-
├── src/ # Test code
202-
├── templates/ # A directory of helm templates that will generate Kubernetes manifest files.
203-
├── Chart.yaml # A YAML file containing information about the helm chart and its dependencies
204-
├── scenarios-matrix.yaml # A YAML file containing configuration and custom values for stress test(s)
205-
├── Dockerfile # A Dockerfile for building the stress test image
206-
├── stress-test-resources.bicep # An Azure Bicep for deploying stress test azure resources
207-
├── pom.xml
208-
└── README.md
209-
```
168+
To understand the project structure, you may also refer to the [Java Stress Testing Documentation](https://msazure.visualstudio.com/One/_wiki/wikis/One.wiki/697419/Java-Stress-Testing-Documentation?anchor=how-it-works).
210169

211170
<!-- links -->
212171
[azure_sdk_stress_test]: https://aka.ms/azsdk/stress
@@ -222,8 +181,9 @@ Below is the current structure of project:
222181
[deploy_stress_test]: https://github.com/Azure/azure-sdk-tools/blob/main/tools/stress-cluster/chaos/README.md#deploying-a-stress-test
223182
[stress_test_layout]: https://github.com/Azure/azure-sdk-tools/blob/main/tools/stress-cluster/chaos/README.md#layout
224183
[http-fault-injector]: https://github.com/Azure/azure-sdk-tools/tree/main/tools/http-fault-injector
225-
[telemtery-helper]: https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/storage/azure-storage-stress/src/main/java/com/azure/storage/stress/TelemetryHelper.java
184+
[telemetry-helper]: https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/storage/azure-storage-stress/src/main/java/com/azure/storage/stress/TelemetryHelper.java
226185
[blob-scenario-base]: https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/storage/azure-storage-blob-stress/src/main/java/com/azure/storage/blob/stress/BlobScenarioBase.java
227-
[storage-workbook]: https://ms.portal.azure.com/#blade/AppInsightsExtension/UsageNotebookBlade/ComponentId/%2Fsubscriptions%2Ffaa080af-c1d8-40ad-9cce-e1a450ca5b57%2FresourceGroups%2Frg-stress-cluster-pg%2Fproviders%2FMicrosoft.Insights%2Fcomponents%2Fstress-pg-ai-s7b6dif73rup6/ConfigurationId/%2Fsubscriptions%2Ffaa080af-c1d8-40ad-9cce-e1a450ca5b57%2Fresourcegroups%2Frg-stress-cluster-pg%2Fproviders%2Fmicrosoft.insights%2Fworkbooks%2Fa6fc3414-4c15-4651-8517-6f74cbe0d0fe/Type/workbook/WorkbookTemplateName/Storage%20stress%20tests
228186
[application-insights-logging]: https://learn.microsoft.com/azure/azure-monitor/app/java-standalone-config#autocollected-logging
229187
[logging-azure-sdk]: https://github.com/Azure/azure-sdk-for-java/wiki/Logging-in-Azure-SDK
188+
[stress-test-dashboard]: https://portal.azure.com/#@TME01.onmicrosoft.com/resource/subscriptions/4d042dc6-fe17-4698-a23f-ec6a8d1e98f4/resourceGroups/rg-stress-cluster-storage/providers/microsoft.insights/workbooks/01d047ea-1c0d-4463-98fa-4f465596401e/workbook
189+
[azure-monitor]: https://portal.azure.com/#view/AppInsightsExtension/WorkbookViewerBlade/ComponentId/azure%20monitor/ConfigurationId/%2Fsubscriptions%2F4d042dc6-fe17-4698-a23f-ec6a8d1e98f4%2FresourceGroups%2Frg-stress-cluster-storage%2Fproviders%2FMicrosoft.Insights%2Fworkbooks%2F1acdb3e6-3135-5dff-b808-85def43fb868/WorkbookTemplateName/Azure%20SDK%20Stress%20Testing%20-%20storage/NotebookParams~/%7B%22TimeRange%22%3A%7B%22durationMs%22%3A604800000%7D%2C%22NamespaceParameter%22%3A%5B%22java%22%5D%2C%22PodUidParameter%22%3A%5B%2267d0dcea-82f1-4992-bc3d-b7ee199aaca3%22%5D%7D

0 commit comments

Comments
 (0)