Skip to content

Commit c7634e4

Browse files
Copilotswathipil
andauthored
Add CLIENT_DEVELOPER.md for azure-schemaregistry with comprehensive developer guidance (#41491)
* Initial plan for issue * Add CLIENT_DEVELOPER.md for azure-schemaregistry with comprehensive developer guidance Co-authored-by: swathipil <[email protected]> * Update live testing instructions in CLIENT_DEVELOPER.md Co-authored-by: swathipil <[email protected]> * Address PR feedback: update text in CLIENT_DEVELOPER.md Co-authored-by: swathipil <[email protected]> * Simplify TypeSpec regeneration section to reference tsp-client docs Co-authored-by: swathipil <[email protected]> * Update CLIENT_DEVELOPER.md: simplify code structure section and remove REST API link Co-authored-by: swathipil <[email protected]> * Update Performance Tests README link to use full GitHub URL Co-authored-by: swathipil <[email protected]> --------- Co-authored-by: copilot-swe-agent[bot] <[email protected]> Co-authored-by: swathipil <[email protected]>
1 parent 52ab15b commit c7634e4

File tree

1 file changed

+239
-0
lines changed

1 file changed

+239
-0
lines changed
Lines changed: 239 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,239 @@
1+
# Azure Schema Registry Client Library Developer Guide
2+
3+
This guide is intended for developers contributing to the Azure Schema Registry Python client library. It provides information on setting up your development environment, regenerating the client from TypeSpec, understanding the code structure, running tests, and contributing to the codebase.
4+
5+
## Setting Up Development Environment
6+
7+
### Prerequisites
8+
9+
- Python version [supported by the client library](https://github.com/Azure/azure-sdk-for-python/wiki/Azure-SDKs-Python-version-support-policy)
10+
- Git
11+
- pip and setuptools
12+
- Node.js and npm (for TypeSpec client generation)
13+
- Azure subscription to create Schema Registry resources
14+
15+
### Setup
16+
17+
1. Clone the repository:
18+
```bash
19+
git clone https://github.com/Azure/azure-sdk-for-python.git
20+
cd azure-sdk-for-python/sdk/schemaregistry/azure-schemaregistry
21+
```
22+
23+
2. Create a virtual environment:
24+
```bash
25+
# Linux/macOS
26+
python -m venv .venv && source .venv/bin/activate && pip install -r dev_requirements.txt
27+
28+
# Windows PowerShell
29+
python -m venv .venv; .\.venv\Scripts\Activate.ps1; pip install -r dev_requirements.txt
30+
```
31+
32+
3. Install the package in development mode:
33+
```bash
34+
pip install -e .
35+
```
36+
37+
## Regenerating the Client from TypeSpec
38+
39+
The Azure Schema Registry client is generated from TypeSpec definitions. The TypeSpec files are maintained in the [Azure REST API Specs repository](https://github.com/Azure/azure-rest-api-specs/tree/main/specification/schemaregistry/SchemaRegistry).
40+
41+
### Prerequisites for Code Generation
42+
43+
1. Install the TypeSpec client generator:
44+
```bash
45+
npm install -g @azure-tools/typespec-client-generator-cli
46+
```
47+
48+
2. Ensure you have the latest TypeSpec definitions by checking the [specification directory](https://github.com/Azure/azure-rest-api-specs/tree/main/specification/schemaregistry/SchemaRegistry).
49+
50+
### Regeneration Process
51+
52+
The client regeneration is controlled by the `tsp-location.yaml` file in the root of this package directory. This file specifies:
53+
- The TypeSpec specification repository (`Azure/azure-rest-api-specs`)
54+
- The specific commit hash to use
55+
- The directory path (`specification/schemaregistry/SchemaRegistry`)
56+
57+
To regenerate the client, follow the [`tsp-client` docs](https://github.com/Azure/azure-sdk-tools/tree/main/tools/tsp-client/README.md).
58+
59+
## Code Structure
60+
61+
The Azure Schema Registry client library follows a hybrid approach with both generated and handwritten code:
62+
63+
### Generated Code
64+
65+
All files under `azure/` are automatically generated from TypeSpec and should **not** be manually edited, except for the handwritten code described below.
66+
67+
### Handwritten Code
68+
69+
The following contain handwritten customizations and extensions:
70+
71+
#### `_patch.py` Files
72+
Any files with the name `_patch.py` contain handwritten code that allows you to:
73+
- Add custom methods to generated classes
74+
- Override generated method implementations
75+
- Add custom models and enums
76+
- Extend functionality while preserving generated code
77+
78+
#### Encoder Implementation
79+
All files under `encoder/jsonencoder/` contain handwritten code that provides schema-based encoding and decoding capabilities using the Schema Registry client.
80+
81+
### Code Customization Guidelines
82+
83+
When adding handwritten code:
84+
1. Use `_patch.py` files for extending generated classes
85+
2. Follow the existing patterns for method signatures and documentation
86+
3. Import required types from generated modules
87+
4. Use the `patch_sdk()` function to register customizations
88+
5. Ensure async and sync implementations remain consistent
89+
90+
## Related Libraries
91+
92+
### Azure Schema Registry Avro Encoder
93+
94+
The [azure-schemaregistry-avroencoder](https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/schemaregistry/azure-schemaregistry-avroencoder) is an extension library that provides Avro-specific encoding and decoding capabilities. It builds on top of this Schema Registry client to provide:
95+
96+
- Avro schema-based serialization and deserialization
97+
- Integration with Apache Avro
98+
- Optimized performance for Avro payloads
99+
100+
For detailed usage and examples, see the [Avro Encoder README](https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/schemaregistry/azure-schemaregistry-avroencoder/README.md).
101+
102+
## Running Tests
103+
104+
### Live Tests
105+
106+
1. Login to Azure:
107+
```bash
108+
az login
109+
```
110+
111+
2. Set required environment variables (or create a `.env` file):
112+
113+
The following need to be set to run live tests locally:
114+
```
115+
SCHEMAREGISTRY_AVRO_FULLY_QUALIFIED_NAMESPACE
116+
SCHEMAREGISTRY_JSON_FULLY_QUALIFIED_NAMESPACE
117+
SCHEMAREGISTRY_CUSTOM_FULLY_QUALIFIED_NAMESPACE
118+
SCHEMAREGISTRY_GROUP
119+
AZURE_TEST_RUN_LIVE=true
120+
```
121+
* Each namespace is Standard tier. Each namespace has one schema group, with schema type set as per namespace.
122+
* The schema group name is the same across namespaces.
123+
124+
If using CLI:
125+
```
126+
AZURE_TEST_USE_CLI_AUTH=true
127+
```
128+
129+
OR
130+
131+
If using pwsh:
132+
```
133+
AZURE_TEST_USE_PWSH_AUTH=true
134+
```
135+
136+
Note: To run tests in playback mode instead of live mode, set:
137+
```
138+
AZURE_TEST_RUN_LIVE=false
139+
```
140+
3. Install test dependencies and the package in editable mode:
141+
```bash
142+
pip install -r dev_requirements.txt
143+
pip install -e .
144+
```
145+
146+
**Note**: If the azure-schemaregistry-avroencoder package is installed, you may encounter import errors due to issues identifying separate packages with shared namespace. To work around this, you will need to install the client library in non-editable mode:
147+
```bash
148+
pip install .
149+
```
150+
Note: You'll need to rerun this command after making changes to the package.
151+
152+
4. Run tests:
153+
```bash
154+
# Run all tests
155+
pytest tests
156+
157+
# Run specific test
158+
pytest tests/test_specific_test.py::test_specific_function
159+
```
160+
161+
5. Updating recordings:
162+
163+
To pull test recordings from assets repo:
164+
```bash
165+
python azure-sdk-for-python/scripts/manage_recordings.py restore -p sdk/schemaregistry/azure-schemaregistry/assets.json
166+
```
167+
168+
To push after recording tests in live mode:
169+
```bash
170+
python scripts/manage_recordings.py push -p sdk/schemaregistry/azure-schemaregistry/assets.json
171+
```
172+
173+
### Common Test Issues
174+
175+
- **Authentication errors**: Ensure Azure CLI is logged in (`az login`) or service principal credentials are correctly set
176+
- **Resource not found**: Verify that the Schema Registry namespace and group exist
177+
- **Permission errors**: Ensure the authentication principal has appropriate Schema Registry permissions
178+
179+
## Performance Testing
180+
181+
Performance tests are located in `tests/perfstress_tests/` and help measure client library performance under various conditions.
182+
183+
### Setup for Performance Tests
184+
185+
1. Install performance test dependencies:
186+
```bash
187+
pip install -r dev_requirements.txt
188+
pip install -e .
189+
```
190+
191+
2. Set environment variables:
192+
```bash
193+
SCHEMAREGISTRY_FULLY_QUALIFIED_NAMESPACE=<your-namespace>.servicebus.windows.net
194+
SCHEMAREGISTRY_GROUP=<your-schema-group>
195+
```
196+
197+
### Running Performance Tests
198+
199+
1. Navigate to the tests directory:
200+
```bash
201+
cd tests/
202+
```
203+
204+
2. List available performance tests:
205+
```bash
206+
perfstress
207+
```
208+
209+
3. Run specific performance tests:
210+
```bash
211+
# Example: Run schema registration test
212+
perfstress RegisterSchemaTest --duration=30 --parallel=5
213+
214+
# Example: Run schema retrieval test
215+
perfstress GetSchemaByIdTest --duration=30 --parallel=10 --num-schemas=100
216+
```
217+
218+
### Performance Test Options
219+
220+
Common options for all performance tests:
221+
- `--duration=30` - Number of seconds to run operations (default: 10)
222+
- `--iterations=1` - Number of test iterations (default: 1)
223+
- `--parallel=1` - Number of parallel operations (default: 1)
224+
- `--warm-up=5` - Warm-up time in seconds (default: 5)
225+
- `--sync` - Run synchronous tests instead of async (default: async)
226+
227+
Schema Registry specific options:
228+
- `--schema-size=150` - Size of each schema in bytes (default: 150)
229+
- `--num-schemas=10` - Number of schemas to use in tests (default: 10)
230+
231+
For detailed performance testing setup and resource requirements, see the [Performance Tests README](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/schemaregistry/azure-schemaregistry/tests/perfstress_tests/README.md).
232+
233+
## Additional Resources
234+
235+
- [Azure Schema Registry Documentation](https://docs.microsoft.com/azure/event-hubs/schema-registry-overview)
236+
- [Azure SDK Design Guidelines](https://azure.github.io/azure-sdk/python_design.html)
237+
- [TypeSpec Documentation](https://typespec.io/)
238+
- [Schema Registry Samples](https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/schemaregistry/azure-schemaregistry/samples)
239+
- [Event Hubs and Schema Registry Quickstart](https://learn.microsoft.com/azure/event-hubs/create-schema-registry)

0 commit comments

Comments
 (0)