|
| 1 | +# Azure Schema Registry Client Library Developer Guide |
| 2 | + |
| 3 | +This guide is intended for developers contributing to the Azure Schema Registry Python client library. It provides information on setting up your development environment, regenerating the client from TypeSpec, understanding the code structure, running tests, and contributing to the codebase. |
| 4 | + |
| 5 | +## Setting Up Development Environment |
| 6 | + |
| 7 | +### Prerequisites |
| 8 | + |
| 9 | +- Python version [supported by the client library](https://github.com/Azure/azure-sdk-for-python/wiki/Azure-SDKs-Python-version-support-policy) |
| 10 | +- Git |
| 11 | +- pip and setuptools |
| 12 | +- Node.js and npm (for TypeSpec client generation) |
| 13 | +- Azure subscription to create Schema Registry resources |
| 14 | + |
| 15 | +### Setup |
| 16 | + |
| 17 | +1. Clone the repository: |
| 18 | + ```bash |
| 19 | + git clone https://github.com/Azure/azure-sdk-for-python.git |
| 20 | + cd azure-sdk-for-python/sdk/schemaregistry/azure-schemaregistry |
| 21 | + ``` |
| 22 | + |
| 23 | +2. Create a virtual environment: |
| 24 | + ```bash |
| 25 | + # Linux/macOS |
| 26 | + python -m venv .venv && source .venv/bin/activate && pip install -r dev_requirements.txt |
| 27 | + |
| 28 | + # Windows PowerShell |
| 29 | + python -m venv .venv; .\.venv\Scripts\Activate.ps1; pip install -r dev_requirements.txt |
| 30 | + ``` |
| 31 | + |
| 32 | +3. Install the package in development mode: |
| 33 | + ```bash |
| 34 | + pip install -e . |
| 35 | + ``` |
| 36 | + |
| 37 | +## Regenerating the Client from TypeSpec |
| 38 | + |
| 39 | +The Azure Schema Registry client is generated from TypeSpec definitions. The TypeSpec files are maintained in the [Azure REST API Specs repository](https://github.com/Azure/azure-rest-api-specs/tree/main/specification/schemaregistry/SchemaRegistry). |
| 40 | + |
| 41 | +### Prerequisites for Code Generation |
| 42 | + |
| 43 | +1. Install the TypeSpec client generator: |
| 44 | + ```bash |
| 45 | + npm install -g @azure-tools/typespec-client-generator-cli |
| 46 | + ``` |
| 47 | + |
| 48 | +2. Ensure you have the latest TypeSpec definitions by checking the [specification directory](https://github.com/Azure/azure-rest-api-specs/tree/main/specification/schemaregistry/SchemaRegistry). |
| 49 | + |
| 50 | +### Regeneration Process |
| 51 | + |
| 52 | +The client regeneration is controlled by the `tsp-location.yaml` file in the root of this package directory. This file specifies: |
| 53 | +- The TypeSpec specification repository (`Azure/azure-rest-api-specs`) |
| 54 | +- The specific commit hash to use |
| 55 | +- The directory path (`specification/schemaregistry/SchemaRegistry`) |
| 56 | + |
| 57 | +To regenerate the client, follow the [`tsp-client` docs](https://github.com/Azure/azure-sdk-tools/tree/main/tools/tsp-client/README.md). |
| 58 | + |
| 59 | +## Code Structure |
| 60 | + |
| 61 | +The Azure Schema Registry client library follows a hybrid approach with both generated and handwritten code: |
| 62 | + |
| 63 | +### Generated Code |
| 64 | + |
| 65 | +All files under `azure/` are automatically generated from TypeSpec and should **not** be manually edited, except for the handwritten code described below. |
| 66 | + |
| 67 | +### Handwritten Code |
| 68 | + |
| 69 | +The following contain handwritten customizations and extensions: |
| 70 | + |
| 71 | +#### `_patch.py` Files |
| 72 | +Any files with the name `_patch.py` contain handwritten code that allows you to: |
| 73 | +- Add custom methods to generated classes |
| 74 | +- Override generated method implementations |
| 75 | +- Add custom models and enums |
| 76 | +- Extend functionality while preserving generated code |
| 77 | + |
| 78 | +#### Encoder Implementation |
| 79 | +All files under `encoder/jsonencoder/` contain handwritten code that provides schema-based encoding and decoding capabilities using the Schema Registry client. |
| 80 | + |
| 81 | +### Code Customization Guidelines |
| 82 | + |
| 83 | +When adding handwritten code: |
| 84 | +1. Use `_patch.py` files for extending generated classes |
| 85 | +2. Follow the existing patterns for method signatures and documentation |
| 86 | +3. Import required types from generated modules |
| 87 | +4. Use the `patch_sdk()` function to register customizations |
| 88 | +5. Ensure async and sync implementations remain consistent |
| 89 | + |
| 90 | +## Related Libraries |
| 91 | + |
| 92 | +### Azure Schema Registry Avro Encoder |
| 93 | + |
| 94 | +The [azure-schemaregistry-avroencoder](https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/schemaregistry/azure-schemaregistry-avroencoder) is an extension library that provides Avro-specific encoding and decoding capabilities. It builds on top of this Schema Registry client to provide: |
| 95 | + |
| 96 | +- Avro schema-based serialization and deserialization |
| 97 | +- Integration with Apache Avro |
| 98 | +- Optimized performance for Avro payloads |
| 99 | + |
| 100 | +For detailed usage and examples, see the [Avro Encoder README](https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/schemaregistry/azure-schemaregistry-avroencoder/README.md). |
| 101 | + |
| 102 | +## Running Tests |
| 103 | + |
| 104 | +### Live Tests |
| 105 | + |
| 106 | +1. Login to Azure: |
| 107 | + ```bash |
| 108 | + az login |
| 109 | + ``` |
| 110 | + |
| 111 | +2. Set required environment variables (or create a `.env` file): |
| 112 | + |
| 113 | + The following need to be set to run live tests locally: |
| 114 | + ``` |
| 115 | + SCHEMAREGISTRY_AVRO_FULLY_QUALIFIED_NAMESPACE |
| 116 | + SCHEMAREGISTRY_JSON_FULLY_QUALIFIED_NAMESPACE |
| 117 | + SCHEMAREGISTRY_CUSTOM_FULLY_QUALIFIED_NAMESPACE |
| 118 | + SCHEMAREGISTRY_GROUP |
| 119 | + AZURE_TEST_RUN_LIVE=true |
| 120 | + ``` |
| 121 | + * Each namespace is Standard tier. Each namespace has one schema group, with schema type set as per namespace. |
| 122 | + * The schema group name is the same across namespaces. |
| 123 | + |
| 124 | + If using CLI: |
| 125 | + ``` |
| 126 | + AZURE_TEST_USE_CLI_AUTH=true |
| 127 | + ``` |
| 128 | + |
| 129 | + OR |
| 130 | + |
| 131 | + If using pwsh: |
| 132 | + ``` |
| 133 | + AZURE_TEST_USE_PWSH_AUTH=true |
| 134 | + ``` |
| 135 | + |
| 136 | + Note: To run tests in playback mode instead of live mode, set: |
| 137 | + ``` |
| 138 | + AZURE_TEST_RUN_LIVE=false |
| 139 | + ``` |
| 140 | +3. Install test dependencies and the package in editable mode: |
| 141 | + ```bash |
| 142 | + pip install -r dev_requirements.txt |
| 143 | + pip install -e . |
| 144 | + ``` |
| 145 | + |
| 146 | + **Note**: If the azure-schemaregistry-avroencoder package is installed, you may encounter import errors due to issues identifying separate packages with shared namespace. To work around this, you will need to install the client library in non-editable mode: |
| 147 | + ```bash |
| 148 | + pip install . |
| 149 | + ``` |
| 150 | + Note: You'll need to rerun this command after making changes to the package. |
| 151 | + |
| 152 | +4. Run tests: |
| 153 | + ```bash |
| 154 | + # Run all tests |
| 155 | + pytest tests |
| 156 | + |
| 157 | + # Run specific test |
| 158 | + pytest tests/test_specific_test.py::test_specific_function |
| 159 | + ``` |
| 160 | + |
| 161 | +5. Updating recordings: |
| 162 | + |
| 163 | + To pull test recordings from assets repo: |
| 164 | + ```bash |
| 165 | + python azure-sdk-for-python/scripts/manage_recordings.py restore -p sdk/schemaregistry/azure-schemaregistry/assets.json |
| 166 | + ``` |
| 167 | + |
| 168 | + To push after recording tests in live mode: |
| 169 | + ```bash |
| 170 | + python scripts/manage_recordings.py push -p sdk/schemaregistry/azure-schemaregistry/assets.json |
| 171 | + ``` |
| 172 | + |
| 173 | +### Common Test Issues |
| 174 | + |
| 175 | +- **Authentication errors**: Ensure Azure CLI is logged in (`az login`) or service principal credentials are correctly set |
| 176 | +- **Resource not found**: Verify that the Schema Registry namespace and group exist |
| 177 | +- **Permission errors**: Ensure the authentication principal has appropriate Schema Registry permissions |
| 178 | + |
| 179 | +## Performance Testing |
| 180 | + |
| 181 | +Performance tests are located in `tests/perfstress_tests/` and help measure client library performance under various conditions. |
| 182 | + |
| 183 | +### Setup for Performance Tests |
| 184 | + |
| 185 | +1. Install performance test dependencies: |
| 186 | + ```bash |
| 187 | + pip install -r dev_requirements.txt |
| 188 | + pip install -e . |
| 189 | + ``` |
| 190 | + |
| 191 | +2. Set environment variables: |
| 192 | + ```bash |
| 193 | + SCHEMAREGISTRY_FULLY_QUALIFIED_NAMESPACE=<your-namespace>.servicebus.windows.net |
| 194 | + SCHEMAREGISTRY_GROUP=<your-schema-group> |
| 195 | + ``` |
| 196 | + |
| 197 | +### Running Performance Tests |
| 198 | + |
| 199 | +1. Navigate to the tests directory: |
| 200 | + ```bash |
| 201 | + cd tests/ |
| 202 | + ``` |
| 203 | + |
| 204 | +2. List available performance tests: |
| 205 | + ```bash |
| 206 | + perfstress |
| 207 | + ``` |
| 208 | + |
| 209 | +3. Run specific performance tests: |
| 210 | + ```bash |
| 211 | + # Example: Run schema registration test |
| 212 | + perfstress RegisterSchemaTest --duration=30 --parallel=5 |
| 213 | + |
| 214 | + # Example: Run schema retrieval test |
| 215 | + perfstress GetSchemaByIdTest --duration=30 --parallel=10 --num-schemas=100 |
| 216 | + ``` |
| 217 | + |
| 218 | +### Performance Test Options |
| 219 | + |
| 220 | +Common options for all performance tests: |
| 221 | +- `--duration=30` - Number of seconds to run operations (default: 10) |
| 222 | +- `--iterations=1` - Number of test iterations (default: 1) |
| 223 | +- `--parallel=1` - Number of parallel operations (default: 1) |
| 224 | +- `--warm-up=5` - Warm-up time in seconds (default: 5) |
| 225 | +- `--sync` - Run synchronous tests instead of async (default: async) |
| 226 | + |
| 227 | +Schema Registry specific options: |
| 228 | +- `--schema-size=150` - Size of each schema in bytes (default: 150) |
| 229 | +- `--num-schemas=10` - Number of schemas to use in tests (default: 10) |
| 230 | + |
| 231 | +For detailed performance testing setup and resource requirements, see the [Performance Tests README](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/schemaregistry/azure-schemaregistry/tests/perfstress_tests/README.md). |
| 232 | + |
| 233 | +## Additional Resources |
| 234 | + |
| 235 | +- [Azure Schema Registry Documentation](https://docs.microsoft.com/azure/event-hubs/schema-registry-overview) |
| 236 | +- [Azure SDK Design Guidelines](https://azure.github.io/azure-sdk/python_design.html) |
| 237 | +- [TypeSpec Documentation](https://typespec.io/) |
| 238 | +- [Schema Registry Samples](https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/schemaregistry/azure-schemaregistry/samples) |
| 239 | +- [Event Hubs and Schema Registry Quickstart](https://learn.microsoft.com/azure/event-hubs/create-schema-registry) |
0 commit comments