|
| 1 | +# Manifest Migrations |
| 2 | + |
| 3 | +This directory contains the logic and registry for manifest migrations in the Airbyte CDK. Migrations are used to update or transform manifest components to newer formats or schemas as the CDK evolves. |
| 4 | + |
| 5 | +## Adding a New Migration |
| 6 | + |
| 7 | +1. **Create a Migration File:** |
| 8 | + - Add a new Python file in the `migrations/` subdirectory. |
| 9 | + - Name the file using the pattern: `<description>_v<major>_<minor>_<patch>__<order>.py`. |
| 10 | + - Example: `http_requester_url_base_to_url_v6_45_2__0.py` |
| 11 | + - The `<order>` integer is used to determine the order of migrations for the same version. |
| 12 | + |
| 13 | +2. **Define the Migration Class:** |
| 14 | + - The migration class must inherit from `ManifestMigration`. |
| 15 | + - Name the class using the pattern: `V_<major>_<minor>_<patch>_ManifestMigration_<Description>`. |
| 16 | + - Example: `V_6_45_2_ManifestMigration_HttpRequesterUrlBaseToUrl` |
| 17 | + - Implement the following methods: |
| 18 | + - `should_migrate(self, manifest: ManifestType) -> bool`: Return `True` if the migration should be applied to the given manifest. |
| 19 | + - `migrate(self, manifest: ManifestType) -> None`: Perform the migration in-place. |
| 20 | + |
| 21 | +3. **Migration Versioning:** |
| 22 | + - The migration version is extracted from the class name and used to determine applicability. |
| 23 | + - Only manifests with a version less than or equal to the migration version will be migrated. |
| 24 | + |
| 25 | +4. **Component Type:** |
| 26 | + - Use the `TYPE_TAG` constant to check the component type in your migration logic. |
| 27 | + |
| 28 | +5. **Examples:** |
| 29 | + - See `migrations/http_requester_url_base_to_url_v6_45_2__0.py` and `migrations/http_requester_path_to_url_v6_45_2__1.py` for reference implementations. |
| 30 | + |
| 31 | +## Migration Registry |
| 32 | + |
| 33 | +- All migration classes in the `migrations/` folder are automatically discovered and registered in `migrations_registry.py`. |
| 34 | +- Migrations are applied in order, determined by the `<order>` suffix in the filename. |
| 35 | + |
| 36 | +## Testing |
| 37 | + |
| 38 | +- Ensure your migration is covered by unit tests. |
| 39 | +- Tests should verify both `should_migrate` and `migrate` behaviors. |
| 40 | + |
| 41 | +## Example Migration Skeleton |
| 42 | + |
| 43 | +```python |
| 44 | +from airbyte_cdk.sources.declarative.migrations.manifest.manifest_migration import TYPE_TAG, ManifestMigration, ManifestType |
| 45 | + |
| 46 | +class V_1_2_3_ManifestMigration_Example(ManifestMigration): |
| 47 | + component_type = "ExampleComponent" |
| 48 | + original_key = "old_key" |
| 49 | + replacement_key = "new_key" |
| 50 | + |
| 51 | + def should_migrate(self, manifest: ManifestType) -> bool: |
| 52 | + return manifest[TYPE_TAG] == self.component_type and self.original_key in manifest |
| 53 | + |
| 54 | + def migrate(self, manifest: ManifestType) -> None: |
| 55 | + manifest[self.replacement_key] = manifest[self.original_key] |
| 56 | + manifest.pop(self.original_key, None) |
| 57 | +``` |
| 58 | + |
| 59 | +## Additional Notes |
| 60 | + |
| 61 | +- Do not modify the migration registry manually; it will pick up all valid migration classes automatically. |
| 62 | +- If you need to skip certain component types, use the `NON_MIGRATABLE_TYPES` list in `manifest_migration.py`. |
| 63 | + |
| 64 | +--- |
| 65 | + |
| 66 | +For more details, see the docstrings in `manifest_migration.py` and the examples in the `migrations/` folder. |
0 commit comments