|
| 1 | +--- |
| 2 | +myst: |
| 3 | + html_meta: |
| 4 | + "description lang=en": | |
| 5 | + Learn how RedisVL index migrations work and which schema changes are supported. |
| 6 | +--- |
| 7 | + |
| 8 | +# Index Migrations |
| 9 | + |
| 10 | +Redis Search indexes are immutable. To change an index schema, you must drop the existing index and create a new one. RedisVL provides a migration workflow that automates this process while preserving your data. |
| 11 | + |
| 12 | +This page explains how migrations work and which changes are supported. For step by step instructions, see the [migration guide](../user_guide/how_to_guides/migrate-indexes.md). |
| 13 | + |
| 14 | +## Supported and blocked changes |
| 15 | + |
| 16 | +The migrator classifies schema changes into two categories: |
| 17 | + |
| 18 | +| Change | Status | |
| 19 | +|--------|--------| |
| 20 | +| Add or remove a field | Supported | |
| 21 | +| Change field options (sortable, separator) | Supported | |
| 22 | +| Change vector algorithm (FLAT, HNSW, SVS-VAMANA) | Supported | |
| 23 | +| Change distance metric (COSINE, L2, IP) | Supported | |
| 24 | +| Tune algorithm parameters (M, EF_CONSTRUCTION) | Supported | |
| 25 | +| Quantize vectors (float32 to float16) | Supported | |
| 26 | +| Change vector dimensions | Blocked | |
| 27 | +| Change key prefix | Blocked | |
| 28 | +| Rename a field | Blocked | |
| 29 | +| Change storage type (hash to JSON) | Blocked | |
| 30 | +| Add a new vector field | Blocked | |
| 31 | + |
| 32 | +**Supported** changes can be applied automatically using `rvl migrate`. The migrator handles the index rebuild and any necessary data transformations. |
| 33 | + |
| 34 | +**Blocked** changes require manual intervention because they involve incompatible data formats or missing data. The migrator will reject these changes and explain why. |
| 35 | + |
| 36 | +## How the migrator works |
| 37 | + |
| 38 | +The migrator uses a plan first workflow: |
| 39 | + |
| 40 | +1. **Plan**: Capture the current schema, classify your changes, and generate a migration plan |
| 41 | +2. **Review**: Inspect the plan before making any changes |
| 42 | +3. **Apply**: Drop the index, transform data if needed, and recreate with the new schema |
| 43 | +4. **Validate**: Verify the result matches expectations |
| 44 | + |
| 45 | +This separation ensures you always know what will happen before any changes are made. |
| 46 | + |
| 47 | +## Migration mode: drop_recreate |
| 48 | + |
| 49 | +The `drop_recreate` mode rebuilds the index in place while preserving your documents. |
| 50 | + |
| 51 | +The process: |
| 52 | + |
| 53 | +1. Drop only the index structure (documents remain in Redis) |
| 54 | +2. For datatype changes, re-encode vectors to the target precision |
| 55 | +3. Recreate the index with the new schema |
| 56 | +4. Wait for Redis to re-index the existing documents |
| 57 | +5. Validate the result |
| 58 | + |
| 59 | +**Tradeoff**: The index is unavailable during the rebuild. The migrator requires explicit acknowledgment of this downtime before proceeding. |
| 60 | + |
| 61 | +## Index only vs document dependent changes |
| 62 | + |
| 63 | +Schema changes fall into two categories based on whether they require modifying stored data. |
| 64 | + |
| 65 | +**Index only changes** affect how Redis Search indexes data, not the data itself: |
| 66 | + |
| 67 | +- Algorithm changes: The stored vector bytes are identical. Only the index structure differs. |
| 68 | +- Distance metric changes: Same vectors, different similarity calculation. |
| 69 | +- Adding or removing fields: The documents already contain the data. The index just starts or stops indexing it. |
| 70 | + |
| 71 | +These changes complete quickly because they only require rebuilding the index. |
| 72 | + |
| 73 | +**Document dependent changes** require modifying the stored data: |
| 74 | + |
| 75 | +- Datatype changes (float32 to float16): Stored vector bytes must be re-encoded. |
| 76 | +- Field renames: Stored field names must be updated in every document. |
| 77 | +- Dimension changes: Vectors must be re-embedded with a different model. |
| 78 | + |
| 79 | +The migrator handles datatype changes automatically. Other document dependent changes are blocked because they require application level logic or external services. |
| 80 | + |
| 81 | +## Vector quantization |
| 82 | + |
| 83 | +Changing vector precision from float32 to float16 reduces memory usage at the cost of slight precision loss. The migrator handles this automatically by: |
| 84 | + |
| 85 | +1. Reading all vectors from Redis |
| 86 | +2. Converting to the target precision |
| 87 | +3. Writing updated vectors back |
| 88 | +4. Recreating the index with the new schema |
| 89 | + |
| 90 | +Typical reductions: |
| 91 | + |
| 92 | +| Metric | Value | |
| 93 | +|--------|-------| |
| 94 | +| Index size reduction | ~50% | |
| 95 | +| Memory reduction | ~35% | |
| 96 | + |
| 97 | +Quantization time is proportional to document count. Plan for downtime accordingly. |
| 98 | + |
| 99 | +## Why some changes are blocked |
| 100 | + |
| 101 | +### Vector dimension changes |
| 102 | + |
| 103 | +Vector dimensions are determined by your embedding model. A 384 dimensional vector from one model is mathematically incompatible with a 768 dimensional index expecting vectors from a different model. There is no way to resize an embedding. |
| 104 | + |
| 105 | +**Resolution**: Re-embed your documents using the new model and load them into a new index. |
| 106 | + |
| 107 | +### Prefix changes |
| 108 | + |
| 109 | +Changing a prefix from `docs:` to `articles:` requires copying every document to a new key. This operation doubles storage temporarily and can leave orphaned keys if interrupted. |
| 110 | + |
| 111 | +**Resolution**: Create a new index with the new prefix and reload your data. |
| 112 | + |
| 113 | +### Field renames |
| 114 | + |
| 115 | +Field names are stored in the documents themselves as hash field names or JSON keys. Renaming requires iterating through every document and updating the field name. |
| 116 | + |
| 117 | +**Resolution**: Create a new index with the correct field name and reload your data. |
| 118 | + |
| 119 | +### Storage type changes |
| 120 | + |
| 121 | +Hash and JSON have different data layouts. Hash stores flat key value pairs. JSON stores nested structures. Converting between them requires understanding your schema and restructuring each document. |
| 122 | + |
| 123 | +**Resolution**: Export your data, transform it to the new format, and reload into a new index. |
| 124 | + |
| 125 | +### Adding a vector field |
| 126 | + |
| 127 | +Adding a vector field means all existing documents need vectors for that field. The migrator cannot generate these vectors because it does not know which embedding model to use or what content to embed. |
| 128 | + |
| 129 | +**Resolution**: Add vectors to your documents using your application, then run the migration. |
| 130 | + |
| 131 | +## Downtime considerations |
| 132 | + |
| 133 | +With `drop_recreate`, your index is unavailable between the drop and when re-indexing completes. Plan for: |
| 134 | + |
| 135 | +- Search unavailability during the migration window |
| 136 | +- Partial results while indexing is in progress |
| 137 | +- Resource usage from the re-indexing process |
| 138 | +- Quantization time if changing vector datatypes |
| 139 | + |
| 140 | +The duration depends on document count, field count, and vector dimensions. For large indexes, consider running migrations during low traffic periods. |
| 141 | + |
| 142 | +## Learn more |
| 143 | + |
| 144 | +- [Migration guide](../user_guide/how_to_guides/migrate-indexes.md): Step by step instructions |
| 145 | +- [Search and indexing](search-and-indexing.md): How Redis Search indexes work |
0 commit comments