Skip to content

Commit 560df9e

Browse files
dwdoughertydwdougherty
committed
Apply review comments.
1 parent 00fa3fa commit 560df9e

File tree

1 file changed

+122
-63
lines changed

1 file changed

+122
-63
lines changed

‎content/develop/interact/search-and-query/best-practices/index-mgmt-best-practices.md‎

Lines changed: 122 additions & 63 deletions
Original file line numberDiff line numberDiff line change
@@ -37,84 +37,143 @@ By following the strategies outlined in this guide, you can:
3737
- Scale efficiently with your growing datasets.
3838

3939
## Plan your indexes strategically
40-
- Understand your query patterns: before creating indexes, analyze your expected query patterns to ensure indexes are optimized for performance.
41-
- Avoid over-indexing: indexing every field increases memory usage and can slow down updates. Only index fields essential for your queries.
42-
- Choose appropriate index types: use the correct field types (`TEXT`, `TAG`, `NUMERIC`, `GEO`, or `VECTOR`) for your data to maximize efficiency.
40+
41+
Planning your indexes strategically requires understanding your application’s query patterns and tailoring indexes to match.
42+
Begin by identifying the types of searches your application performs—such as full-text search, range queries, or geospatial lookups—and the fields involved.
43+
Categorize fields based on their purpose: searchable fields (e.g., [`TEXT`]({{< relref "/develop/interact/search-and-query/basic-constructs/field-and-type-options#text-fields" >}}) for full-text searches), filterable fields (e.g., [`TAG`]({{< relref "/develop/interact/search-and-query/basic-constructs/field-and-type-options#tag-fields" >}}) for exact match searches), and sortable fields (e.g., [`NUMERIC`]({{< relref "/develop/interact/search-and-query/basic-constructs/field-and-type-options#numeric-fields" >}}) for range queries or sorting).
44+
Match field types to their intended use and avoid indexing fields that are rarely queried to conserve resources. Here's the list of index types:
45+
46+
- [`TEXT`]({{< relref "/develop/interact/search-and-query/basic-constructs/field-and-type-options#text-fields" >}}): use `TEXT` for free-text searches and set weights if some fields are more important.
47+
- [`TAG`]({{< relref "/develop/interact/search-and-query/basic-constructs/field-and-type-options#tag-fields" >}}): use `TAG` for categorical data (e.g., product categories) that benefit from exact matching and filtering.
48+
- [`NUMERIC`]({{< relref "/develop/interact/search-and-query/basic-constructs/field-and-type-options#numeric-fields" >}}): use `NUMERIC` for numeric ranges (e.g., prices, timestamps).
49+
- [`GEO`]({{< relref "/develop/interact/search-and-query/basic-constructs/field-and-type-options#geo-fields" >}}): use `GEO` for geospatial coordinates (e.g., latitude/longitude).
50+
- [`GEOSHAPE`]({{< relref "/develop/interact/search-and-query/basic-constructs/field-and-type-options#geoshape-fields" >}}): use `GEOSHAPE` to represent locations as points, but also to define shapes and query the interactions between points and shapes (e.g., to find all points that are contained within an enclosing shape).
51+
- [`VECTOR`]({{< relref "/develop/interact/search-and-query/basic-constructs/field-and-type-options#vector-fields" >}}): use `VECTOR` for high-dimensional similarity searches.
52+
53+
See [these pages]({{< relref "/develop/interact/search-and-query/query" >}}) for discussions and examples on how best to use these index types.
54+
55+
Next, simulate queries on a sample dataset to identify potential bottlenecks.
56+
Use tools like [`FT.PROFILE`]({{< baseurl >}}/commands/ft.profile) to analyze query execution and refine your schema if needed.
57+
For example, assign weights to `TEXT` fields for prioritizing results or use the `PREFIX` option of [`FT.CREATE`]({{< baseurl >}}/commands/ft.create) to limit indexing to specific key patterns. Note: you can use multiple `PREFIX` clauses when you create an index.
58+
After creating the index, validate its performance with real queries and monitor usage with the available tools:
59+
60+
- [`FT.EXPLAIN`]({{< baseurl >}}/commands/ft.explain) and [`FT.EXPLAINCLI`]({{< baseurl >}}/commands/ft.explaincli) allow you to see how Redis Query Engine parses a given search query. `FT.EXPLAIN` returns a structured breakdown of the query execution plan, while `FT.EXPLAINCLI` presents a more readable, tree-like format for easier interpretation. These commands are useful for diagnosing query structure and ensuring it aligns with the intended logic.
61+
- [`FT.INFO `]({{< baseurl >}}/commands/ft.info) provides detailed statistics about an index, including the number of indexed documents, memory usage, and configuration settings. It helps in monitoring index growth, assessing memory consumption, and verifying index structure to detect potential inefficiencies.
62+
- [`FT.PROFILE`]({{< baseurl >}}/commands/ft.profile) runs a query while capturing execution details, revealing query performance bottlenecks. It provides insights into processing time, key accesses, and filter application, making it a crucial tool for fine-tuning complex queries and optimizing search efficiency.
63+
64+
Avoid over-indexing. Indexing every field increases memory usage and can slow down updates.
65+
Only index the fields that are essential for your planned queries.
4366

4467
## Index creation
4568
- Use the [`FT.CREATE`]({{< baseurl >}}/commands/ft.create) command to define an index schema.
46-
- Field weighting: assign weights to `TEXT` fields to prioritize certain fields in full-text search results.
47-
- Prefix optimization: leverage the `PREFIX` option to restrict indexing to keys with specific patterns.
69+
- Assign weights to `TEXT` fields to prioritize certain fields in full-text search results.
70+
- Leverage the `PREFIX` option to restrict indexing to keys with specific patterns.
71+
Using multiple PREFIX clauses when creating an index allows you to index multiple key patterns under a single index. This is useful in several scenarios:
72+
- If your Redis database stores different types of entities under distinct key prefixes (e.g., `user:123`, `order:456`), a single index can cover both by specifying multiple prefixes. For example:
73+
74+
```bash
75+
FT.CREATE my_index ON HASH PREFIX 2 "user:" "order:" SCHEMA name TEXT age NUMERIC status TAG
76+
```
77+
78+
This approach enables searching across multiple entity types without needing separate indexes.
79+
80+
- Instead of querying multiple indexes separately, you can search across related data structures using a single query. This is particularly helpful when data structures share common fields, such as searching both customer and vendor records under a unified contacts index.
81+
82+
- Maintaining multiple indexes for similar data types can be inefficient in terms of memory and query performance. By consolidating data under one index with multiple prefixes, you reduce overhead while still allowing for distinct key organization.
83+
84+
- If your data model evolves and new key patterns are introduced, using multiple `PREFIX` clauses from the start ensures future compatibility without requiring a full reindexing.
4885
- Data loading strategy: load data into Redis before creating an index when working with large datasets. Use the `ON HASH` or `ON JSON` options to match the data structure.
4986

5087
## Index aliasing
51-
- What is index aliasing?
52-
- Aliases act as abstracted names for indexes, allowing applications to reference the alias instead of the actual index name. This simplifies schema updates and index management.
53-
- Use cases for index aliasing:
54-
- Seamless schema updates: point the alias to a new index without changing application code.
55-
- Version control: assign aliases like `products_live` to track the active index version.
56-
- Testing and rollback: temporarily assign aliases to test indexes, and revert if needed.
57-
- How to manage aliases:
58-
- Assign an alias: `FT.ALIASADD my_alias my_index`
59-
- Update an alias: `FT.ALIASUPDATE my_alias new_index`
60-
- Remove an alias: `FT.ALIASDEL my_alias`
61-
62-
## Monitoring index population
63-
- Check document count:
64-
- Use the `FT.INFO` command to monitor the `num_docs` field, to check that all expected documents are indexed.
65-
- Example:
66-
```bash
67-
FT.INFO my_new_index
68-
```
69-
- Run test queries:
70-
- Validate data with sample queries to ensure proper indexing:
71-
```bash
72-
FT.SEARCH my_new_index "*"
73-
```
74-
- Query profiling:
75-
- Use `FT.PROFILE` to analyze query plans and validate performance:
76-
```bash
77-
FT.PROFILE my_new_index SEARCH QUERY "your_query"
78-
```
79-
- Automate checks:
80-
- Implement scripts to periodically verify document counts and query results. For example, in Python:
81-
```python
82-
import redis
83-
84-
def check_index_readiness(index_name, expected_docs):
85-
r = redis.StrictRedis(host='localhost', port=6379, decode_responses=True)
86-
info = r.execute_command('FT.INFO', index_name)
87-
num_docs = int(info[info.index('num_docs') + 1])
88-
return num_docs >= expected_docs
89-
90-
if check_index_readiness('my_new_index', 100000):
91-
print("Index is fully populated!")
92-
else:
93-
print("Index is still populating...")
94-
```
88+
89+
Index aliasing allows you to assign an alias to an index. Aliases act as abstracted names for the underlying indexes, enabling applications to reference the alias instead of the actual index name. This approach simplifies schema updates and index management.
90+
91+
There are several use cases for index aliasing, including:
92+
93+
- Schema updates: when updating an index schema, create a new index and associate the same alias with it. This allows a seamless transition without requiring application-level changes.
94+
- Version control: use aliases to manage different versions of an index. For example, assign the alias products to `products_v1` initially and later to `products_v2` when the schema evolves.
95+
- Testing and rollback: assign an alias to a test index during staged deployments. If issues arise, quickly switch the alias back to the stable index.
96+
97+
Best practices for aliasing:
98+
99+
- Default aliases: always create an alias for your indexes during initial setup, even if you don’t anticipate immediate schema changes.
100+
- Consistent naming conventions: use clear and descriptive alias names to avoid confusion (e.g., `users_current` or `orders_live`).
101+
- Avoid overlapping aliases: ensure that an alias points to only one index at a time to maintain predictable query results.
102+
- Use aliases to provide tenant-specific access. For example, assign tenant-specific aliases like `tenant1_products` and `tenant2_products` to different indexes for isolated query performance.
103+
104+
Tools for managing aliases:
105+
106+
- Assign an alias: [`FT.ALIASADD`]({{< baseurl >}}/commands/ft.aliasadd) `my_alias my_index`
107+
- Update an alias: [`FT.ALIASUPDATE`]({{< baseurl >}}/commands/ft.aliasupdate) `my_alias new_index`
108+
- Remove an alias: [`FT.ALIASDEL`]({{< baseurl >}}/commands/ft.aliasdel) `my_alias`
109+
110+
Monitoring and troubleshooting aliases:
111+
112+
- Use the `FT.INFO` command to check which aliases are associated with an index.
113+
- Make sure your aliases always points to valid indexes and are correctly updated during schema changes.
114+
115+
## Monitor index population
116+
117+
- Use the `FT.INFO` command to monitor the `num_docs` and `indexing` fields, to check that all expected documents are indexed.
118+
```bash
119+
FT.INFO my_new_index
120+
```
121+
- Validate data with sample queries to ensure proper indexing:
122+
```bash
123+
FT.SEARCH my_new_index "*"
124+
```
125+
- Use `FT.PROFILE` to analyze query plans and validate performance:
126+
127+
```bash
128+
FT.PROFILE my_new_index SEARCH QUERY "your_query"
129+
```
130+
- Implement scripts to periodically verify document counts and query results. For example, in Python:
131+
132+
```python
133+
import re
134+
def check_index_readiness(index_name, expected_docs):
135+
r = redis.StrictRedis(host='localhost', port=6379, decode_responses=True)
136+
info = r.execute_command('FT.INFO', index_name)
137+
num_docs = int(info[info.index('num_docs') + 1])
138+
return num_docs >= expected_d
139+
if check_index_readiness('my_new_index', 100000):
140+
print("Index is fully populated!")
141+
else:
142+
print("Index is still populating...")
143+
```
95144
96145
## Monitoring index performance
97-
- Query profiling: use the `FT.PROFILE` command to analyze query performance and identify bottlenecks.
98-
- Memory usage: regularly monitor memory usage with the `INFO memory` and `FT.INFO` commands to detect growth patterns and optimize resource allocation.
99-
- Search query logs: enable query logging for better insights into how indexes are utilized.
146+
147+
- Use the `FT.PROFILE` command to analyze query performance and identify bottlenecks.
148+
- Regularly monitor memory usage with the [`INFO`]({{< baseurl >}}/commands/info) `memory` and `FT.INFO` commands to detect growth patterns and optimize resource allocation.
100149
101150
## Index maintenance
102-
- Reindexing: if schema changes are required, create a new index with the updated schema and reassign the alias once the index is ready.
103-
- Expire old data: use Redis key expiration or TTLs to automatically remove outdated records and keep indexes lean.
151+
152+
- If schema changes are required, create a new index with the updated schema and reassign the alias once the index is ready.
153+
- Use [Redis key expiration]({{< relref "/develop/use/keyspace#key-expiration" >}}) to automatically remove outdated records and keep indexes lean.
154+
155+
When is it appropriate to use [`FT.ALTER`]({{< baseurl >}}/commands/ft.alter) and when is it best to use aliasing?
156+
157+
Use `FT.ALTER` when you need to add new fields to an existing index without rebuilding it, minimizing downtime and resource usage. However, `FT.ALTER` cannot remove or modify existing fields, limiting its flexibility.
158+
159+
Use index aliasing when making schema changes that require reindexing, such as modifying field types or removing fields. In this case, create a new index with the updated schema, populate it, and then use `FT.ALIASUPDATE` to seamlessly switch queries to the new index without disrupting application functionality.
104160
105161
## Scaling and high availability
106-
- Sharding considerations: in a clustered Redis setup, ensure indexes are designed with key distribution in mind to prevent query inefficiencies.
107-
- Replication: test how indexes behave under replica promotion to ensure consistent query behavior across nodes.
108-
- Active-Active support: if using Redis in an active-active setup, validate how index updates propagate to avoid inconsistencies.
162+
163+
- In a clustered Redis setup, make sure indexes are designed with key distribution in mind to prevent query inefficiencies.
164+
- Test how indexes behave under replica promotion to ensure consistent query behavior across nodes.
109165
110166
## Versioning and testing
111-
- Index versioning: when changing schemas, create a new version of the index alongside the old one and migrate data progressively.
112-
- Staging environment: test index changes in a staging environment before deploying them to production.
167+
168+
- When changing schemas, create a new version of the index alongside the old one and migrate data progressively.
169+
- Test index changes in a staging environment before deploying them to production.
113170
114171
## Cleaning up
115-
- Index deletion: use the `FT.DROPINDEX` command to remove unused indexes and free up memory. Be cautious with the `DD` (Delete Documents) flag to avoid unintended data deletion.
116-
- Monitoring orphaned keys: Ensure no keys remain that were previously associated with dropped indexes.
172+
173+
- Use the [`FT.DROPINDEX`]({{< baseurl >}}/commands/ft.dropindex) command to remove unused indexes and free up memory. Be cautious with the `DD` (Delete Documents) flag to avoid unintended data deletion.
174+
- Make sure no keys remain that were previously associated with dropped indexes if the data is no longer relevant.
117175
118176
## Documentation and automation
119-
- Maintain clear index schemas: document your index configurations to facilitate future maintenance.
120-
- Automate index management: use scripts or orchestration tools to automate index creation, monitoring, and cleanup.
177+
178+
- Document your index configurations to facilitate future maintenance.
179+
- Use scripts or orchestration tools to automate index creation, monitoring, and cleanup.

0 commit comments

Comments
 (0)