Skip to content

Commit f47cd8a

Browse files
fh-msfh-ms
authored
Refactor VectorIndexConfiguration by removing redundant flags (#544)
* Refactor `VectorIndexConfiguration`: remove redundant background persistence and optimization flags. * Update docs
1 parent e4ffcc3 commit f47cd8a

File tree

8 files changed

+119
-212
lines changed

8 files changed

+119
-212
lines changed

‎docs/modules/gigamap/pages/indexing/jvector/advanced.adoc‎

Lines changed: 4 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -36,14 +36,12 @@ VectorIndexConfiguration config = VectorIndexConfiguration.builder()
3636
// On-disk storage
3737
.onDisk(true)
3838
.indexDirectory(Path.of("/data/vectors"))
39-
// Background persistence (async, non-blocking)
40-
.backgroundPersistence(true)
41-
.persistenceIntervalMs(30_000) // Check every 30 seconds
39+
// Background persistence (async, non-blocking, enabled by setting interval > 0)
40+
.persistenceIntervalMs(30_000) // Enable, check every 30 seconds
4241
.minChangesBetweenPersists(100) // Only persist if >= 100 changes
4342
.persistOnShutdown(true) // Persist on close()
44-
// Background optimization (periodic cleanup)
45-
.backgroundOptimization(true)
46-
.optimizationIntervalMs(60_000) // Check every 60 seconds
43+
// Background optimization (periodic cleanup, enabled by setting interval > 0)
44+
.optimizationIntervalMs(60_000) // Enable, check every 60 seconds
4745
.minChangesBetweenOptimizations(1000) // Only optimize if >= 1000 changes
4846
.optimizeOnShutdown(false) // Skip for faster shutdown
4947
.build();

‎docs/modules/gigamap/pages/indexing/jvector/configuration.adoc‎

Lines changed: 8 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -87,18 +87,15 @@ VectorIndexConfiguration config = VectorIndexConfiguration.builder()
8787
== Background Persistence
8888

8989
Enable automatic asynchronous persistence to avoid blocking operations during writes.
90+
Setting `persistenceIntervalMs` to a value greater than 0 enables background persistence.
9091

9192
[options="header",cols="1,1,3"]
9293
|===
9394
|Parameter |Default |Description
9495

95-
|`backgroundPersistence`
96-
|`false`
97-
|Enable automatic background persistence.
98-
9996
|`persistenceIntervalMs`
100-
|`30000`
101-
|Check interval in milliseconds.
97+
|`0`
98+
|Check interval in milliseconds. A value > 0 enables background persistence, 0 disables it.
10299

103100
|`minChangesBetweenPersists`
104101
|`100`
@@ -118,8 +115,7 @@ VectorIndexConfiguration config = VectorIndexConfiguration.builder()
118115
.similarityFunction(VectorSimilarityFunction.COSINE)
119116
.onDisk(true)
120117
.indexDirectory(Path.of("/data/vectors"))
121-
.backgroundPersistence(true)
122-
.persistenceIntervalMs(30_000) // Check every 30 seconds
118+
.persistenceIntervalMs(30_000) // Enable, check every 30 seconds
123119
.minChangesBetweenPersists(100) // Only persist if >= 100 changes
124120
.persistOnShutdown(true) // Persist on close()
125121
.build();
@@ -128,18 +124,15 @@ VectorIndexConfiguration config = VectorIndexConfiguration.builder()
128124
== Background Optimization
129125

130126
Enable periodic graph optimization to maintain query performance as the index grows.
127+
Setting `optimizationIntervalMs` to a value greater than 0 enables background optimization.
131128

132129
[options="header",cols="1,1,3"]
133130
|===
134131
|Parameter |Default |Description
135132

136-
|`backgroundOptimization`
137-
|`false`
138-
|Enable automatic graph optimization.
139-
140133
|`optimizationIntervalMs`
141-
|`60000`
142-
|Check interval in milliseconds.
134+
|`0`
135+
|Check interval in milliseconds. A value > 0 enables background optimization, 0 disables it.
143136

144137
|`minChangesBetweenOptimizations`
145138
|`1000`
@@ -159,8 +152,7 @@ VectorIndexConfiguration config = VectorIndexConfiguration.builder()
159152
.similarityFunction(VectorSimilarityFunction.COSINE)
160153
.onDisk(true)
161154
.indexDirectory(Path.of("/data/vectors"))
162-
.backgroundOptimization(true)
163-
.optimizationIntervalMs(60_000) // Check every 60 seconds
155+
.optimizationIntervalMs(60_000) // Enable, check every 60 seconds
164156
.minChangesBetweenOptimizations(1000) // Only optimize if >= 1000 changes
165157
.optimizeOnShutdown(false) // Skip for faster shutdown
166158
.build();

‎docs/modules/gigamap/pages/indexing/jvector/use-cases.adoc‎

Lines changed: 0 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -146,7 +146,6 @@ VectorIndexConfiguration config = VectorIndexConfiguration.builder()
146146
.beamWidth(200)
147147
.onDisk(true)
148148
.indexDirectory(Path.of("/data/product-embeddings"))
149-
.backgroundPersistence(true)
150149
.persistenceIntervalMs(60_000)
151150
.build();
152151
@@ -220,7 +219,6 @@ VectorIndexConfiguration config = VectorIndexConfiguration.builder()
220219
.indexDirectory(Path.of("/data/customer-embeddings"))
221220
.enablePqCompression(true) // Large customer bases benefit from compression
222221
.pqSubspaces(32)
223-
.backgroundPersistence(true)
224222
.persistenceIntervalMs(30_000)
225223
.build();
226224
@@ -293,7 +291,6 @@ VectorIndexConfiguration config = VectorIndexConfiguration.builder()
293291
.beamWidth(300)
294292
.onDisk(true)
295293
.indexDirectory(Path.of("/data/ticket-embeddings"))
296-
.backgroundPersistence(true)
297294
.persistenceIntervalMs(10_000)
298295
.minChangesBetweenPersists(5)
299296
.build();
@@ -354,7 +351,6 @@ VectorIndexConfiguration config = VectorIndexConfiguration.builder()
354351
.beamWidth(200)
355352
.onDisk(true)
356353
.indexDirectory(Path.of("/data/content-embeddings"))
357-
.backgroundOptimization(true)
358354
.optimizationIntervalMs(300_000)
359355
.build();
360356
@@ -414,7 +410,6 @@ VectorIndexConfiguration config = VectorIndexConfiguration.builder()
414410
.beamWidth(200)
415411
.onDisk(true)
416412
.indexDirectory(Path.of("/data/semantic-index"))
417-
.backgroundPersistence(true)
418413
.persistenceIntervalMs(60_000)
419414
.minChangesBetweenPersists(50)
420415
.build();
@@ -452,10 +447,8 @@ VectorIndexConfiguration config = VectorIndexConfiguration.builder()
452447
.indexDirectory(Path.of("/data/rag-index"))
453448
.enablePqCompression(true)
454449
.pqSubspaces(64) // 768 / 64 = 12 dims per subspace
455-
.backgroundPersistence(true)
456450
.persistenceIntervalMs(30_000)
457451
.minChangesBetweenPersists(100)
458-
.backgroundOptimization(true)
459452
.optimizationIntervalMs(300_000) // Every 5 minutes
460453
.minChangesBetweenOptimizations(500)
461454
.build();
@@ -491,7 +484,6 @@ VectorIndexConfiguration config = VectorIndexConfiguration.builder()
491484
.beamWidth(100)
492485
.onDisk(true)
493486
.indexDirectory(Path.of("/data/recommendations"))
494-
.backgroundPersistence(true)
495487
.persistenceIntervalMs(10_000) // Frequent persistence
496488
.minChangesBetweenPersists(20)
497489
.persistOnShutdown(true)
@@ -530,10 +522,8 @@ VectorIndexConfiguration config = VectorIndexConfiguration.builder()
530522
.indexDirectory(Path.of("/data/large-scale"))
531523
.enablePqCompression(true)
532524
.pqSubspaces(48) // 768 / 48 = 16 dims per subspace
533-
.backgroundPersistence(true)
534525
.persistenceIntervalMs(60_000)
535526
.minChangesBetweenPersists(1000)
536-
.backgroundOptimization(true)
537527
.optimizationIntervalMs(600_000) // Every 10 minutes
538528
.minChangesBetweenOptimizations(5000)
539529
.optimizeOnShutdown(false) // Faster shutdown
@@ -571,7 +561,6 @@ VectorIndexConfiguration config = VectorIndexConfiguration.builder()
571561
.neighborOverflow(1.5f)
572562
.onDisk(true)
573563
.indexDirectory(Path.of("/data/precision-index"))
574-
.backgroundOptimization(true)
575564
.optimizationIntervalMs(120_000)
576565
.minChangesBetweenOptimizations(200)
577566
.build();
@@ -612,7 +601,6 @@ VectorIndexConfiguration config = VectorIndexConfiguration.builder()
612601
.indexDirectory(Path.of("/data/image-index"))
613602
.enablePqCompression(true)
614603
.pqSubspaces(32) // 512 / 32 = 16 dims per subspace
615-
.backgroundPersistence(true)
616604
.persistenceIntervalMs(30_000)
617605
.minChangesBetweenPersists(100)
618606
.build();
@@ -705,7 +693,6 @@ VectorIndexConfiguration config = VectorIndexConfiguration.builder()
705693
.beamWidth(150)
706694
.onDisk(true)
707695
.indexDirectory(Path.of("/data/sensor-data"))
708-
.backgroundPersistence(true)
709696
.persistenceIntervalMs(5_000) // Frequent for streaming data
710697
.minChangesBetweenPersists(10)
711698
.build();
@@ -738,7 +725,6 @@ VectorIndexConfiguration config = VectorIndexConfiguration.builder()
738725
.beamWidth(150)
739726
.onDisk(true)
740727
.indexDirectory(Path.of("/data/mips-index"))
741-
.backgroundPersistence(true)
742728
.persistenceIntervalMs(30_000)
743729
.build();
744730
----

‎gigamap/jvector/README.md‎

Lines changed: 6 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -90,17 +90,15 @@ List<Document> topDocs = result.stream()
9090

9191
| Parameter | Default | Description |
9292
|-----------|---------|-------------|
93-
| `backgroundPersistence` | `false` | Enable automatic background persistence |
94-
| `persistenceIntervalMs` | `30000` | Check interval in milliseconds |
93+
| `persistenceIntervalMs` | `0` | Check interval in milliseconds. A value > 0 enables background persistence, 0 disables it |
9594
| `minChangesBetweenPersists` | `100` | Minimum changes before persisting |
9695
| `persistOnShutdown` | `true` | Persist pending changes on close() |
9796

9897
### Background Optimization
9998

10099
| Parameter | Default | Description |
101100
|-----------|---------|-------------|
102-
| `backgroundOptimization` | `false` | Enable automatic graph optimization |
103-
| `optimizationIntervalMs` | `60000` | Check interval in milliseconds |
101+
| `optimizationIntervalMs` | `0` | Check interval in milliseconds. A value > 0 enables background optimization, 0 disables it |
104102
| `minChangesBetweenOptimizations` | `1000` | Minimum changes before optimizing |
105103
| `optimizeOnShutdown` | `false` | Optimize pending changes on close() |
106104

@@ -136,14 +134,12 @@ VectorIndexConfiguration config = VectorIndexConfiguration.builder()
136134
// On-disk storage
137135
.onDisk(true)
138136
.indexDirectory(Path.of("/data/vectors"))
139-
// Background persistence (async, non-blocking)
140-
.backgroundPersistence(true)
141-
.persistenceIntervalMs(30_000) // Check every 30 seconds
137+
// Background persistence (enabled by setting interval > 0)
138+
.persistenceIntervalMs(30_000) // Enable, check every 30 seconds
142139
.minChangesBetweenPersists(100) // Only persist if >= 100 changes
143140
.persistOnShutdown(true) // Persist on close()
144-
// Background optimization (periodic cleanup)
145-
.backgroundOptimization(true)
146-
.optimizationIntervalMs(60_000) // Check every 60 seconds
141+
// Background optimization (enabled by setting interval > 0)
142+
.optimizationIntervalMs(60_000) // Enable, check every 60 seconds
147143
.minChangesBetweenOptimizations(1000) // Only optimize if >= 1000 changes
148144
.optimizeOnShutdown(false) // Skip for faster shutdown
149145
.build();

‎gigamap/jvector/src/main/java/org/eclipse/store/gigamap/jvector/VectorIndex.java‎

Lines changed: 7 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -136,26 +136,25 @@
136136
* }</pre>
137137
*
138138
* <h2>Background Persistence</h2>
139-
* Automatically persist the index at regular intervals:
139+
* Automatically persist the index at regular intervals (enabled by setting interval &gt; 0):
140140
* <pre>{@code
141141
* VectorIndexConfiguration config = VectorIndexConfiguration.builder()
142142
* .dimension(768)
143143
* .onDisk(true)
144144
* .indexDirectory(Path.of("/data/vectors"))
145-
* .backgroundPersistence(true)
146-
* .persistenceIntervalMs(30_000) // Check every 30 seconds
145+
* .persistenceIntervalMs(30_000) // Enable, check every 30 seconds
147146
* .minChangesBetweenPersists(100) // Only persist if >= 100 changes
148147
* .persistOnShutdown(true) // Persist on close()
149148
* .build();
150149
* }</pre>
151150
*
152151
* <h2>Background Optimization</h2>
153-
* Periodically clean up the graph to reduce memory and improve query latency:
152+
* Periodically clean up the graph to reduce memory and improve query latency
153+
* (enabled by setting interval &gt; 0):
154154
* <pre>{@code
155155
* VectorIndexConfiguration config = VectorIndexConfiguration.builder()
156156
* .dimension(768)
157-
* .backgroundOptimization(true)
158-
* .optimizationIntervalMs(60_000) // Check every 60 seconds
157+
* .optimizationIntervalMs(60_000) // Enable, check every 60 seconds
159158
* .minChangesBetweenOptimizations(1000) // Only optimize if >= 1000 changes
160159
* .optimizeOnShutdown(false) // Skip on close() for faster shutdown
161160
* .build();
@@ -386,11 +385,10 @@ public default VectorSearchResult<E> search(final E queryEntity, final int k)
386385
*
387386
* <h4>Background Optimization</h4>
388387
* Instead of calling this method manually, you can enable automatic background
389-
* optimization via {@link VectorIndexConfiguration.Builder#backgroundOptimization(boolean)}:
388+
* optimization by setting {@code optimizationIntervalMs} to a value greater than 0:
390389
* <pre>{@code
391390
* VectorIndexConfiguration config = VectorIndexConfiguration.builder()
392391
* .dimension(768)
393-
* .backgroundOptimization(true)
394392
* .optimizationIntervalMs(60_000)
395393
* .minChangesBetweenOptimizations(1000)
396394
* .build();
@@ -453,13 +451,12 @@ public default VectorSearchResult<E> search(final E queryEntity, final int k)
453451
*
454452
* <h4>Background Persistence</h4>
455453
* Instead of calling this method manually, you can enable automatic background
456-
* persistence via {@link VectorIndexConfiguration.Builder#backgroundPersistence(boolean)}:
454+
* persistence by setting {@code persistenceIntervalMs} to a value greater than 0:
457455
* <pre>{@code
458456
* VectorIndexConfiguration config = VectorIndexConfiguration.builder()
459457
* .dimension(768)
460458
* .onDisk(true)
461459
* .indexDirectory(Path.of("/data/vectors"))
462-
* .backgroundPersistence(true)
463460
* .persistenceIntervalMs(30_000)
464461
* .minChangesBetweenPersists(100)
465462
* .persistOnShutdown(true)

0 commit comments

Comments
 (0)