You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+45-45Lines changed: 45 additions & 45 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -11,7 +11,7 @@ To add more relevance and practicality, we provide cost-effectiveness reports pa
11
11
12
12
Closely mimicking real-world production environments, we've set up diverse testing scenarios including insertion, searching, and filtered searching. To provide you with credible and reliable data, we've included public datasets from actual production scenarios, such as [SIFT](http://corpus-texmex.irisa.fr/), [GIST](http://corpus-texmex.irisa.fr/), [Cohere](https://huggingface.co/datasets/Cohere/wikipedia-22-12/tree/main/en), and a dataset generated by OpenAI from an opensource [raw dataset](https://huggingface.co/datasets/allenai/c4). It's fascinating to discover how a relatively unknown open-source database might excel in certain circumstances!
13
13
14
-
Prepare to delve into the world of VDBBench, and let it guide you in uncovering your perfect vector database match.
14
+
Prepare to delve into the world of VDBBench, and let it guide you in uncovering your perfect vector database match.
15
15
16
16
VDBBench is sponsored by Zilliz,the leading opensource vectorDB company behind Milvus. Choose smarter with VDBBench - start your free test on [zilliz cloud](https://zilliz.com/) today!
The vectordbbench command can optionally read some or all the options from a yaml formatted configuration file.
321
321
322
-
By default, configuration files are expected to be in vectordb_bench/config-files/, this can be overridden by setting
323
-
the environment variable CONFIG_LOCAL_DIR or by passing the full path to the file.
322
+
By default, configuration files are expected to be in vectordb_bench/config-files/, this can be overridden by setting
323
+
the environment variable CONFIG_LOCAL_DIR or by passing the full path to the file.
324
324
325
325
The required format is:
326
326
```yaml
@@ -349,16 +349,16 @@ milvushnsw:
349
349
drop_old: False
350
350
load: False
351
351
```
352
-
> Notes:
352
+
> Notes:
353
353
> - Options passed on the command line will override the configuration file*
354
354
> - Parameter names use an _ not -
355
355
356
356
#### Using a batch configuration file.
357
357
358
358
The vectordbbench command can read a batch configuration file to run all the test cases in the yaml formatted configuration file.
359
359
360
-
By default, configuration files are expected to be in vectordb_bench/config-files/, this can be overridden by setting
361
-
the environment variable CONFIG_LOCAL_DIR or by passing the full path to the file.
360
+
By default, configuration files are expected to be in vectordb_bench/config-files/, this can be overridden by setting
361
+
the environment variable CONFIG_LOCAL_DIR or by passing the full path to the file.
362
362
363
363
The required format is:
364
364
```yaml
@@ -387,7 +387,7 @@ milvushnsw:
387
387
drop_old: False
388
388
load: False
389
389
```
390
-
> Notes:
390
+
> Notes:
391
391
> - Options can only be passed through configuration files
392
392
> - Parameter names use an _ not -
393
393
@@ -402,11 +402,11 @@ To facilitate the presentation of test results and provide a comprehensive perfo
402
402
403
403
### Scoring Rules
404
404
405
-
1. For each case, select a base value and score each system based on relative values.
406
-
- For QPS and QP$, we use the highest value as the reference, denoted as `base_QPS` or `base_QP$`, and the score of each system is `(QPS/base_QPS) * 100` or `(QP$/base_QP$) * 100`.
407
-
- For Latency, we use the lowest value as the reference, that is, `base_Latency`, and the score of each system is `(base_Latency + 10ms)/(Latency + 10ms) * 100`.
405
+
1. For each case, select a base value and score each system based on relative values.
406
+
- For QPS and QP$, we use the highest value as the reference, denoted as `base_QPS` or `base_QP$`, and the score of each system is `(QPS/base_QPS) * 100` or `(QP$/base_QP$) * 100`.
407
+
- For Latency, we use the lowest value as the reference, that is, `base_Latency`, and the score of each system is `(base_Latency + 10ms)/(Latency + 10ms) * 100`.
408
408
409
-
We want to give equal weight to different cases, and not let a case with high absolute result values become the sole reason for the overall scoring. Therefore, when scoring different systems in each case, we need to use relative values.
409
+
We want to give equal weight to different cases, and not let a case with high absolute result values become the sole reason for the overall scoring. Therefore, when scoring different systems in each case, we need to use relative values.
410
410
411
411
Also, for Latency, we add 10ms to the numerator and denominator to ensure that if every system performs particularly well in a case, its advantage will not be infinitely magnified when latency tends to 0.
412
412
@@ -467,7 +467,7 @@ All standard benchmark results are generated by a client running on an 8 core, 3
467
467
1. Initially, you select the systems to be tested - multiple selections are allowed. Once selected, corresponding forms will pop up to gather necessary information for using the chosen databases. The db_label is used to differentiate different instances of the same system. We recommend filling in the host size or instance type here (as we do in our standard results).
468
468
2. The next step is to select the test cases you want to perform. You can select multiple cases at once, and a form to collect corresponding parameters will appear.
469
469
3. Finally, you'll need to provide a task label to distinguish different test results. Using the same label for different tests will result in the previous results being overwritten.
@@ -508,11 +508,11 @@ We have strict requirements for the data set format, please follow them.
508
508
- Vectors data files: The file must be named `train.parquet` and should have two columns: `id` as an incrementing `int` and `emb` as an array of `float32`.
509
509
- Query test vectors: The file must be named `test.parquet` and should have two columns: `id` as an incrementing `int` and `emb` as an array of `float32`.
510
510
- We recommend limiting the number of test query vectors, like 1,000.
511
-
When conducting concurrent query tests, Vdbbench creates a large number of processes.
512
-
To minimize additional communication overhead during testing,
511
+
When conducting concurrent query tests, Vdbbench creates a large number of processes.
512
+
To minimize additional communication overhead during testing,
513
513
we prepare a complete set of test queries for each process, allowing them to run independently.
514
-
However, this means that as the number of concurrent processes increases,
515
-
the number of copied query vectors also increases significantly,
514
+
However, this means that as the number of concurrent processes increases,
515
+
the number of copied query vectors also increases significantly,
516
516
which can place substantial pressure on memory resources.
517
517
- Ground truth file: The file must be named `neighbors.parquet` and should have two columns: `id` corresponding to query vectors and `neighbors_id` as an array of `int`.
518
518
@@ -542,10 +542,10 @@ VDBBench aims to provide a more comprehensive, multi-faceted testing environment
542
542
543
543
**Step 2: Implement new_client.py and config.py**
544
544
545
-
1. Open new_client.py and define the NewClient class, which should inherit from the clients/api.py file's VectorDB abstract class. The VectorDB class serves as the API for benchmarking, and all DB clients must implement this abstract class.
545
+
1. Open new_client.py and define the NewClient class, which should inherit from the clients/api.py file's VectorDB abstract class. The VectorDB class serves as the API for benchmarking, and all DB clients must implement this abstract class.
546
546
Example implementation in new_client.py:
547
547
new_client.py
548
-
```python
548
+
```python
549
549
from ..api import VectorDB
550
550
classNewClient(VectorDB):
551
551
# Implement the abstract methods defined in the VectorDB class
@@ -574,7 +574,7 @@ class NewDBCaseConfig(DBCaseConfig):
574
574
575
575
In this final step, you will import your DB client into clients/__init__.py and update the initialization process.
576
576
1. Open clients/__init__.py and import your NewClient from new_client.py.
577
-
2. Add your NewClient to the DB enum.
577
+
2. Add your NewClient to the DB enum.
578
578
3. Update the db2client dictionary by adding an entry for your NewClient.
1. Add database specific options as an Annotated TypedDict, see ZillizTypedDict above.
675
+
1. Add database specific options as an Annotated TypedDict, see ZillizTypedDict above.
676
676
2. Add index configuration specific options as an Annotated TypedDict. (example: vectordb_bench/backend/clients/pgvector/cli.py)
677
677
1. May not be needed if there is only one index config.
678
-
2. Repeat for each index configuration, nesting them if possible.
678
+
2. Repeat for each index configuration, nesting them if possible.
679
679
2. Add a index config specific function for each index type, see Zilliz above. The function name, in lowercase, will be the command name passed to the vectordbbench command.
680
680
3. Update db_config and db_case_config to match client requirements
681
681
4. Continue to add new functions for each index config.
682
-
5. Import the client cli module and command to vectordb_bench/cli/vectordbbench.py (for databases with multiple commands (index configs), this only needs to be done for one command)
682
+
5. Import the client cli module and command to vectordb_bench/cli/vectordbbench.py (for databases with multiple commands (index configs), this only needs to be done for one command)
683
683
6. Import the `get_custom_case_config` function from `vectordb_bench/cli/cli.py` and use it to add a new key `custom_case` to the `parameters` variable within the command.
684
684
685
685
@@ -697,7 +697,7 @@ For the system under test, we use the default server-side configuration to maint
697
697
For the Client, we welcome any parameter tuning to obtain better results.
698
698
### Incomplete Results
699
699
Many databases may not be able to complete all test cases due to issues such as Out of Memory (OOM), crashes, or timeouts. In these scenarios, we will clearly state these occurrences in the test results.
700
-
### Mistake Or Misrepresentation
700
+
### Mistake Or Misrepresentation
701
701
We strive for accuracy in learning and supporting various vector databases, yet there might be oversights or misapplications. For any such occurrences, feel free to [raise an issue](https://github.com/zilliztech/VectorDBBench/issues/new) or make amendments on our GitHub page.
702
702
## Timeout
703
703
In our pursuit to ensure that our benchmark reflects the reality of a production environment while guaranteeing the practicality of the system, we have implemented a timeout plan based on our experiences for various tests.
0 commit comments