update some docs

README.md

@@ -3,17 +3,17 @@
 [![version](https://img.shields.io/pypi/v/vectordb-bench.svg?color=blue)](https://pypi.org/project/vectordb-bench/)
 [![Downloads](https://pepy.tech/badge/vectordb-bench)](https://pepy.tech/project/vectordb-bench)
 
-## What is VectorDBBench
-VectorDBBench(VDBBench) is not just an offering of benchmark results for mainstream vector databases and cloud services, it's your go-to tool for the ultimate performance and cost-effectiveness comparison. Designed with ease-of-use in mind, VectorDBBench is devised to help users, even non-professionals, reproduce results or test new systems, making the hunt for the optimal choice amongst a plethora of cloud services and open-source vector databases a breeze.
+## What is VDBBench
+VDBBench is not just an offering of benchmark results for mainstream vector databases and cloud services, it's your go-to tool for the ultimate performance and cost-effectiveness comparison. Designed with ease-of-use in mind, VDBBench is devised to help users, even non-professionals, reproduce results or test new systems, making the hunt for the optimal choice amongst a plethora of cloud services and open-source vector databases a breeze.
 
 Understanding the importance of user experience, we provide an intuitive visual interface. This not only empowers users to initiate benchmarks at ease, but also to view comparative result reports, thereby reproducing benchmark results effortlessly.
 To add more relevance and practicality, we provide cost-effectiveness reports particularly for cloud services. This allows for a more realistic and applicable benchmarking process.
 
 Closely mimicking real-world production environments, we've set up diverse testing scenarios including insertion, searching, and filtered searching. To provide you with credible and reliable data, we've included public datasets from actual production scenarios, such as [SIFT](http://corpus-texmex.irisa.fr/), [GIST](http://corpus-texmex.irisa.fr/), [Cohere](https://huggingface.co/datasets/Cohere/wikipedia-22-12/tree/main/en), and a dataset generated by OpenAI from an opensource [raw dataset](https://huggingface.co/datasets/allenai/c4). It's fascinating to discover how a relatively unknown open-source database might excel in certain circumstances!
 
-Prepare to delve into the world of VectorDBBench, and let it guide you in uncovering your perfect vector database match.  
+Prepare to delve into the world of VDBBench, and let it guide you in uncovering your perfect vector database match.  
 
-VectorDBBench is sponsered by Zilliz，the leading opensource vectorDB company behind Milvus. Choose smarter with VectorDBBench- start your free test on [zilliz cloud](https://zilliz.com/) today!
+VDBBench is sponsered by Zilliz，the leading opensource vectorDB company behind Milvus. Choose smarter with VDBBench - start your free test on [zilliz cloud](https://zilliz.com/) today!
 
 **Leaderboard:** https://zilliz.com/benchmark
 ## Quick Start
@@ -420,7 +420,7 @@ make format
 ## How does it work?
 ### Result Page
 ![image](https://github.com/zilliztech/VectorDBBench/assets/105927039/8a981327-c1c6-4796-8a85-c86154cb5472)
-This is the main page of VectorDBBench, which displays the standard benchmark results we provide. Additionally, results of all tests performed by users themselves will also be shown here. We also offer the ability to select and compare results from multiple tests simultaneously.
+This is the main page of VDBBench, which displays the standard benchmark results we provide. Additionally, results of all tests performed by users themselves will also be shown here. We also offer the ability to select and compare results from multiple tests simultaneously.
 
 The standard benchmark results displayed here include all 15 cases that we currently support for 6 of our clients (Milvus, Zilliz Cloud, Elastic Search, Qdrant Cloud, Weaviate Cloud and PgVector). However, as some systems may not be able to complete all the tests successfully due to issues like Out of Memory (OOM) or timeouts, not all clients are included in every case.
 
@@ -454,7 +454,7 @@ We've developed lots of comprehensive benchmark cases to test vector databases'
 - **Int-Filter Cases:** Evaluates search performance with int-based filter expression (e.g.  "id >= 2,000").
 - **Label-Filter Cases:** Evaluates search performance with label-based filter expressions (e.g., "color == 'red'"). The test includes randomly generated labels to simulate real-world filtering scenarios.
 #### Streaming Cases
-- **Insertion-Under-Load Case:** Evaluates search performance while maintaining a constant insertion workload. VectorDBBench applies a steady stream of insert requests at a fixed rate to simulate real-world scenarios where search operations must perform reliably under continuous data ingestion.
+- **Insertion-Under-Load Case:** Evaluates search performance while maintaining a constant insertion workload. VDBBench applies a steady stream of insert requests at a fixed rate to simulate real-world scenarios where search operations must perform reliably under continuous data ingestion.
 
 Each case provides an in-depth examination of a vector database's abilities, providing you a comprehensive view of the database's performance.
 
@@ -480,15 +480,15 @@ We have strict requirements for the data set format, please follow them.
 
 - `Train File Count` - If the vector file is too large, you can consider splitting it into multiple files. The naming format for the split files should be `train-[index]-of-[file_count].parquet`. For example, `train-01-of-10.parquet` represents the second file (0-indexed) among 10 split files.
 
-- `Use Shuffled Data` - If you check this option, the vector data files need to be modified. VectorDBBench will load the data labeled with `shuffle`. For example, use `shuffle_train.parquet` instead of `train.parquet` and `shuffle_train-04-of-10.parquet` instead of `train-04-of-10.parquet`. The `id` column in the shuffled data can be in any order.
+- `Use Shuffled Data` - If you check this option, the vector data files need to be modified. VDBBench will load the data labeled with `shuffle`. For example, use `shuffle_train.parquet` instead of `train.parquet` and `shuffle_train-04-of-10.parquet` instead of `train-04-of-10.parquet`. The `id` column in the shuffled data can be in any order.
 
 
 ## Goals
 Our goals of this benchmark are:
 ### Reproducibility & Usability
-One of the primary goals of VectorDBBench is to enable users to reproduce benchmark results swiftly and easily, or to test their customized scenarios. We believe that lowering the barriers to entry for conducting these tests will enhance the community's understanding and improvement of vector databases. We aim to create an environment where any user, regardless of their technical expertise, can quickly set up and run benchmarks, and view and analyze results in an intuitive manner.
+One of the primary goals of VDBBench is to enable users to reproduce benchmark results swiftly and easily, or to test their customized scenarios. We believe that lowering the barriers to entry for conducting these tests will enhance the community's understanding and improvement of vector databases. We aim to create an environment where any user, regardless of their technical expertise, can quickly set up and run benchmarks, and view and analyze results in an intuitive manner.
 ### Representability & Realism
-VectorDBBench aims to provide a more comprehensive, multi-faceted testing environment that accurately represents the complexity of vector databases. By moving beyond a simple speed test for algorithms, we hope to contribute to a better understanding of vector databases in real-world scenarios. By incorporating as many complex scenarios as possible, including a variety of test cases and datasets, we aim to reflect realistic conditions and offer tangible significance to our community. Our goal is to deliver benchmarking results that can drive tangible improvements in the development and usage of vector databases.
+VDBBench aims to provide a more comprehensive, multi-faceted testing environment that accurately represents the complexity of vector databases. By moving beyond a simple speed test for algorithms, we hope to contribute to a better understanding of vector databases in real-world scenarios. By incorporating as many complex scenarios as possible, including a variety of test cases and datasets, we aim to reflect realistic conditions and offer tangible significance to our community. Our goal is to deliver benchmarking results that can drive tangible improvements in the development and usage of vector databases.
 
 ## Contribution
 ### General Guidelines

pyproject.toml

@@ -220,3 +220,9 @@ builtins-ignorelist = [
     # "dict", # TODO
     # "filter",
 ]
+
+[tool.ruff.lint.per-file-ignores]
+"vectordb_bench/backend/clients/*" = ["PLC0415"]
+"vectordb_bench/cli/batch_cli.py" = ["PLC0415"]
+"vectordb_bench/backend/data_source.py" = ["PLC0415"]
+

vectordb_bench/__main__.py

@@ -17,7 +17,7 @@ def run_streamlit():
     cmd = [
         "streamlit",
         "run",
-        f"{pathlib.Path(__file__).parent}/frontend/vdb_benchmark.py",
+        f"{pathlib.Path(__file__).parent}/frontend/vdbbench.py",
         "--logger.level",
         "info",
         "--theme.base",

vectordb_bench/backend/clients/api.py

@@ -132,7 +132,7 @@ class VectorDB(ABC):
     """
 
     "The filtering types supported by the VectorDB Client, default only non-filter"
-    supported_filter_types: list[FilterOp] = [FilterOp.NonFilter, FilterOp.NumGE]
+    supported_filter_types: list[FilterOp] = [FilterOp.NonFilter]
 
     @classmethod
     def filter_supported(cls, filters: Filter) -> bool:

vectordb_bench/backend/clients/aws_opensearch/config.py

@@ -75,6 +75,20 @@ def __eq__(self, obj: any):
             and self.quantization_type == obj.quantization_type
         )
 
+    def __hash__(self) -> int:
+        return hash(
+            (
+                self.engine,
+                self.M,
+                self.efConstruction,
+                self.number_of_shards,
+                self.number_of_replicas,
+                self.number_of_segments,
+                self.use_routing,
+                self.quantization_type,
+            )
+        )
+
     def parse_metric(self) -> str:
         log.info(f"User specified metric_type: {self.metric_type_name}")
         self.metric_type = MetricType[self.metric_type_name.upper()]

vectordb_bench/backend/clients/elastic_cloud/config.py

@@ -48,6 +48,18 @@ def __eq__(self, obj: any):
             and self.M == obj.M
         )
 
+    def __hash__(self) -> int:
+        return hash(
+            (
+                self.index,
+                self.number_of_shards,
+                self.number_of_replicas,
+                self.use_routing,
+                self.efConstruction,
+                self.M,
+            )
+        )
+
     def parse_metric(self) -> str:
         if self.metric_type == MetricType.L2:
             return "l2_norm"

vectordb_bench/backend/clients/milvus/milvus.py

@@ -29,7 +29,7 @@ def __init__(
         dim: int,
         db_config: dict,
         db_case_config: MilvusIndexConfig,
-        collection_name: str = "VectorDBBenchCollection",
+        collection_name: str = "VDBBench",
         drop_old: bool = False,
         name: str = "Milvus",
         with_scalar_labels: bool = False,

vectordb_bench/backend/clients/qdrant_cloud/config.py

@@ -63,6 +63,20 @@ def __eq__(self, obj: any):
             and self.default_segment_number == obj.default_segment_number
         )
 
+    def __hash__(self) -> int:
+        return hash(
+            (
+                self.m,
+                self.payload_m,
+                self.create_payload_int_index,
+                self.create_payload_keyword_index,
+                self.is_tenant,
+                self.use_scalar_quant,
+                self.sq_quantile,
+                self.default_segment_number,
+            )
+        )
+
     def parse_metric(self) -> str:
         if self.metric_type == MetricType.L2:
             return "Euclid"

vectordb_bench/backend/clients/zilliz_cloud/zilliz_cloud.py

@@ -10,7 +10,7 @@ def __init__(
         dim: int,
         db_config: dict,
         db_case_config: DBCaseConfig,
-        collection_name: str = "ZillizCloudVectorDBBench",
+        collection_name: str = "ZillizCloudVDBBench",
         drop_old: bool = False,
         name: str = "ZillizCloud",
         **kwargs,

vectordb_bench/backend/dataset.py

@@ -242,6 +242,9 @@ def __eq__(self, obj: any):
             return self.data.name == obj.data.name and self.data.label == obj.data.label
         return False
 
+    def __hash__(self) -> int:
+        return hash((self.data.name, self.data.label))
+
     def set_reader(self, reader: DatasetReader):
         self.reader = reader
 

vectordb_bench/backend/task_runner.py

@@ -59,6 +59,17 @@ def __eq__(self, obj: any):
             )
         return False
 
+    def __hash__(self) -> int:
+        """Hash method to maintain consistency with __eq__ method."""
+        return hash(
+            (
+                self.ca.label,
+                self.config.db,
+                self.config.db_case_config,
+                self.ca.dataset,
+            )
+        )
+
     def display(self) -> dict:
         c_dict = self.ca.dict(
             include={

vectordb_bench/frontend/components/check_results/headerIcon.py

@@ -4,7 +4,7 @@
 def drawHeaderIcon(st):
     st.markdown(
         f"""
-    <a href="/vdb_benchmark" target="_self">
+    <a href="/vdbbench" target="_self">
         <div class="headerIconContainer"></div>
     </a>
 
@@ -16,8 +16,10 @@ def drawHeaderIcon(st):
         width: 100%;
         border-bottom: 2px solid #E8EAEE;
         background-image: url({HEADER_ICON});
+        background-size: contain;
+        background-position: left top;
         background-repeat: no-repeat;
-        cursor: pointer; 
+        cursor: pointer;
     }}
     </style>
     """,

vectordb_bench/frontend/components/welcome/explainPrams.py

@@ -1,16 +1,16 @@
 def explainPrams(st):
     st.markdown("## descriptions")
-    st.markdown("### 1.Overview")
+    st.markdown("### 1. Overview")
     st.markdown(
         """
-- **VectorDBBench** is an open-source benchmarking tool designed specifically for vector databases. Its main features include:
+- **VectorDBBench(VDBBench)** is an open-source benchmarking tool designed specifically for vector databases. Its main features include:
     - (1) An easy-to-use **web UI** for configuration of tests and visual analysis of results.
     - (2) A comprehensive set of **standards for testing and metric collection**.
     - (3) Support for **various scenarios**, including additional support for **Filter** and **Streaming** based on standard tests.
-- VectorDBBench embraces open-source and welcome contributions of code and test result submissions. The testing process and extended scenarios of VectorDBBench, as well as the intention behind our design will be introduced as follows.
+- VDBBench embraces open-source and welcome contributions of code and test result submissions. The testing process and extended scenarios of VDBBench, as well as the intention behind our design will be introduced as follows.
 """
     )
-    st.markdown("### 2.Dataset")
+    st.markdown("### 2. Dataset")
     st.markdown(
         """
 - We provide two embedding datasets:
@@ -19,7 +19,7 @@ def explainPrams(st):
     - (3)*OpenAI 1536dim*, generated using the **OpenAI** model based on the [C4 corpus](https://huggingface.co/datasets/legacy-datasets/c4).
 """
     )
-    st.markdown("### 3.Standard Test")
+    st.markdown("### 3. Standard Test")
     st.markdown(
         """
 The test is actually divided into 3 sub-processes
@@ -42,19 +42,19 @@ def explainPrams(st):
 """,
         unsafe_allow_html=True,
     )
-    st.markdown("### 4.Filter Search Test")
+    st.markdown("### 4. Filter Search Test")
     st.markdown(
         """
 - Compared to the Standard Test, the **Filter Search** introduces additional scalar constraints (e.g. **color == red**) during the Search Test. Different **filter_ratios** present varying levels of challenge to the VectorDB's search performance.
 - We provide an additional **string column** containing 10 labels with different distribution ratios (50%,20%,10%,5%,2%,1%,0.5%,0.2%,0.1%). For each label, we conduct both a **Serial Test** and a **Concurrency Test** to observe the VectorDB's performance in terms of **QPS, latency, and recall** under different filtering conditions.
 """
     )
-    st.markdown("### 5.Streaming Search Test")
+    st.markdown("### 5. Streaming Search Test")
     st.markdown(
         """
 Different from Standard's load and search separation, Streaming Search Test primarily focuses on **search performance during the insertion process**. 
 Different **base dataset sizes** and varying **insertion rates** set distinct challenges to the VectorDB's search capabilities.
-VectorDBBench will send insert requests at a **fixed rate**, maintaining consistent insertion pressure. The search test consists of three steps as follows:
+VDBBench will send insert requests at a **fixed rate**, maintaining consistent insertion pressure. The search test consists of three steps as follows:
 - 1.**Streaming Search** 
     - Users can configure **multiple search stages**. When the inserted data volume reaches a specified stage, a **Serial Test** and a **Concurrent Test** will be conducted, recording qps, latency, and recall performance.
 - 2.**Streaming Final Search**

vectordb_bench/frontend/components/welcome/welcomePrams.py

@@ -21,7 +21,7 @@ def get_image_as_base64(image_path):
 
 
 def welcomePrams(st):
-    st.title("Welcome to VectorDB Benchmark!")
+    st.title("Welcome to VDBBench!")
     options = [
         {
             "title": "Standard Test Results",

vectordb_bench/frontend/config/dbCaseConfigs.py

@@ -520,7 +520,7 @@ class CaseConfigInput(BaseModel):
     inputConfig={
         "min": 4,
         "max": 64,
-        "value": 30,
+        "value": 16,
     },
     isDisplayed=lambda config: config.get(CaseConfigParamType.IndexType, None)
     in [
@@ -550,7 +550,7 @@ class CaseConfigInput(BaseModel):
     inputConfig={
         "min": 8,
         "max": 512,
-        "value": 360,
+        "value": 256,
     },
     isDisplayed=lambda config: config[CaseConfigParamType.IndexType]
     in [
@@ -1441,7 +1441,7 @@ class CaseConfigInput(BaseModel):
     label=CaseConfigParamType.use_partition_key,
     inputType=InputType.Option,
     inputHelp="whether to use partition_key for label-filter cases. only works in label-filter cases",
-    inputConfig={"options": [True, False]},
+    inputConfig={"options": [False, True]},
 )
 
 

vectordb_bench/frontend/config/styles.py

@@ -33,8 +33,8 @@ def getPatternShape(i):
 MAX_AUTO_REFRESH_INTERVAL = 5000  # 5s
 
 PAGE_TITLE = "VectorDB Benchmark"
-FAVICON = "https://assets.zilliz.com/favicon_f7f922fe27.png"
-HEADER_ICON = "https://assets.zilliz.com/vdb_benchmark_db790b5387.png"
+FAVICON = "https://assets.zilliz.com/VDB_Bench_icon_d3276bedc4.png"
+HEADER_ICON = "https://assets.zilliz.com/VDB_Bench_text_icon_6c5f52a458.png"
 
 # RedisCloud icon: https://assets.zilliz.com/Redis_Cloud_74b8bfef39.png
 # Elasticsearch icon: https://assets.zilliz.com/elasticsearch_beffeadc29.png

vectordb_bench/frontend/pages/results.py

@@ -30,10 +30,12 @@ def main():
 
     st.title("Vector Database Benchmark")
     st.caption(
-        "Except for zillizcloud-v2024.1, which was tested in _January 2024_, all other tests were completed before _August 2023_."
+        "Choose your desired test results to display from the sidebar. "
+        "For your reference, we've included two standard benchmarks tested by our team. "
+        "Note that `standard_2025` was tested in 2025; the others in 2023. "
+        "Unless explicitly labeled as distributed multi-node, test with single-node mode by default."
     )
-    st.caption("All tested milvus are in _standalone_ mode.")
-
+    st.caption("We welcome community contributions for better results, parameter configurations, and optimizations.")
     # results selector and filter
     resultSelectorContainer = st.sidebar.container()
     shownData, failedTasks, showCaseNames = getshownData(resultSelectorContainer, allResults)