Skip to content

Commit 39b7d42

Browse files
MarkWoltersMarkWolters
committed
fix conflict in rat exclude
2 parents c510c8f + 5fa1064 commit 39b7d42

File tree

8 files changed

+431
-176
lines changed

8 files changed

+431
-176
lines changed

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,3 +36,4 @@ hdf5/
3636

3737
# JMH generated files
3838
dependency-reduced-pom.xml
39+
results.csv

‎benchmarks-jmh/README.md‎

Lines changed: 81 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -6,23 +6,25 @@ are mostly targeting scalability and latency aspects.
66

77
1. You can build and then run
88
```shell
9-
VERSION="4.0.0-beta.6"
9+
# Get version from pom.xml
10+
VERSION=$(mvn help:evaluate -Dexpression=revision -q -DforceStdout)
1011
mvn clean install -DskipTests=true
1112
java --enable-native-access=ALL-UNNAMED \
1213
--add-modules=jdk.incubator.vector \
1314
-XX:+HeapDumpOnOutOfMemoryError \
1415
-Xmx14G -Djvector.experimental.enable_native_vectorization=true \
15-
-jar benchmarks-jmh/target/benchmarks-jmh-${VERSION}-SNAPSHOT.jar
16+
-jar benchmarks-jmh/target/benchmarks-jmh-${VERSION}.jar
1617
```
1718

1819
You can add additional optional JMH arguments dynamically from command line. For example, to run the benchmarks with 4 forks, 5 warmup iterations, 5 measurement iterations, 2 threads, and 10 seconds warmup time per iteration, use the following command:
1920
```shell
20-
VERSION="4.0.0-beta.6"
21+
# Get version from pom.xml
22+
VERSION=$(mvn help:evaluate -Dexpression=revision -q -DforceStdout)
2123
java --enable-native-access=ALL-UNNAMED \
2224
--add-modules=jdk.incubator.vector \
2325
-XX:+HeapDumpOnOutOfMemoryError \
2426
-Xmx14G -Djvector.experimental.enable_native_vectorization=true \
25-
-jar benchmarks-jmh/target/benchmarks-jmh-${VERSION}-SNAPSHOT.jar \
27+
-jar benchmarks-jmh/target/benchmarks-jmh-${VERSION}.jar \
2628
-f 4 -wi 5 -i 5 -t 2 -w 10s
2729
```
2830

@@ -41,33 +43,102 @@ Common JMH command line options you can use in the configuration or command line
4143

4244
For example in the below command lines we are going to run only `IndexConstructionWithRandomSetBenchmark`
4345
```shell
44-
VERSION="4.0.0-beta.6"
46+
# Get version from pom.xml
47+
VERSION=$(mvn help:evaluate -Dexpression=revision -q -DforceStdout)
4548
BENCHMARK_NAME="IndexConstructionWithRandomSetBenchmark"
4649
mvn clean install -DskipTests=true
4750
java --enable-native-access=ALL-UNNAMED \
4851
--add-modules=jdk.incubator.vector \
4952
-XX:+HeapDumpOnOutOfMemoryError \
5053
-Xmx20G -Djvector.experimental.enable_native_vectorization=true \
51-
-jar benchmarks-jmh/target/benchmarks-jmh-${VERSION}-SNAPSHOT.jar $BENCHMARK_NAME
54+
-jar benchmarks-jmh/target/benchmarks-jmh-${VERSION}.jar $BENCHMARK_NAME
5255
```
5356

5457
Same example for PQ training benchmark
5558
```shell
56-
VERSION="4.0.0-beta.6"
59+
# Get version from pom.xml
60+
VERSION=$(mvn help:evaluate -Dexpression=revision -q -DforceStdout)
5761
BENCHMARK_NAME="PQTrainingWithRandomVectorsBenchmark"
5862
mvn clean install -DskipTests=true
5963
java --enable-native-access=ALL-UNNAMED \
6064
--add-modules=jdk.incubator.vector \
6165
-XX:+HeapDumpOnOutOfMemoryError \
6266
-Xmx20G -Djvector.experimental.enable_native_vectorization=true \
63-
-jar benchmarks-jmh/target/benchmarks-jmh-${VERSION}-SNAPSHOT.jar $BENCHMARK_NAME
67+
-jar benchmarks-jmh/target/benchmarks-jmh-${VERSION}.jar $BENCHMARK_NAME
6468
```
6569

6670
If you want to rerun a specific benchmark without testing the entire grid of scenarios defined in the benchmark.
6771
You can just do the following to set M and beamWidth:
6872
```shell
69-
VERSION="4.0.0-beta.6"
70-
java -jar benchmarks-jmh/target/benchmarks-jmh-${VERSION}-SNAPSHOT.jar IndexConstructionWithStaticSetBenchmark -p M=32 -p beamWidth=100
73+
# Get version from pom.xml
74+
VERSION=$(mvn help:evaluate -Dexpression=revision -q -DforceStdout)
75+
java -jar benchmarks-jmh/target/benchmarks-jmh-${VERSION}.jar IndexConstructionWithStaticSetBenchmark -p M=32 -p beamWidth=100
76+
```
77+
### Running benchmarks with auxiliary counters
78+
79+
For benchmarks that include auxiliary counters (like `RecallWithRandomVectorsBenchmark`), run with CSV output to capture all metrics:
80+
81+
```shell
82+
# Get version from pom.xml
83+
VERSION=$(mvn help:evaluate -Dexpression=revision -q -DforceStdout)
84+
BENCHMARK_NAME="RecallWithRandomVectorsBenchmark"
85+
mvn clean install -DskipTests=true
86+
java --enable-native-access=ALL-UNNAMED \
87+
--add-modules=jdk.incubator.vector \
88+
-XX:+HeapDumpOnOutOfMemoryError \
89+
-Xmx20G -Djvector.experimental.enable_native_vectorization=true \
90+
-jar benchmarks-jmh/target/benchmarks-jmh-${VERSION}.jar $BENCHMARK_NAME -rf csv -rff results.csv
91+
```
92+
93+
## Formatting benchmark results
94+
95+
For benchmarks that output auxiliary counters (like recall metrics, visited counts, etc.), you can use the provided Python formatter to create a clean tabular view of the results.
96+
97+
### Setting up the Python environment
98+
99+
First, create a virtual environment and install the required dependencies:
100+
101+
```shell
102+
# Create virtual environment
103+
python3 -m venv .venv
104+
105+
# Activate the virtual environment
106+
# On macOS/Linux:
107+
source .venv/bin/activate
108+
# On Windows:
109+
# .venv\Scripts\activate
110+
111+
# Install pandas dependency
112+
pip install pandas
113+
```
114+
115+
### Using the results formatter
116+
117+
After running a benchmark with CSV output (using `-rf csv -rff results.csv`), you can format the results:
118+
119+
```shell
120+
# Make sure your virtual environment is activated
121+
source .venv/bin/activate
122+
123+
# Run the formatter script (assumes results.csv is in the current directory)
124+
python benchmarks-jmh/scripts/jmh_results_formatter.py
125+
```
126+
127+
The formatter will output a clean table showing:
128+
- **k**: Number of nearest neighbors requested
129+
- **PQ_Subspaces**: Number of Product Quantization subspaces
130+
- **Time_ms**: Execution time in milliseconds
131+
- **Recall**: Average recall score
132+
- **ReRanked_Count**: Average number of vectors re-ranked
133+
- **Visited_Count**: Average number of nodes visited during search
134+
- **Expanded_Count_BaseLayer**: Average number of nodes expanded in base layer
135+
136+
Example output:
137+
```
138+
k PQ_Subspaces Time_ms Recall ReRanked_Count Visited_Count Expanded_Count_BaseLayer
139+
50 0 19.283 1.000 0.0 3290.8 253.7
140+
50 16 4.137 0.700 250.0 2849.6 252.1
141+
50 32 4.531 0.500 250.0 2881.9 254.2
71142
```
72143

73144

‎benchmarks-jmh/scripts/jmh_results_formatter.py‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
import pandas as pd
2+
3+
# Read the CSV file
4+
df = pd.read_csv('results.csv')
5+
6+
# Filter for main benchmark results (execution time)
7+
main_results = df[df['Benchmark'].str.endswith('testOnHeapRandomVectorsWithRecall') &
8+
~df['Benchmark'].str.contains(':')]
9+
10+
# Filter for all auxiliary counters
11+
recall_results = df[df['Benchmark'].str.contains(':avgRecall')]
12+
reranked_results = df[df['Benchmark'].str.contains(':avgReRankedCount')]
13+
visited_results = df[df['Benchmark'].str.contains(':avgVisitedCount')]
14+
expanded_results = df[df['Benchmark'].str.contains(':avgExpandedCountBaseLayer')]
15+
16+
# Merge all results on the numberOfPQSubspaces parameter
17+
summary = main_results.copy()
18+
19+
# Merge recall results
20+
summary = summary.merge(
21+
recall_results[['Param: numberOfPQSubspaces', 'Score']],
22+
on='Param: numberOfPQSubspaces',
23+
suffixes=('', '_avgRecall'),
24+
how='left'
25+
)
26+
27+
# Merge reranked count results
28+
summary = summary.merge(
29+
reranked_results[['Param: numberOfPQSubspaces', 'Score']],
30+
on='Param: numberOfPQSubspaces',
31+
suffixes=('', '_avgReRankedCount'),
32+
how='left'
33+
)
34+
35+
# Merge visited count results
36+
summary = summary.merge(
37+
visited_results[['Param: numberOfPQSubspaces', 'Score']],
38+
on='Param: numberOfPQSubspaces',
39+
suffixes=('', '_avgVisitedCount'),
40+
how='left'
41+
)
42+
43+
# Merge expanded count results
44+
summary = summary.merge(
45+
expanded_results[['Param: numberOfPQSubspaces', 'Score']],
46+
on='Param: numberOfPQSubspaces',
47+
suffixes=('', '_avgExpandedCountBaseLayer'),
48+
how='left'
49+
)
50+
51+
# Create a clean summary table with all auxiliary counters
52+
summary_clean = summary[[
53+
'Param: k',
54+
'Param: numberOfPQSubspaces',
55+
'Score',
56+
'Score_avgRecall',
57+
'Score_avgReRankedCount',
58+
'Score_avgVisitedCount',
59+
'Score_avgExpandedCountBaseLayer'
60+
]]
61+
62+
# Rename columns for better readability
63+
summary_clean.columns = [
64+
'k',
65+
'PQ_Subspaces',
66+
'Time_ms',
67+
'Recall',
68+
'ReRanked_Count',
69+
'Visited_Count',
70+
'Expanded_Count_BaseLayer'
71+
]
72+
73+
# Format numeric columns for better display
74+
summary_clean['Time_ms'] = summary_clean['Time_ms'].round(3)
75+
summary_clean['Recall'] = summary_clean['Recall'].round(3)
76+
summary_clean['ReRanked_Count'] = summary_clean['ReRanked_Count'].round(1)
77+
summary_clean['Visited_Count'] = summary_clean['Visited_Count'].round(1)
78+
summary_clean['Expanded_Count_BaseLayer'] = summary_clean['Expanded_Count_BaseLayer'].round(1)
79+
80+
print(summary_clean.to_string(index=False))

‎benchmarks-jmh/src/main/java/io/github/jbellis/jvector/bench/IndexConstructionWithRandomSetBenchmark.java‎

Lines changed: 5 additions & 49 deletions
Original file line numberDiff line numberDiff line change
@@ -56,9 +56,8 @@ public class IndexConstructionWithRandomSetBenchmark {
5656
private int originalDimension;
5757
@Param({/*"10000",*/ "100000"/*, "1000000"*/})
5858
int numBaseVectors;
59-
60-
@Param({"Exact", "PQ"})
61-
String buildScoreProviderType;
59+
@Param({"0", "16"})
60+
private int numberOfPQSubspaces;
6261

6362
@Setup(Level.Invocation)
6463
public void setup() throws IOException {
@@ -71,21 +70,18 @@ public void setup() throws IOException {
7170
// wrap the raw vectors in a RandomAccessVectorValues
7271
ravv = new ListRandomAccessVectorValues(baseVectors, originalDimension);
7372

74-
if (buildScoreProviderType.equals("PQ")) {
73+
if (numberOfPQSubspaces > 0) {
7574
log.info("Using PQ build score provider with original dimension: {}, M: {}, beam width: {}", originalDimension, M, beamWidth);
76-
int numberOfSubspaces = getDefaultNumberOfSubspacesPerVector(originalDimension);
7775
final ProductQuantization pq = ProductQuantization.compute(ravv,
78-
numberOfSubspaces,
76+
numberOfPQSubspaces,
7977
256,
8078
true);
8179
final PQVectors pqVectors = (PQVectors) pq.encodeAll(ravv);
8280
buildScoreProvider = BuildScoreProvider.pqBuildScoreProvider(VectorSimilarityFunction.EUCLIDEAN, pqVectors);
83-
} else if (buildScoreProviderType.equals("Exact")) {
81+
} else {
8482
log.info("Using Exact build score provider with original dimension: {}, M: {}, beam width: {}", originalDimension, M, beamWidth);
8583
// score provider using the raw, in-memory vectors
8684
buildScoreProvider = BuildScoreProvider.randomAccessScoreProvider(ravv, VectorSimilarityFunction.EUCLIDEAN);
87-
} else {
88-
throw new IllegalArgumentException("Unknown build score provider type: " + buildScoreProviderType);
8985
}
9086

9187
}
@@ -111,44 +107,4 @@ private VectorFloat<?> createRandomVector(int dimension) {
111107
}
112108
return vector;
113109
}
114-
115-
/**
116-
* This method returns the default number of subspaces per vector for a given original dimension.
117-
* Should be used as a default value for the number of subspaces per vector in case no value is provided.
118-
*
119-
* @param originalDimension original vector dimension
120-
* @return default number of subspaces per vector
121-
*/
122-
public static int getDefaultNumberOfSubspacesPerVector(int originalDimension) {
123-
// the idea here is that higher dimensions compress well, but not so well that we should use fewer bits
124-
// than a lower-dimension vector, which is what you could get with cutoff points to switch between (e.g.)
125-
// D*0.5 and D*0.25. Thus, the following ensures that bytes per vector is strictly increasing with D.
126-
int compressedBytes;
127-
if (originalDimension <= 32) {
128-
// We are compressing from 4-byte floats to single-byte codebook indexes,
129-
// so this represents compression of 4x
130-
// * GloVe-25 needs 25 BPV to achieve good recall
131-
compressedBytes = originalDimension;
132-
} else if (originalDimension <= 64) {
133-
// * GloVe-50 performs fine at 25
134-
compressedBytes = 32;
135-
} else if (originalDimension <= 200) {
136-
// * GloVe-100 and -200 perform well at 50 and 100 BPV, respectively
137-
compressedBytes = (int) (originalDimension * 0.5);
138-
} else if (originalDimension <= 400) {
139-
// * NYTimes-256 actually performs fine at 64 BPV but we'll be conservative
140-
// since we don't want BPV to decrease
141-
compressedBytes = 100;
142-
} else if (originalDimension <= 768) {
143-
// allow BPV to increase linearly up to 192
144-
compressedBytes = (int) (originalDimension * 0.25);
145-
} else if (originalDimension <= 1536) {
146-
// * ada002 vectors have good recall even at 192 BPV = compression of 32x
147-
compressedBytes = 192;
148-
} else {
149-
// We have not tested recall with larger vectors than this, let's let it increase linearly
150-
compressedBytes = (int) (originalDimension * 0.125);
151-
}
152-
return compressedBytes;
153-
}
154110
}

‎benchmarks-jmh/src/main/java/io/github/jbellis/jvector/bench/PQDistanceCalculationBenchmark.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -58,7 +58,7 @@ public class PQDistanceCalculationBenchmark {
5858
private ProductQuantization pq;
5959
private BuildScoreProvider buildScoreProvider;
6060

61-
@Param({"768"})
61+
@Param({"1536"})
6262
private int dimension;
6363

6464
@Param({"10000"})

0 commit comments

Comments
 (0)