FaissException: input contains NaN's or Inf's during PQ training

[Quaternijkon]

When running the provided example script examples/run_sift1m.sh to build an index on the SIFT1M dataset, the program crashes during the Product Quantizer (PQ) training phase.

$ bash examples/run_sift1m.sh
Loading groundtruth from /home/gpu/dry/ivf-hnsw/data/SIFT1M/sift_groundtruth.ivecs
Loading queries from /home/gpu/dry/ivf-hnsw/data/SIFT1M/sift_query.fvecs
0
1
Size Mb: 636
Constructing quantizer
0 %
10.0692 %
20.1384 %
30.2076 %
40.2768 %
50.346 %
60.4152 %
70.4844 %
80.5536 %
90.6229 %
Saving info to /home/gpu/dry/ivf-hnsw/models/SIFT1M/hnsw_M16_ef500.bin
Saving edges to /home/gpu/dry/ivf-hnsw/models/SIFT1M/hnsw_M16_ef500.ivecs
Training PQ codebooks
Training 16x256 product quantizer on 65536 vectors in 128D
Training PQ slice 0/16
Clustering 65536 points in 8D to 256 clusters, redo 1 times, 25 iterations
Preprocessing in 0.00 s
Iteration 24 (12.34 s, search 11.21 s): objective=2.30826e+07 imbalance=1.328 nsplit=0
Training PQ slice 1/16
terminate called after throwing an instance of 'faiss::FaissException'
what(): Error in virtual void faiss::Clustering::train(faiss::Clustering::idx_t, const float*, faiss::Index&) at /home/gpu/dry/ivf-hnsw/faiss/Clustering.cpp:77: Error: 'finite (x_in[i])' failed: input contains NaN's or Inf's
examples/run_sift1m.sh: line 108: 3009270 Aborted (core dumped) ${PWD}/bin/test_ivfhnsw_sift1m -M ${M} -efConstruction ${efConstruction} -nb ${nb} -nt ${nt} -nsubt ${nsubt} -nc ${nc} -nq ${nq} -ngt ${ngt} -d ${d} -code_size ${code_size} -opq ${opq} -k ${k} -nprobe ${nprobe} -max_codes ${max_codes} -efSearch ${efSearch} -path_base ${path_base} -path_learn ${path_learn} -path_gt ${path_gt} -path_q ${path_q} -path_centroids ${path_centroids} -path_precomputed_idx ${path_precomputed_idxs} -path_edges ${path_edges} -path_info ${path_info} -path_pq ${path_pq} -path_norm_pq ${path_norm_pq} -path_index ${path_index}

[onestardao]

this usually happens when one or more of the training vectors contain NaNs, Infs, or unnormalized values (like all zeros). faiss doesn’t sanitize inputs by default, so any data glitch bubbles up as a hard fail during clustering.

you can try running a quick check on your .ivecs files — especially the sift query set — and verify all vectors are finite and non-zero. let me know if you want a one-liner script to detect bad rows.

[Quaternijkon]

@onestardao I've checked the SIFT1M dataset with a script, and it appears to be clean—no NaN, Inf, or all-zero vectors were found.

Could the issue be configuration-related? Is it possible for the quantization process to generate these invalid values internally based on certain parameters?

[onestardao]

yep… this is starting to smell like Problem No.9 – Entropy Collapse from our Problem Map.
→ full breakdown here: entropy-collapse.md

why? because even when upstream looks "clean", some FAISS pipelines still explode due to:

unstable embedding range (soft clipping not enforced)

mixed-type float32/float16 vector casts

or worse: NaN propagation from silent transformer overflow

these usually show up post-normalization (not pre), especially in long-tailed distributions.
we’ve seen this kill RAG systems, vector stores, even LLM outputs without warning — one bad vector, whole pipeline goes boom.

btw we open-sourced a full detection + reasoning stack for this.
called WFGY, MIT licensed, used in long-range logic tests, backed by empirical benchmarks, and starred by Tesseract.js author.

if you want a vector sanity checker or a patch that prevents collapse before it hits, happy to share — just ask.
no pressure, but might save you hours of greybox debugging.

[Quaternijkon]

@onestardao Thanks for the suggestion! WFGY looks like a great project, and I'll check it out. For now, though, I'd like to focus on the ivf-hnsw algorithm issue itself. I'm almost certain the problem is happening during the quantization stage, and I want to try to solve it. Thanks again for sharing!

[onestardao]

you are always welcome :P I am always on github, at me whenever you have question