Fail to use this library

[MrWWheat]

Hi,

I am trying to run the example, but when I run go get github.com/DataIntelligenceCrew/go-faiss it shows:

# github.com/DataIntelligenceCrew/go-faiss
../../autotune.go:12:6: could not determine kind of name for C.FaissParameterSpace
../../autotune.go:39:2: could not determine kind of name for C.faiss_ParameterSpace_free
../../autotune.go:18:10: could not determine kind of name for C.faiss_ParameterSpace_new
../../autotune.go:29:7: could not determine kind of name for C.faiss_ParameterSpace_set_index_parameter
../../autotune.go:27:8: could not determine kind of name for C.free

I use local Mac env, and I have copy libfaiss_c.so to /usr/local/lib/

By any chance you know the reason? Thanks!

Just tried again, the error becomes:

# github.com/DataIntelligenceCrew/go-faiss
./autotune.go:5:10: fatal error: 'faiss/c_api/AutoTune_c.h' file not found
#include <faiss/c_api/AutoTune_c.h>
         ^~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.

[p-ouellette]

Did you install the header files? If you did but they aren't being found, you can set the environment variable CGO_CFLAGS to -I <path_to_include_dir>.

[MrWWheat]

Do I need to install the header files? I thought just copy libfaiss_c.so to /usr/local/lib/ is enough.

How should I install this faiss/c_api/AutoTune_c.h header file?

[p-ouellette]

Yes, CGO needs the header files. They should be installed if you use make install after you compile faiss. You can use the cmake option -DCMAKE_INSTALL_PREFIX= to set the install location.

[MrWWheat]

When I run the make -C build install command after make -C build -j faiss, it shows:

(base) qi@Tians-MacBook-Pro faiss % make -C build install
[ 61%] Built target faiss
[ 62%] Built target swigfaiss_swig_compilation
[ 64%] Linking CXX shared library libfaiss_python_callbacks.dylib
Undefined symbols for architecture x86_64:
  "_PyGILState_Ensure", referenced from:
      (anonymous namespace)::PyThreadLock::PyThreadLock() in python_callbacks.cpp.o
  "_PyGILState_Release", referenced from:
      (anonymous namespace)::PyThreadLock::~PyThreadLock() in python_callbacks.cpp.o
  "_PyObject_CallFunction", referenced from:
      PyCallbackIOWriter::operator()(void const*, unsigned long, unsigned long) in python_callbacks.cpp.o
      PyCallbackIOReader::operator()(void*, unsigned long, unsigned long) in python_callbacks.cpp.o
  "_PyString_AsString", referenced from:
      PyCallbackIOReader::operator()(void*, unsigned long, unsigned long) in python_callbacks.cpp.o
  "_PyString_FromStringAndSize", referenced from:
      PyCallbackIOWriter::operator()(void const*, unsigned long, unsigned long) in python_callbacks.cpp.o
  "_PyString_Size", referenced from:
      PyCallbackIOReader::operator()(void*, unsigned long, unsigned long) in python_callbacks.cpp.o
  "faiss::FaissException::FaissException(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, char const*, char const*, int)", referenced from:
      PyCallbackIOWriter::operator()(void const*, unsigned long, unsigned long) in python_callbacks.cpp.o
      PyCallbackIOReader::operator()(void*, unsigned long, unsigned long) in python_callbacks.cpp.o
  "faiss::IOReader::fileno()", referenced from:
      vtable for PyCallbackIOReader in python_callbacks.cpp.o
  "faiss::IOWriter::fileno()", referenced from:
      vtable for PyCallbackIOWriter in python_callbacks.cpp.o
  "typeinfo for faiss::FaissException", referenced from:
      PyCallbackIOWriter::operator()(void const*, unsigned long, unsigned long) in python_callbacks.cpp.o
      PyCallbackIOReader::operator()(void*, unsigned long, unsigned long) in python_callbacks.cpp.o
  "typeinfo for faiss::IOReader", referenced from:
      typeinfo for PyCallbackIOReader in python_callbacks.cpp.o
  "typeinfo for faiss::IOWriter", referenced from:
      typeinfo for PyCallbackIOWriter in python_callbacks.cpp.o
  "vtable for faiss::FaissException", referenced from:
      faiss::FaissException::~FaissException() in python_callbacks.cpp.o
  NOTE: a missing vtable usually means the first non-inline virtual member function has no definition.
  "vtable for faiss::IOReader", referenced from:
      faiss::IOReader::IOReader() in python_callbacks.cpp.o
      faiss::IOReader::~IOReader() in python_callbacks.cpp.o
  NOTE: a missing vtable usually means the first non-inline virtual member function has no definition.
  "vtable for faiss::IOWriter", referenced from:
      faiss::IOWriter::IOWriter() in python_callbacks.cpp.o
      faiss::IOWriter::~IOWriter() in python_callbacks.cpp.o
  NOTE: a missing vtable usually means the first non-inline virtual member function has no definition.
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [faiss/python/libfaiss_python_callbacks.dylib] Error 1
make[1]: *** [faiss/python/CMakeFiles/faiss_python_callbacks.dir/all] Error 2
make: *** [all] Error 2

Maybe this is why it didn't install the header files....By any chance you know the reason?...

[p-ouellette]

Try `make -C install` without the build. It looks like it's trying to build the python bindings.

…

On Thu, Jun 17, 2021, 7:57 PM xiaomai ***@***.***> wrote: When I run the make -C build install command after make -C build -j faiss, it shows: (base) ***@***.*** faiss % make -C build install [ 61%] Built target faiss [ 62%] Built target swigfaiss_swig_compilation [ 64%] Linking CXX shared library libfaiss_python_callbacks.dylib Undefined symbols for architecture x86_64: "_PyGILState_Ensure", referenced from: (anonymous namespace)::PyThreadLock::PyThreadLock() in python_callbacks.cpp.o "_PyGILState_Release", referenced from: (anonymous namespace)::PyThreadLock::~PyThreadLock() in python_callbacks.cpp.o "_PyObject_CallFunction", referenced from: PyCallbackIOWriter::operator()(void const*, unsigned long, unsigned long) in python_callbacks.cpp.o PyCallbackIOReader::operator()(void*, unsigned long, unsigned long) in python_callbacks.cpp.o "_PyString_AsString", referenced from: PyCallbackIOReader::operator()(void*, unsigned long, unsigned long) in python_callbacks.cpp.o "_PyString_FromStringAndSize", referenced from: PyCallbackIOWriter::operator()(void const*, unsigned long, unsigned long) in python_callbacks.cpp.o "_PyString_Size", referenced from: PyCallbackIOReader::operator()(void*, unsigned long, unsigned long) in python_callbacks.cpp.o "faiss::FaissException::FaissException(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, char const*, char const*, int)", referenced from: PyCallbackIOWriter::operator()(void const*, unsigned long, unsigned long) in python_callbacks.cpp.o PyCallbackIOReader::operator()(void*, unsigned long, unsigned long) in python_callbacks.cpp.o "faiss::IOReader::fileno()", referenced from: vtable for PyCallbackIOReader in python_callbacks.cpp.o "faiss::IOWriter::fileno()", referenced from: vtable for PyCallbackIOWriter in python_callbacks.cpp.o "typeinfo for faiss::FaissException", referenced from: PyCallbackIOWriter::operator()(void const*, unsigned long, unsigned long) in python_callbacks.cpp.o PyCallbackIOReader::operator()(void*, unsigned long, unsigned long) in python_callbacks.cpp.o "typeinfo for faiss::IOReader", referenced from: typeinfo for PyCallbackIOReader in python_callbacks.cpp.o "typeinfo for faiss::IOWriter", referenced from: typeinfo for PyCallbackIOWriter in python_callbacks.cpp.o "vtable for faiss::FaissException", referenced from: faiss::FaissException::~FaissException() in python_callbacks.cpp.o NOTE: a missing vtable usually means the first non-inline virtual member function has no definition. "vtable for faiss::IOReader", referenced from: faiss::IOReader::IOReader() in python_callbacks.cpp.o faiss::IOReader::~IOReader() in python_callbacks.cpp.o NOTE: a missing vtable usually means the first non-inline virtual member function has no definition. "vtable for faiss::IOWriter", referenced from: faiss::IOWriter::IOWriter() in python_callbacks.cpp.o faiss::IOWriter::~IOWriter() in python_callbacks.cpp.o NOTE: a missing vtable usually means the first non-inline virtual member function has no definition. ld: symbol(s) not found for architecture x86_64 clang: error: linker command failed with exit code 1 (use -v to see invocation) make[2]: *** [faiss/python/libfaiss_python_callbacks.dylib] Error 1 make[1]: *** [faiss/python/CMakeFiles/faiss_python_callbacks.dir/all] Error 2 make: *** [all] Error 2 Maybe this is why it didn't install the header files....By any chance you know the reason?... — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#7 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFB5PWNXY7VKUHVKNB62LUDTTKDXFANCNFSM464RNCOQ> .

[MrWWheat]

make -C install shows make: *** install: No such file or directory. Stop.

[p-ouellette]

Sorry, that command is wrong. You may need to pass the cmake option to disable building the python bindings.

…

On Thu, Jun 17, 2021, 9:11 PM Paul Ouellette ***@***.***> wrote: Try `make -C install` without the build. It looks like it's trying to build the python bindings. On Thu, Jun 17, 2021, 7:57 PM xiaomai ***@***.***> wrote: > When I run the make -C build install command after make -C build -j faiss, > it shows: > > (base) ***@***.*** faiss % make -C build install > [ 61%] Built target faiss > [ 62%] Built target swigfaiss_swig_compilation > [ 64%] Linking CXX shared library libfaiss_python_callbacks.dylib > Undefined symbols for architecture x86_64: > "_PyGILState_Ensure", referenced from: > (anonymous namespace)::PyThreadLock::PyThreadLock() in python_callbacks.cpp.o > "_PyGILState_Release", referenced from: > (anonymous namespace)::PyThreadLock::~PyThreadLock() in python_callbacks.cpp.o > "_PyObject_CallFunction", referenced from: > PyCallbackIOWriter::operator()(void const*, unsigned long, unsigned long) in python_callbacks.cpp.o > PyCallbackIOReader::operator()(void*, unsigned long, unsigned long) in python_callbacks.cpp.o > "_PyString_AsString", referenced from: > PyCallbackIOReader::operator()(void*, unsigned long, unsigned long) in python_callbacks.cpp.o > "_PyString_FromStringAndSize", referenced from: > PyCallbackIOWriter::operator()(void const*, unsigned long, unsigned long) in python_callbacks.cpp.o > "_PyString_Size", referenced from: > PyCallbackIOReader::operator()(void*, unsigned long, unsigned long) in python_callbacks.cpp.o > "faiss::FaissException::FaissException(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, char const*, char const*, int)", referenced from: > PyCallbackIOWriter::operator()(void const*, unsigned long, unsigned long) in python_callbacks.cpp.o > PyCallbackIOReader::operator()(void*, unsigned long, unsigned long) in python_callbacks.cpp.o > "faiss::IOReader::fileno()", referenced from: > vtable for PyCallbackIOReader in python_callbacks.cpp.o > "faiss::IOWriter::fileno()", referenced from: > vtable for PyCallbackIOWriter in python_callbacks.cpp.o > "typeinfo for faiss::FaissException", referenced from: > PyCallbackIOWriter::operator()(void const*, unsigned long, unsigned long) in python_callbacks.cpp.o > PyCallbackIOReader::operator()(void*, unsigned long, unsigned long) in python_callbacks.cpp.o > "typeinfo for faiss::IOReader", referenced from: > typeinfo for PyCallbackIOReader in python_callbacks.cpp.o > "typeinfo for faiss::IOWriter", referenced from: > typeinfo for PyCallbackIOWriter in python_callbacks.cpp.o > "vtable for faiss::FaissException", referenced from: > faiss::FaissException::~FaissException() in python_callbacks.cpp.o > NOTE: a missing vtable usually means the first non-inline virtual member function has no definition. > "vtable for faiss::IOReader", referenced from: > faiss::IOReader::IOReader() in python_callbacks.cpp.o > faiss::IOReader::~IOReader() in python_callbacks.cpp.o > NOTE: a missing vtable usually means the first non-inline virtual member function has no definition. > "vtable for faiss::IOWriter", referenced from: > faiss::IOWriter::IOWriter() in python_callbacks.cpp.o > faiss::IOWriter::~IOWriter() in python_callbacks.cpp.o > NOTE: a missing vtable usually means the first non-inline virtual member function has no definition. > ld: symbol(s) not found for architecture x86_64 > clang: error: linker command failed with exit code 1 (use -v to see invocation) > make[2]: *** [faiss/python/libfaiss_python_callbacks.dylib] Error 1 > make[1]: *** [faiss/python/CMakeFiles/faiss_python_callbacks.dir/all] Error 2 > make: *** [all] Error 2 > > Maybe this is why it didn't install the header files....By any chance you > know the reason?... > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <#7 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AFB5PWNXY7VKUHVKNB62LUDTTKDXFANCNFSM464RNCOQ> > . >

[MrWWheat]

Oh, it works! But I can not find the libfaiss.so after the build...It is not in the /build/c_api or /build/lib

Currently make -C build install shows:

(base) qi@Tians-MacBook-Pro faiss % make -C build install
[ 64%] Built target faiss
[ 81%] Built target faiss_c
[ 83%] Built target gtest
[ 85%] Built target gtest_main
[100%] Built target faiss_test
Install the project...
-- Install configuration: ""
-- Up-to-date: /usr/local/lib/libfaiss.dylib
-- Up-to-date: /usr/local/include/faiss/AutoTune.h
-- Up-to-date: /usr/local/include/faiss/Clustering.h
-- Up-to-date: /usr/local/include/faiss/IVFlib.h
-- Up-to-date: /usr/local/include/faiss/Index.h
-- Up-to-date: /usr/local/include/faiss/Index2Layer.h
-- Up-to-date: /usr/local/include/faiss/IndexBinary.h
-- Up-to-date: /usr/local/include/faiss/IndexBinaryFlat.h
-- Up-to-date: /usr/local/include/faiss/IndexBinaryFromFloat.h
-- Up-to-date: /usr/local/include/faiss/IndexBinaryHNSW.h
-- Up-to-date: /usr/local/include/faiss/IndexBinaryHash.h
-- Up-to-date: /usr/local/include/faiss/IndexBinaryIVF.h
-- Up-to-date: /usr/local/include/faiss/IndexFlat.h
-- Up-to-date: /usr/local/include/faiss/IndexHNSW.h
-- Up-to-date: /usr/local/include/faiss/IndexIVF.h
-- Up-to-date: /usr/local/include/faiss/IndexIVFFlat.h
-- Up-to-date: /usr/local/include/faiss/IndexIVFPQ.h
-- Up-to-date: /usr/local/include/faiss/IndexIVFPQFastScan.h
-- Up-to-date: /usr/local/include/faiss/IndexIVFPQR.h
-- Up-to-date: /usr/local/include/faiss/IndexIVFSpectralHash.h
-- Up-to-date: /usr/local/include/faiss/IndexLSH.h
-- Up-to-date: /usr/local/include/faiss/IndexLattice.h
-- Up-to-date: /usr/local/include/faiss/IndexNNDescent.h
-- Up-to-date: /usr/local/include/faiss/IndexNSG.h
-- Up-to-date: /usr/local/include/faiss/IndexPQ.h
-- Up-to-date: /usr/local/include/faiss/IndexPQFastScan.h
-- Up-to-date: /usr/local/include/faiss/IndexPreTransform.h
-- Up-to-date: /usr/local/include/faiss/IndexRefine.h
-- Up-to-date: /usr/local/include/faiss/IndexReplicas.h
-- Up-to-date: /usr/local/include/faiss/IndexResidual.h
-- Up-to-date: /usr/local/include/faiss/IndexScalarQuantizer.h
-- Up-to-date: /usr/local/include/faiss/IndexShards.h
-- Up-to-date: /usr/local/include/faiss/MatrixStats.h
-- Up-to-date: /usr/local/include/faiss/MetaIndexes.h
-- Up-to-date: /usr/local/include/faiss/MetricType.h
-- Up-to-date: /usr/local/include/faiss/VectorTransform.h
-- Up-to-date: /usr/local/include/faiss/clone_index.h
-- Up-to-date: /usr/local/include/faiss/index_factory.h
-- Up-to-date: /usr/local/include/faiss/index_io.h
-- Up-to-date: /usr/local/include/faiss/impl/AdditiveQuantizer.h
-- Up-to-date: /usr/local/include/faiss/impl/AuxIndexStructures.h
-- Up-to-date: /usr/local/include/faiss/impl/FaissAssert.h
-- Up-to-date: /usr/local/include/faiss/impl/FaissException.h
-- Up-to-date: /usr/local/include/faiss/impl/HNSW.h
-- Up-to-date: /usr/local/include/faiss/impl/LocalSearchQuantizer.h
-- Up-to-date: /usr/local/include/faiss/impl/NNDescent.h
-- Up-to-date: /usr/local/include/faiss/impl/NSG.h
-- Up-to-date: /usr/local/include/faiss/impl/PolysemousTraining.h
-- Up-to-date: /usr/local/include/faiss/impl/ProductQuantizer-inl.h
-- Up-to-date: /usr/local/include/faiss/impl/ProductQuantizer.h
-- Up-to-date: /usr/local/include/faiss/impl/ResidualQuantizer.h
-- Up-to-date: /usr/local/include/faiss/impl/ResultHandler.h
-- Up-to-date: /usr/local/include/faiss/impl/ScalarQuantizer.h
-- Up-to-date: /usr/local/include/faiss/impl/ThreadedIndex-inl.h
-- Up-to-date: /usr/local/include/faiss/impl/ThreadedIndex.h
-- Up-to-date: /usr/local/include/faiss/impl/io.h
-- Up-to-date: /usr/local/include/faiss/impl/io_macros.h
-- Up-to-date: /usr/local/include/faiss/impl/lattice_Zn.h
-- Up-to-date: /usr/local/include/faiss/impl/platform_macros.h
-- Up-to-date: /usr/local/include/faiss/impl/pq4_fast_scan.h
-- Up-to-date: /usr/local/include/faiss/impl/simd_result_handlers.h
-- Up-to-date: /usr/local/include/faiss/invlists/BlockInvertedLists.h
-- Up-to-date: /usr/local/include/faiss/invlists/DirectMap.h
-- Up-to-date: /usr/local/include/faiss/invlists/InvertedLists.h
-- Up-to-date: /usr/local/include/faiss/invlists/InvertedListsIOHook.h
-- Up-to-date: /usr/local/include/faiss/utils/AlignedTable.h
-- Up-to-date: /usr/local/include/faiss/utils/Heap.h
-- Up-to-date: /usr/local/include/faiss/utils/WorkerThread.h
-- Up-to-date: /usr/local/include/faiss/utils/distances.h
-- Up-to-date: /usr/local/include/faiss/utils/extra_distances-inl.h
-- Up-to-date: /usr/local/include/faiss/utils/extra_distances.h
-- Up-to-date: /usr/local/include/faiss/utils/hamming-inl.h
-- Up-to-date: /usr/local/include/faiss/utils/hamming.h
-- Up-to-date: /usr/local/include/faiss/utils/ordered_key_value.h
-- Up-to-date: /usr/local/include/faiss/utils/partitioning.h
-- Up-to-date: /usr/local/include/faiss/utils/quantize_lut.h
-- Up-to-date: /usr/local/include/faiss/utils/random.h
-- Up-to-date: /usr/local/include/faiss/utils/simdlib.h
-- Up-to-date: /usr/local/include/faiss/utils/simdlib_avx2.h
-- Up-to-date: /usr/local/include/faiss/utils/simdlib_emulated.h
-- Up-to-date: /usr/local/include/faiss/utils/simdlib_neon.h
-- Up-to-date: /usr/local/include/faiss/utils/utils.h
-- Up-to-date: /usr/local/include/faiss/invlists/OnDiskInvertedLists.h
-- Up-to-date: /usr/local/share/faiss/faiss-config.cmake
-- Up-to-date: /usr/local/share/faiss/faiss-config-version.cmake
-- Up-to-date: /usr/local/share/faiss/faiss-targets.cmake
-- Up-to-date: /usr/local/share/faiss/faiss-targets-noconfig.cmake
-- Up-to-date: /usr/local/include/faiss/c_api/AutoTune_c.h
-- Up-to-date: /usr/local/include/faiss/c_api/Clustering_c.h
-- Up-to-date: /usr/local/include/faiss/c_api/IndexFlat_c.h
-- Up-to-date: /usr/local/include/faiss/c_api/IndexIVFFlat_c.h
-- Up-to-date: /usr/local/include/faiss/c_api/IndexIVF_c.h
-- Up-to-date: /usr/local/include/faiss/c_api/IndexLSH_c.h
-- Up-to-date: /usr/local/include/faiss/c_api/IndexPreTransform_c.h
-- Up-to-date: /usr/local/include/faiss/c_api/IndexScalarQuantizer_c.h
-- Up-to-date: /usr/local/include/faiss/c_api/IndexShards_c.h
-- Up-to-date: /usr/local/include/faiss/c_api/Index_c.h
-- Up-to-date: /usr/local/include/faiss/c_api/MetaIndexes_c.h
-- Up-to-date: /usr/local/include/faiss/c_api/VectorTransform_c.h
-- Up-to-date: /usr/local/include/faiss/c_api/clone_index_c.h
-- Up-to-date: /usr/local/include/faiss/c_api/error_c.h
-- Up-to-date: /usr/local/include/faiss/c_api/error_impl.h
-- Up-to-date: /usr/local/include/faiss/c_api/faiss_c.h
-- Up-to-date: /usr/local/include/faiss/c_api/impl/AuxIndexStructures_c.h
-- Up-to-date: /usr/local/include/faiss/c_api/index_factory_c.h
-- Up-to-date: /usr/local/include/faiss/c_api/index_io_c.h
-- Up-to-date: /usr/local/include/faiss/c_api/macros_impl.h
-- Up-to-date: /usr/local/include/faiss/c_api/utils/distances_c.h

[p-ouellette]

Is it in build/faiss?

[MrWWheat]

No, I can only find libfaiss.dylib, I searched all files in faiss, and there is no libfaiss.so...

[p-ouellette]

Do you have libfaiss_c.so in build/c_api? If not, you could try building again with -DFAISS_ENABLE_PYTHON=OFF passed to cmake. You got the same error as #5.

[MrWWheat]

Yes, I tried again, my commands are:

cmake -B build -DFAISS_ENABLE_GPU=OFF -DFAISS_ENABLE_PYTHON=OFF -DFAISS_ENABLE_C_API=ON -DBUILD_SHARED_LIBS=ON .
make -C build
make -C build install

But still no libfaiss.so in build...

[p-ouellette]

Oh I think MacOS uses .dylib instead of .so. It should work with the dylib.

[MrWWheat]

Yes, but when I run go get github.com/DataIntelligenceCrew/go-faiss, it shows:

# github.com/DataIntelligenceCrew/go-faiss
ld: library not found for -lfaiss_c
clang: error: linker command failed with exit code 1 (use -v to see invocation)

And libfaiss.dylib is already in the /usr/local/lib

[p-ouellette]

The error indicates that it can't find libfaiss_c, not libfaiss. Also /usr/local/lib might not be searched. You could try installing the library in /usr/lib instead or adding /usr/local/lib to DYLD_LIBRARY_PATH. I don't use MacOS so I'm not sure.