Updates for macOS Stability with NN Backend Options

[williamsyy]

Due to maintenance issues with the annoy library affecting macOS environments (#94, #136), we are introducing more flexibility in nearest-neighbor computations. We’ve added a new nn_backend parameter to PaCMAP and LocalMAP. You can now choose between:

• annoy (Default)

• faiss

• voyager

This ensures that macOS users have reliable alternatives when annoy fails. This update is the result of a fantastic community effort—special thanks to @Belluxx, @brurucy, @hyhuang00, @gauravrparikh, @jolespin, @BabaSanfour, @quentino-58, and @mathematicalmichael for their contributions to the discussion, and to @Tyrannas for providing the fix in #137.

[mathematicalmichael]

thanks for the acknowledgement and work here.
please consider putting these changes atop my earlier PR #105, in particular so that you can leverage the optional dependencies to manage this feature. Putting all three as requirements actually still poses installation issues for users (in particular the annoy wheels on windows with newer python versions).

I am happy to help with CI/CD here.

[jolespin]

I'd like to try out the dev branch for this. Is there a certain branch you recommend?

[williamsyy]

@jolespin Thanks for your interest. You can try on the nn_backend branch for now if you are interested. We are still figuring out the dependency issues between voyager and faiss in python 3.13 and 3.8 (mainly due to the unreleased voyager 2.1.1 (which supports python 3.13, and the updated nanobind package version that voyager used in python 3.8)

[williamsyy]

Hi all, this PR still needs some work on the dependency issue for 3.8 nanobind for voyager. We are making the decision on whether to still maintain the 3.8 version or we need to set up stricter constraints for it. We will try to make the decision as soon as possible so hopefullly this PR can be done this week or next.

[BabaSanfour]

would there be experiments on the difference between using voyager; faiss and annoy ?

[williamsyy]

@BabaSanfour If you run pytest within the nn_backend branch for now you can see all the results in ./test/output directory. Here is one of the tests that we have done for a subset of MNIST dataset:

We will do more tests soon, but you can first try on the pytest results. There are no significant difference based on visuals.

[BabaSanfour]

@williamsyy ! amazing work! thanks for the work.

[williamsyy]

@mathematicalmichael Thanks for your work, we will also take that into consideration.

[mathematicalmichael]

Hi all, this PR still needs some work on the dependency issue for 3.8 nanobind for voyager. We are making the decision on whether to still maintain the 3.8 version or we need to set up stricter constraints for it. We will try to make the decision as soon as possible so hopefullly this PR can be done this week or next.

again, stuff like this would be easier to manage in pyproject.toml, you can simply omit voyager as an optional dependency if the python version is just 3.8, without impacting your overall compatibility with the older version. setup.py is a regression from using setup.cfg, and pyproject.toml is the current norm for all intents and purposes.

@williamsyy just say the word and I will jump in and help merge the two branches together (plus you'll get automated publishing to pypi instead of manual uploads). The only reason I'm hesitant to do so without your consent is because my previous PR has sat open and ready to merge for a long time without response.

I've been dealing with the pains of python packaging for the better part of 8 years now... help me help you. =)

[williamsyy]

@mathematicalmichael I get your idea. The point is that whehter we would like to support voyager in 3.8 for pacmap (as it should work but since they haven't updated the code for a while so they might not be aware of those updates). It is not just about setting up things to optional, but to see if any of our current users still relies on a lower version of python and pacmap in their pipelines. We are still making the decision, but I think most likely we will just depreciated the support for 3.8 to make things easier.

[mathematicalmichael]

I understand. but even if you choose to drop 3.8 support (I think that's fine, it's been long enough), you still should not be putting all three libraries in install_requires, as it (a) creates redundancy, (b) bloats the installation, and (c) does not solve annoy installation issues for Python>3.12. This creates more problems than it solves.

just put faiss as the required dep, as it's the most well-maintained / has wheels from 3.8 onwards.

I suggest using optional deps based on the names of the alternatives ANN options:
pacmap[voyager], pacmap[annoy], (possibly even include pacmap[annoy-mm] for my fork of it, which has wheels through Python 3.13 + wheels for Windows).

and again, it's a lot easier to define these in pyproject.toml.

also I think you should bump your minor version number to indicate the change in dependencies, so you won't mess up installations that use ~= pins against pacmap.

[williamsyy]

Yes, that part is not fixed yet, but will be done before it is released to pypi and conda. There are more considerations about performance, dependency support for other packages that relies on our package from our side. We don't want to mess them up as well. Thanks for your suggestions and we really appreciate your work and interest towards pacmap.

[gauravrparikh]

I think @hyhuang00 has raised some good refactoring suggestions to improve readability but apart from that this code looks good to merge.