Consider other build options

[hinerm]

Issue 1: License

Details

I am 75% sure that [curvelops](https://github.com/PyLops/curvelops) is in violation of the CurveLab license. Specifically:

LICENSEE may incorporate FDCT with other source code developed by LICENSEE (Incorporated Work) for the sole purpose of using FDCT for research internal to LICENSEE and not for any other redistribution. LICENSEE may not distribute any Incorporated Work outside of LICENSEE, and specifically LICENSEE may not provide FDCT or any Incorporated Work under an open source license such as the Gnu Public License

1. I believe curvelops is an incorporated work, as it:
   • includes fdct headers
   • links to the .a fdct libraries
2. So curvelops probably should not be distributed, and should not be MIT licensed.

Q: Do we have any evidence that curvelops has permission from CurveLab to be distributed?

A: We have the OK for using curvelops!

Issue 2: Build complexity

Details

Putting the license issue aside, using curvelops requires a lot of manual installation: * building fftw from source or acquiring the pre-built libraries (only available on windows?) * building fdct from source * building curvelops itself from source

This likely requires managing conda environments as well.

This feels very fragile to me. I am wondering what our options might be to simplify things:

• Build scripts?
• Containers?

It would be ideal if tme-quant worked without curvelops but directed users to install the required libraries, with any environment variable management being transparent through user configuration.

Requirements

• This is approximately what I want the user experience to be:
  1. Download our "product"
  2. Agree to CurveLab license and download their bundle folder
  3. Run our downloaded "product". It asks them where the CurveLab download folder is (and links them to https://curvelet.org/download.php)
  4. Our "product" outputs a compiled, runnable binary

To achieve this, I think we need, at the least, a script which:

• installs necessary required software (e.g. python)
• Downloads and builds fftw source/binaries, curvelops source
• Sets the environment variables approriately
• Builds tme-quant/pycurvelets

I would like to try to go further. Ideally we would create something like a platform-specific virtual environment (e.g. docker container) that is capable of doing these steps in a controlled and consistent way: consuming the CurveLab folder from, and generating the compiled binary to,the user's local machine.

I do not know what the current container landscape is like so some exploration needs to be done here. In the past, Docker images could be annoying to set up and required extra software to be installed. Plus, file I/O between them was very annoying to configure. Ultimately we want as painless a process as we can create for the user, ideally not requiring any additional software installs or environment setup beyond the steps outline above.

A script is a great starting point since we probably can't use containers for Mac anyway. Just make sure that the script checks for existing tools (like conda) and installs them if missing.

NB: one technology I am interested in exploring here is Bilayers

[dlee489]

Regarding Issue 1:
The only documentation I was able to find relating to the licensing is this. That being said, I'm unsure as to whether curvelops actually has permission for distribution. I'm curious what the workaround for this would be though, as using the CurveLab library will always be a bottleneck as it stands when it comes to setup and distribution.

Regarding Issue 2:
I'm slightly confused by what you mean by

It would be ideal if tme-quant worked without curvelops but directed users to install the required libraries

If tme-quant wasn't using curvelops and we use Python, we'd still have to use the FDCT headers to write a Python wrapper for the C++ functions, no? Unless this is more relating to the friction in dev setup.

I also do like the idea of using a container with a Dockerfile bundling every download except for FDCT. If the user manually downloads the FDCT, I think we should be able to make a script that helps set up the FDCT library too.

[hinerm]

If tme-quant wasn't using curvelops and we use Python, we'd still have to use the FDCT headers to write a Python wrapper for the C++ functions, no?

Yeah I wrote that over-simplified. When I say "worked without curvelops" I meant in the sense "doesn't throw exceptions without curvelops". If a curvelet routine was called but CurveLab/curvelops isn't present then I would want tme-quant to inform the user how to add CurveLab appropriately.

[hinerm]

Thinking about this a bit more, I am thinking we should definitely distribute:

• Curvelops source
• FFTW source or binaries

Ideally we want to get to a "one-click" solution that we point to a CurveLab location:

• Either a virtual environment that contains all the other source/binaries and build tools, allowing a consistent build process
• Or a build script that installs necessary tools/utils and sets everything up

What might be cool is a virtual environment that contains everything for building but "outputs" into the user's environment a compiled binary ready for use (so you don't have to use the virtual env for actual runs)

[dlee489]

I'm not entirely familiar with how the process of creating such a "one click" solution can be done, but I'm down for the idea and willing to help as the difficulty in setting up FDCT is definitely a difficulty for users.

Also sounds like something interesting to implement 😄

[hinerm]

I have updated this issue based on our current knowledge and goals. @Phil-Dua will start looking into container-style solutions (or alternatives - I'm open to anything that satisfies the outlined requirements) in the near future.

[Phil-Dua]

Newest commit: e864d00, updated automated setup, added automated one-click installation script and updated documentation (with Napari-plugin this time)

[hinerm]

In commit messages we can say "Closes xxx" with either #issue_number or a full link to the issue. Then when these commits hit main the issue auto-closes. I prefer that to manually closing issue.

If we do manually close an issue and link to the closing commit/branch, it should be something on main.

[Phil-Dua]

Should this issue be closed by PR 26 as well? @ctrueden

[ctrueden]

@Phil-Dua I think so, yeah. #26 adds the one-stop bin/install.sh script that does everything it can do, resulting in a constructed Python environment in .venv that includes a working curvelops. The user does need to download CurveLab as discussed above, but that's a legal requirement.

About Docker: my understanding is that we don't have permission legally to redistribute a binary build of curvelops? Therefore, we cannot push a Docker image containing that built binary to DockerHub or other public Docker image repository. I would advise not to bother with Docker, and in fact to remove the Docker stuff from this project unless there is a clear use case I'm not seeing.