Publish as conda package

[benjello]

Hi there!

I really enjoy OpenFisca, but I recently encountered an issue.

Here is what I did:

I want to advocate openfisca to non-dev (mainly economists) Windows users relying on (standard) Anaconda for Python stuff.

Here is what I expected to happen:

Install the conda package at least for openfisca-core.
Install using pip breaks because missing windows libraries.

Here is what actually happened:

I solved the problem by installing using conda as much packages (numpy etc) as i could then pip installing the openfisca-core package.

It would be nice to be able to install openfisca-core using conda.

Context

I identify more as a:

• Analyst (I make macroscopic computations on real populations).
• Business expert (I create tests and model legislation).
• Developer (I create tools that use the existing OpenFisca code).

[MattiSG]

Thanks @benjello for creating this issue! I also encountered this issue when installing on an M1 setup, and had to fix it with #990.I'm not sure I fully understand what creating a conda package entails, in terms of maintenance, technical consequences, and benefits for reusers. Do we have someone here with enough experience to make a recap? 😃

[bonjourmauko]

I have never used conda, but a couple of weeks ago I was trying to see how could be deliver Windows, Linux, and universal distributions with each version.

Situation

I did so because I wrote a C-Extension for OpenFisca and I wanted to see how difficult would it be for reusers and contributors if we decided to ship one.

C-Extensions, done properly, could provide a 2-5x performance improvement in costly operations.

Complications

It was already hard to make the thing compile in my own local 😄 .

When I did, I tried to package the whole thing for Linux and Windows, and this is what I found: you need Linux and Windows… so the CI will do fine for Linux, but I don't know how to package automatically for Windows.

Things I've explored so far

The prerequisite for me would be that a solution has to be integrated into the CI workflow:

• Using python setup.py bdist_wininst bdist_msi with wine (couldn't make it work)
• Using pyinstaller (couldn't make it work with the C-Ext, but haven't tried standalone)
• Using conda, haven't tried it yet but I found "Converting a package for use on all platforms" that might be what we would be trying to achieve.

Hope it helps :)

[sandcha]

I also have never used conda but from the Building conda packages from scratch page, creating a package looks like creating 3 files (meta.yaml, bld.bat and build.sh) that would use our setup.py to get the package dependencies and could target Pypi (or our GitHub repositories).

If this description isn't missing some step, the 3 files shouldn't need much maintenance and uploading the package to anaconda.org could be done by one command (+ install of this command + account on Anaconda.org). So, it looks like a step that we can add to the CI (and I agree that it should be integrated to the CI if we are looking for a sustainable solution).

As for the benefits for the users/economists on Windows, from what I understand, it would mean bringing openfisca to their usual environment instead of making them adapt the developers way of getting openfisca on another platform with all the mishaps that it comes with. And, from a perspective where installing a new version regularly is already painful and comes before any "real work", having to adapt is even a barrier to consider using openfisca.

So, I would say that it's time to find a non exotic solution for users on Windows. ^^'
Conda is one of them. Poetry might be another option 🤔

[benjello]

I would favor conda over Poetry for the kind of user I target.
It worked pretty well with my students. I asked them to install conda create ann env with a few important packages that requires Windows specific libs (numpy pandas pytables etc).
Since I only used openfisca within notebooks, the openfisca packages were installed by pip within those notebooks.

Having a conda package would definitely ease the first steps.

I tried Poetry and I still have problem using it ...

[benoit-cty]

Hello,
For the CASD users Conda is preferred.

In addition, don't having Openfisca on Conda prevent others to push packages that rely on openfisca on conda.

I'm a contributor of CodeCarbon where we use github action to push on Conda.
And for Leximpact I use NBDev that have a system to push to conda using the pip package.

I could have a look at this issue.

[benoit-cty]

I achieve to upload OpenFisca-Core on Anaconda : https://anaconda.org/leximpact/openfisca-core

What I've done:

• Add conda/meta.yaml in the project
• docker run --rm -t -i -v $PWD:/src continuumio/miniconda3 /bin/bash
  • cd /src
  • conda install conda-build
  • conda build -c conda-forge --output-folder ./conda-out/ ./conda/
  • conda install anaconda-client
  • anaconda login
  • anaconda upload ./conda-out/noarch/openfisca-core-0.0.1-py_0.tar.bz2

BUT this not the best way to do it. See below.

[benoit-cty]

The way above publish the package in a specific "channel" (~ repositoruy) of Anaconda, to be on conda-forge, we need to use another procedure:
For publishing to Conda-Forge, we have to fork a project and made a PR back to it. There is automated build in Linux, OSX and Windows: conda-forge/staged-recipes#17429

• "Propose the change as a pull request. Your recipe will automatically be built on Windows, Linux, and OSX to test that it works, but the distribution will not yet be available on the conda-forge channel."
• "Once the recipe is ready it will be merged and a new "feedstock" repository will automatically be created for the recipe. The build and upload processes take place in the feedstock, and once complete the package will be available on the conda-forge channel."

A chat is available for conda-forge integration at https://gitter.im/conda-forge/conda-forge.github.io

The fork is here : https://github.com/openfisca/staged-recipes

The documentation on how to do it for OpenFisca is here : https://github.com/openfisca/staged-recipes/blob/main/recipes/openfisca-core/README.md