Releasing ISF vs using ISF in the lab

[bgmeulem]

@abast recently hit the nail on the head by saying that "releasing ISF and using ISF in the lab are two diverging objectives" (loosely paraphrased).

The more I work with pixi, the more I feel like this could solve our issue of these two diverging objectives. A small recap of what the issue exactly is.

The problem

For use within our lab, we need to pin down some package versions. msgpack being the most prevalent, but it also requires us to pin down e.g. neuron, flask, pyyaml and numpy. Essentially, a large part of the Python ecosystem needs to be frozen for us to be able to work with older code and older data. Dropping support for our older code/data is not really an option. We have terabytes of useful data, and reproducing results is quite high on the priority list.

On the other hand, releasing ISF requires me to make it user-friendly, and preferably also future-proof. I am mostly worried about the "future-proof" part. Sure, we can enforce users to only use ISF in the specific same way that we do it, but this has proven to be a bit of a pain at best, and nigh impossible at worst. Some examples:

1. installing ISF cross-platform (MacOS) is a request I've had twice. Many older package version (like neuron) were only available through the usual repo's at later versions. Simply installing ISF on more recent versions of linux, or recent versions of gcc is also not possible with the usual installation method (see issue Installation breaks on newer distros when using anaconda #258). This is already solved with pixi though.
2. Building documentation with sphinx-autoapi requires a version of flask that is less ancient.
3. I am not a fan of upgrading just for the sake if things being newer. However, not being able to upgrade is a different issue entirely. Whatever new or useful functionality comes out, there's a good chance we may not be able to use it because we're running old versions of integral packages that we can not upgrade: numpy, pandas and NEURON. Having access to package upgrades, if possible, can massively improve functionality, performance, and usability. Simply updating our neuron version could already provide a simulation speedup without too much hassle.

The solution

pixi allows to define separate environments through the use of separate features. One feature can e.g. be "building documentation". One feature can be "actually running ISF code" (which would then be the default feature). I'm already using this feature to specify an older version of flask for actually running ISF code (i.e. the default environment), and a newer version for building the documentation. And here comes my idea: what if one feature was "being able to read in old data and run older code".

The way this would work is as follows:

1. We define a "core" feature for ISF, which includes all the core packages. As many packages as possible go here.
2. We define a "default" or "isf" or "release" feature, with (if we want) updated code for neuron, pandas, numpy etc., and without pandas-msgpack.
3. We define an "ibs" feature for reading msgpack data and running older code. We can even define a "ibs-legacy" (name is a WIP) for Py2 code if we want to.
4. Only the packages that need to be different between the two, go in either category. E.g. the "release" feature will include a new numpy, and the "legacy" feature will contain pandas 1.1.3 and pandas-msgpack.
5. We define an environment for "release" or "default"by including ["core", "release"], we define an environment for IBS by including ["core", "ibs"], and we define a documentation env with ["core", "docs"] features.
6. We adapt the tests so that msgpack tests only need to pass for the "ibs" environment, and reproducibility tests can be relaxed for numerical truncation for the "release" version, but must be exact (like it is now) for the "ibs" version.

If this works, the IBS version would not need to be separated from an ISF release. IBS and ISF-"release" can happily co-exist. I argue that this way:

1. allows us to exactly control which aspects of ISF are specific to our use-case, and which aspects are ready for release
2. The end-user never needs to worry about this, until they implement a code-change that messes up our usecase. At this point, the test suite will complain.
3. we do not need to maintain two separate codebases

I can make a PR of this if we agree that this is the way forward.

Related issues:

• Installation breaks on newer distros when using anaconda #258
• Deprecate msgpack #252

[abast]

Pretty cool! I want to point out that you also have adressed this issue to some degree with isf_db, which doesn't rely on pickle for metadata. In many cases, we were locked in to old versions of pandas because past Arco thought saving metadata as a pandas data frame using pickle within some of the dask dumpers is a good idea (misled by the fact that dask dataframes specified metadata as a pandas data frame).As long as there is an installer to get access to a legacy environment, the ibs lab should be fine, and for the future remove pickle in as many places as possible. There still may be issues with the simulator / evaluator objects for which I didn't design a specific dumper.When working with collaborators, it would be ideal if an identical version to the one used by the respective person in IBS can easily be set up on their end. That's what I was using the offline installer for. It makes collaborations much easier if data can flow without having to worry about versions. Finally, my experience was that treating ISF with the mindset of 'LTS' software that should be kept stable was a much smoother experience than incorporating the newest version of everything. I remember one person who obsessively upgraded and had to spend a majority of time on version conflicts. That was at least one of my major goals when I created isf: make it such that people can cooperate without having to deal with that, such that I can reproduce my old stuff, such that data formats are decoupled from the way data is accessed, such that data formats can evolve without breaking code.On Jan 8, 2025, at 20:01, Bjorge Meulemeester ***@***.***> wrote: @abast recently hit the nail on the head by saying that "releasing ISF and using ISF in the lab are two diverging objectives" (loosely paraphrased). The more I work with pixi, the more I feel like this could solve our issue of these two diverging objectives. A small recap of what the issue exactly is. The problem For use within our lab, we need to pin down some package versions. msgpack being the most prevalent, but it also requires us to pin down e.g. neuron, flask, pyyaml and numpy. Essentially, a large part of the Python ecosystem needs to be frozen for us to be able to work with older code and older data. Dropping support for our older code/data is not really an option. We have terabytes of useful data, and reproducing results is quite high on the priority list. On the other hand, releasing ISF requires me to make it user-friendly, and preferably also future-proof. I am mostly worried about the "future-proof" part. Sure, we can enforce users to only use ISF in the specific same way that we do it, but this has proven to be a bit of a pain at best, and nigh impossible at worst. Exhibit 1: installing ISF cross-platform (MacOS) is a request I've had twice. Many older package version (like neuron) were only available through the usual repo's at later versions. Just installing ISF on more recent OS's is also not possible with the usual installation method (see issue #258). Exhibit 2: Building documentation with sphinx-autoapi requires a version of flask that is less ancient. Exhibit 3: Having access to package upgrades, if possible, can massively improve functionality, performance, and usability. Simply updating our neuron version could already provide a simulation speedup without too much hassle. Exhibit 4: I am not a fan of upgrading just for the sake if things being newer. However, not being able to upgrade is a different issue entirely. Whatever new or useful functionality comes out, there's a good chance we may not be able to use it because we're running old versions of integral packages that we can not upgrade: numpy, pandas and NEURON. The solution pixi allows to define separate environments through the use of separate features. One feature can e.g. be "building documentation". One feature can be "actually running ISF code" (which would then be the default feature). I'm already using this feature to specify an older version of flask for actually running ISF code (i.e. the default environment), and a newer version for building the documentation. And here comes my idea: what if one feature was "being able to read in old data and run older code". The way this would work is as follows: We define a "core" feature for ISF, which includes all the core packages. As many packages as possible go here. We define a "default" or "isf" or "release" feature, with (if we want) updated code for neuron, pandas, numpy etc., and without pandas-msgpack. We define an "ibs" feature for reading msgpack data and running older code. We can even define a "ibs-legacy" (name is a WIP) for Py2 code if we want to. Only the packages that need to be different between the two, go in either category. E.g. the "release" feature will include a new numpy, and the "legacy" feature will contain pandas 1.1.3 and pandas-msgpack. We define an environment for "release" or "default"by including ["core", "release"], we define an environment for IBS by including ["core", "ibs"], and we define a documentation env with ["core", "docs"] features. We adapt the tests so that msgpack tests only need to pass for the "ibs" environment, and reproducibility tests can be relaxed for numerical truncation for the "release" version, but must be exact (like it is now) for the "ibs" version. If this works, the IBS version would not need to be separated from an ISF release. IBS and ISF-"release" can happily co-exist. I argue that this way: allows us to exactly control which aspects of ISF are specific to our use-case, and which aspects are ready for release The end-user never needs to worry about this, until they implement a code-change that messes up our usecase. At this point, the test suite will complain. we do not need to maintain two separate codebases I can make a PR of this if we agree that this is the way forward. Related issues: #258 #252 —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[bgmeulem]

I agree with everything here! I think you'll be happy to know that this is the same philosphy as pixi. They define a manifest file that describes which versions should in principle be compatible, as well as a lockfile as soon as it has solved the environment. The lockfile specifies the exact versions of all packages installed (version, build number and repo). It also saves the resulting environment within the project, but this is not version tracked (although it can be made to be version tracked). Both files are version tracked, so if you know the commit that produced some result, you have the environment that produced some result (assuming you didn't mess with the env and didn't commit it, but at that point it's a bit your own fault I'd say).

I can keep around the original Anaconda installers (for Py3, and Py2 in case it's ever needed) - they don't do any harm. However, it may be possible that future versions of ISF implement some change that breaks an Anaconda installation.

If the only reason to keep these around is purely for reproducibility, I can point out that I saved a handful of releases of ISF: v0.1.0-alpha, v0.2.0-alpha and v0.3.0-beta all still support the original Py2 and Py3 installers.
Even the updated pixi installer planned for release 0.4.0 should in principle be perfectly backwards compatible in the sense that it exactly reproduces old results, as long as they were produced on linux and with Python 3. If we ever need to reproduce old results, simply git checkout v0.[1-4].0 does the trick. I can adapt the release notes so that they explicitly specify which versions/backwards compatibility they still support.

I propose that every major change (like changing the dependencies or removing/changing some API) should be subject to a proper release schedule, where we decide on the impact it will have, and plan accordingly. This is what you essentially already did with ISFDB: remove the option to write out to msgpack, but keep the option to read it, while adding deprecation warnings. If we ever plan to remove it entirely, we should ensure that at least one release exists where it is clearly noted that it will be removed.

At the same time, I have now also experienced the hassle of changing API, and I'm not planning on swapping stuff around nilly-willy. That being said, I want the API to be as LTS as possible right now, before this goes public. If all goes well and ISF catches on, we will be much less flexible to change API in the future compared to now. Hence why I'm pushing a bit to make somewhat "bigger" changes now, I don't think we will be as flexible in the future.

[bgmeulem]

We could (should?) move model_data_base to IBS_projects.

The current compatibility patch simply adds the entire package to the pythonpath. As a consequence, it doesn't really matter where this package is originally located. Arguably, end users don't need to see or know about this, and it does make the database package more difficult to understand.

Edit: in hindsight, this does not improve anything. I thought we could have a single database system this way instead if having to differentiate between isfdb and mdb in ISFs source code, but we would have to keep the directory structure as it is anyways. Fin.

[bgmeulem]

How would we feel about moving dendrite_thickness and spike_analysis to ibs_projects?

ibs_projects makes extensive use of spike_analysis for hot_zone, simulations_for_RM_generation, but ISF does not internally rely on it.