Why don't you use X instead?

"Why don't you use the X package instead?"

This is a question that gets asked of us all the time. Most of the time, we the answer is simple: we haven't had a chance to give it a thorough test yet.

This page collects tools that have been suggested for inclusion in easydata.

It also outlines known problems with the tools we currently use.

Virtual Environments

Conda

We do, most of the time. There are a few headaches. First, the conda binary is no longer added to the path when Anaconda is installed. Instead, some shell magic is used to invoke conda. This doesn't work in a Makefile, so we have to ask the user for the path to their conda binary (which there's no reason they should have to know).

Also, we rely extensively on conda-env, and Anaconda is threatening to replace this with an entirely new mechanism.

Next, the lock files we generate from conda-env's environment.yml files are not platform independent, so we can't claim reproducibility across platforms. "One nightmare at a time" is our only response at the moment.

Pipenv

We tried. We really did. In the end, while it had some nice features (notably pipenv shell for use in Makefiles), it made some decisions which made it less useful for our application. In particular, there's no way to specify the version of python required. It's also explicity designed for managing application dependencies, not libraries. These are pretty much fatal flaws for our use cases.

And for the record, no, it isn't python's recommended packaging tool. We got bit by that claim, too.

Poetry

We like the .toml format. We may give it a try soon.

DAG Managers

DVC

DAG Version Control. Essentially a git-for-datasets, plus the plumbing to create DAGS for data refinement or model generation. We are looking at this one.

Dataset Discovery

Intake

Yep. Looking at it. Would love a DataSet.from_intake() constructor.