Have one supported dev setup path?

[mfisher87]

I think this is resurrecting an old conversation, but we should totally do that every so often :)

cc @chuckwondo

The idea of removing our environment.yml file as a dev environment setup pattern came up again here #1244 (comment)

I fully support this.

Why not (IMO)?

The cognitive load of using a conda-on-top-of-pip (or is it better to say it's "underneath" pip?) setup is very high (less so with pixi, but let's not go there). You need to know about virtual environments, how to activate them, how to avoid accidentally installing things in the base environment, you need to remember to deactivate to avoid cross-project pollution, you probably need to know how to blow away and recreate the environment, you need to know that using pip in an activated conda environment will install dependencies in that environment (IFF you installed pip in the environment, and you should probably know to which pip to be sure) , and most annoying to me, you need to remember and type out the names of environments.

We absolutely cannot guarantee success or happiness with conda.

And I think, in most cases, if a contributor is experienced enough with conda to insist on it for development, they know how to proceed.

Counterarguments

I believe the argument I've heard before is that this is helping to meet developers where they are. I think this is a noble goal and important.

But I think it's more nuanced, we need to ask "at what cost?" If we meet a user where they are and they're unhappy, did we succeed? Can we lead users to happiness while respecting where they're at now? (e.g. making the transition enjoyable and satisfying and demonstrating how it pays off. You know what I mean! Openscapes things!)

the uv rave you didn't ask for

And I think we should further decide on uv as the "one true path" for setting up a dev environment for many reasons. I think uv is easier for new developers because it reduces the number of concepts the developer must understand to work. You don't need to know how to install Python, it just does it; you don't need to know how to set up a venv, or even that you need a venv, or how to activate it, or concepts like source/. (new developer: "wtf is this magic incantation I'm doing??"), it just does it. And you don't need to worry about dev dependencies, it just does it. All with uv sync. One concept! And then you need to know to use uv run to run commands specific to this project. Two concepts total.

Most importantly, we can more-or-less guarantee success solely by writing solid config (e.g. using the dev dependency group), and through success, happiness. The pitfalls of environment management are gone! 🪄 ✨ 💨

Oh yeah, and it's fast! 😁 That's just an afterthought, far less important IMO than the above ☝️

[itcarroll]

I wish GitHub allowed the 🍿 reaction.

Can't wait for @jhkennedy to chime in (who I think I'll be inclined to agree with 😁).

[jhkennedy]

I think this is resurrecting an old conversation, but we should totally do that every so often :)

Yes, we hashed this out multiple times already (e.g., #614, #741) and we currently suggest: venv, conda/mamba and pipx/nox.

Notably, as hinted at in the issue, there's more tools here than just the environment manager; we also should consider the task-runner (e.g., how do we invoke tests? how do analyze our code?) and the things the task-runner executes (the testing framework).

Currently, for development tasks we use nox, which actually uses uv or virtualenv on the back end and sets up independent environments for testing and docs. The completely abstracts away the necessary environments much the same way uv run does.

So, we actually don't require a development environment to develop earthaccess. We describe our dependencies using normal PEP-621 compliant pyproject.toml and users are able to install and set up their development environment however they choose (or to not set one up at all).

The proposal here seems to be to only endorse a single environment manager for the development environment, even though one isn't required currently. And not requiring one would still be true if we switched to uv for task running.

The cognitive load of using a conda-on-top-of-pip (or is it better to say it's "underneath" pip?) setup is very high (less so with pixi, but let's not go there). You need to know about virtual environments, how to activate them, how to avoid accidentally installing things in the base environment, you need to remember to deactivate to avoid cross-project pollution, you probably need to know how to blow away and recreate the environment, you need to know that using pip in an activated conda environment will install dependencies in that environment (IFF you installed pip in the environment, and you should probably know to which pip to be sure) , and most annoying to me, you need to remember and type out the names of environments.

All of this is also true of pip, pipx, venv, uv, pixi, etc. If you're using a development environment you do, in fact, need to know how it's environment manager works.

Supporting development with conda/mamba so far amounts to including a single environment.yml file that just pip installs everything from all the testing/docs environments into a mamba environment so users can do:

mamba env create -f environment.yml
mamba activate earthaccess

instead of

mamba create -n earthaccess python pip
mamba activate earthaccess
python -m pip install -e ".[dev,test,docs]"

and notably, having to "just know" what our optional dependency groups are in this case, or which ones matter.

If you don't want to use conda/mamba, that file has no effect on you, because again, dependencies are described in the pyproject.toml.

But I think it's more nuanced, we need to ask "at what cost?" If we meet a user where they are and they're unhappy, did we succeed? Can we lead users to happiness while respecting where they're at now? (e.g. making the transition enjoyable and satisfying and demonstrating how it pays off. You know what I mean! Openscapes things!)

Is there a cost here? There is no real measurable cost described in this issue.

Maybe it's because my work and research is in the SAR and Earth System Modeling spaces, but I use conda primarily because that actual data processing tools I use and develop are largely not pure python. It make me "happy" and reduces my "cognitative load" to not have to use a different environment/development pattern for earthaccess. I've been slowly transitioning to pixi for new projects for many of the uv-reasons stated here, but balancing two different ways of setting up my development environments and getting developer tools to understand them is annoying. I'm not interested in a third.

Earthaccess serve a broad community, many of them are not pure python communities. I don't think earthaccess should care if you develop it with pip, uv, pixi, conda, mamba, pyenv, etc. And this was one of the main drivers to switch from poetry -- standards compliant dependencies so any tool can set up the development environment.

And I think we should further decide on uv as the "one true path"

I'd be in favor of switching to uv for task running over nox. uv's uptake in the community is huge, it's easier to install, and does handle more things in the same space. Or pixi, because all the same arguments apply.

I'm not in favor of removing the environment.yml.

I am not in favor of removing documentation on installing the necessary development dependencies into an environment and telling developerss how they should work instead of working with them.

I would be fine with streamlining the docs to recommend uv as long as we still tell developers what they need to do to install the necessary development dependencies (e.g., the pip command).

[mfisher87]

First, I know I've hit on a topic that's perhaps a bit sensitive. I've tried to boil down the useful part of my reply to the bolded section. I'd like to request that we focus on where we can find agreement, because I tend to find myself drawn to responding point-by-point and I acknowledge sometimes that's not productive. Nonetheless, I've kinda done that 😬 Mostly because I want to be sure we're understanding each other and connecting with each other's ideas and being generous in our interpretation of each others' positions. But maybe most of this post is not actually useful. I'm entertained by Ian's 🍿 comment, but I really want this conversation to be boring af :) Sorry, Ian ;)

So, we actually don't require a development environment to develop earthaccess

I'm not sure I understand. We do, it's just a question of what tool the user uses (i.e. nox creates environments that are used during development) and how much the tool abstracts away -- is that what you mean?

The cognitive load of using a conda-on-top-of-pip (or is it better to say it's "underneath" pip?) setup is very high (less so with pixi, but let's not go there). You need to know about virtual environments, how to activate them, how to avoid accidentally installing things in the base environment, you need to remember to deactivate to avoid cross-project pollution, you probably need to know how to blow away and recreate the environment, you need to know that using pip in an activated conda environment will install dependencies in that environment (IFF you installed pip in the environment, and you should probably know to which pip to be sure) , and most annoying to me, you need to remember and type out the names of environments.

All of this is also true of pip, pipx, venv, uv, pixi, etc. If you're using a development environment you do, in fact, need to know how it's environment manager works.

My argument boils down to what extent the user is required to understand environment management, not whether they need to think about it at all. The uv sync / uv run model (i.e. the "project" model, vs the traditional "environment" model) requires no understanding of where the environment is, what it's called, whether it's activated or not. Are you in the project directory? uv sync will do the right thing. Would you agree that pip/venv/pyenv/conda have a larger cognitive load than uv? Cognitive load is where I'm coming from.

Is there a cost here? There is no real measurable cost described in this issue.

I would assert "yes", and that costs aren't necessarily measurable. The complexity of the docs both for authors and readers is a cost. And the complexity, cognitive load, and pitfalls of the toolsets we encourage users to use has a cost, and I'd argue this is the greater cost that can make or break the experience that could convert someone to a contributor and eventually leader in the project! This is very hard to measure. So, to be fair, you're right about "measurable", but that doesn't mean I didn't describe any costs :)

I would be fine with streamlining the docs to recommend uv as long as we still tell developers what they need to do to install the necessary development dependencies (e.g., the pip command).

I definitely recognize I'm a dog with a bone 🐶 🦴 on this particular issue. But I'm definitely not so _dog_matic (😁) that I would argue we need to go straight for what I believe is the ideal. I still do believe that "one way that is in line with the larger community and will minimize cognitive load and pitfalls" is the ideal, but I feel what you're suggesting is a strong step in the right direction.

How would you feel about replacing the venv tab in our development guide docs with "uv (recommended)"? I cringe at the idea of adding one more tab, but I would not complain too loudly if we can't agree on removing venv.

To me, "streamlining" means removing some of the branching paths, though. Is this what you meant? I think maybe it's more likely that you mean something like front-loading the recommended path and moving the other options further down in the docs. I.e. "recommended way" first then "other ways" separate, as opposed to the approach we currently take with tabs?

I'd be in favor of switching to uv for task running over nox.

Unfortunately, uv does not yet have pixi-like task running functionality. astral-sh/uv#5903 I wish ... 😭 And there's no PEP for task specification yet, afaik.

Maybe it's because my work and research is in the SAR and Earth System Modeling spaces, but I use conda primarily because that actual data processing tools I use and develop are largely not pure python. It make me "happy" and reduces my "cognitative load" to not have to use a different environment/development pattern for earthaccess.

I don't like the framing of "have to" here. Not trying to be nitpicky, I just want to make sure we're not talking past each other, misunderstanding each other's claims, and interpreting each other generously. Contributors of course don't have to use a different pattern, I'm just suggesting that we only teach people how to use the pattern that has the fewest pitfalls. If they know another pattern they are comfortable with, by all means, they can use it! But I feel the degree to which we support those other patterns should be more limited than our recommended pattern.