Ideas for updated CLI in `conda-project`

[mattkram]

In preparation of upcoming hackdays, I'm collecting some ideas around how the CLI could be refined when converting anaconda-project to a potential conda project extension.

This table summarizes potential renaming's of sub-commands, which would all be prefixed by conda project or `conda-project.

After typing this, my personal preference is Alternative 2, where essentially many of the hyphenated subcommands are further broken down into higher level sub-commands such as service with operations like add, remove, list, etc.

If would also be great if somehow the conda install PACKAGE command could somehow be intercepted from within an activated environment to add to anaconda-project.yaml using the current behavior of anaconda-project add-packages.

Existing	New Alt 1	New Alt 2	Description
init			Initialize a directory with default project configuration
run			Run the project, setting up requirements first
prepare	install	install	Set up the project requirements, but does not run the project
clean			Removes generated state (stops services, deletes environment files, etc)
archive			Create a .zip, .tar.gz, or .tar.bz2 archive with project files in it
unarchive			Unpack a .zip, .tar.gz, or .tar.bz2 archive with project files in it
upload			Upload the project to Anaconda Cloud
download			Download the project from Anaconda Cloud
add-variable	add --variable	variable add	Add a required environment variable to the project
remove-variable	remove --variable	variable remove	Remove an environment variable from the project
list-variables	list --variable	variable list	List all variables on the project
set-variable	set --variable	variable set	Set an environment variable value in anaconda-project-local.yml
unset-variable	unset --variable	variable unset	Unset an environment variable value from anaconda-project-local.yml
add-download	add --data	data add	Add a URL to be downloaded before running commands
remove-download	remove --data	data remove	Remove a download from the project and from the filesystem
list-downloads	list --data	data list	List all downloads on the project
add-service	add --service	service add	Add a service to be available before running commands
remove-service	remove --service	service remove	Remove a service from the project
list-services	list --service	service list	List services present in the project
add-env-spec	?	env add	Add a new environment spec to the project
remove-env-spec	?	env remove	Remove an environment spec from the project
list-env-specs	?	env list	List all environment specs for the project
export-env-spec	?	env export	Save an environment spec as a conda environment file
lock			Lock all packages at their current versions
unlock			Remove locked package versions
update			Update all packages to their latest versions
add-packages	add	package add	Add packages to one or all project environments
remove-packages	remove	package remove	Remove packages from one or all project environments
list-packages	list	package list	List packages for an environment on the project
add-platforms	add --platform	platform add	Add platforms to one or all project environments
remove-platforms	remove --platform	platform remove	Remove platforms from one or all project environments
list-platforms	list --platform	platform list	List platforms for an environment on the project
add-command	add --command	command add	Add a new command to the project
remove-command	remove --command	command remove	Remove a command from the project
list-default-command	list --default-command	command list --default	List only the default command on the project
list-commands	list --command	command list	List the commands on the project

[AlbertDeFusco]

Nice! I've been thinking about adding some optional arguments to init. Right now it would create an empty dependencies list. We could do

conda project init [-c CHANNEL] [package_spec [package_spec ...]]

so it feels a little more like conda create

[mattkram]

I really like that idea. Every new project now I do conda env create -p ./env -f environment.yml. Actually can't remember if I'm using conda create or conda env create. I always mess it up.

So I think it'd be awesome if init basically did that, and you could pass in a path for your environment and also load a package spec quickly from an existing file.

Another feature would be an --interactive flag, which would give a list of questions to fill in the parts of anaconda-project.yml. Can't remember if that exists already.

[mattkram]

Another stretch goal would be to move config to pyproject.toml and somehow register conda as a build backend, but that's a lot harder to do I think. The benefit is the project spec would be similar to those by other tools like poetry and flit.

[jlstevens]

Why not make conda project init env more like conda env create by renaming it to conda project create?

[jlstevens]

Also, I do think the sub-command proposal above (Alternative 2) would indeed help tidy things up!

[jlstevens]

With this proposal, I count 18 'top level' conda-project commands (11 without subcommands, 7 with).

Here are the ones currently proposed without subcommands, though I have some suggestions for more subcommands which would bring it down to 16 commands total. I have also attempted to order these by how often I use them personally:

1. init (rename to create for consistency with conda env create: it would be conda project create?)
2. run
3. prepare (rename to install?),
4. lock
5. unlock (how about a lock remove subcommand?)
6. archive
7. unarchive (how about an archive extract subcommand?)
8. upload
9. download (any way to combine with upload?)
10. update
11. clean (I've never used this!)

Proposed subcommands (also roughly ordered by my frequency of usage):

12. command X
13. data X
14. env X
15. variable X
16. package X
17. platform X
18. service X (I've never used this either!)

Of these, I could do away with upload and download (unless we integrate with Nucleus?) and clean. I also have never seen services used either...

Lastly, I never use add commands as I don't like commands that mutate the project.yml (though I know some people do use them, e.g. @AlbertDeFusco when giving tutorials).

[mattkram]

Interesting perspective re: add. I have precisely the opposite perspective, where I avoid changing the project.yml manually and try to rely on CLI commands (when I use poetry).

I think clean is actually nice. One thing that is missing IMO from poetry is no easy way to tear down the created environment (especially since that venv gets stored centrally).

upload/download feel like Nucleus/AE5 features. Unsure whether those would better be included in a separate CLI for Nucleus (which we've talked about before).

[AlbertDeFusco]

Clean also removes downloaded data (defined in the downloads: key) and any locally set variables (anaconda-project-local.yml)

[AlbertDeFusco]

The service feature was never fully developed and maybe it doesn't really need to be there. For example the conda package redis no longer provides the redis server, but only the python bindings.

https://anaconda-project.readthedocs.io/en/latest/user-guide/reference.html#services

[AlbertDeFusco]

And don't forget dockerize

[kenodegard]

I concur, I think there's value in following conventions and usage from npm & poetry to lower the conversion barrier from those tools, so I vote to keep the install and add commands. That said I also agree with being consistent with other conda commands, so aliasing create to install would also be valuable. Docs/tutorials teaching npm/poetry -> conda project could use the install alias and docs/tutorials teaching conda env -> conda project could use the create alias.

Given the plugin efforts in conda it wouldn't be crazy to move upload/download (and other "adjacent" features) into separate plugins, help keep each plugin "pure".

Originally I wanted to vote for the proposed alt 2 CLI (bc it followed the subcommand logic used by git) but tucking the add, remove, and list subcommands that far down seems highly undesirable. I'd expect to be able to run conda project add PACKAGE so my vote is for alt 1 CLI.

I would also like to see us including a cpm alias for conda project, even less typing!

[mattkram]

+1 for cpm! I was already trying to think of how I'd add to my .bashrc, but cp is obviously ripe w/ problems :P

[jbednar]

What's the "m" stand for?

I don't use npm or poetry, and am instead coming from a background of expecting to edit an environment.yml or requirements.txt file in a text editor. I'm quite distressed when I run some anaconda-project commands and find that the .yml file has been modified for no apparent reason, e.g. due to a recent bug that is somehow duplicating the contents of the user-fields section every time I run some command that has nothing to do with it. (Not sure what command that is, but it's definitely not one that I'd expect to modify the file, and the fact that it modifies it at all, even apart from the buggy way it modifies it, is deeply distressing to me.)

So it sounds like I want the opposite of what some people here want -- I'd either want no circumstance where [ana]conda-project modifies the file, or for all such commands to be organized under an edit submenu or a fully separate command that I could then safely avoid. I know I'm more text-file-centric than most people, so maybe I'm an outlier, but to me such modification violates a contract with the user that the user is in charge of the text file, not the program; it's up to me to write it and the program to respect it! Once the program itself is modifying it in any way, I'm immediately suspicious and ill at ease -- what happens to my edits? Will comments be dropped? Will the file be reformatted? If I make a syntax error, will it silently delete all my work? How can I be sure? I deeply do not like the feeling that I'm not fully in charge of this config file. Again, maybe I'm an outlier. :-/