Environment File for Workshop

[lsetiawan]

Hi! I wanna start a list of all the packages that we will need for the workshop so that I can create an environment file. Thanks!

For client-only environment:

- ODM2PythonAPI
- YODA Tools
- requests 
- folium
- geopandas
- ulmo
- owslib
- scipy
- xarray
# Managing environments in jupyter notebook
- ipykernel (for environment switching)
- jupyter
- nb_conda_kernels

For server environment:

- WOFpy
# Managing environments in jupyter notebook
- ipykernel (for environment switching)
- jupyter
- nb_conda_kernels

[emiliom]

Do we need a single environment, or can we have 2-3 environments?

Mainly, I'd prefer to separate an environment that doesn't include any server software (WOFpy, ODM2 REST API, etc) from one that does.

I'll add to your list directly.

[lsetiawan]

Mainly, I'd prefer to separate an environment that doesn't include any server software (WOFpy, ODM2 REST API, etc) from one that does.

Sure! It'll be fun to figure out 😜 Go ahead and have separate environments. Thanks!

[ocefpaf]

Just to give my two cents here. We have tried multiple envs with IOOS and we gave up due to the cost of maintaining them. Even thought that is good practice, and sometimes necessary b/c some server packages are not available on Windows, we try to create a single one whenever possible.

TL;DR I am +0.5 to have separate envs b/c practicality beats purity, if multiple envs are hard to figure out they are not worth it.

[emiliom]

Thanks, @ocefpaf. Your advice is always very much welcome. In this case, for the workshop, the environment are not necessarily intended to be maintained for the long term, but are rather a near term convenience and to minimize the possibility of conflicts/problems. But more broadly, we definitely have discussed what envs and how many we should maintain, so your input helps.

[emiliom]

@lsetiawan, no rush on this (no need to respond this week!), but I'm posting this question to keep track of it and as documentation.

We want the environment file(s) to be usable to people locally, too. But jupyter is not listed. I'm guessing it's installed by default as a dependency to ipykernel?

I went ahead and created the odm2client env locally (conda env create -f clientenvironment.yml) on my laptop. jupyter notebook worked as expected; I guess I'll find out if some jupyter component is missing!

BTW, on my local install, I got these warnings:

dbus post-link :: /etc/machine-id not found ..
dbus post-link :: .. using /proc/sys/kernel/random/boot_id

I assume they're nothing to worry about.

[emiliom]

Regarding jupyter, I found the answer: jupyter is not fully functioning (it was broken in a subtle way). I had to install it (conda install jupyter) after creating the env, in order to have a functioning jupyter.

So, the question now will be: if jupyter is included in the environment file, does it cause any problems when using the env in jupyterhub?

Again, @lsetiawan, no need to reply this week. Just documenting what I find.

[lsetiawan]

Hi Emilio, Good to know! There are two ways that we can do. People install jupyter ipykernel, and nb_conda_kernels on their root conda environment. Install the odm2client environment. Then run jupyter notebook from root. In this case you'll see the environment as an option. Or add jupyter in the odm2client environment, activating the environment, then run jupyter notebook. In this case, you won't be able to switch environments defaulting to odm2client.

[emiliom]

Hmm. Let's discuss next week when you're back. I'm good with the set up this week, for the remaining tests and R&D I intend to do. eg, my progress on #5 -- pretty darn cool, if you ask me! I'll see if tomorrow I can test on JupyterHub the two notebooks I've got running already. I just pushed all my updates to the repo.

[ocefpaf]

People install jupyter
ipykernel, and nb_conda_kernels on their root conda environment

I usually do that. (In fact I go for a brute-force approach to always add those every where, even in envs b/c I never know if they are available in the root env or not and I do not want to write instructions to install something in the root env and then create the env. It is an overkill but it works 😬)

[lsetiawan]

I usually do that. (In fact I go for a brute-force approach to always add those every where, even in envs b/c I never know if they are available in the root env or not and I do not want to write instructions to install something in the root env and then create the env. It is an overkill but it works 😬)

Wow. I would need more hard drive space to do that! That's a lot of copy! 😛

[ocefpaf]

Wow. I would need more hard drive space to do that! That's a lot of copy! 😛

Not really b/c conda packages are hardlinks, so only a few extra bytes for the file metadata but they point to the same disk space.

[lsetiawan]

@ocefpaf I didn't know that. Are the file metadata supposed to be big files? I have about 16 environments currently in my laptop and it's taking 13GB of space. It seems like each environment created, with similar packages, copies everything that package needs.

[ocefpaf]

@ocefpaf I didn't know that. Are the file metadata supposed to be big files?

They are supposed to be tiny.

I have about 16 environments currently in my laptop and it's taking 13GB of space. It seems like each environment created, with similar packages, copies everything that package needs.

Unless something change that could be b/c the new env has updated packages and not a link to the old env :-/

I just tested creating the same env with different names and the space saved was not as good as I expected:

150M    TEST
110M    TEST0

~40 MB I am not sure why, maybe some packages, like python itself, are not linked.

[emiliom]

People install jupyter, ipykernel, and nb_conda_kernels on their root conda environment

I usually do that. (In fact I go for a brute-force approach to always add those every where, even in envs b/c I never know if they are available in the root env or not and I do not want to write instructions to install something in the root env and then create the env. It is an overkill but it works 😬)

I actually didn't know about that practice with jupyter + friends on conda root. It seems like an extra step for users (for local env installation), which is a potential hassle in the context of the workshop. Alternatively:

Or add jupyter in the odm2client environment, activating the environment, then run jupyter notebook. In this case, you won't be able to switch environments defaulting to odm2client.

That's a real bummer. But it seems "cleaner" to include these 3 packages in the env file itself.

So, which approach do you guys recommend we follow?