[NASA-GHG][Workshop] Summer School Workshop July 8-18

[batpad]

Context

The GHG Summer School will be holding a workshop with about 35 participants from July 8 - July 18. They ran a similar workshop last year, and most things seem smooth. Just creating a ticket here for visibility and for making sure we're fully prepared.

The instructors have created a custom image for the workshop and we've already deployed a custom profile for workshop participants that sets the default image and default resource requirements. Note: each container is allocated 61GB of RAM, so these are fairly large individual instances.

@yuvipanda has increased the IOPS on the EBS volume used for the home folders to (hopefully) mitigate any issues with the shared disk.

The workshop will begin at 9am Mountain Time on July 8th (realistically, post-registration, etc. people will start using the hub likely around 10am MT). @aprilmj - would you be able to help advising on who they can reach out to just in case of any emergency situations or issues with the hub?

What we need to do

We're mostly setup, but there's a few questions -

• It will be ~35 people each using instances with ~61GB RAM. Are we certain we won't face any issues with any arbitrary AWS limits or max nodepool sizes or anything that might prevent nodes from coming up? They seem ok with the longer startup time when new nodes come online, so I don't think we need to preprovision, I just want to make sure we don't hit any limits.
• They will be using local /tmp folders for some stuff. They are curious what the limit on the size of /tmp would be. They don't need more than 10GB per user, so I think there's enough space, but they would like to know what the limit is just in case.
• Home folders / NFS: we should double-check total available size and what they need. (cc @wildintellect )

cc @aprilmj

TODO:

• Double check how containers are getting placed on nodes and ensure the placement is sane for the instance size we are using for the workshop. - @sunu may need your help on this - we essentially want to double-check that we aren't leaving a lot of room on instances, etc.

Definition of Done

• Answers are communicated to workshop organizers
• Workshop runs smoothly

[sunu]

We tested spinning up multiple user pods but that always created a new node because of insufficient CPU availability in the existing nodes. I created #6309 to tweak the CPU requests a bit hoping that will help accommodate two user pods per node without leaving half the memory unused.

If this is not the best way to do it then we may have to use a separate node type to make sure we optimize the resource usage on nodes.

[aprilmj]

@batpad If they have issues with the hub, an email to support@2i2c.org will come to our engineering group. That's the fastest and clearest path to 2i2c help in an emergency.

[batpad]

Adding here: it seems folks will need more space in home directories. The current EBS volume size seems to be 200GB. I think we should plan for upto ~50GB per user, so we're looking at upping the EBS volume to about 1.8TB.

Sorry, this is a bit unfortunately and hopefully we can do things differently a bit next time - we will need to plan for some cleanup here post-workshop.

[batpad]

They will be using local /tmp folders for some stuff. They are curious what the limit on the size of /tmp would be. They don't need more than 10GB per user, so I think there's enough space, but they would like to know what the limit is just in case.

Doing a df -h after spinning up an instance and I see there is 80GB available on /. I'm assuming that's the total storage on the node, and if we are fitting two instances per node, that means each user will have roughly 40GB they can use. Accounting for system files, etc. - the answer I gave them was 30GB of /tmp/ space per user. @aprilmj would be good to confirm from someone at 2i2c if that seems about right.

cc @sunu

[jnywong]

Adding here: it seems folks will need more space in home directories. The current EBS volume size seems to be 200GB. I think we should plan for upto ~50GB per user, so we're looking at upping the EBS volume to about 1.8TB.

This request is noted! I'll let you know when that change gets deployed.

the answer I gave them was 30GB of /tmp/ space per user ... would be good to confirm from someone at 2i2c if that seems about right.

Yep, you are correct that's the node storage and I agree 30GB of /tmp per user sounds right. There is no quota placed here, so definitely an item to communicate to the workshop organisers to pass to users.

[jnywong]

[nasa-ghg, prod] Increase EBS volume size #6316 increases the EBS volume to 1.8T as requested, though there is a little less than that available since 178G is used.

The quota limits of 50G per user are still in place. Let me know if there's anything else you need!

[batpad]

This looks great to me, thank you @jnywong !

[jnywong]

Hey @batpad how did the first day go? Anything else you need from our side?

[batpad]

Whoops - sorry for the late response here -

We had no news, which was definitely good news :) . Things seem good with the infrastructure. Yesterday, received an email from Sourish about some glitches they were facing. The glitches seem to be frontend related and not infrastructure, but pasting the email below in case y'all have run into this on other hubs and if there are any known solutions:

Hi all,

Today is the third day of the summer school, and I’m noticing some small glitches in our notebooks with the python kernel. For example, I use tqdm to display a progress bar widget in my notebooks for some operations, which has worked flawlessly for the past year. However, today I’m getting this error occasionally:

<img width="379" height="186" alt="Image" src="https://github.com/user-attachments/assets/25537d46-9dcb-4e5d-99fc-76318c1a86c3" />
 
i.e., the tqdm widget does not display, and throws an error. Restarting the kernel doesn’t help either, the only way to fix this is to stop and restart the server. Occasionally, we are also seeing glitchy displays, like when scrolling up or down the display in the browser will start “shaking up and down” (sorry, not sure there’s a better way to describe it), like a sync problem on an old CRT monitor.


Was anything changed recently in the image?


Best,

Sourish

Of course, nothing has changed in the image, and this seems like occasional frontend glitchiness. Alex's response:

TQDM error, widget errors like that are usually some race condition with javascript libraries loading. Often a Browser refresh will fix the issue. Do you have a notebook we could try to repeat with? To confirm, it sometimes works? Just want to check because I see some historical issues where ipywidgets and tqdm widget get out of sync with their versions.

Glitchy displays, there have been many reports over the years of similar issues in the Jupyter ecosystem. A really common one is the notebook jumps down to a cell at the bottom randomly. I don't think the causes have been satisfactorily tracked down, and it seems to randomly show up in some very specific versions of Jupyter.
https://discourse.jupyter.org/t/scrolling-bug-makes-my-screen-shake-erratically/32421
One fix might be noted here https://discourse.jupyter.org/t/jupyterlab-version-4-2-1-barely-useable-due-to-a-variety-of-glitches/26535/5

We'll see if the rest of the team comes up with other ideas overnight.

These problems do seem intermittent and hard to reliably reproduce, and are fixed with a restart. Putting it here just in case anyone has any ideas / has seen this before.

cc @sunu

[jnywong]

That's super awesome news, thanks! We'll also review #6323 once the workshop concludes.