Skip to content

updates to cronjob to no longer use image name/size #20

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
DanNiESh merged 1 commit into from
Aug 28, 2025
Merged

updates to cronjob to no longer use image name/size #20

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 2 additions & 4 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,11 +38,9 @@ Alternatively, to run the script immediately:

### nb-culler

This cronjob runs once every hours at the top of the hour, exclusively applied to notebooks associated with specific user group and will not impact other notebooks within the rhods-notebooks namespace. The cronjob performs the following actions:
This cronjob runs once every hours at the top of the hour, exclusively applied to notebooks associated with specific user group and will not impact other notebooks within the rhods-notebooks namespace. The cronjob performs the following action:

1. **Shuts down notebooks exceeding X hours of runtime**: any notebook found to have been running for more than X hours will be gracefully shut down to conserve resources. PVCs persist the shutdown process.
2. **Deletes notebooks with wrong images**: students are allowed to launch notebook instances with their class image. Notebooks that are running images that are not approved for use will be deleted along with their associated PVCs.
3. **Deletes notebooks with wrong container size**: notebooks that are configured with container sizes other than **X Small** will be deleted, including their PVCs.

To add resources to the rhods-notebooks namespace:

Expand All @@ -52,7 +50,7 @@ To add resources to the rhods-notebooks namespace:
oc project rhods-notebooks
```

3. Ensure the environment variables for `GROUP_NAME`, `CUTOFF_TIME` (seconds), `IMAGE_NAME` are correctly set.
3. Ensure the environment variables for `GROUP_NAME`, and `CUTOFF_TIME` (seconds) are correctly set.

4. From cronjobs/nb-culler/ directory run:
```
Expand Down
56 changes: 23 additions & 33 deletions cronjobs/nb-culler/cronjob.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,43 +62,37 @@ spec:
user_in_group2=true
fi

if $user_in_group1 && $user_in_group2; then
# Handle both group conditions
if [[ $image != *$IMAGE_NAME_1* && $image != *$IMAGE_NAME_2* && $image != *$IMAGE_NAME_3* ]]; then
echo "$nb is not using the correct images, deleting the notebook"
oc delete notebook $nb -n $ns
oc delete pvc $pvc -n $ns
elif [[ $image == *$IMAGE_NAME_1* && $size != "X Small" ]]; then
echo "$nb resource size is not correct for $IMAGE_NAME_1, deleting the notebook"
oc delete notebook $nb -n $ns
oc delete pvc $pvc -n $ns
elif [[ ($image == *$IMAGE_NAME_2* || $image == *$IMAGE_NAME_3*) && $size != "Small" ]]; then
echo "$nb resource size is not correct for $IMAGE_NAME_2 or $IMAGE_NAME_3, deleting the notebook"
oc delete notebook $nb -n $ns
oc delete pvc $pvc -n $ns
elif [[ $image == *$IMAGE_NAME_1* && $difference -gt $CUTOFF_TIME_1 ]]; then
echo "$nb is more than $(($CUTOFF_TIME_1 / 3600)) hours old, stopping the notebook"
# USER IS IN BOTH CLASSES
if [[ $user_in_group1 == true && $user_in_group2 == true ]]; then

if [[ $CUTOFF_TIME_1 -ge $CUTOFF_TIME_2 ]]; then
LONGER_CUTOFF=$CUTOFF_TIME_1
else
LONGER_CUTOFF=$CUTOFF_TIME_2
fi

if [[ $difference -gt $LONGER_CUTOFF ]]; then
echo "$nb is more than $(($LONGER_CUTOFF / 3600)) hours old, stopping the notebook"
oc patch notebook $nb -n $ns --type merge -p '{"metadata":{"annotations":{"kubeflow-resource-stopped":"'$(date -u +"%Y-%m-%dT%H:%M:%SZ")'"}}}'
elif [[ $image == *$IMAGE_NAME_2* || $image == *$IMAGE_NAME_3* ]]; then
echo "$nb is using $IMAGE_NAME_2 or $IMAGE_NAME_3, no shutdown applied."
fi

# USER IS IN CLASS ONE
elif $user_in_group1; then
# Handle group1 conditions
if [[ $image != *$IMAGE_NAME_1* || $size != "X Small" ]]; then
echo "$nb is not using the correct image or size, deleting the notebook"
oc delete notebook $nb -n $ns
oc delete pvc $pvc -n $ns
elif [ $difference -gt $CUTOFF_TIME_1 ]; then
if [ $difference -gt $CUTOFF_TIME_1 ]; then
echo "$nb is more than $(($CUTOFF_TIME_1 / 3600)) hours old, stopping the notebook"
oc patch notebook $nb -n $ns --type merge -p '{"metadata":{"annotations":{"kubeflow-resource-stopped":"'$(date -u +"%Y-%m-%dT%H:%M:%SZ")'"}}}'
fi

# USER IS IN CLASS TWO
elif $user_in_group2; then
# Handle group2 conditions
if [[ ($image != *$IMAGE_NAME_2* && $image != *$IMAGE_NAME_3*) || $size != "Small" ]]; then
echo "$nb is not using the correct image or size, deleting the notebook"
oc delete notebook $nb -n $ns
oc delete pvc $pvc -n $ns
if [ $difference -gt $CUTOFF_TIME_2 ]; then
echo "$nb is more than $(($CUTOFF_TIME_2 / 3600)) hours old, stopping the notebook"
oc patch notebook $nb -n $ns --type merge -p '{"metadata":{"annotations":{"kubeflow-resource-stopped":"'$(date -u +"%Y-%m-%dT%H:%M:%SZ")'"}}}'
fi

# USER IS IN NEITHER CLASS
else
echo "user $user does not belong to $GROUP_NAME_1 or $GROUP_NAME_2, deleting the notebook"
oc delete notebook $nb -n $ns
Expand All @@ -116,12 +110,8 @@ spec:
- name: CUTOFF_TIME_1
value: "43200"
# EDIT VALUE HERE BEFORE RUNNING
- name: IMAGE_NAME_1
value: "ucsls-f24"
- name: IMAGE_NAME_2
value: "jupyter-rust"
- name: IMAGE_NAME_3
value: "vscode-rust"
- name: CUTOFF_TIME_2
value: "43200"
resources:
limits:
cpu: 100m
Expand Down