Update Blog “open-sourcing-workshops-on-demand-part4-managing-the-backend”

pattdepanick · pattdepanick · commit 404199194f1e · 2023-04-05T12:24:14.000Z
diff --git a/content/blog/open-sourcing-workshops-on-demand-part4-managing-the-backend.md b/content/blog/open-sourcing-workshops-on-demand-part4-managing-the-backend.md
@@ -19,7 +19,7 @@ S﻿imple tree view of the wod-backend directory:
 
 ![](/img/wod-blogserie2-tree4.png "Tree view of wod-backend directory")
 
-T﻿he `ansible` folder contains all the necessary playbooks and variables files to support the main functions of the backend server. It provides playbooks for installation of the server or appliances, setup of the servers (backend, frontend, api-db),of appliances or workshops as well as maintenance tasks.
+T﻿he `ansible` folder contains all the necessary playbooks and variables files to support the main functions of the backend server. It provides playbooks for a minimal installation of the server or appliances, setup of the servers (backend, frontend, api-db),of appliances or workshops as well as maintenance tasks.
 
 A﻿t the root of this directory can be found:
 
@@ -54,13 +54,13 @@ T﻿he `conf` folder hosts configuration files in a jinja format. Once expanded,
 
 A﻿s part of the refacturing work to open source the project, we reaaranged the different scripts locations. We have created an install folder to handle the different installation scripts either from a Jupyterhub 's perpective or from an appliance 's standpoint too.
 
-We separated the workshops related scripts from the pure system ones. When one creates a workshop, one needs to provide a series of notebooks and in some cases some scripts to manage the creation, setup of a related appliance along with additional scripts to manage its lifecycle in the overall workshops-on-Demand architecture (Create, Cleanup, Reset scripts at deployment or Cleanup times). These scripts need to be located in the script folder. On the other hand, the system scripts are located in the sys folder.
+We separated the workshops related scripts from the system ones. When one creates a workshop, one needs to provide a series of notebooks and in some cases some scripts to manage the creation, setup of a related appliance along with additional scripts to manage its lifecycle in the overall workshops-on-Demand architecture (Create, Cleanup, Reset scripts at deployment or Cleanup times). These scripts need to be located in the `scripts` folder. On the other hand, the system scripts are located in the `sys` folder.
 
 ![](/img/tree-wkshop2.png "Tree view of the sys directory")
 
-t﻿his directory hosts important configuration files for the system, for Jupyterhub. You can see for instance `fail2ban` configuration files. Some jinja templates are present here too. These templates will be expanded trough the deliver mechanism allowing the creation of files customized with ansible variables. All the wod related tasks are prefixed with wod for better understanding and ease of use.  
+T﻿his directory hosts important configuration files for both the system and Jupyterhub. You can see for instance `fail2ban` configuration files. Some jinja templates are present here too. These templates will be expanded trough the `deliver` mechanism allowing the creation of files customized with ansible variables. All the wod related tasks are prefixed with wod for better understanding and ease of use.
 
-They can refer to some Jupyterhub kernel needs like wod-build-evcxr.sh.j2 that aims at creating a script allowing the rust kernel installation. Some other templates are related to the system and Jupyterhub. `wod-kill-processes.pl.j2` has been created after discovering the harsh reality of online mining...In a ideal world, I would not have to explain further as the script would not be needed. Unfortunately, this is not the case, when one offers access to some harware freely online, he can expect sooner or later to  see his original and pure idea to be highjacked...Let's say that you want to provide some AI/ML 101 type of workshops, you may consider providing servers with some GPUs. Any twisted minded cryptominer discovering your ressources will definitely think he hits the jackpot! This little anecdot actually happened to us and not only on GPU based servers, some regular servers got hit as well. We found out that performance on some servers became very poor and wehen looking into it, we found some scripts that were not supposed to be here and to run here...As a result, we implemented monitors to check load on our server and make sure that to  kill any suspicious process before kicking out the misbehaving student.
+They can refer to some Jupyterhub kernel needs like `wod-build-evcxr.sh.j2` that aims at creating a script allowing the rust kernel installation. Some other templates are related to the system and Jupyterhub. `wod-kill-processes.pl.j2` has been created after discovering the harsh reality of online mining...In a ideal world, I would not have to explain further as the script would not be needed. Unfortunately, this is not the case, when one offers access to some hardware freely online, he can expect sooner or later to  see his original and pure idea to be highjacked...Let's say that you want to provide some AI/ML 101 type of workshops, you may consider providing servers with some GPUs. Any twisted minded cryptominer discovering your ressources will definitely think he hits the jackpot! This little anecdot actually happened to us and not only on GPU based servers, some regular servers got hit as well. We found out that performance on some servers became very poor and when looking into it, we found some scripts that were not supposed to run there...As a result, we implemented monitors to check load on our server and make sure that to  kill any suspicious process before kicking out the misbehaving student.
 
 `wod-test-action.sh.j2` is another interesting template that will create a script that we use for testing. This script mimics the procmail API and allows you to  test the deployment of a workshop from the shell. 
 
@@ -112,7 +112,7 @@ T﻿here are two types of activities that can occur on the backend server: punct
 
 T﻿he backend server hosts all the necessary content for delivering workshops: it implies notebooks and scripts and playbooks to deploy and personalize them. It also hosts some services that are needed by the overall architecture solution (Jupyterhub, Procmail, Fail2ban among others).
 
-S﻿ervices are installed once and for all at the installation time. These services may evolve over time. One may need to update the jupyterhub application to fix a bug or get new features. In the same fashion, you may consider bumping from one python version to a new major one. If you are willing to update these services or add additional ones, you will need to update the relevant installation playbooks in wod-backend/ansible directory.
+S﻿ervices are installed once and for all at the installation time. These services may evolve over time. One may need to update the jupyterhub application to fix a bug or get new features. In the same fashion, you may consider bumping from one python version to a new major one. If you are willing to update these services or add new ones, you will need to update the relevant installation playbooks in `wod-backend/ansible` directory.
 
 h﻿ere is a small extract of the `install_backend.yml` playbook: Full version [here](https://github.com/Workshops-on-Demand/wod-backend/blob/main/ansible/install_backend.yml)
 

-Original file line number
+Diff line change
 ![](/img/wod-blogserie2-tree4.png "Tree view of wod-backend directory")
 -The `ansible` folder contains all the necessary playbooks and variables files to support the main functions of the backend server. It provides playbooks for installation of the server or appliances, setup of the servers (backend, frontend, api-db),of appliances or workshops as well as maintenance tasks.
 +The `ansible` folder contains all the necessary playbooks and variables files to support the main functions of the backend server. It provides playbooks for a minimal installation of the server or appliances, setup of the servers (backend, frontend, api-db),of appliances or workshops as well as maintenance tasks.
 At the root of this directory can be found:
 As part of the refacturing work to open source the project, we reaaranged the different scripts locations. We have created an install folder to handle the different installation scripts either from a Jupyterhub 's perpective or from an appliance 's standpoint too.
 -We separated the workshops related scripts from the pure system ones. When one creates a workshop, one needs to provide a series of notebooks and in some cases some scripts to manage the creation, setup of a related appliance along with additional scripts to manage its lifecycle in the overall workshops-on-Demand architecture (Create, Cleanup, Reset scripts at deployment or Cleanup times). These scripts need to be located in the script folder. On the other hand, the system scripts are located in the sys folder.
 +We separated the workshops related scripts from the system ones. When one creates a workshop, one needs to provide a series of notebooks and in some cases some scripts to manage the creation, setup of a related appliance along with additional scripts to manage its lifecycle in the overall workshops-on-Demand architecture (Create, Cleanup, Reset scripts at deployment or Cleanup times). These scripts need to be located in the `scripts` folder. On the other hand, the system scripts are located in the `sys` folder.
 ![](/img/tree-wkshop2.png "Tree view of the sys directory")
 -this directory hosts important configuration files for the system, for Jupyterhub. You can see for instance `fail2ban` configuration files. Some jinja templates are present here too. These templates will be expanded trough the deliver mechanism allowing the creation of files customized with ansible variables. All the wod related tasks are prefixed with wod for better understanding and ease of use.
 +This directory hosts important configuration files for both the system and Jupyterhub. You can see for instance `fail2ban` configuration files. Some jinja templates are present here too. These templates will be expanded trough the `deliver` mechanism allowing the creation of files customized with ansible variables. All the wod related tasks are prefixed with wod for better understanding and ease of use.
 -They can refer to some Jupyterhub kernel needs like wod-build-evcxr.sh.j2 that aims at creating a script allowing the rust kernel installation. Some other templates are related to the system and Jupyterhub. `wod-kill-processes.pl.j2` has been created after discovering the harsh reality of online mining...In a ideal world, I would not have to explain further as the script would not be needed. Unfortunately, this is not the case, when one offers access to some harware freely online, he can expect sooner or later to  see his original and pure idea to be highjacked...Let's say that you want to provide some AI/ML 101 type of workshops, you may consider providing servers with some GPUs. Any twisted minded cryptominer discovering your ressources will definitely think he hits the jackpot! This little anecdot actually happened to us and not only on GPU based servers, some regular servers got hit as well. We found out that performance on some servers became very poor and wehen looking into it, we found some scripts that were not supposed to be here and to run here...As a result, we implemented monitors to check load on our server and make sure that to  kill any suspicious process before kicking out the misbehaving student.
 +They can refer to some Jupyterhub kernel needs like `wod-build-evcxr.sh.j2` that aims at creating a script allowing the rust kernel installation. Some other templates are related to the system and Jupyterhub. `wod-kill-processes.pl.j2` has been created after discovering the harsh reality of online mining...In a ideal world, I would not have to explain further as the script would not be needed. Unfortunately, this is not the case, when one offers access to some hardware freely online, he can expect sooner or later to  see his original and pure idea to be highjacked...Let's say that you want to provide some AI/ML 101 type of workshops, you may consider providing servers with some GPUs. Any twisted minded cryptominer discovering your ressources will definitely think he hits the jackpot! This little anecdot actually happened to us and not only on GPU based servers, some regular servers got hit as well. We found out that performance on some servers became very poor and when looking into it, we found some scripts that were not supposed to run there...As a result, we implemented monitors to check load on our server and make sure that to  kill any suspicious process before kicking out the misbehaving student.
 `wod-test-action.sh.j2` is another interesting template that will create a script that we use for testing. This script mimics the procmail API and allows you to  test the deployment of a workshop from the shell.
 The backend server hosts all the necessary content for delivering workshops: it implies notebooks and scripts and playbooks to deploy and personalize them. It also hosts some services that are needed by the overall architecture solution (Jupyterhub, Procmail, Fail2ban among others).
 -Services are installed once and for all at the installation time. These services may evolve over time. One may need to update the jupyterhub application to fix a bug or get new features. In the same fashion, you may consider bumping from one python version to a new major one. If you are willing to update these services or add additional ones, you will need to update the relevant installation playbooks in wod-backend/ansible directory.
 +Services are installed once and for all at the installation time. These services may evolve over time. One may need to update the jupyterhub application to fix a bug or get new features. In the same fashion, you may consider bumping from one python version to a new major one. If you are willing to update these services or add new ones, you will need to update the relevant installation playbooks in `wod-backend/ansible` directory.
 here is a small extract of the `install_backend.yml` playbook: Full version [here](https://github.com/Workshops-on-Demand/wod-backend/blob/main/ansible/install_backend.yml)