compute servers: massively improve the development instructions; document why allow_other is required for FUSE mounts

Author: williamstein

src/compute/compute/dev/4-startup-script.sh

@@ -17,19 +17,21 @@
 
 set -v
 
-export api_server=`cat conf/api_server`
-
+. env.sh
 
 function setState {
-  id=`cat conf/compute_server_id`
+  id=$COMPUTE_SERVER_ID
   name=$1
   state=${2:-'ready'}
   extra=${3:-''}
   timeout=${4:-0}
   progress=${5:-100}
+  project_id=$PROJECT_ID
 
   echo "$name is $state"
-  curl -sk -u `cat conf/api_key`:  -H 'Content-Type: application/json' -d "{\"id\":$id,\"name\":\"$name\",\"state\":\"$state\",\"extra\":\"$extra\",\"timeout\":$timeout,\"progress\":$progress}" $api_server/api/v2/compute/set-detailed-state
+  PAYLOAD="{\"id\":$id,\"name\":\"$name\",\"state\":\"$state\",\"extra\":\"$extra\",\"timeout\":$timeout,\"progress\":$progress,\"project_id\":\"$project_id\"}"
+  echo $PAYLOAD
+  curl -sk -u $API_KEY:  -H 'Content-Type: application/json' -d $PAYLOAD $API_SERVER/api/v2/compute/set-detailed-state
 }
 
 

src/compute/compute/dev/README.md

@@ -23,17 +23,27 @@ This is potentially confusing, and when developing this it was 10x worse... Mayb
 4-startup-script.sh
 ```
 
-However, a bunch of things are likely to go wrong. The scripts `1-websocketfs.sh` and `2-syncfs.sh` will definitely fail if support for FUSE isn't enabled for normal users where you are working! Test bindfs locally. You probably  have to add `user_allow_other` to `/etc/fuse.conf`. For `2-syncfs.sh`, you must also install unionfs-fuse via `sudo apt install unionfs-fuse` . Also, you need to do the following so that testing of tmp being a "fast local data directory that isn't sync"'d can be done:
+However, a bunch of things are likely to go wrong. 
 
-```
-~/cocalc/src/compute/compute/dev$ sudo mkdir /data/tmp
-~/cocalc/src/compute/compute/dev$ sudo chown `whoami`:`whoami` /data/tmp
+**Problem:** Regarding the id of the compute server in the file [conf/compute\_server\_id](./conf/compute_server_id), create a self\-hosted compute server in the project on your dev server, then find the record in the postgresql database by querying the `compute_servers` table, and copy the id field from that.  Note that the displayed id in the UI starts from 1 for each project, but `compute_server_id` must be the id in the database.
+
+**Problem:** Get the [conf/api_key](./conf/api_key) by clicking start on the self\-hosted compute server, inspect the URL, and copy it from there.  If you stop the server explicitly, then the api key is deleted from the project, so you need to make it again.
+
+**Problem:** The scripts `1-websocketfs.sh` and `2-syncfs.sh` will definitely fail if support for FUSE isn't enabled for normal users where you are working! Test bindfs locally. 
+
+**Problem:** For `2-syncfs.sh`, you must also install unionfs\-fuse via `sudo apt install unionfs-fuse,`  since the cocalc package @cocalc/sync\-fs assumes unionfs\-fuse is installed.   
+
+**Problem:** You need to do the following so that you can fully test the scratch functionality \(see [conf/exclude_from_sync](./conf/exclude_from_sync)\):
+
+```sh
+sudo mkdir -p /data/scratch && sudo chown -R `whoami`:`whoami` /data
 ```
 
-Once you get the 4 scripts above to run, the net result is basically the same as using a compute server, but you can run it all locally, and debugging is massively easier. Without something like this, development is impossible, and even figuring out what configuration goes where could cost me days of confusion (even though I wrote it all!). It's complicated.
+Once you get the 4 scripts above to run, the net result is basically the same as using a compute server, but you can run it all locally, and development and debugging is ~~massively easier~~ possible! Without something like this, development is impossible, and even figuring out what configuration goes where could cost me days of confusion \(even though I wrote it all!\). It's complicated.
 
 For debugging set the DEBUG env variable to different things according to the debug npm module. E.g.,
 
 ```sh
-DEBUG=* 2-syncfs.sh
+DEBUG_CONSOLE=yes DEBUG=* ./2-syncfs.sh
 ```
+

src/compute/compute/dev/start-filesystem.js

@@ -47,7 +47,8 @@ async function main() {
     exports.fs = await mountProject({
       project_id: process.env.PROJECT_ID,
       path: PROJECT_HOME,
-      options: { mountOptions: { allowOther: true, nonEmpty: true } },
+      // NOTE: allowOther is disabled by default on Ubuntu and we do not need it. 
+      options: { mountOptions: { allowOther: false, nonEmpty: true } },
       unionfs,
       readTrackingFile: process.env.READ_TRACKING_FILE,
       exclude,

src/compute/compute/lib/filesystem.ts

@@ -148,9 +148,9 @@ export async function mountProject({
           ...options.connectOptions,
         },
         mountOptions: {
-          allowOther: true,
-          nonEmpty: true,
           ...options.mountOptions,
+          allowOther: true, // this is critical to allow for fast bind mounts of scratch etc. as root.
+          nonEmpty: true,
         },
         cacheTimeout,
         hidePath: "/.unionfs",
@@ -175,6 +175,11 @@ export async function mountProject({
         );
         websocketfsMountOptions.mountOptions.allowOther = false;
         ({ unmount } = await mount(websocketfsMountOptions));
+
+        // This worked so the problem is allow_other.
+        throw Error(
+          "fusermount: option allow_other only allowed if 'user_allow_other' is set in /etc/fuse.conf\n\n\nFix this:\n\n    sudo sed -i 's/#user_allow_other/user_allow_other/g' /etc/fuse.conf\n\n\n",
+        );
       }
 
       pingInterval = setInterval(async () => {

src/packages/server/compute/cloud/install.ts

@@ -125,6 +125,9 @@ systemctl restart docker
 `;
 }
 
+// NOTE: we absolutely DO need "# Allow root to use FUSE mount of user" below.
+// This is needed so that we can do a very fast bind mount as root of fast
+// scratch directories on top of the slower fuse mounted home directory.
 export function installUser() {
   return `
 # Create the "user" if they do not already exist:
@@ -138,7 +141,7 @@ if ! id -u user >/dev/null 2>&1; then
   # Allow to be root
   echo '%user ALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers
 
-  # Allow to use FUSE
+  # Allow root to use FUSE mount of user
   sed -i 's/#user_allow_other/user_allow_other/g' /etc/fuse.conf
 
 fi

src/packages/sync-fs/lib/index.ts

@@ -354,6 +354,8 @@ class SyncFS {
   };
 
   private mountUnionFS = async () => {
+    // NOTE: allow_other is essential to allow bind mounted as root
+    // of fast scratch directories into HOME!
     // unionfs-fuse -o allow_other,auto_unmount,nonempty,large_read,cow,max_files=32768 /upper=RW:/home/user=RO /merged
     await execa("unionfs-fuse", [
       "-o",