Merge branch 'master' of github.com:sagemathinc/cocalc into openapi/obsolete-legacy-docs

Author: schrodingersket

.github/workflows/make-and-test.yml

@@ -40,7 +40,7 @@ jobs:
       - name: Install python3 requests
         run: sudo apt-get install python3-requests
       - name: Check doc links
-        run: cd src/scripts && python3 check_doc_urls.py
+        run: cd src/scripts && python3 check_doc_urls.py || sleep 5 || python3 check_doc_urls.py
 
   build:
     runs-on: ubuntu-latest

src/README.md

@@ -180,12 +180,12 @@ which installs exactly the right packages, and builds the code.
 
 See `packages/backend/data.ts` . In particular, you can set BASE_PATH, DATA, PGHOST, PGDATA, PROJECTS, SECRETS to override the defaults. Data is stored in `cocalc/src/data/` by default.
 
-#### Filesystem Build Caching
+#### File System Build Caching
 
-There are two types of filesystem build caching. These greatly improve the time to compile typescript or start webpack between runs. However, in rare cases bugs may lead to weird broken behavior. Here's where the caches are, so you know how to clear them to check if this is the source of trouble. _As of now, I'm_ _**not**_ _aware of any bugs in filesystem caching._
+There are two types of file system build caching. These greatly improve the time to compile typescript or start webpack between runs. However, in rare cases bugs may lead to weird broken behavior. Here's where the caches are, so you know how to clear them to check if this is the source of trouble. _As of now, I'm_ _**not**_ _aware of any bugs in file system caching._
 
 - In the `dist/` subdirectory of a package, there's a file `tsconfig.tsbuildinfo` that caches incremental typescript builds, so running `tsc` is much faster. This is enabled by setting `incremental: true` in `tsconfig.json`. I've never actually seen a case where caching of this file caused a problem (those typescript developers are careful).
-- Webpack caches its builds in `/tmp/webpack` . This is configured in `packages/static/webpack.config.js` , and we use `/tmp` since random access file system performance is critical for this **large** GB+ cache -- otherwise, it's almost slower than no cache. (I also benchmarked tsc, and it works fine on a potentially slow local filesystem.) I did sees bugs with this cache when I had some useless antd tree shaking plugin enabled, but I have never seen any problems with it since I got rid of that.
+- Webpack caches its builds in `/tmp/webpack` . This is configured in `packages/static/webpack.config.js` , and we use `/tmp` since random access file system performance is critical for this **large** GB+ cache -- otherwise, it's almost slower than no cache. (I also benchmarked tsc, and it works fine on a potentially slow local file system.) I did sees bugs with this cache when I had some useless antd tree shaking plugin enabled, but I have never seen any problems with it since I got rid of that.
 
 #### Creating an admin user
 

src/compute/compute/README.md

@@ -23,9 +23,9 @@ pnpm build
   - You \(and collabs\) can then use this power from cocalc on your laptop.
 - A web browser with WebGPU [providing PyTorch](https://praeclarum.org/2023/05/19/webgpu-torch.html) \(say\).
 
-## The filesystem
+## The File System
 
-The filesystem from the project will get mounted via [WebSocketFS](https://github.com/sagemathinc/websocketfs). This will initially only be for FUSE, but later could also use WASI in the browser.
+The file system from the project will get mounted via [WebSocketFS](https://github.com/sagemathinc/websocketfs). This will initially only be for FUSE, but later could also use WASI in the browser.
 
 ## Status
 

src/compute/compute/lib/filesystem.ts

@@ -1,5 +1,5 @@
 /*
-Mount a remote CoCalc project's filesystem locally over a websocket using FUSE.
+Mount a remote CoCalc project's file system locally over a websocket using FUSE.
 
  await require('.').mount({remote:'wss://cocalc.com/10f0e544-313c-4efe-8718-2142ac97ad11/raw/.smc/websocketfs',path:process.env.HOME + '/dev2', connectOptions:{perMessageDeflate: false,  headers: {Cookie: require('cookie').serialize('api_key', 'sk-at7ALcGBKMbzq7Vc00000P')}}})
 
@@ -39,9 +39,9 @@ interface Options {
   unionfs?: {
     upper: string;
     lower: string;
-    // If true, doesn't do anything until the type of the filesystem that lower is
+    // If true, doesn't do anything until the type of the file system that lower is
     // mounted on is of this type, e.g., "fuse". This is done *INSTEAD OF* just
-    // trying to mount that filesystem.  Why? because in docker we hit a deadlock
+    // trying to mount that file system.  Why? because in docker we hit a deadlock
     // when trying to do both in the same process (?), which I can't solve -- maybe
     // a bug in node.  In any case, separating the unionfs into a separate container
     // is nice anyways.
@@ -160,7 +160,7 @@ export async function mountProject({
         // modified = ignore any file modified with this many seconds (at least);
         //            also ignores any file not in the stat cache.
         readTrackingFile: readTrackingFile,
-        readTrackingExclude: [".*", ...exclude],
+        readTrackingExclude: exclude,
         // metadata file
         metadataFile,
       }));

src/compute/compute/lib/manager.ts

@@ -40,11 +40,11 @@ interface Options {
   // it should be the id in the database from the compute_servers table.
   compute_server_id: number;
   // HOME = local home directory.  This should be a network mounted (or local)
-  // filesystem that is identical to the home directory of the target project.
+  // file system that is identical to the home directory of the target project.
   // The ipynb file will be loaded and saved from here, and must exist, and
   // process.env.HOME gets set to this.
   home: string;
-  // If true, doesn't do anything until the type of the filesystem that home is
+  // If true, doesn't do anything until the type of the file system that home is
   // mounted on is of this type, e.g., "fuse".
   waitHomeFilesystemType?: string;
 }
@@ -117,7 +117,7 @@ class Manager {
     this.state = "init";
     // Ping to start the project and ensure there is a hub connection to it.
     await pingProjectUntilSuccess(this.project_id);
-    // wait for home direcotry filesystem to be mounted:
+    // wait for home direcotry file system to be mounted:
     if (this.waitHomeFilesystemType) {
       this.reportComponentState({
         state: "waiting",

src/dev/minimal/start_postgres.py

@@ -16,7 +16,7 @@
     # Lock down authentication so it is ONLY via unix socket
     open(os.path.join(PG_DATA,'pg_hba.conf'), 'w').write(
 """
-# This is safe since we only enable a socket protected by filesystem permissions:
+# This is safe since we only enable a socket protected by file system permissions:
 local all all trust
 
 # You can uncomment this and comment out the above if you want to test password auth.

src/dev/project/start_postgres.py

@@ -18,7 +18,7 @@
 
     # Lock down authentication so it is ONLY via unix socket
     open(os.path.join(PG_DATA, 'pg_hba.conf'), 'w').write("""
-# This is safe since we only enable a socket protected by filesystem permissions:
+# This is safe since we only enable a socket protected by file system permissions:
 local all all trust
 
 # You can uncomment this and comment out the above if you want to test password auth.

src/packages/assets/misc/talk-20121004/index.html

@@ -1,9 +1,9 @@
-<!-- 
+<!--
   This file is part of CoCalc: Copyright © 2020 Sagemath, Inc.
   License: AGPLv3 s.t. "Commons Clause" – see LICENSE.md for details
  -->
 
-<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" 
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
 	"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
 
 <html xmlns="http://www.w3.org/1999/xhtml">
@@ -106,7 +106,7 @@ <h1>sage-support mailing list a few days ago</h1>
 <font size=+3>
 <pre>
 On Wed, Sep 26, 2012 at 9:54 AM, Chris Seberino <cseberino@gmail.com> wrote:
-> I have a Xen Ubuntu Linux 1 Gb RAM virtual machine  from a provider 
+> I have a Xen Ubuntu Linux 1 Gb RAM virtual machine  from a provider
 > called Linode.
 >
 
@@ -314,18 +314,18 @@ <h1>Architecture Diagram</h1>
      /|\
       |       (sockjs)
    Stunnels ... (https://salv.us)
-      |     
       |
-     \|/ 
+      |
+     \|/
  HAProxy Load Balancers  ...  Nginx static servers ...
  /|\       /|\      /|\      /|\
   |         |        |        |           [tinc vpn]
-  |https1.1 |        |        |                                     
- \|/       \|/      \|/      \|/                                      
+  |https1.1 |        |        |
+ \|/       \|/      \|/      \|/
 Tornado  Tornado  Tornado  Tornado    <------------------>   Cassandra   ...
            /|\      /|\                                         /|\
             |        |        -----------------------------------|
-   ---------|        |        |                                         
+   ---------|        |        |
    |                 |        |
    |                 |        |
   \|/               \|/      \|/
@@ -365,7 +365,7 @@ <h1>Virtualization:  libvirt + kvm</h1>
     disk images).</li>
 <li> <b>The <i>only</i> persistent data:</b> in all of Salvus is in the distributed
     database.  This will sit in a disk partition that is
-    encrypted using the Linux ecryptfs filesystem and a passphrase
+    encrypted using the Linux ecryptfs file system and a passphrase
     that I type in (exactly once from my management interface) when
     launching the database. </li>
 </ul>
@@ -384,7 +384,7 @@ <h1>Private Cloud Distributions</h1>
     making noise about implementing something open source kind of like
     what Amazon and VMware already did, but with a lot of the truly hard
     real work being doing by libvirt.  (This will likely be useful for
-    something in a year or two, and now if you <a href="http://www.ubuntu.com/cloud/solutions">pay companies money</a>, 
+    something in a year or two, and now if you <a href="http://www.ubuntu.com/cloud/solutions">pay companies money</a>,
 but now it is worse than useless...)
 </li>
 <li>    The real work is done by libvirt, kvm (or xen).  Evidently, Redhat put a ton
@@ -406,7 +406,7 @@ <h1>VPN: Motivation</h1>
 
   <li>By using a VPN, Salvus can easily combine cloud computers, my own
     computers, etc., and don't have to worry about ip addresses
-    changing. 
+    changing.
 </li>
 <li> We don't have to worry about securing
     communication between the different services or vm's in salvus.
@@ -434,7 +434,7 @@ <h1>VPN I chose: tinc</h1>
          <li> Public-facing machines on the VPN also acts as
            servers, so clients can connect to VPN as long as at least one public-facing
            machine is up.   </li>
-         <li> It's P2P -- traffic does not all go through one 
+         <li> It's P2P -- traffic does not all go through one
            master node, which could be very inefficient. </li>
          <li> All traffic is encrypted.</li>
          <li> Easy/sensible; to add a machine to the VPN, you just
@@ -444,7 +444,7 @@ <h1>VPN I chose: tinc</h1>
 </ul>
 </li>
 </ul></div>
-      
+
 
 
 
@@ -465,14 +465,14 @@ <h1>Database: the memory of Salvus</h1>
     databases based on usernames.  (I also read a lot about
     SQLalchemy, used PostgreSQL a lot, etc.)</li>
 <li>Then the Padelford server room blew up for the <i>n</i>-th time, and I
-    firmly decided on no SPOF's as an <i>Axiom</i>. 
+    firmly decided on no SPOF's as an <i>Axiom</i>.
 </ul>
 </div>
 
 <div class="slide">
 <h1>Database: noSQL</h1>
 <ul>
-<li> NO SPOFS!:  
+<li> NO SPOFS!:
     Relational databases like PostgreSQL become even more difficult to
     use with this constraint. You have to have a master database, at least one other slave
     databases replicating it, and some sort of automatic failover that
@@ -492,7 +492,7 @@ <h1>Database: noSQL</h1>
     how redundant they are.  I had learned a lot about MongoDB (a
     noSQL database) a few years ago, and after getting LMFDB.org to
     use it a lot (and using it for my own database work), I think it
-    sucks.... for me at least. 
+    sucks.... for me at least.
 
       <a href="http://news.ycombinator.com/item?id=4570790">"Does everyone hate MongoDB?"</a>
       </li>
@@ -555,7 +555,7 @@ <h1>SSL: stunnel</h1>
 <h1>Load balancing: haproxy</h1>
 The stunnel process directs all requests to another program called
 haproxy (which can be running on a different computer or the same
-computer). 
+computer).
 <ul>
 <li>Haproxy is pretty much the canonical open source load balancer
 program.</li>
@@ -630,7 +630,7 @@ <h1>Math/compute: forking tcp-based Sage server</h1>
 with various things pre-imported such as graphics, Maxima, etc.,
 listens on a socket (running as root), accepts a connection, forks,
 lowers privileges, executes blocks of sage code, and streams results
-back over the connection.  
+back over the connection.
 </li>
 <li>The design is much different than with the Sage notebook, where one
 process watches another.  Here, as code executes, the running process
@@ -661,7 +661,7 @@ <h1>Service Management: approaches</h1>
 software on a bunch of machine.  I found one called Ansible that I
 kind of liked at first, but found it to be too immature and not so
 widely used.  There were some other solutions, but they seemed like
-massive overkill.  I spent quite a bit of time investigating them. 
+massive overkill.  I spent quite a bit of time investigating them.
 But for every actual application, there is a bunch of subtle logic
 that just has to be figured out and coded, and I had a lot of that down
 from my code mentioned above.</li>

src/packages/backend/data.ts

@@ -1,11 +1,11 @@
 /*
 Where Data is Stored:
 
-We centralize here determination of all directories on the filesystem
+We centralize here determination of all directories on the file system
 where data is stored for any of the components of CoCalc, run in any way.
 
 All information here must be determinable when this module is initialized,
-e.g., from environment variables or heuristics involving the filesystem.
+e.g., from environment variables or heuristics involving the file system.
 In particular, nothing here can be impacted by command line flags
 or content of a database.
 */

src/packages/backend/files/delete-files.ts

@@ -33,7 +33,7 @@ export async function delete_files(
   //    - that results in weird race conditions that can make files immediately
   //      reappear after deletion
   //    - we MUST make deletion detection fully work based entirely on what happens
-  //      on the filesystem, e.g., due to git checkout and people using the terminal
+  //      on the file system, e.g., due to git checkout and people using the terminal
   log.debug("extra = ", extra);
   await rimraf(paths.concat(extra), { maxRetries: 2 });
 }