Add deployment notes and update docs on storage

Author: ohsayan

deploy.sh

@@ -2,12 +2,14 @@
 id: persistence
 title: Persistence
 ---
+
 Skytable supports the persistent storage of data, which is an inherently obvious need for any database. In this document we explore how Skytable's persistence works.
 
 ## Data directory structure
 
 Whenever you start Skytable, it will create a number of directories under a root 'data' directory. This is what the
 data directory structure looks like:
+
 ```
 |__user-files (your other files)
 |__data
@@ -18,34 +20,44 @@ data directory structure looks like:
          |___<snapshotname>.snapshot (many)
 ```
 
-The `data.bin` file is primarily used for persistence while the snapshots folder contains automatically created 
+The `data.bin` file is primarily used for persistence while the snapshots folder contains automatically created
 snapshots and remotely created snapshots.
 
 ## How Skytable stores data
 
-As soon as you start Skytable, it will look for a `data.bin` file in the data directory and then fall back to the
-`data.bin` file in the current directory for backwards compatibility. The database daemon will then attempt to 
-migrate the older data into the structure noted above if required.
-:::note
-This backward compatibility will possibly be removed in future releases
-:::
-If the data file is found and successfully read, then the previous data is moved into the in-memory table. If no
+As soon as you start Skytable, it will look for a `data.bin` file in the data directory;
+if the data file is found and successfully read, then the previous data is moved into the in-memory table. If no
 file was found, then the database server will create one. Once you terminate the daemon, it will flush all data
-to this file. There are more features like [BGSAVE](#automated-background-saving) and [snapshots](/snapshots) that
-can be configured and used by users.
+to this file. All writes are `fsync`ed by the time they complete.
+
+### Save failure during termination
 
+The server would attempt to do a _final_ save operation before it terminates and if this fails, the
+server would enter into a retry loop. It will try the save operation after every 10 seconds.
+Exponential backoff was not chosen because it could increase to extremely large values that may hurt
+a sysadmin's time and productivity.
+
+You might be interested in more features like [BGSAVE](#automated-background-saving) and [snapshots](/snapshots) that
+can be configured and used by users.
 
 ## Automated background saving
 
 Skytable supports automated saving of data in the background, via `BGSAVE`. `BGSAVE`, runs every two minutes to flush all the data in the in-memory table onto disk, unless customized through the [configuration file](config-files/#an-example-configuration). BGSAVE is enabled by default and we don't recommend disabling it until you're sure that
 your hardware will never fail; it is likely that this will never be the case. First BGSAVE will create a temporary
-file and then flush the current in-memory table onto disk. It will then replace the old `data.bin` file.
+file and then flush the current in-memory table onto disk. It will then replace the old `data.bin` file. The daemon automatically `fsync`s after every successful write (whether to the buffers or
+to the actual disk).
 
 ### Reliability of BGSAVE
 
-It can happen that BGSAVE fails to flush data to the disk due to some unforeseen system issues (such as lack of 
+It can happen that BGSAVE fails to flush data to the disk due to some unforeseen system issues (such as lack of
 empty disk space, permission errors, etc). But if we continue to accept modifications to the data, it is a bad idea
 because this data may never get updated! This is why if BGSAVE fails, it will automatically _poison_ the in-memory
 table preventing it from accepting any write/update operations. Poisioning is nothing but a global flag set in the
 database that indicates that the DB shouldn't accept any more updates/writes to data and in such a poisoned state,
-only reads are permitted. 
+only reads are permitted.
+
+### BGSAVE Recovery
+
+BGSAVE will automatically try to recover on every 120s (or whatever duration was set). If the problem
+was corrected (say it was a permissions issue), then the database server will automatically resume
+writes and _unpoison_ the database.

docs/4.persistence.md

@@ -2,34 +2,36 @@
 id: config-files
 title: Configuration Files
 ---
+
 Skytable supports custom configuration files to let users customize the functioning of Sky. Arguably, Sky has one of the simplest configuration files around. Skytable also allows configuration via [command line arguments](config-cmd).
 
 ## An example configuration
 
-A configuration file is a TOML file, which looks like:
+A configuration file is a TOML file, which has the following basic structure:
 
-``` toml
+```toml
 [server]
 host = "127.0.0.1"
 port = 2003
-noart = false
+noart = false # optional
 
 [bgsave]
 enabled = true
 every = 120 # Every 120 seconds
 ```
 
-This is the default configuration used by Sky when you don't specify a configuration file. Let's understand what each of the keys mean:
+This is the default configuration used by Sky when you don't specify a configuration file. Let's understand what each of the keys mean along with some other keys that can be used for more
+advanced configuration:
 
-* `server` :
-    - `host` : This is the IP address to which you want the database server to bind to. It can be any valid IPv4 *or* IPv6 address, as a quoted string
-    - `port` : This is the port to which you want Sky to bind to
-    - `noart` (OPTIONAL): This is **an optional argument** and is recommended for secure environments where displaying terminal artwork might cause problems
-* `bgsave`:
-    - `enabled` : This is an optional key, which is to be set to true to enable BGSAVE or false to disable it. If this key is not specified, Sky will enable BGSAVE by default
-    - `every` : Run BGSAVE `every` seconds. So, for example, if you set this to 120, BGSAVE will run every two minutes. This is also an optional key, and if you don't provide it, the default BGSAVE duration of 120 seconds is used
-* `snapshot` (OPTIONAL): This key can be used to configure snapshots and is not enabled by default. See [this](snapshots) for more information.
-* `ssl`: This key can be used to configure SSL/TLS options. See [this](ssl) for more information.
+- `server` :
+  - `host` : This is the IP address to which you want the database server to bind to. It can be any valid IPv4 _or_ IPv6 address, as a quoted string
+  - `port` : This is the port to which you want Sky to bind to
+  - `noart`: This is **an optional argument** and is recommended for secure environments where displaying terminal artwork might cause problems
+- `bgsave`:
+  - `enabled`: This is an optional key, which is to be set to true to enable BGSAVE or false to disable it. If this key is not specified, Sky will enable BGSAVE by default
+  - `every` : Run BGSAVE `every` seconds. So, for example, if you set this to 120, BGSAVE will run every two minutes. This is also an optional key, and if you don't provide it, the default BGSAVE duration of 120 seconds is used
+- `snapshot` (OPTIONAL): This key can be used to configure snapshots and is not enabled by default. See [this](snapshots) for more information.
+- `ssl`: This key can be used to configure SSL/TLS options. See [this](ssl) for more information.
 
 ## Using a configuration file
 
@@ -44,11 +46,15 @@ If you're confused about creating a configuration file, we always recommend you
 That's all that's there for using configuration files!
 :::tip Bonus tip
 If you're using a custom host/port, then you can bind `skysh` to a custom host/port by starting `skysh` like:
+
 ```shell
 skysh -h [HOST] -p [PORT]
 ```
+
 You can do the same for `sky-bench`:
+
 ```shell
 sky-bench -h [HOST] -p [PORT]
 ```
-:::
+
+:::

docs/5.cfg-files.md

@@ -0,0 +1,16 @@
+---
+id: deployment-notes
+title: Deployment notes
+---
+
+Here are some _good to know_ things about deploying Skytable:
+
+- Skytable is under active development. If you do find any rough edges, [please open an issue](https://github.com/skytable/skytable/issues)
+- The daemon will create a `.sky_pid` file in its working directory which functions as a PID file
+  and indicates other processes to not use the data directory. If the daemon is somehow forcefully
+  stopped, the file may not be removed. In that case, you should manually remove the file
+- Skytable currently has a hardcoded limit of 50000 connections on a single daemon instance. This limit will be user accessible in the (near) future
+- Skytable is inherently multithreaded. As of now, there is no way to stop Skytable from using
+  multiple threads
+- The best way to deploy Skytable is as a service (and passing `--noart` to avoid terminal artwork
+  in logs)

docs/deployment-notes.md

@@ -2,23 +2,27 @@
 id: snapshots
 title: Snapshots
 ---
-Skytable supports automated snapshots that can be used for periodic backups. 
+
+Skytable supports automated snapshots that can be used for periodic backups.
 Skytable's snapshotting system is dead simple and works in a similar way to [BGSAVE](persistence).
 
 ## Enabling snapshots
 
-Snapshots aren't enabled by default - you have to enable them by using the configuration file  or [command line arguments](config-cmd) To your existing configuration file, just add the following block:
+Snapshots aren't enabled by default - you have to enable them by using the configuration file or [command line arguments](config-cmd) To your existing configuration file, just add the following block:
 
-``` toml
+```toml
 [snapshot]
 every = 3600
 atmost = 4
+failsafe = true # optional
 ```
 
 Here's what these values mean:
 
-* `every` - Number of seconds to wait before creating another snapshot
-* `atmost` - The maximum number of snapshots to keep
+- `every` - Number of seconds to wait before creating another snapshot
+- `atmost` - The maximum number of snapshots to keep
+- `failsafe` - This indicates whether the database should stop accepting write operations if
+  snapshotting fails
 
 ## Storage of snapshots
 
@@ -32,4 +36,4 @@ As mentioned earlier, snapshots work just like `BGSAVE` . A task is spawned that
 
 ## Methods of creating snapshots
 
-Snapshots can be created automatically by using the configuration file. However, if you want to create snapshots remotely, you can use the [ `MKSNAP` ](actions/mksnap) action. Do note that this **requires snapshotting to be enabled** before it can create snapshots.
+Snapshots can be created automatically by using the configuration file. However, if you want to create snapshots remotely, you can use the [ `MKSNAP` ](actions/mksnap) action. Do note that this **requires snapshotting to be enabled** before it can create snapshots.

docs/snapshots.md

@@ -11,6 +11,7 @@ module.exports = {
     "snapshots",
     "ssl",
     "benchmarking",
+    "deployment-notes",
     "building-from-source",
     {
         "type": "category",
@@ -28,6 +29,7 @@ module.exports = {
             "actions/mksnap",
             "actions/mset",
             "actions/mupdate",
+            "actions/pop",
             "actions/sdel",
             "actions/set",
             "actions/sset",