|
| 1 | +--- |
| 2 | +id: persistence |
| 3 | +title: Persistence |
| 4 | +--- |
| 5 | + |
| 6 | +Skytable supports the persistent storage of data, which is an inherently obvious need for any database. In this document we explore how Skytable's persistence works. |
| 7 | + |
| 8 | +## Data directory structure |
| 9 | + |
| 10 | +Whenever you start Skytable, it will create a number of directories under a root 'data' directory. This is what the |
| 11 | +data directory structure looks like: |
| 12 | + |
| 13 | +``` |
| 14 | +|__user-files (your other files) |
| 15 | +|__data |
| 16 | + |__data.bin |
| 17 | + |__snapshots |
| 18 | + |__<YYYYMMDD>-<HHMMSS>.snapshot (many) |
| 19 | + |__remote |
| 20 | + |___<snapshotname>.snapshot (many) |
| 21 | +``` |
| 22 | + |
| 23 | +The `data.bin` file is primarily used for persistence while the snapshots folder contains automatically created |
| 24 | +snapshots and remotely created snapshots. |
| 25 | + |
| 26 | +## How Skytable stores data |
| 27 | + |
| 28 | +As soon as you start Skytable, it will look for a `data.bin` file in the data directory; |
| 29 | +if the data file is found and successfully read, then the previous data is moved into the in-memory table. If no |
| 30 | +file was found, then the database server will create one. Once you terminate the daemon, it will flush all data |
| 31 | +to this file. All writes are `fsync`ed by the time they complete. |
| 32 | + |
| 33 | +### Save failure during termination |
| 34 | + |
| 35 | +The server would attempt to do a _final_ save operation before it terminates and if this fails, the |
| 36 | +server would enter into a retry loop. It will try the save operation after every 10 seconds. |
| 37 | +Exponential backoff was not chosen because it could increase to extremely large values that may hurt |
| 38 | +a sysadmin's time and productivity. |
| 39 | + |
| 40 | +You might be interested in more features like [BGSAVE](#automated-background-saving) and [snapshots](/snapshots) that |
| 41 | +can be configured and used by users. |
| 42 | + |
| 43 | +## Automated background saving |
| 44 | + |
| 45 | +Skytable supports automated saving of data in the background, via `BGSAVE`. `BGSAVE`, runs every two minutes to flush all the data in the in-memory table onto disk, unless customized through the [configuration file](config-files/#an-example-configuration). BGSAVE is enabled by default and we don't recommend disabling it until you're sure that |
| 46 | +your hardware will never fail; it is likely that this will never be the case. First BGSAVE will create a temporary |
| 47 | +file and then flush the current in-memory table onto disk. It will then replace the old `data.bin` file. The daemon automatically `fsync`s after every successful write (whether to the buffers or |
| 48 | +to the actual disk). |
| 49 | + |
| 50 | +### Reliability of BGSAVE |
| 51 | + |
| 52 | +It can happen that BGSAVE fails to flush data to the disk due to some unforeseen system issues (such as lack of |
| 53 | +empty disk space, permission errors, etc). But if we continue to accept modifications to the data, it is a bad idea |
| 54 | +because this data may never get updated! This is why if BGSAVE fails, it will automatically _poison_ the in-memory |
| 55 | +table preventing it from accepting any write/update operations. Poisioning is nothing but a global flag set in the |
| 56 | +database that indicates that the DB shouldn't accept any more updates/writes to data and in such a poisoned state, |
| 57 | +only reads are permitted. |
| 58 | + |
| 59 | +### BGSAVE Recovery |
| 60 | + |
| 61 | +BGSAVE will automatically try to recover on every 120s (or whatever duration was set). If the problem |
| 62 | +was corrected (say it was a permissions issue), then the database server will automatically resume |
| 63 | +writes and _unpoison_ the database. |
0 commit comments