FileStorage

To avoid too memoy consume, LiteDB has a limit document size in 1Mb. To text documents, this is a huge size. But for store files, 1Mb is too small. So, LiteDB implements FileStorage, a custom colleciton to store files and streams.

LiteDB use 2 special collections to split file content in chuncks.

• _files collection store file reference and metadata only

{
    _id: "my-photo",
    filename: "my-photo.jpg",
    mimeType: "image/jpg",
    length: { $numberLong: "2340000" },
    uploadDate: { $date: "2015-01-01T00:00:00Z" },
    metadata: { "key1": "value1", "key2": "value2" }
}

• _chunks collection stores binary data in 1MB each chunck.

{
    _id: "my-photo\00001",
    data: { $binary: "VHlwZSAob3Igc ... GUpIGhlcmUuLi4" }
}
{
    _id: "my-photo\00002",
    data: { $binary: "pGaGhlcmUuLi4 ... VHlwZSAob3Igc" }
}
{
   ...
}

Files are identified by an _id string value, with following rules:

• Starts with a letter, number, _, -, $, @, !, +, %, ; or .
• If contains a /, must be sequence with chars above

To better organize many files, you can use _id as a directory/file_id. This will be a great solution to quick find all files in a directory using Find method.

Example: $/photos/2014/picture-01.jpg

To access files, FileStorage collection contains simple methods like:

• Upload: Send file or stream to database. Can be used with file or Stream. If file already exits, file content is override.
• Download: Get your file from database and copy to Stream parameter
• Delete: Delete a file reference and all data chunks
• Find: Find one or many files in _files collection. Returns LiteFileInfo class, that can be download data after.
• SetMetadata: Update a stored file metadata. This method doesn't change data bytes, affect only _files.metadata
• OpenRead: Find file by _id and returns a LiteFileStream to read file content as stream

// Upload a file from file system
db.FileStorage.Upload("$/photos/2014/picture-01.jpg", @"C:\Temp\picture-01.jpg");

// Upload a file from a Stream
db.FileStorage.Upload("$/photos/2014/picture-01.jpg", strem);

// Find file reference only (no binary data downloaded yet) - returns null if not found
LiteFileInfo file = db.FileStoage.FindById("$/photos/2014/picture-01.jpg");

// Now, load binary data and save to file system
file.SaveAs(@"C:\Temp\new-picture.jpg");

// Or get binary data as Stream and copy to another Stream
file.CopyTo(Response.OutputStream);

// Find all files references in a "directory" - use `_id` index with `StartsWith` operator
var files = db.FileStoage.Find("$/photos/2014/");

FileStorage do not support transaction. The reason for that is to avoid put all file in memory before store in disk. Transactions are used per chunck. Each uploaded chunck are in a single transaction.

FileStorage keeps _files.length in 0 during _chuncks uploads. When all chuncks finish FileStorage updates _files.length to real data size. If you try to download a zero bytes length file, the uploaded file are corrupted. Corrupted files doesn't mean corrupt database.