-
Notifications
You must be signed in to change notification settings - Fork 10
In Memory File System
In real time mode, LunarMax maintains an in-memory light-weighted file system for quick access of data, avoiding most of the random disk access. Apparently, Real time mode consumes more memory than normal mode. LunarMax allocates memory by its internal lmalloc, a thing quite similar to the Linux default implemented malloc.
Of course data in memory is compressed, but if there are no sufficient memory to use, lmalloc uses external disk space for data swapping. In this situation, the performance will be slow down, so please make sure you have enough memory space for this mode. We have an estimation of consumption:
one column, e.g. int:payment, and each record has size limitation up to 32 KB:
1M records (with size up to 32GB) need about 50M memory space
10M records (with size up to 320GB) need about 500M memory space
100M reords (with size up to 3200GB) need about 5000M memory space
...
More columns definitely cost more memory space, but linearly. You shall prepare a server with, say 100GB, memory if you will have billions of records to process in real time.
LunarMax supports using extra disk as memory just like what modern operation system does. Consult this mode (rt_virtual_mem_enabled = on)[https://github.com/LunarBaseEngin/Application/wiki/Real-Time-Configuration] for
how to replace the default OS virtual memory system, and use an extra SSD space as much more cheaper memory.
1 Home
1.1 summary
1.2 System Preparation
1.3 When LunarBase is your best choice
1.4 Benchmark
1.5 Power consumption
2 Data Model And Process
2.1 Why internal big cache
2.2 Memory Management: LunarMMU
2.3 Garbage Collection
2.4 Transaction Log
2.5 JOIN via materialized view
3 Real Time Computation: LunarMax
3.1 In-Memory File System: Memory Estimation
3.2 Configuration
3.3 Use SSD as a cheaper memory
3.4 Data Safety
3.5 HE Server VS. Cluster
3.6 High Availability
4 Create a database
4.1 Three modes
4.2 creation.conf settings
4.3 Table space
4.4 Multiple Instance
4.5 Database Status
4.6 Remove and Restore a table
5 Insertion
5.1 Insert as normal record
5.2 Insert to search engine
6 Query
6.1 Point Query
6.2 Result Handler: register your own event handler
6.3 Interpreter Pattern: complex query conditions
6.4 Range Query
6.5 Full-text Search
6.6 Algebraic Logical Query
8 Deletion
9 Materialized view
9.1 Eventual consistency
9.2 Update
9.3 MVCC in LunarBase
9.4 Easy JOIN via denormalization
9.5 CRUD in view
10 Distributed integration with
10.1 Kafka
10.2 Storm
10.3 Spark
11 Storage: Lunar Virtual File System
13 Roadmap of LunarBase future
15 FAQ