Skip to content

Commit 2bb254f

Browse files
prrao87prrao87
committed
Update architecture section
1 parent 8610967 commit 2bb254f

File tree

1 file changed

+112
-32
lines changed

1 file changed

+112
-32
lines changed

‎docs/enterprise/architecture.mdx‎

Lines changed: 112 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -1,51 +1,131 @@
11
---
22
title: "Architecture"
33
sidebarTitle: "Architecture"
4-
description: "Learn about LanceDB Enterprise architecture and system design."
4+
description: "Learn how LanceDB Enterprise separates control, compute, and storage to serve remote tables at scale."
55
icon: "cubes"
66
---
77

8-
LanceDB Enterprise consists of the following key components:
8+
LanceDB Enterprise is a remote, cluster-backed service built for teams that need low-latency search, predictable operations, and durable storage beyond a single machine. Instead of tying query serving, indexing, compaction, and data persistence to the same process, Enterprise separates those concerns so each part of the system can scale and evolve independently.
99

10-
- Query Fleet
11-
- Plan Execution Fleet
12-
- Indexer Fleet
10+
At a high level, it helps to think of LanceDB Enterprise as a set of layers that interoperate with one another. Users connect to **remote tables** over the network. The **data plane** serves reads, writes and background jobs such as indexing and compaction. The **control plane** manages configuration, identity, policy, and cluster lifecycle. **Indexers** build indexes and compact data outside the request path. **Object storage** holds the durable table data, manifests, and index artifacts independently of the machines serving traffic.
1311

14-
![architecture](/static/assets/images/enterprise/architecture.png)
12+
This architecture matters because enterprise workloads are rarely shaped like a single benchmark. Some need thousands of concurrent queries. Others need large-scale ingestion, continuous indexing, or strict operational boundaries between user traffic and background work. LanceDB Enterprise is designed so those workloads do not all compete for the same machine, disk, or process.
1513

16-
### Query Execution
14+
```mermaid
15+
flowchart TB
16+
RT[Remote tables]
1717
18-
The LanceDB stateless query fleet is capable of managing **tens of thousands** of queries per second (QPS) per table with minimal latency.
19-
This level of throughput satisfies the requirements of even the most demanding production environments.
18+
subgraph DP[Data plane]
19+
QS[Query serving]
20+
IX[Indexers]
21+
end
2022
21-
Each query is compiled into a distributed query plan and executed on the Plan Execution Fleet in parallel.
22-
Additionally, each query is auto-vectorized for recent generations of `x86_64` and `ARM`
23-
CPUs for enhanced hardware efficiency.
23+
CP[Control plane]
24+
OS[(Object storage)]
2425
25-
### Plan Execution Fleet
26+
RT --> DP
27+
DP --> OS
28+
IX --> OS
29+
CP -.->|Govern and configure| DP
30+
CP -.->|Govern and configure| IX
31+
```
2632

27-
Each plan execution node is equipped with high-performance NVMe SSDs that act as
28-
a hybrid cache for cloud object storage systems like AWS S3,
29-
Google Cloud Storage, and Azure Blob Storage.
33+
## Compute-storage separation
3034

31-
The distributed query plan enforces cache locality for both data and indices using a variant of
32-
the **consistent hashing** algorithm with a low cache miss rate.
33-
LanceDB can serve warm queries with latency in **the single-digit to low double-digit milliseconds** range.
35+
In LanceDB Enterprise, storage and compute are deliberately decoupled. Table data and index artifacts live in object storage, while query-serving and background workers read from and write to that shared durable layer. This means compute can be replaced, scaled, or specialized without making any individual node the owner of the dataset.
3436

35-
### Write Path
37+
This design has practical consequences. Query fleets can scale for interactive traffic without also scaling background indexing capacity. Heavy indexing and compaction work can run on dedicated workers instead of stealing resources from user-facing queries. Caches can accelerate hot reads without becoming the source of truth. And because the data remains in object storage, durability does not depend on the lifecycle of a particular server or local disk.
3638

37-
LanceDB Enterprise is engineered for high-throughput data ingestion and indexing.
38-
The system ensures data persistence on durable object storage before confirming any write request.
39+
## Architecture
3940

40-
An extensive indexing fleet, enhanced with hardware acceleration, operates asynchronously to
41-
perform partial or full indexing, data compaction, and cleanup.
42-
Furthermore, we achieve high-throughput indexing operations without compromising query performance.
41+
At a high level, the control plane governs the system and the data plane executes the work. The control plane is responsible for configuration, service discovery, identity integration, policy, and cluster lifecycle. It determines how the system should behave, but it is not the layer serving table data or executing user queries -- that's the role of the data plane.
4342

44-
<Note>
45-
Customer data does not go through the event queue. The queue sends events such as
46-
"create an index" to the indexers to trigger actions.
47-
</Note>
43+
Within the data plane, serving is separated into two kinds of nodes that are provisioned independently. Query nodes are the client-facing layer. They receive requests against remote tables, validate them, resolve the target table, plan the work, and return results.
4844

49-
<Info>
50-
Indexing scales down to zero when there is no activity on the table.
51-
</Info>
45+
Plan executors are the read-execution layer behind the query nodes. For read-heavy query paths, they execute cache-backed reads against object storage, which helps reduce repeated remote reads and makes performance more predictable as load grows.
46+
47+
Indexers handle heavyweight background work such as building indexes, merging index state, compacting data, and updating the table’s stored index and layout artifacts. Together, these components let LanceDB Enterprise scale request handling, read execution, and index-building independently instead of forcing them to compete for the same compute resources.
48+
49+
```mermaid
50+
flowchart TB
51+
Client[Client application]
52+
Remote[Remote table / API boundary]
53+
54+
subgraph DP[Data plane]
55+
QN[Query nodes]
56+
PE[Plan executors<br/>cache + distributed execution]
57+
IX[Indexers / background workers]
58+
end
59+
60+
subgraph CP[Control plane]
61+
CFG[Configuration, identity,<br/>policy, lifecycle]
62+
end
63+
64+
OS[(Object storage<br/>table data, manifests, indexes)]
65+
66+
Client -->|Read / write requests| Remote
67+
Remote --> QN
68+
QN -->|Query sub-plans| PE
69+
QN -->|Commits writes| OS
70+
PE <-->|Read data| OS
71+
IX <-->|Build indexes / compact data| OS
72+
QN -.->|Indexing / compaction signals| IX
73+
CFG -.->|Govern and configure| QN
74+
CFG -.->|Govern and configure| PE
75+
CFG -.->|Govern and configure| IX
76+
```
77+
78+
## Remote tables
79+
80+
A [remote table](/tables-and-namespaces#understanding-tables) is the user-facing abstraction over this architecture. From the client side, you connect to a logical storage layer and table over the network by providing a `db://...` connection identifier. The system then resolves that logical name to the underlying storage-backed table and executes the operation inside the cluster.
81+
82+
This is why Enterprise feels familiar at the API level while operationally behaving differently. Your application still issues table operations and queries, but it is no longer coupled to a local storage path or a single host. Instead, the cluster takes responsibility for execution, coordination, and background upkeep. In SDK terms, `open_table(...)` returns a `RemoteTable`. Architecturally, a remote table is the bridge between the client-facing API and the storage-backed system behind it.
83+
84+
This design makes LanceDB Enterprise suitable for catalog-backed layouts, see [Namespaces and the Catalog Model](/namespaces) for more details. For the basic application flow, see the [Enterprise quickstart](/enterprise/quickstart).
85+
86+
## Read path
87+
88+
When a client issues a query against a remote table, the path is straightforward:
89+
90+
1. The request reaches a query node in the data plane.
91+
2. The query node validates the request, resolves the table, and plans the work.
92+
3. For read-heavy queries, the query node can send part of the read work to plan executors.
93+
4. Plan executors read the required table data from object storage, using cache where it helps reduce repeated remote reads.
94+
5. Results are returned to the query node, assembled, and sent back to the client.
95+
96+
This separation is what lets Enterprise combine a clean remote API with a serving layer that can scale horizontally and keep hot data close to execution.
97+
98+
## Write path
99+
100+
Writes follow a different path because durability comes first:
101+
102+
1. A client writes to a remote table.
103+
2. The query node validates the request and commits the new table state.
104+
3. After the write succeeds, the system emits follow-up signals for indexing, compaction, or cleanup.
105+
106+
Keeping the commit path centered on object storage ensures that the durable record of the table lives outside any single query node. Regardless of whether you're using Lance namespaces or an external catalog, the catalog's role is mainly to resolve table names and provide access details -- the table’s actual data and index artifacts remain in object storage.
107+
108+
## Background work
109+
110+
Indexing, compaction, and cleanup are intentionally moved off the user request path. After table changes are committed, the system can determine that additional work is needed and assign it to background workers built for heavyweight processing.
111+
112+
In practice, that usually looks like this:
113+
114+
1. Table changes create follow-up signals for indexing, compaction, or cleanup.
115+
2. A background coordinator agent evaluates the state of the table and decides what should run next.
116+
3. Indexers read the relevant table state from object storage, produce updated artifacts, and write the results back.
117+
118+
This separation is one of the clearest architectural reasons to use LanceDB Enterprise: the same query-serving infrastructure does not have to handle every expensive indexing or compaction task itself.
119+
120+
## What this means for users
121+
122+
For teams using LanceDB Enterprise, the architecture changes the _operational model_ more than the programming model. You still work with tables and queries, but the cluster now takes responsibility for distributed execution, cache-aware reads, and long-running background jobs.
123+
124+
The result is a system that is easier to run under production pressure:
125+
126+
- You interact with remote tables instead of managing physical storage layout directly.
127+
- Query serving, indexing, and compaction can scale independently.
128+
- Durable state lives in object storage rather than on individual machines.
129+
- Background indexing and compaction improve performance over time without forcing that work into the foreground request path.
130+
131+
This architecture gives ML and AI teams a strong storage-backed platform for training, retrieval, search, and analytics workloads that can scale beyond a single machine while still providing a familiar API and predictable performance.

0 commit comments

Comments
 (0)