Skip to content

Commit 43db26c

Browse files
Ambient Code Botclaude
Ambient Code Bot
and
claude
committed
docs: Add ARCHITECTURE.md for codebase overview
Add top-level architecture document describing the Feast system design, component relationships, and codebase structure. Uses the existing architecture diagram from docs/assets/feast_marchitecture.png consistent with the README. Covers: system overview, core concepts, key abstractions, feature server, permissions, CLI, Kubernetes operator, directory structure, data flow diagrams, and extension points. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Ambient Code Bot <bot@ambient-code.local>
1 parent c3332dc commit 43db26c

File tree

1 file changed

+223
-0
lines changed

1 file changed

+223
-0
lines changed

‎ARCHITECTURE.md‎

Lines changed: 223 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,223 @@
1+
# Feast Architecture
2+
3+
This document describes the high-level architecture of Feast, the open-source feature store for machine learning. It is intended for contributors, AI agents, and anyone who needs to understand how the codebase is organized.
4+
5+
## System Overview
6+
7+
Feast manages the lifecycle of ML features: from batch and streaming data sources through transformations, storage, registration, and low-latency serving for real-time inference. The system is designed around pluggable backends—every storage layer, compute engine, and registry can be swapped independently.
8+
9+
![Feast Architecture](docs/assets/feast_marchitecture.png)
10+
11+
The above diagram shows Feast's core flow:
12+
13+
- **Request Sources**, **Stream Sources** (Kafka, Kinesis), and **Batch Sources** (Snowflake, BigQuery, S3, Redshift, GCS, Parquet) feed into Feast
14+
- **Transform** — feature transformations applied at ingestion or request time
15+
- **Store** — features are stored in offline (historical) and online (low-latency) stores
16+
- **Serve** — features are served for real-time model inference
17+
- **Register** — feature definitions are registered in a metadata catalog
18+
- **Online Features** — served for real-time model inference
19+
- **Offline Features** — used for model training and batch scoring
20+
21+
For the full deployment guide (Snowflake/GCP/AWS), see [How-To Guides](https://docs.feast.dev/how-to-guides/feast-snowflake-gcp-aws).
22+
23+
### Component Overview
24+
25+
| Layer | Component | Implementations |
26+
|-------|-----------|-----------------|
27+
| **SDK / CLI** | `feast.feature_store` | Python, Go, Java |
28+
| **Registry** | Metadata catalog | File (S3/GCS), SQL (Postgres/MySQL/SQLite) |
29+
| **Provider** | Orchestrator | PassthroughProvider |
30+
| **Offline Store** | Historical retrieval | BigQuery, Snowflake, Redshift, Spark, DuckDB, Postgres, Trino, Athena |
31+
| **Online Store** | Low-latency serving | Redis, DynamoDB, Bigtable, Postgres, SQLite, Cassandra, Milvus, Qdrant |
32+
| **Compute Engine** | Materialization jobs | Local, Spark, Kubernetes, Ray, Snowflake, AWS Lambda |
33+
34+
## Core Concepts
35+
36+
| Concept | Description | Definition File |
37+
|---------|-------------|-----------------|
38+
| **Entity** | A real-world object (user, product) that features describe | `sdk/python/feast/entity.py` |
39+
| **FeatureView** | A group of features sourced from a single data source | `sdk/python/feast/feature_view.py` |
40+
| **OnDemandFeatureView** | Features computed at request time via transformations | `sdk/python/feast/on_demand_feature_view.py` |
41+
| **StreamFeatureView** | Features derived from streaming data sources | `sdk/python/feast/stream_feature_view.py` |
42+
| **FeatureService** | A named collection of feature views for a use case | `sdk/python/feast/feature_service.py` |
43+
| **DataSource** | Connection to raw data (file, warehouse, stream) | `sdk/python/feast/data_source.py` |
44+
| **Permission** | Authorization policy controlling access to resources | `sdk/python/feast/permissions/permission.py` |
45+
46+
## Key Abstractions
47+
48+
### FeatureStore (`sdk/python/feast/feature_store.py`)
49+
50+
The main entry point for all SDK operations. Users interact with Feast through this class:
51+
52+
- `apply()` — register feature definitions in the registry
53+
- `get_historical_features()` — point-in-time correct feature retrieval for training
54+
- `get_online_features()` — low-latency feature retrieval for inference
55+
- `materialize()` / `materialize_incremental()` — copy features from offline to online store
56+
- `push()` — push features directly to the online store
57+
- `teardown()` — remove infrastructure
58+
59+
### Provider (`sdk/python/feast/infra/provider.py`)
60+
61+
Orchestrates the offline store, online store, and compute engine. All cloud providers (GCP, AWS, Azure, local) use `PassthroughProvider`, which delegates directly to the configured store implementations.
62+
63+
### OfflineStore (`sdk/python/feast/infra/offline_stores/offline_store.py`)
64+
65+
Abstract base class for historical feature retrieval. Key methods:
66+
67+
- `get_historical_features()` — point-in-time join of features with entity timestamps
68+
- `pull_latest_from_table_or_query()` — extract latest entity rows for materialization
69+
- `pull_all_from_table_or_query()` — extract all rows in a time range
70+
- `offline_write_batch()` — write features to the offline store
71+
72+
Implementations: BigQuery, Snowflake, Redshift, Spark, Dask, DuckDB, Postgres, Trino, Athena, and more under `infra/offline_stores/contrib/`.
73+
74+
### OnlineStore (`sdk/python/feast/infra/online_stores/online_store.py`)
75+
76+
Abstract base class for low-latency feature serving. Key methods:
77+
78+
- `online_read()` — read features by entity keys
79+
- `online_write_batch()` — write materialized features
80+
- `update()` — create/update cloud resources
81+
- `retrieve_online_documents()` — vector similarity search (for embedding stores)
82+
83+
Implementations: Redis, DynamoDB, Bigtable, Snowflake, SQLite, Postgres, Cassandra, MongoDB, MySQL, Elasticsearch, Milvus, Qdrant, and a HybridOnlineStore that combines multiple backends.
84+
85+
### Registry (`sdk/python/feast/infra/registry/`)
86+
87+
The metadata catalog that stores all feature definitions (entities, feature views, feature services, permissions). Two main implementations:
88+
89+
- **FileRegistry** (`registry.py`) — serializes the entire registry as a single protobuf file, stored on local disk, S3, GCS, or Azure Blob. Uses `RegistryStore` backends for storage.
90+
- **SqlRegistry** (`sql.py`) — stores metadata in a SQL database (PostgreSQL, MySQL, SQLite).
91+
92+
### ComputeEngine (`sdk/python/feast/infra/compute_engines/base.py`)
93+
94+
Abstract base class for materialization — the process of copying features from the offline store to the online store. Key method:
95+
96+
- `materialize()` — execute materialization tasks, each representing a (feature_view, time_range) pair
97+
98+
Implementations: Local (single-machine), Spark, Kubernetes (K8s Jobs), Ray, Snowflake (SQL-based), AWS Lambda.
99+
100+
### Feature Server (`sdk/python/feast/feature_server.py`)
101+
102+
A FastAPI application that exposes Feast operations over HTTP:
103+
104+
- `POST /get-online-features` — retrieve online features
105+
- `POST /push` — push features to online/offline stores
106+
- `POST /materialize` — trigger materialization
107+
- `POST /materialize-incremental` — incremental materialization
108+
109+
Started via `feast serve` CLI command.
110+
111+
## Permissions and Authorization
112+
113+
The permissions system (`sdk/python/feast/permissions/`) provides fine-grained access control:
114+
115+
| Component | File | Purpose |
116+
|-----------|------|---------|
117+
| `Permission` | `permission.py` | Policy definition (resource type + action + roles) |
118+
| `SecurityManager` | `security_manager.py` | Runtime permission enforcement |
119+
| `AuthManager` | `auth/auth_manager.py` | Token extraction and parsing |
120+
| `AuthConfig` | `auth_model.py` | Auth configuration (OIDC, Kubernetes, NoAuth) |
121+
122+
Auth flow: Client sends token → AuthManager extracts identity → SecurityManager checks Permission policies → access granted or denied.
123+
124+
## CLI
125+
126+
The Feast CLI (`sdk/python/feast/cli/cli.py`) is built with Click and provides commands for:
127+
128+
- `feast apply` — register feature definitions
129+
- `feast materialize` / `feast materialize-incremental` — run materialization
130+
- `feast serve` — start the feature server
131+
- `feast plan` — preview changes before applying
132+
- `feast teardown` — remove infrastructure
133+
- `feast init` — scaffold a new feature repository
134+
135+
## Kubernetes Operator
136+
137+
The Feast Operator (`infra/feast-operator/`) is a Go-based Kubernetes operator built with controller-runtime (Kubebuilder):
138+
139+
| Component | Location | Purpose |
140+
|-----------|----------|---------|
141+
| CRD (`FeatureStore`) | `api/v1/featurestore_types.go` | Custom Resource Definition |
142+
| Reconciler | `internal/controller/featurestore_controller.go` | Main control loop |
143+
| Service handlers | `internal/controller/services/` | Manage Deployments, Services, ConfigMaps |
144+
| AuthZ | `internal/controller/authz/` | RBAC/authorization setup |
145+
146+
The operator watches `FeatureStore` custom resources and reconciles Deployments, Services, ConfigMaps, Secrets, CronJobs, and HPAs to run Feast components in Kubernetes.
147+
148+
## Directory Structure
149+
150+
```
151+
feast/
152+
├── sdk/python/feast/ # Python SDK (primary implementation)
153+
│ ├── cli/ # CLI commands (Click)
154+
│ ├── infra/ # Infrastructure abstractions
155+
│ │ ├── offline_stores/ # Offline store implementations
156+
│ │ ├── online_stores/ # Online store implementations
157+
│ │ ├── compute_engines/ # Materialization engines
158+
│ │ ├── registry/ # Registry implementations
159+
│ │ ├── feature_servers/ # Feature server deployments
160+
│ │ └── common/ # Shared infra code
161+
│ ├── permissions/ # Authorization system
162+
│ ├── transformation/ # Feature transformations
163+
│ └── feature_store.py # Main FeatureStore class
164+
├── go/ # Go SDK
165+
├── java/ # Java SDK (serving + client)
166+
├── protos/ # Protocol Buffer definitions
167+
├── ui/ # React/TypeScript web UI
168+
├── infra/ # Infrastructure and deployment
169+
│ ├── feast-operator/ # Kubernetes operator (Go)
170+
│ ├── charts/ # Helm charts
171+
│ └── terraform/ # Cloud infrastructure (IaC)
172+
├── docs/ # Documentation (GitBook)
173+
├── examples/ # Example feature repositories
174+
└── Makefile # Build targets
175+
```
176+
177+
## Data Flow
178+
179+
### Training (Offline)
180+
181+
```
182+
Data Source → OfflineStore.get_historical_features() → Point-in-Time Join → Training DataFrame
183+
```
184+
185+
1. User defines `FeatureView` + `Entity` + `DataSource`
186+
2. User calls `store.get_historical_features(entity_df, features)`
187+
3. OfflineStore performs point-in-time join against the data source
188+
4. Returns a `RetrievalJob` that materializes to a DataFrame or Arrow table
189+
190+
### Serving (Online)
191+
192+
```
193+
OfflineStore → ComputeEngine.materialize() → OnlineStore → FeatureServer → Inference
194+
```
195+
196+
1. `feast materialize` triggers the compute engine
197+
2. ComputeEngine reads latest values from the offline store
198+
3. Values are written to the online store via `OnlineStore.online_write_batch()`
199+
4. Feature server or SDK reads from online store via `OnlineStore.online_read()`
200+
201+
### Push-Based Ingestion
202+
203+
```
204+
Application → FeatureStore.push() → OnlineStore (+ optionally OfflineStore)
205+
```
206+
207+
Features can be pushed directly without materialization, useful for streaming or real-time features.
208+
209+
## Extension Points
210+
211+
Feast is designed for extensibility. To add a new backend:
212+
213+
1. **Offline Store**: Subclass `OfflineStore` and `OfflineStoreConfig` in `infra/offline_stores/contrib/`
214+
2. **Online Store**: Subclass `OnlineStore` and `OnlineStoreConfig` in `infra/online_stores/`
215+
3. **Compute Engine**: Subclass `ComputeEngine` in `infra/compute_engines/`
216+
4. **Registry Store**: Subclass `RegistryStore` in `infra/registry/`
217+
218+
Register the new implementation in `RepoConfig` (see `repo_config.py` for the class resolution logic).
219+
220+
## Related Documents
221+
222+
- [Development Guide](docs/project/development-guide.md) — build, test, and debug instructions
223+
- [Operator README](infra/feast-operator/README.md) — Kubernetes operator documentation

0 commit comments

Comments
 (0)