Skip to content

Commit f5554b9

Browse files
committed
update readme and version etc
1 parent b8d3f11 commit f5554b9

File tree

3 files changed

+45
-1
lines changed

3 files changed

+45
-1
lines changed

README.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,14 @@ Transform data and create rich visualizations iteratively with AI 🪄. Try Data
2222

2323
## News 🔥🔥🔥
2424

25+
- [04-23-2025] Data Formulator 0.2.1: External Data Loader
26+
- We introduced an external data loader to make import data easier.
27+
- Example data loaders from MySQL and Azure Data Explorer (Kusto) are provided.
28+
- Call for action:
29+
- Users: let us know which data source you'd like to load data from.
30+
- Developers: help us build more data loaders.
31+
- Created a discord channel for discussions.
32+
2533
- [04-23-2025] Data Formulator 0.2: working with *large* data 📦📦📦
2634
- Explore large data by:
2735
1. Upload large data file to the local database (powered by [DuckDB](https://github.com/duckdb/duckdb)).
Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
## Data Loader Module
2+
3+
This module provides a framework for loading data from various external sources into DuckDB. It follows an abstract base class pattern to ensure consistent implementation across different data sources.
4+
5+
### Building a New Data Loader
6+
7+
The abstract class `ExternalDataLoader` defines the data loader interface. Each concrete implementation (e.g., `KustoDataLoader`, `MySQLDataLoader`) handles specific data source connections and data ingestion.
8+
9+
To create a new data loader:
10+
11+
1. Create a new class that inherits from `ExternalDataLoader`
12+
2. Implement the required abstract methods:
13+
- `list_params()`: Define required connection parameters
14+
- `__init__()`: Initialize connection to data source
15+
- `list_tables()`: List available tables/views
16+
- `ingest_data()`: Load data from source
17+
- `view_query_sample()`: Preview query results
18+
- `ingest_data_from_query()`: Load data from custom query
19+
3. Register the new class into `__init__.py` so that the front-end can automatically discover the new data loader.
20+
21+
The UI automatically provide the query completion option to help user generate queries for the given data loader (from NL or partial queries).
22+
23+
### Example Implementations
24+
25+
- `KustoDataLoader`: Azure Data Explorer (Kusto) integration
26+
- `MySQLDataLoader`: MySQL database integration
27+
28+
### Testing
29+
30+
Ensure your implementation:
31+
- Handles connection errors gracefully
32+
- Properly sanitizes table names
33+
- Respects size limits for data ingestion
34+
- Returns consistent metadata format
35+
36+
Launch the front-end and test the data loader.

pyproject.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
44

55
[project]
66
name = "data_formulator"
7-
version = "0.2.1.1"
7+
version = "0.2.1.2"
88

99
requires-python = ">=3.9"
1010
authors = [

0 commit comments

Comments
 (0)