[deploy] Data Formulator v0.2 -- working with large dataset (with database support)

data formulator v0.2, with database support to handle large sized data

Author: Chenglong-MS

.env.template

@@ -1,2 +1,20 @@
 # Provide frontend configuration settings from environment variables
-SHOW_KEYS_ENABLED=true
+# You can override these settings when lauching the app as well:
+#   python -m data_formulator -p 5000 --exec-python-in-subprocess true --disable-display-keys true
+
+DISABLE_DISPLAY_KEYS=false # if true, the display keys will not be shown in the frontend
+EXEC_PYTHON_IN_SUBPROCESS=false # if true, the python code will be executed in a subprocess to avoid crashing the main app, but it will increase the time of response
+
+LOCAL_DB_DIR= # the directory to store the local database, if not provided, the app will use the temp directory
+
+# External atabase connection settings
+# check https://duckdb.org/docs/stable/extensions/mysql.html 
+# and https://duckdb.org/docs/stable/extensions/postgres.html
+USE_EXTERNAL_DB=false # if true, the app will use an external database instead of the one in the app
+DB_NAME=mysql_db # the name to refer to this database connection
+DB_TYPE=mysql # mysql or postgresql
+DB_HOST=localhost 
+DB_PORT=0 
+DB_DATABASE=mysql 
+DB_USER=root 
+DB_PASSWORD=

DEVELOPMENT.md

@@ -20,9 +20,25 @@ How to set up your local machine.
     ```
 - **Configure environment variable (optional)s**
     - copy `api-keys.env.example` to `api-keys.env` and add your API keys.
-    - required fields for different providers are different, please refer to the [LiteLLM setup](https://docs.litellm.ai/docs#litellm-python-sdk) guide for more details.
-        - currently only endpoint, model, api_key, api_base, api_version are supported.
-    - this helps data formulator to automatically load the API keys when you run the app, so you don't need to set the API keys in the app UI.
+        - required fields for different providers are different, please refer to the [LiteLLM setup](https://docs.litellm.ai/docs#litellm-python-sdk) guide for more details.
+            - currently only endpoint, model, api_key, api_base, api_version are supported.
+        - this helps data formulator to automatically load the API keys when you run the app, so you don't need to set the API keys in the app UI.
+
+    - set `.env` to configure server properties:
+        - copy `.env.template` to `.env`
+        - configure settings as needed:
+            - DISABLE_DISPLAY_KEYS: if true, API keys will not be shown in the frontend
+            - EXEC_PYTHON_IN_SUBPROCESS: if true, Python code runs in a subprocess (safer but slower), you may consider setting it true when you are hosting Data Formulator for others
+            - LOCAL_DB_DIR: directory to store the local database (uses temp directory if not set)
+            - External database settings (when USE_EXTERNAL_DB=true):
+                - DB_NAME: name to refer to this database connection
+                - DB_TYPE: mysql or postgresql (currently only these two are supported)
+                - DB_HOST: database host address
+                - DB_PORT: database port
+                - DB_DATABASE: database name
+                - DB_USER: database username
+                - DB_PASSWORD: database password
+
 
 - **Run the app**
     - **Windows**

README.md

@@ -22,9 +22,18 @@ Transform data and create rich visualizations iteratively with AI 🪄. Try Data
 
 ## News 🔥🔥🔥
 
+- [04-23-2025] Data Formulator 0.2: working with *large* data 📦📦📦
+  - Explore large data by:
+    1. Upload large data file to the local database (powered by [DuckDB](https://github.com/duckdb/duckdb)).
+    2. Use drag-and-drop to specify charts, and Data Formulator dynamically fetches data from the database to create visualizations (with ⚡️⚡️⚡️ speeds).
+    3. Work with AI agents: they generate SQL queries to transform the data to create rich visualizations!
+    4. Anchor the result / follow up / create a new branch / join tables; let's dive deeper. 
+  - Checkout the demos at [[https://github.com/microsoft/data-formulator/releases/tag/0.2]](https://github.com/microsoft/data-formulator/releases/tag/0.2)
+  - Improved overall system performance, and enjoy the updated derive concept functionality.
+
 - [03-20-2025] Data Formulator 0.1.7: Anchoring ⚓︎
   - Anchor an intermediate dataset, so that followup data analysis are built on top of the anchored data, not the original one.
-  - Clean a data and work with only the cleaned data; create a subset from the original data or join multiple data, and then focus your analysis from there. The AI agent will be less likely to get confused and work faster. ⚡️⚡️
+  - Clean a data and work with only the cleaned data; create a subset from the original data or join multiple data, and then go from there. AI agents will be less likely to get confused and work faster. ⚡️⚡️
   - Check out the demos at [[https://github.com/microsoft/data-formulator/releases/tag/0.1.7]](https://github.com/microsoft/data-formulator/releases/tag/0.1.7)
   - Don't forget to update Data Formulator to test it out!