Data Formulator 0.2

Data Formulator 0.2 now supports working with large datasets, powered by the backend database!

Demonstration: Exploration of Metacritic's Best Games and Reviews - 2025

• This Kaggle dataset contains 13k+ games and 1.6M+ reviews of best games based on Metacritics reviews.
• Data source: https://www.kaggle.com/datasets/davutb/metacritic-games
• Exploration:
  • What's the relation between user scores and critic scores?
  • What are games where user reviews are really high but critic's scores are really low?
  • How does the score distribution compare between critics and users?

df-demo-game-reviews.mp4

Release details: data visualization with large sized data.

Data Formulator integrates DuckDB as the backend local database to support data exploration with large datasets (million rows). It is also possible to connect external database with DuckDB, not all connection are supported at the moment, but that's the beginning!

• Upload large sized data to the local database, or connect to existing databases (mysql or postgres) to work with large data.
  • A subset of sample data will be pulled to the frontend to explore, you can roll the dice 🎲 or sort the data by different columns to view different samples.
  • Manage local database with the Database manager.
• Interaction with Data Formulator as usual:
  • Use drag and drop to specify a chart, and Data Formulator can dynamically generate SQL query to fetch data to instantiate data. This process is quite fast!
  • Specify new visualization fields / provide NL instructions as usual, and the newly introduced NL2SQL agents can generate SQL queries based on your instruction to prepare the data, and create visualizations.
  • Anchor a dataset, followup, join some tables, can you can dive deep pretty fast into insights!
• (Minor feature updates)
  • Updated how derived concept works in Data Formulator -- data transformation is executed in the backend and updated data is appended to the new dataset. New concepts can be applied directly to new dataset in one click.
  • Improved system performance with configurable sandboxing options (main process versus subprocess) for LLM generated code (~3s interaction time reduction).
  • Configurable default visualization size in the main panel.

More explorations on the demo dataset:

• What's your favorite games and how their review change over time?
• What's the franchise that consistently improved reviews?
• What are games that have most different reviews in different platforms?
• What are games with many positive critic reviews but no user bother to play?
• What about reviews trends for the No Man's sky?

Well, it is time to upgrade Data Formulator and play with it! Let us know what you come up with :)

Data Formulator 0.1.7

With Data Anchor, we can anchor an intermediate data to isolate it's derivation context from it's predecessors. Tables created from the anchor will take the anchored table as direct input (not the original data).

This could be helpful for cleaning initial input data (so we always work with cleaned data afterwards), or when we want to focus our analysis into a subset of dataset.

Example 1: Clean table

Use anchor to clean the table, so that follow-up analysis are all build on top of the clean data. Analysis of director profit is based on the filtered data.

anchor-clean-data.mp4

Example 2: Analyze a subset

Create a subset from the original table to focus analysis. The AI agent will be less likely to be confused, analysis will be faster. The anchored asian-energy dataset includes only countries from Asia.

anchor-subset-analysis.mp4

Illustration

The anchored thread has it's own context --- no more access to the original data. Though, you have the option to add the original data back using "multi-table" approach from the previous release. You can also go back to the original data to create another branch there.

Data Formulator 0.1.6

Highlight

It was supposed to only be some improvements and bug fixes over 0.1.5, but ended up getting much better --- Data Formulator now supports working with multiple tables! 🔥🔥🔥

When you add multiple tables to Data Formulator, you can select which base tables Data Formulator will use to derive the data (in the chart builder). This means Data Formulator can flexibly decide how to join or combine multiple tables together to create a visualization or answer your question.

In this demo below, we have a datasets of UK wheats production.

• To visualize wheats production by UK monarch, we can load a second table (here I ask GPT-4o to generate the table out of nowhere since it has knowledge about history :)).
• Then, we can drag a field from the second table to indicate that we want Data Formulator to leverage both tables to generate the chart, and it does.
• In the second demo, we can manually tell Data Formulator needs to consider both tables to answer "average wheat production per monarch", and it will also join the two tables for create the answer.

df-multi-table-demo.mp4

Besides this feature, we have improved and fixed various UI and model selection issues from the community, thanks everyone for your suggestions! Let us know what you would like to see in Data Formulator next. :)

What's Changed

• Add and manipulate multiple table (dev) by @noless3011 in #89
• Dev: multi-table support & fixes by @Chenglong-MS in #99
• Bump vega from 5.23.0 to 5.26.0 by @dependabot in #95

New Contributors

• @noless3011 made their first contribution in #89

Full Changelog: 0.1.5.1...0.1.6

Data Formulator 0.1.5

What's New

Support more models!

• you can provide model in the model_selection dialog as follows, we have tested with openai, azure, anthropic, ollama (codellama:7b, llama3.2)
• to preload credential to data formulator, provide them via api-keys.env, checkout https://github.com/microsoft/data-formulator/blob/main/api-keys.env.template. Data Formulator will test and load them into the tool on start.

Still, check out this Data Formulator experience vidoe:

data-formulator-ms-year-report-demo.mp4

0.1.5.1 -- fix the file upload bug

Data Formulator 0.1.4

This is the updates to the previous version with better error message display to help users debug what's going on if Data Formulator fails to run. Also introduces the direct conversation with table, could be useful for data cleaning.

• We also improved data visualization challenges with data formulator -- can you complete them all?
• Comment in the issue when you did, or share your results/questions with others! [comment here]

Enjoy this version! If there is any feedback, let us know.

data-formulator-ms-year-report-demo.mp4

Data Formulator v0.1.3.3

This is the updates to the previous version with better error message display to help users debug what's going on if Data Formulator fails to run.

Update in 0.1.3.2: also include port option to run data formulator on a different port if the default one is occupied.
Update in 0.1.3.3: also to provide cleaning instruction when uploading an image.

Enjoy this version! If there is any feedback, let us know.

Here is a demo of this new version!

data-formulator-ms-year-report-demo.mp4

Data Formulator v0.1.2 Release

Data Formulator v0.1.2 is released, featuring:

• Python package release, so that you can install Data Formulator with pip install data_formulator locally, and run it with data_formulator.
• Codespace script is updated to use the python package, for faster start up time.
• Experimental feature on loading images or messy texts as inputs.