Dolt format for version control and incremental downloads

[Javdat]

Hi Jeroen,

First off, thank you for putting this project together! It's very useful and well structured.

I wanted to throw out an idea that might be worth exploring: moving this dataset (or maintaining a mirror of it) using Dolt. It’s like Git, but for SQL databases — so it lets you version control structured data, run SQL queries on it, and even track row-level changes over time.

This could be a nice fit for FinanceDatabase since:
• The data is already highly structured (CSV tables)
• It gets updated periodically
• There’s potential for collaboration and even SQL-based querying from the community

Some practical benefits:
• Anyone can query the data directly with SQL (e.g. find all ETFs by sector or country)
• Changes are tracked at the row level, not just file-level diffs. So allows incremental downloads vs redownloading the whole files every time.
• It supports branching, merging, and versioning just like Git, but for tables

Totally understand if this is out of scope or more overhead than it’s worth — just thought it might be a cool enhancement down the road, or maybe something to experiment with alongside the main repo.

Thanks again for all the work you’ve put into this!

Kind regards,
Javdat

[wolfferine]

👍 Sounds as an excellent idea to me! Hope Jeroen agrees!

[JerBouma]

Thank you for the suggestion. I can see the clear benefit this has but I will have to look into the implementation. Also, given their relative small team at most I'd make a mirror available given that it is not a guarantee their business model will stay afloat.