Application to organize data tables

[OAGr]

Right now in

Right now in effective altruism, there seem to be a whole bunch of different Airtable tables and other sorts of data tables. These are typically spread out through a bunch of different websites. It's become a pain.

In particular, this is a bottleneck to forecasting on structured data. If there were better table infrastructure, then that could be used as the foundation for scalable forecasting. We could both have forecasts on all items of existing data tables and on items of future data tables.

If we could import/synchronize these into one system, that could help with:

• Discoverability
• Metadata additions, like comments, and ratings of accuracy.
• A very fast interface for going through them.
• An API to access each part of it (very useful for some forecasting challenges)

Relevant data includes:

• Much of QURI's stuff, like our list of projects (future and previous)
• Our World In Data data tables
• Much of Vipul Naik's websites
• There are a bunch of one-off effective altruism tables out there.
• Michael Aird has a lot of databases/collections. Here's one.

I'm not sure how to best do this. Some options include:

• Host everything using custom instances of Datasette
• Use some enterprise solution that ingests data, like Snowflake.
• Make a more hacky solution with PostGres, directly.

It's possible there's some neat solution to this, like posting all of the data to Github repos, or to some other "Github but for data" alternative. dolthub is one option. This Hacker News Thread discusses other options, with one relevant table being linked to here.

Datahub is also sort of neat: https://datahub.io/

https://www.datopian.com/ does consulting. They might be a good group to reach out to.

[uvafan]

I'd be wary of QURI trying this out in the near-term even though it seems like the type of thing that seems like it might be super valuable if done well, because:

1. If it doesn't go super well it could be a pain, and could drag on to finish a satisfactory version. In particular, the XKCD on "competing standards" (can find link if you're not familiar but 95% confident that you are) concern.
2. QURI is pretty small + young, and something like this would need to be consistently maintained and updated to be valuable. I think when possible QURI should avoid projects like this to allow flexibility for pivoting, and in case QURI doesn't work out.

Some of the concerns I laid out in #18 (comment) also apply here.

[OAGr]

Quick thoughts:

1. One big question is "How much work would this be". I was thinking of an experiment that would take 1-4 weeks. I have a lot of uncertainty on how technically doable this would be. It's possible it would be pretty easy with some clever thinking, and it's possible it would be a nightmare.
2. "If it doesn't go super well it could be a pain" -> Hopefully we could find ways to be okay killing projects that stop performing. I think so far this hasn't been a big problem for us (arguably we've made the opposite mistake), but maybe that will change.
3. "need to be consistently maintained" -> I'm not sure. I agree it needs to be maintained, but hope that it might be fairly lightweight to maintain. Not sure though.

[OAGr]

Rows.com might help a lot here.
https://rows.com/