Thought experiment: index partitions

[pudo]

Imagine a yente index in which each dataset carries a partition ID (which is default if unset). The partition ID for the dataset is then baked into the index alias name, so instead of yente-entities, the alias is now called yente-entities-default. This is reflected in the indexer as a definition of managing the index lifecycle (rollover, GC). When a collection is used as the main query scope, the partition it is linked to is then used to build the alias name for all queries.

Concerns:

• Overall complexity
• Do the dataset indexes then need to also carry that prefix? We're running into the ES index max len limit (64. yente-entities-default-default-xxxx is probably ok, but yente-entities-default-icij_offshoreleaks-xxxx) is pushing it.

[pudo]

Fun times: so far, there's a reasonable expectation that the metadata of each dataset name (e.g. us_ofac_sdn) is linked to one copy of the data - so metadata used is the one released during the last update of default, which is within a couple of minutes also the version of the data used. In this scenario, however, both partitions main datasets could be released at very different times, so the metadata may be much more out of sync.