[datastore] Improve store/datastore implementation

[the-glu]

The current datastore/store implementation is quite complex with some 'loops' and database specific elements in generic classes.

Here is the current schema of interfaces/packages:

Other packages work with instances of the only type in the schema.

I propose change it it like this:

• The common store.Store takes care of initialization in a generic way, without knowing about specific databases.
• No intermediate stores in specific packages.
• Specific database implementations moved to dedicated structs (with explicit naming), including initialization, and are embedded as needed in the package's repository implementations. Common struct used for function needed during initialization by store.Store.
• Should a new type of storage be used, only the implementation of the generic store + the implementation of the repository would be added + a way to initialize the correct "sub-store." This is shown with low opacity on the schema.
• Other packages still use the only type in the schema.

[mickmis]

From what I can gather, the (aux|rid|scd).repos.Repository interfaces are the ones actually abstracting operations to be made over entities, so those are the abstractions we need to keep around to be future-proof. A lot that is on top of that can be significantly simplified.

[BenjaminPelletier]

The specific future need I have in mind is being able to support a data backing that isn't a database -- the most near-term specific example would be persistence via application-level consensus of a Raft group. In this case, whole logical operations (like "create this operational intent reference along with this subscription") would be proposed to the group which would adopt them by consensus. There wouldn't be any source-of-truth database, dialing, etc, just a shared log of DAR Entity (e.g., op intent, constraint) creation, mutation, and deletion.

In the existing architecture, the top-level resource handlers have to interact with the data backing is a Store. Possession of a Store[RepoType] allows the handler to obtain a RepoType instance that they can attempt to perform a series of atomic operations on, and the whole series of operations can automatically be retried according to Store-specific logic. In this paradigm, it doesn't seem like anything would change from the perspective of the handlers when we implement the Raft-based Store. The implementation of Store.Transact would presumably involve preparing to queue up new log entries based on the log state from the most recently accepted batch of log entries, and then expecting that new batch to be accepted if and only if all its appended log entries were consistent. The implementation of the SCD RepoType would presumably involve constructing and enqueuing log messages that would accomplish the DAR Entry CRUD operations. I would expect effectively none of the Store.Transact nor SCD RepoType implementations to have anything substantive in common with parallel implementations that use a database as the data backing.

I believe the purpose of datastore.Store is to move up one level of specificity from store.Store. While store.Store must support every possible data backing (since it is what is presented to the handlers in all cases), datastore.Store supports data backings that use a database. Therefore, everything common to all possible data backings is defined on store.Store, and then everything common to all data backings that use a database is defined on datastore.Store (perhaps a clearer name would be databasestore.Store).

So, the concept mapping would be something like:

Interface/type/struct	Concept
store.Store	Means to access and interact with any type of data backing the DSS may ever use
store.Store[RepoType]	Means to obtain a means to perform RepoType-specific operations on any type of data backing the DSS may ever use
rid.store.Store	Means to obtain a means to perform RID-specific operations on any type of data backing the DSS may ever use (this is what handlers for RID operations need at the top level)
rid.repos.Repository	RID-specific RepoType that is the immediate means to perform RID-specific operations on any type of data backing the DSS may ever use (obtainable from rid.store.Store)
datastore.Store	Partial implementation of store.Store when the data backing is a database
datastore.Datastore	Information about database that a datastore.Store is using (just a subset of datastore.Store fields with methods that only need those fields)
rid.store.datastore.Store	An rid.store.Store when the data backing is a database

Under this paradigm, I would perhaps expect something like the below when implementing an application-level consensus data backing using Raft:

Interface/type/struct	Concept
raft.Store	Partial implementation of store.Store when the data backing is a Raft group's log
raft.Group	Information about Raft group that a raft.Store is using (analogous to datastore.Datastore for database-backed storage)
rid.store.raft.Store	An rid.store.Store when the data backing is a Raft group's log

With that in mind, the following things confuse me:

---

The common store.Store takes care of initialization in a generic way, without knowing about specific databases.

What would the generic store.Store initialization do in the case of a non-database data backing like Raft consensus? I would expect database-agnostic but not-data-backing-agnostic initialization to be taken care of by datastore.Store.

---

What methods would this interface define?

---

How would this interact with other elements on the diagram?

---

Overall, I'm struggling to understand what is changing. If I ignore the names, it seems like the structure of the blocks lines up pretty much identically to what we already have except that the concrete thing that provides the rid.repos.Repository (rid.store.datastore.Store in current, store.Store[R] in proposed) moves from specific package to common package (purple line):

---

The main things that confuse me about that move are: How would this single store.Store[R] implementation implement both of the arrows circled above, and how would the right one be chosen? Or, is the intent that there would be one green store.Store[rid.stores.Postgres] and a separate store.Store[rid.stores.NewStore20]? And, how would a common-package object (store.Store[R]) generate/provide specific-package objects (rid.stores.Postgres, rid.stores.NewStore20, presumably scd.stores.Postgres, presumably scd.stores.NewStore20)?

Let me know if we should set up a real-time chat.

I think regardless of outcome, I think it would be very useful to add documentation of what we expect all these interfaces/types/structs to represent to the code. I'd be happy to make such a PR reflecting the content of this comment, but I wouldn't want to do that if we were going to immediately change a lot of things :)