Module/Services System Loader and communication interface

[AlfredoG87]

I have tried to capture here all the notes from @jsync-swirlds about the current discussion.

In a nutshell: We want an interface for communication between the different services of the block node, also, we want a module loader mechanism for them, so the system is decoupled and extensible.

Services (so far) of the Block Node:

1. Streaming Service
   a. Inbound (Consume from producers)
   b. Unverified Outbound Stream (to subscribers)
   (live and historical)
   c. Verified Outbound Stream (to subscribers)
   (live and historical)
2. Block Storage
3. Block Verification
4. State Builder
5. State Proof Creator Generator
6. ...?

@jsync-swirlds comments about this:

We probably need to meet as a team and design the interfaces for services. A Handler is reasonable within a service, but I would suggest that we need a general interface so the JVM can dynamically add and remove services at deployment (using the service loader subsystem) rather than relying on configuration or hard-coded connections.

The following is what I understand as the fundamental design goal of the "services" structure of the Block Node.

Nothing in the block node; not a single component, should be dependent on, or even aware of, any other component (or service). By designing everything asynchronous with ring buffers as the "message queue" between services, we gain a system with much improved ability to be deployed cleanly.

Example required deployment options:

• If we deploy the API service for block stream (publish and subscribe calls), and nothing else, will the system function correctly?
  • This would be a "pass-through" service that just feeds downstream consumers, but does so with the minimum of other processing.
• If we deploy block stream API and verification (without storage), will the system function correctly?
  • This represents a "verified stream" node that passes on a verified stream, but does not store the stream or process state.
• If we deploy persistence and verification, without block stream and block access service API, will that work correctly?
  • This represents a "history-only node" that reads from shared storage (perhaps S3 style) and delivers verified blocks on demand.
• If we deploy persistence, state proof, and the proof service API, and no other service, will that work correctly?
  • This represents a proof-only node that provides state and block proofs based on shared (possibly remote/S3) storage, but does not otherwise manage blocks or state.
• If we deploy Block Stream service (API), with subscribe disabled, verification service, state management, and query services, but not persistence, will that work correctly?
  • This might be a node that only manages a live query service, which has almost no local storage.

We should not be getting lists of items anywhere except the publisher API component.
That component is supposed to extract all the individual items, and pass those to the ring buffer.
If we created an interface that assumes lists, we designed that incorrectly (in my opinion), and this is exactly the correct venue to change that.

#384 (comment)_

[jsync-swirlds]

Quick note, We don't need a separate module loader; that's already built into the JVM.
We just need to use the service loading mechanism.

Also, I would suggest that inbound and outbound unverified stream is a single service.

Item (5), I would name "State Proofs" in general, as it's not just creator/generator, but also management and potentially verification.

We could add "Block Item Proofs" as another service; or we could combine that with "State Proofs".

We should add "Status Service" to the list of composable services.

I would rename "State Builder" to "Live State Management", and add "State Snapshot" as an additional service.

[AlfredoG87]

[jsync-swirlds]

I agree with a List for now. I suspect that there will be less impact than we think, and running single item is entirely acceptable. We need to get to a point where we can reasonably test performance, however, and there are a lot of additional variables currently that prevent getting any clear information about the (relatively subtle) impact of list size.

I would argue strongly it needs to be a simple List, however, not a Response message.

The other items in the main discussion post remain to be discussed.

[mattp-swirldslabs]

The Response message is meant to reuse the mediator RingBuffer for broadcasts from components (like persistence) to all consumers in the event of a the-server-is-stopping-now event. What is the alternative to provide the same functionality?

[jsync-swirlds]

Have the consumers build their responses themselves from the List.
Also, do not try to reuse the same buffer for two completely different purposes; responses belong on their own ring buffers or other structures.
Passing an output around a ring buffer forces every service to act like an external consumer using an external API message and include completely unnecessary checks for status responses and other structures that have nothing to do with the core List<BlockItem> processing. This is confusing and overcomplicated design for internal processing.
Building the output message is a trivial cost and we should not, in my opinion, hack our internal data processing to try to remove that effort.

[mattp-swirldslabs]

Consumers are unaware that an error occurred. How would they build an error Response based solely on a List? Subscribing consumers to the mediator RB is how we maintain the Helidon connections. Are you suggesting a 3rd, dedicated error RB which we have to duplicate subscriptions?

[jsync-swirlds]

Possibly. We shouldn't be publishing errors from any component on the same ring buffer with the main data flow.
If an error occurs that impacts a subscriber (many will not), then it should be sent via a dedicated channel, which might be a general "errors" channel, or might be specific to the subscribers.
One of the more important principles of messaging, in my view, is that any given channel has a singular purpose and singular content. If it's data, then only send data; if it's errors, then only send errors; if it's results, then only send results.

I need to draw this out and present that, but I am thinking that we should have a set of messaging channels that any service may subscribe to (possibly reactive, but ideally in a blocking-poll approach where feasible, to keep state on the service and limit the work needed in the messaging channel). Any service might also publish on one or more channels; depending on service and the channel purpose.

Overall this would be used to completely decouple the various services (here "channel" may be a ring buffer or any other asynchronous broadcast mechansim).
A very rough sketch:

• Publishing service receives data and pushes a List to a "Incoming Data" channel.
• Publishing service subscribes to a "Results" channel and sends responses to publishers when
  verified and/or stored results (depending on configuration for "verificationRequired"
  and "persistenceRequired" values) are received for the same block.
• Subscribers subscribe to the "Incoming Data" channel and send streams to subscribers.
  • Subscriber errors (internal or received from client) are local; they should not affect
    other services
  • Subscribers mostly don't care about errors from other services, but might send errors to
    the client if messages stop arriving for a while.
• Verification subscribes to "Incoming Data" and publishes results
  (success/failure/internal error) to "Results".
• Persistence subscribes to "Incoming Data" and publishes results (stored/error) to
  "Results".
• Subscribe (verified) is a Subscriber, except it also subscribes to "Results" and holds
  each block until it sees a result from Verification (sending the block on success, or
  possibly an error response on failed verification).
• getBlock subscribes to "Read Data" and publishes a request to "readBlock" to a "Client Request"
  channel.
• Persistence also subscribes to "Client Request" and when a "readBlock" request comes in,
  reads the block and publishes on the "Read Data" channel.
• Status Service subscribes to "Results", "Incoming Data", and possibly other channels.
  This service just watches data and keeps track of the most recent block number, the most
  recent verified block, the most recent stored block, etc...
  • This service also supplies the serverStatus gRPC endpoint.
• State Management listens on "Incoming Data" and just updates a live state.
  • This service also stores state snapshots on a schedule.
  • This service may publish read requests and have other interactions to handle
    missing data or other error conditions.
• State Management also listens on "Client Request" for a "readStateSnapshot"
  and publishes a "stateSnapshot" result on the "Read Snapshots" channel.
  • Note, this response probably will include an input stream rather than the actual
    data, because reading snapshots into memory is likely excessive.
• Reconnect Service publishes a "readStateSnapshot" request to "Client Request" and
  listens for a "stateSnapshot" result on the "Read Snapshots" channel.
  • This is also very likely how the "request snapshot" API Service will work.
• Monetization is special, and likely hooks into the gRPC layer as an authentication
  provider, but I think we need a bit more research in that area.

There are others, but the above gives an idea of how I think things should work.

No one service is aware of, much less tied to, any other service. Each service focuses entirely on its own narrow function, receives generic data on channels, and sends generic data on other channels. If a service is not present, then it's functions do not happen, but everything else works the same. The whole system is completely composable and loosely coupled.

This is not the only way to build the Block Node, but I think it is a good way to do so.

[jsync-swirlds]

I believe the recent refactor has completed the items discussed here.