API Design: OpenStream and CertificatePushed

[Freyskeyd]

This proposal is here to define a starting point on how to handle the different kind of input that can be sent to the TCE gRPC server from any client.

We will try to define a unique API for any clients using this gRPC entrypoint. The goal is to allow anyone to define a context to drive what certificates are pushing to the stream.

For simplicity, I’ll only talk about target stream, meaning that only certificate which targets a particular subnet are going to be pushed.

Opening the stream

After negotiating and handshake between the client and the server, the stream needs to know what certificates send. To do so, the client needs to use the OpenStream message in order to define a context that can be used by the server.

The OpenStream message is defining a TargetCheckpoint that contains various information:

• target_subnet_ids: A list of targeted subnet ids that the client is interested in receiving the certificates. This field can't be empty and need at least one subnet_id.
• positions: A list of TargetSubnetPosition that define the local state of the client.

In the next sections, I’m going to use a state representing the state of a TCE node internally, such as:

The subnet_A is having 7 certificates targeting it, it produced 2 certificates.

The subnet_B is having 2 certificates targeting it, it produced 3 certificates.

The subnet_C is having 1 certificate targeting it.

The subnet_D is having 0 certificate targeting it, it produced 3 certificates.

The certificate 0x07 is targeting both subnet_A and subnet_C.

In this example, the certificate_id represent some kind of "ordering" across certificates. Meaning that 0x03 is coming before 0x04 and after 0x02.

OpenStream without position

OpenStream {
    target_checkpoint: TargetCheckpoint {
        target_subnet_ids: [subnet_A],
        positions: []
    }
}

When a client sends this OpenStream the server will translate it that way:

• This client doesn’t have any position context
• This client is interested in any certificates targeting the subnet_A

The server needs to respond with a StreamOpened and will start pushing certificates.

The response for the current query will be:

| target   | source   | P | cert_id |
|----------|----------|---|---------|
| subnet_A | subnet_B | 0 | 0x01    |
| subnet_A | subnet_B | 1 | 0x03    |
| subnet_A | subnet_B | 2 | 0x07    |
| subnet_A | subnet_C | 0 | 0x04    |
| subnet_A | subnet_D | 0 | 0x02    |
| subnet_A | subnet_D | 1 | 0x05    |
| subnet_A | subnet_D | 2 | 0x06    |

OpenStream with position

OpenStream {
    target_checkpoint: TargetCheckpoint {
        target_subnet_ids: [subnet_A],
        positions: [
            TargetStreamPosition {
                target_subnet_id: subnet_A,
                source_subnet_id: subnet_B,
                position: 1,
                certificate_id: 0x03,
            },
            TargetStreamPosition {
                target_subnet_id: subnet_A,
                source_subnet_id: subnet_D,
                position: 1,
                certificate_id: 0x05,
            }
        ]
    }
}

When a client sends this OpenStream the TCE will translate it that way:

• This client is interested in any certificates targeting the subnet_A
• This client has already received every certificate to the position 1 of source subnet subnet_B
• This client has already received every certificate to the position 1 of source subnet subnet_D

The server needs to respond with a StreamOpened and will start pushing certificates.

The response for the current query will be:

| target   | source   | P | cert_id |
|----------|----------|---|---------|
| subnet_A | subnet_B | 2 | 0x07    |
| subnet_A | subnet_C | 0 | 0x04    |
| subnet_A | subnet_D | 2 | 0x06    |

OpenStream with multiple subnet_id

OpenStream {
    target_checkpoint: TargetCheckpoint {
        target_subnet_ids: [subnet_A, subnet_C],
        positions: []
    }
}

Requesting an OpenStream with multiple target_subnet_ids will return certificates targeting every subnet defined. Duplication will be removed. In our example, the certificate 0x07 is both targeting subnet_A and subnet_C but the TCE is going to push the certificate only one time.

CertificatePushed

The pushed certificate message is containing only the certificate. In order to allow the client to track the progress, we have two options:

• Give this information to the client through another arguments of the CertificatePushed message, containing a list of TargetStreamPosition
• Let the client handle this, meaning that upon receiving a CertificatePushedit needs to infer the position

I would prefer the first option.

[hadjiszs]

Nice write-up 👌
Looks good to me overall

I just see one non-critical question: on whether we want to refuse the handshake in case that one element in positions is incorrect for whatever reason
Or if we still accept to open the stream solely for the source subnets that are consistent with the given checkpoint

I guess we'll prefer at first to accept the stream even in case that some positions are incorrect for some source subnet (e.g, mismatch between given cert id and position, or the position is way too far in the future, etc)

So that we avoid stalling the entire testing system in case of issue with the history of one subnet

[Freyskeyd]

The positions are optionals (at least for now). We can design the checking that way:

The target_subnet_ids field is the source of truth. Meaning that a TargetStreamPosition referring to a subnet in the target_subnet_ids is valid. A TargetStreamPosition referring to a subnet which isn't in the target_subnet_ids is invalid and ignored. A TargetStreamPosition need to have a position field but the certificate_id field is optional, if provided the TargetStreamPosition is validated using it.(?). If a certificate_id is not provided, no verification is done and we just keep sending from the defined position.

Having a TargetStreamPosition far in the past regarding our current state is not a problem, we'll send certificates accordingly. Having a TargetStreamPosition far in the future regarding our current state is a different kind of issue. We need to define the far parameter. We can either accept the TargetStreamPosition, the stream will receive the certificate when the position is reached or we can deny the TargetStreamPosition. In the latter, does it mean we fail the whole stream?

Remaining questions are:

• What to do if the validation of the certificate_id + position fails ? Closing the stream?
• What to do if the TargetStreamPosition is far in the future? Closing the stream?

[hadjiszs]

Remaining questions are:

• What to do if the validation of the certificate_id + position fails ? Closing the stream?
• What to do if the TargetStreamPosition is far in the future? Closing the stream?

yep, I was referring to that here:

I guess we'll prefer at first to accept the stream even in case that some positions are incorrect for some source subnet (e.g, mismatch between given cert id and position, or the position is way too far in the future, etc)

Would it be possible to not closing the stream by continuing to stream the certificates only for the correct TargetStreamPosition ?

As for the incorrect TargetStreamPosition (incorrect in the sense we mentioned above: mismatch, or way too far), we just send a notification error.

If we close the whole stream because of only one history mismatch on one couple (target, source), then one subnet would become completely stuck, which is not so convenient for a testing setup I guess, specifically because this case may happen a lot when we will introduce conflicting cert

Regarding the 2nd question, I think it can be totally arbitrary for now, and it would be figured out most likely empirically

---

Alternatively, I wonder if we want to remove the certificate_id field from TargetStreamPosition and rely only on the field position? But it may be better to know from the get go that there is mismatch without receiving anything in that case, then doing our cleaning on the client side and reestablishing a stream for this TargetStreamPosition