Support for new blob message type

[glopesdev]

• Proposed
• Prototype: Not Started
• Implementation: Not Started
• Specification: Not Started

Summary

This proposal introduces a new message type Blob to represent variable-length, register-specific binary payloads with a custom header, and describes additional constraints on the payload structure for implementing the request-reply pattern on this message type.

Motivation

Uploading firmware blobs or long stimulus waveforms to DAC devices present a challenging edge case to handle in the Harp protocol. Specifically:

• Both cases typically require variable-length registers.
• The size of these blobs often exceeds the maximum 8-bit Length (254 bytes) and even the 16-bit ExtendedLength (64 kB) allowed by standard Harp messages.
• When using very large payloads in the request-reply pattern, the recommendation to echo the entire message in the reply body would place excessive pressure on transmission buffers.
• Custom metadata headers and structure other than flat arrays are usually required.

This specification also indirectly provides a principled approach to nesting other data exchange protocols inside a Harp message.

Detailed Design

New message type: Blob

Below we detail the structure for this new message type:

• The Type field is expanded to 3 bits and a new message type Blob is allowed with the value 4.
• Both Length and ExtendedLength are included, and interpreted as a single 24-bit value, allowing for a maximum message length of 16 MB.
• A new PayloadLength field is included after ExtendedLength, specifying the number of words in the Payload field.
• A new Blob field is included after Payload with size equal to the number of bytes remaining in the message excluding Payload and other message headers.

In the request-reply pattern, the Device MUST reply with a Blob message containing only the Payload field.

With this approach, logging of reply messages could keep to a fixed-length contract containing blob metadata, and transmission bandwidth pressure would be removed by avoiding repetition of long data buffers.

Drawbacks

TBD

Alternatives

Avoid extra length field

An alternative was considered where Length would be used for Payload size and ExtendedLength for Blob size. This would make it less ambiguous to know where Blob begins but reduces maximum blob size, unless ExtendedLength is expanded to 24-bit.

The advantage of this alternative is not requiring any new header fields, e.g. no need for a PayloadLength field.

Add Blob as a separate flag rather than a separate type

The advantage here would be allowing Blob to be applied to all other types, possibly interesting to allow both Event blobs and Read blobs in addition to the request-reply pattern for uploading blobs.

Unresolved Questions

• How best to separate between Payload and Blob?
• Should Blob be a flag or a type?

Design Meetings

[Poofjunior]

Open Questions

• How do we handle sending a BlobType message to a register that does not support it?
  • I think it should be undefined. If we start sending a BlobType message, there will be potentially a very large number of bytes that we need to discard before we start looking for valid messages again. Even if encode the byte count in the message that follows, we are now asking the device to skip over a specified number of bytes again before it can receive legitimate messages again. In a nuthshell, we should just say: "Don't do this."
• How do we handle sending a BlobType message to a core that does not support it?
  • I think this should also be undefined for the same reasons above.
• Why specify payload word length? Can we instead just send the number of bytes over and leave the interpretation of the bytes up to the user? (This makes the idea of sending a blob message type more akin to a memcpy)

Previous Questions

Should Blob be a flag or a type?

Implementation-wise, I don't have strong opinions other than to demarcate that a message contains "blob-like" data as early as possible in the packet. This simplifies the process of parsing the package on the device side since we can interpret the meaning of the bytes that follow as early as possible without needing to buffer an excessive number of bytes before we can decide how to deal with the data that follows.

How best to separate between Payload and Blob?

If we demarcate that a message contains "blob-like" data early in the packet, can we give ourselves more flexibility and just define subsequent fields with lengths message-type-specific? In other words, if BlobType is encoded in the first byte, then a proposed packet fields could be identical to that of a normal harp message except field lengths would be longer for the blob message type.

<msg_type> | <raw_byte_length> | <address> | <port> | <payload_type> | <payload>

# raw_byte_length -> is 3-4 bytes instead of 1 byte
# payload -> is as long as it needs to be.

[glopesdev]

Why specify payload word length? Can we instead just send the number of bytes over and leave the interpretation of the bytes up to the user? (This makes the idea of sending a blob message type more akin to a memcpy)

This was done to keep the original symmetric reply semantics as closely as possible. In general, all requests to a device are echoed back with a timestamp. This allows both reconstructing state changes and the timing of those changes from a single binary stream coming from the device.

For reasons of bandwidth and storage, I felt we probably should not echo full 16 MB messages back to the host, as it would put undue pressure on both device computation and transmission bandwidth. Here I was thinking mostly of data like video streams, and imagining having to place the burden on the device to echo the whole stream back, which seemed excessive to me.

However, I do feel we still want to have some "echo" that there was a change done to the device state, together with the timestamp of that change, and some useful information. The proposal is to standardize the notion of what this "useful information" is by splitting the blob into a "standard" payload (defined in payload type words) which is included in the reply echo, and blob "bytes", which are left entirely up to the device.

Essentially this introduces a natural notion of blob metadata and blob data, where only the metadata is echoed back. The second advantage of having this explicit is we could bake this echo directly into each core, so that applications don't have to worry about implementing compliance with this standard. We can discuss in the next SRM the full idea and pros/cons, etc.

[Poofjunior]

I may be missing something here. It sounds like the rationale is: we want devices to issue a reply back after the request, but we don't want them to reply back with the entire payload in this case, so they instead reply back with the new field: <payload words>.

If that's the case, we're breaking symmetry anyways, so my preference would be to:

• keep the same fields but extend their lengths (instead of adding new fields)
• blob reply would be of type: U32 and the payload would be an unsigned int containing the number of bytes received.

i.e:

Blob Request:           <blob_type> | <raw_byte_length> | <address> | <port> | U8 | <payload>

Blob Reply: <timestamped_blob_type> | <raw_byte_length> | <address> | <port> | U32 | <timestamp> | <number_of_bytes_received>

vs the original proposal:

Blob Request:           <blob_type> | <raw_byte_length> | <extended_length> | <payload_word_length> | <address> | <port> | <payload_type> | <payload> <blob>

Blob Reply: <timestamped_blob_type> | <raw_byte_length> | <extended_length> | <payload_word_length> | <address> | <port> | <payload_type> | <timestamp> | <payload>

(LMK if the above is accurate.)

Since message type is the first byte in the packet, I don't think it's an issue to just take the existing fields and extend how many bytes they're worth. I think doing this makes the parsing and explanation more readable vs adding new fields,

[glopesdev]

Ideas:

• Blob is flag on message type
• Writing with Blob=HIGH gets a reply with SHA-1 + Blob=LOW
• Read with Blob=LOW gets a reply with SHA-1
• Read with Blob=HIGH gets a reply with full payload
• Events could be left up to the board, i.e. board could in theory send stream of events with SHA-1 and Blob=LOW (e.g. streaming report heartbeat) or send full payloads with Blob=HIGH (e.g. video frames)