Harp Protocol: Support For New Blob Message Type

by Admin 49 views
Harp Protocol: Support for New Blob Message Type

This article discusses a proposal to introduce a new Blob message type within the Harp protocol. This enhancement aims to efficiently handle variable-length, register-specific binary payloads, particularly for use cases like firmware updates and complex waveform transmissions to DAC devices. Let's dive into the details!

Summary

The proposed feature centers around introducing a new message type called Blob. This Blob message type is designed to handle variable-length, register-specific binary payloads. It includes a custom header and outlines specific constraints on the payload structure to facilitate the request-reply pattern. This enhancement addresses limitations in the current Harp protocol when dealing with large binary data, making it more efficient and versatile.

Motivation

Why are we even considering this new Blob type, you might ask? Well, there are several compelling reasons. Currently, uploading firmware blobs or sending long stimulus waveforms to DAC (Digital-to-Analog Converter) devices presents some tricky challenges within the existing Harp protocol. Let's break down the key motivations:

  • Variable-Length Registers: Both firmware blobs and stimulus waveforms often require variable-length registers, which the current Harp protocol struggles to manage efficiently.
  • Size Limitations: The size of these blobs frequently exceeds the maximum allowed by standard Harp messages. The current protocol limits Length to 8 bits (254 bytes) and ExtendedLength to 16 bits (64 kB). This is often insufficient for larger payloads.
  • Request-Reply Bottleneck: When using very large payloads in a request-reply pattern, the recommendation to echo the entire message in the reply body puts significant strain on transmission buffers. This can lead to performance issues and inefficiencies.
  • Custom Metadata Requirements: Often, custom metadata headers and specific structures (beyond simple flat arrays) are needed. The existing Harp protocol doesn't readily support this.

In essence, this specification offers a structured approach to integrating other data exchange protocols within a Harp message, making the system more adaptable and robust. It will improve the way how firmware updates and complex waveform transmissions are handled in the Harp protocol, addressing limitations related to variable-length registers, size constraints, request-reply inefficiencies, and custom metadata requirements.

Detailed Design

Alright, let's get into the nitty-gritty of the proposed design. This is where we'll break down the structure of the new Blob message type and how it integrates with the existing Harp protocol. Remember, the Harp standard spans device hardware, microcontroller firmware, high-level software interfaces, and data formats, so we need to consider the impact on all these levels.

New Message Type: Blob

Here's the breakdown of the new Blob message type structure:

  • Type Field Expansion: The Type field is expanded to 3 bits, allowing for a new message type Blob with the value 4. This expansion accommodates the new message type without disrupting existing ones.
  • Combined Length Fields: Both Length and ExtendedLength fields are included and interpreted as a single 24-bit value. This allows for a maximum message length of 16 MB, significantly increasing the capacity for large payloads.
  • PayloadLength Field: A new PayloadLength field is introduced after ExtendedLength. This field specifies the number of words in the Payload field. This addition allows for precise control over the amount of data in the payload.
  • Blob Field: A new Blob field is included after Payload. Its size is equal to the number of bytes remaining in the message, excluding Payload and other message headers. This field contains the actual binary data and any custom metadata.

In the request-reply pattern, the device must reply with a Blob message containing only the Payload field. This reduces transmission bandwidth by avoiding repetition of long data buffers.

With this design, logging reply messages could stick to a fixed-length contract containing blob metadata. This reduces bandwidth pressure by avoiding repetition of long data buffers. The Blob message type is a structured approach to efficiently handle large binary payloads in the Harp protocol, addressing limitations in variable-length registers, size constraints, request-reply inefficiencies, and custom metadata requirements.

Drawbacks

Of course, no proposal is without its potential downsides. So, what are the possible drawbacks of introducing this new Blob message type? Let's consider some potential issues:

  • Increased Complexity: Adding a new message type inevitably increases the complexity of the Harp protocol. This could make it more difficult to implement and maintain.
  • Compatibility Issues: Introducing a new message type might create compatibility issues with existing Harp devices and software. Careful consideration needs to be given to backward compatibility.
  • Overhead: The additional header fields (like PayloadLength) add overhead to the message. While this overhead is likely small, it could impact performance in some scenarios.

Further analysis and testing are needed to fully assess these potential drawbacks and determine the best way to mitigate them.

Alternatives

Before settling on this design, it's important to consider alternative approaches. What other options were explored, and what are the implications of not implementing this Blob type at all?

Avoid Extra Length Field

One alternative considered using the existing Length field for the Payload size and the ExtendedLength field for the Blob size. This would make it clearer where the Blob begins but would reduce the maximum blob size unless ExtendedLength is expanded to 24 bits.

The advantage of this alternative is that it wouldn't require any new header fields, such as a PayloadLength field.

Add Blob as a Separate Flag Rather Than a Separate Type

Another alternative would be to add Blob as a separate flag rather than a separate type. The advantage here would be allowing Blob to be applied to all other types. This could be interesting for allowing both Event blobs and Read blobs, in addition to the request-reply pattern for uploading blobs. This would provide more flexibility but might also add complexity.

Ultimately, not implementing this proposal would mean continuing to struggle with the limitations of the current Harp protocol when dealing with large binary payloads. This could hinder the development of new features and applications that require efficient handling of such data.

Unresolved Questions

Before we can finalize this proposal, there are still some unanswered questions that need to be addressed:

  • Payload and Blob Separation: What is the best way to distinguish between the Payload and Blob fields? Should there be a clear delimiter or a more sophisticated method?
  • Flag vs. Type: Should Blob be implemented as a flag that can be applied to other message types, or should it remain a separate message type? Each approach has its own advantages and disadvantages.

Design Meetings

Links to standard review board meetings where this proposal has been discussed will be added here.