Stellar XDR Changes and Support for Feature Flags

[leighmcculloch]

Hi all,

The way that the curr and next branches of stellar/stellar-xdr get produced is changing.

However, if you consume the curr or next branch, nothing is changing for you. The curr and next branches will continue to exist and work as they do today.

What's Changing

Today development and changes to the XDR happens directly on the curr and next branches. For direct development the curr and next branches are being replaced by the main branch, which will use feature flags with #ifdef...#else...#endif C pre-processor compatible directives in the .x files to gate any features for the next protocol.

The curr branch will be code generated from the main branch through the preprocessor with a final feature set for the current protocol.

The next branch will be code generated from the main branch through the preprocessor with an all-features-enabled feature set.

Example

Here's what this will look like in practice for .x files on main:

union TransactionMeta switch (int v)
{
...
case 4:
    TransactionMetaV4 v4;
#ifdef XDR_TRANSACTION_META_V5
case 5:
    TransactionMetaV5 v5;
#endif
}

curr

If feature XDR_TRANSACTION_META_V5 is not yet included in the final feature set for the current protocol, the curr branch can be expected to contain:

union TransactionMeta switch (int v)
{
...
case 4:
    TransactionMetaV4 v4;
}

next

The next branch will always have all features on and can be expected to contain:

union TransactionMeta switch (int v)
{
...
case 4:
    TransactionMetaV4 v4;
case 5:
    TransactionMetaV5 v5;
}

Why

Today, two independently-maintained versions of the XDR definitions are maintained under the curr and next branches of the stellar/stellar-xdr repository. Some projects, such as stellar/stellar-core and stellar/rs-stellar-xdr, consume both of these, and some projects, such as stellar/go-stellar-sdk, stellar/stellar-rpc, and stellar/stellar-horizon consume one or the other depending on the phase of development, but most other projects consume only curr.

We've been experimenting with this new approach to:

• Reduce the XDR to a single definition. So that there are not two versions to keep updated.
• Decouple next independent XDR changes. To allow projects to adopt and then release a subset of upcoming XDR changes. To reduce coordination across products during protocol development. To allow longer lived experiments to share XDR without upcoming protocol releases needing to navigate them.

What This Means if You Use curr

Nothing. Continue to consume the curr branch.

What This Means if You Use next

Tooling that consumes the next branch can continue to consume the next branch as it will be code generated from the main branch with all features enabled. Alternatively, tooling can transition to consuming the main branch directly and either handling the #ifdef directives in their XDR code generator, or pre-process them using the tool below.

We plan to transition stellar/rs-stellar-xdr and stellar/stellar-core to pass through feature flags so they can be built with a subset of upcoming XDR. This is a work-in-progress that is only just beginning, so exactly what that will look like, and how granular those feature flags are, is still being worked out.

To support tooling that doesn't understand #ifdef directives, the stellar-xdr CLI (embedded in stellar-cli under the xdr sub-command) will be getting a new utility to pre-process .x files given a desired feature set.

Caveats

This work is a work-in-progress and may change and adapt as it takes place, as well as feedback is heard. We'll keep this thread updated with any changes that diverge significantly from this. Subscribe to follow.