Will the Firedancer client work in Kubernetes (i.e. inside a Docker container)?

[alexolaruhf]

A bit of context: we are building an SPE with just a few nodes, using the existing solana-validator cli (Solana Labs client). This SPE is deployed inside a Kubernetes cluster (as a statefulset) with the intention to allow overtime validator nodes outside of this K8s cluster to join the same network (e.g. from other Kubernetes clusters, etc).

Since this network is to be used in an enterprise setting the total amount of validator nodes and number of transactions would be much smaller than what is observed on the public solana networks (mainnet, testnet, etc) or even other SPEs (like the Pyth network). At least for many/initial enterprise use cases.

As such the validators hardware requirements would be much smaller, hence our relatively high confidence that we could run validators in a virtualized environment and hopefully gain a lot more operational automation through the tooling offered by Kubernetes.

We'd like to understand if the Firedancer client will be able to run in such environment. Could you please let us know?

[ripatel-fd]

Hi @alexolaruhf, apologies for the late response. With no Kubernetes users in prod so far, the Firedancer team makes no active efforts to support Kubernetes.

Firedancer will perform worse in Kubernetes deployments than bare-metal deployments.
The main problem is that Firedancer performs best with AF_XDP which is largely incomatible with Kubernetes pod networking.
Even when using the host network namespace, Firedancer has known problems working with routing table entries generated by Kubernetes (#2575)

The upcoming Firedancer v0.2 release adds support for sockets-based networking which will fix host network namespace problems (at the cost of worse performance). Pod/service network support is a larger effort that requires the peer-to-peer code to become aware of NAT.

Please let us know how attempts to run Firedancer on Kubernetes go. We are happy to accept contributions fixing any bugs you find along the road.

[alexolaruhf]

Thank you for the reply. I understand why you would not focus on K8s given the project's goals. We hope though to be able to leverage Firedancer (along with the Agave client) in enterprise settings.

Given the performance aspects (for enterprise deployments) I mentioned above I would not be too concerned about a worse performance - as long as the software works. So far Agave mostly works in K8s - we are looking though into some instability issues and we are not yet in production. If you'd have any hunch about quantifying a bit what worse means, that would be sufficient for us for now.

Do you happen to know if there is a published product/release roadmap where we could get an idea when v0.2 would be released? Do you know which version (0.1.x or 0.2) will be first deployed to the mainnet? I am guessing from your comments we should not even try v0.1.x on K8s but rather wait for 0.2, correct?

[ripatel-fd]

If you'd have any hunch about quantifying a bit what worse means, that would be sufficient for us for now.

@alexolaruhf Only AF_XDP deployments in the host network namespace are supported for now. Those have no performance problems but have compatibility issues. For example, if you upgrade kubelet or the CNI, Firedancer might suddenly break. (Unfortunately the AF_XDP APIs are quite limited here, the fix would involve no less than to fully reproduce the Linux kernel's routing behavior)

The sockets driver is still in development. This driver will predictably work in a K8s environment but it is slower. I expect it to ingest approx 500k packets per second per core in a typical GKE cluster (around 6 Gbps per core with max size packets). For comparison, we've seen AF_XDP ingest 20Mpps/core (small packets) or tens of Gbps per core (large packets).

These high rates matter only when getting flooded (DoS attack) or when sustaining high throughput (1 million transactions per second). For rates below 100k TPS the socket driver should be sufficient.

Do you know which version (0.1.x or 0.2) will be first deployed to the mainnet?

We expect Firedancer v0.1 to be ready for mainnet in late September.

I am guessing from your comments we should not even try v0.1.x on K8s

We've heard of two companies running Firedancer v0.1 successfully on Kubernetes, so maybe it's worth a try. Their names are Kiln and Monogon. I think Monogon required some workarounds.

Do you happen to know if there is a published product/release roadmap where we could get an idea when v0.2 would be released?

We haven't started work on the Firedancer v0.2 release cycle yet, so unfortunately I don't have dates.

[alexolaruhf]

@ripatel-fd we appreciate the detailed follow-up. Took me a bit to get an understanding of the AF_XDP deployments you mention.

If I understand correctly deployments that will use the sockets driver will lead to deployments that should not experience the possibility of sudden breaks in K8s but they will come with performance penalties.

The kind of numbers you mention are very impressive though and even with the performance penalty, to my knowledge most financial systems work today within the degraded limits you mention - even the TPS for executed trades on most US (stocks, options, etc.) exchange or clearing venues would be under that. The place that I think could benefit the most from the higher limits is the processing of quotes/ticks which is much higher than executed trades. In any case, for use cases that would require the higher TPS, bare metal vs K8s would be the natural choice.

We know the folks at Monogon and will contact them to see if they could share some experiences/findings. And the v0.1 mainnet availability timeline should match well with our ability to start deploying the Firedancer client in our K8s deployments for our own testing.

Thank you much again for your help.

[ripatel-fd]

Hello @alexolaruhf, I have a quick update. Firedancer now supports a sockets mode that will work fine in Kubernetes.
It can be configured by adjusting the config as follows.

[development.net]
provider = "socket"