Jepsen for JetStream

[jessekv]

Earlier this year Redpanda went through the Jepsen audit process: http://jepsen.io/analyses/redpanda-21.10.1

Wondering if there is a plan to do something similar for JetStream?

[derekcollison]

Yes there is.

[jessekv]

That's great, I'm really looking forward to reading about it! Is there a rough time frame yet for when it will happen?

[derekcollison]

Possibly 2.9, or 3.0 release. 2.9 is scheduled for next week, fingers crossed.

The testing process is quite pricey as well, so need to budget.

[hwinkel]

Is there any progress in this regards?

[derekcollison]

Yes we have been working with a stealth startup for almost a year now that is incredible. Will let @mprimi divulge more.

[mprimi]

I'm not going to comment on the official Jepsen audit, but I can tell you that for some time we have been hammering NATS and JS with a Jepsen-like approach (e.g. invariant testing with fault injection). Jepsen(-like) testing is great to uncover bugs in code that deals with fault-tolerance and recovery, that are hard to test otherwise.
We've also been relying on Antithesis (https://antithesis.com/) find even more insidious issues. Antithesis is like Jepsen on steroids and can help to seek and destroy one-in-a-million heisenbugs that are so hard to track down in distributed systems.

[atropos112]

With it being couple months, have there been any news about audits?

[mprimi]

We keep continue expanding on testing.

What I can tell you is we're putting NATS/JetStreams through scenarios and trials much more evil, complex and comprehensive than what Jepsen (the software) would do.
Happy to share privately, but we're not ready to be completely open about this just yet.

On the other side, you probably want to see a third-party report and stamp of approval (i.e. Jepsen the blog).
I don't have comments or an ETA on that.

[bbigras]

It would be nice to have something soon, even if as basic as Jepsen.

I'm not sure how to interpret this comment from the Jepsen dev but it kinda sounds like jetstream has a bad reputation or something.

[hpvd]

@bbigras this link gives also the right context/question (its the parent of yours) https://news.ycombinator.com/item?id=42117108

[roded]

@bbigras @hpvd I'm still not sure what conclusion to draw from that discussion. I read it as @bbigras did.

[derekcollison]

We have invested heavily into Antithesis for this type of testing, since it is deterministic and Jepsen is not. We have been working with them as one of their first customers for over 2 years now and have made great progress.

If a Jepsen dev wants to run NATS / JetStream through Jepsen that would be great, by for our investment we will stick with Antithesis.

[roded]

Hi @derekcollison , any plans on making an Antithesis report public or the like?
Thanks

[derekcollison]

Antithesis is a tool, it does not run tests and generate reports on its own. We do have plans to shed more light on our own extensive testing with Antithesis.

[roded]

Thanks @derekcollison . Looking forward for any such publication.