Documentation for iggy (SDK) and iggy-bench

[haubur]

Hi all,

TL;DR Should we have elaborate crate docs including details, examples etc. or keep it simple and put the rest on the website?

I was lately working with the iggy and iggy-bench libraries, while being fairly new to the project and message-streaming in general.
So while I learn a lot of exciting lesson I thought I could as well document those for my reference in a later point in time and others joining the party in the future.

A lot of crates have really nice rust docs (rust std being a good example).
So I would like to work on rustdocs for iggy and iggy-bench.
A couple of questions quickly arose when I started, so I think a discussion might be a decent starting point, before I go off with a documenting style that might be not of interest for others.

Look at my first couple of lines for the starting page of iggy (core/sdk/lib.rs)
This obviously is WIP but illustrates what I am thinking about.

//! Apache Iggy is a high-performance, persistent message streaming platform written in Rust, capable of processing millions of messages per second with ultra-low latency.
//! It is part of the [`Incubating Program`] of the [`Apache Software Foundation`] (ASF).
//!
//! If you are new to message streaming a very good introduction is [`Designing Data-Intensive Applications`] by Martin Kleppmann, especially Chapter 11.
//! A Iggy specific introduction along with an architecture overview can be found on the [`website`].
//! After reading both, you should be familiar with the key ideas and concepts behind message-streaming. Following up with this documention you have everything to build your first message-streaming application.
//!
/*
//! If you are a more advanced user and want to know how Apach Iggy differentiates itself from other exisiting solutions have a look at this [`overview`]. todo(haubur)
*/
//!
//! **This library is the Apache Iggy SDK.** It exposes a low-level and a high-level API for the Apache Iggy message streaming infrastructure for the Rust programming language.
//! SDKs for other programming languages can be found in [`core/foreign`] of the root repository on GitHub.
//!
//! # Iggy Server
//! The core of Iggy is a persisted append-only log data structure.
//! The Iggy server is build around this data structure and consists of logic to allow consumers and producers to read and write data to and from that log.[^note]
//! Interacting with the server is facilitated via commands that are exposed by the server. These commands allow administrative tasks, such as handling users, permissions and setting up streams and topics
//! or writing and reading messages from the log.
//! A comprehensive overview of commands can be found in the [`schema spec`] on the website or checking the [`server command enum`] within the source code.
//!
//! # Low-level API
//! Interfaces with the server commands via the Command trait using binary. 
//! Different transport clients implement the BinaryTransport trait.
//! todo(haubur): Elaborate.
//!
//! # High-level API
//! Builds on top of the low-level API and uses builder pattern to implement common features (e.g. retry-logic, compression etc.)
//! todo(haubur): Elaborate.
//!
//! [^note]: The details of how that is done internally are not discussed here. Refer, e.g. to [`the blog`].
//!
//! [`Apache Incubating Program`]: https://incubator.apache.org/
//! [`Apache Software Foundation`]: https://www.apache.org/
//! [`blog`]: https://iggy.apache.org/blogs/2025/11/17/websocket-io-uring/
//! [`core/foreign`]: https://github.com/apache/iggy/tree/master/foreign
//! [`Designing Data-Intensive Applications`]: https://www.oreilly.com/library/view/designing-data-intensive-applications/9781491903063/
/*
//! [`overview`]: https://google.com/ todo(haubur)
*/
//! [`schema spec`]: https://iggy.apache.org/docs/server/schema/
//! [`server command enum`]: https://github.com/apache/iggy/blob/3e27ebc8dd5dbf257b816993908dc0747c4f8849/core/server/src/binary/command.rs#L74
//! [`website`]: https://iggy.apache.org/docs/introduction/architecture/

So here are of couple of points for discussion that might inform a general style of documentation for the project.

• I personally like the idea of briefly explaining concepts or contexts where it is helpful to bridge understanding of why something is implemented. Do you think that would be to verbose for rustdocs?
• Should we decouple the documentation of code from the overall picture? I would prefer to avoid that, because it also might lead to maintaining docs in two places with high risk both ending up not being in sync.
• Therefore the follow-up question: A lot of documentation currently lives on the website, triggering a lot of references to details. Should either (code documentation or website) take precedence?
• Examples are another example that can be found in crate docs and currently live on the website. Would it be fine to add them to the crate (they can be also compiled and therefore checked if they are still correct after changes.
• I can also see adding some details on the server implementation where it could help. Should we do that?
• While iggy is under fast-pace development, should we even spent much time on documentation after all just now?

I hope you see where I am going. These are questions that immediately came up for me.
I suspect you have your own opinions, thoughts and wishes.

Let's discuss.
I would love to see the best docs experience we can provide to attract and keep as many users as we can.

All the best

[hubcio]

Hi @haubur, great initiative!
We avoided extensive docs earlier because the API not-so-stable and maintaining it was a burden that slowed us down. Now that API is still unstable but LLMs are getting better and better, I think they are good enough to keep docs aligned with code (and have acceptable quality!).

My general thoughts:

• Crate docs should be the source of truth for API usage, examples, and code-level concepts. doctested examples are great (I did that in IggyMessage)
• Website for high-level architecture, comparisons, anything related that doesn't belong in rustdoc.
• Keep explanations concise in rustdoc - link to website for deep dives.
• Your draft looksgood. The low-level/high-level API distinction is exactly what users need.

Looking forward to seeing this evolve. See what needs to be documented in prelude.rs to have a good overview of whole Iggy Rust API.

[spetz]

Hey,
Thank you for bringing this up, I think that especially with SDK, things are getting more and more stable, and while some parts of it might still change (e.g. check #2554), the overall status quo could be documented already, and it looks like most of its parts are under prelude.

Speaking of the website and our custom docs, we host them via https://github.com/apache/iggy-website/ so feel free to contribute here as well :)

Draft looks really good, would be great to see full docs like this!