Storage Landscape Whitepaper V3

Signed-off-by: xing-yang <xingyang105@gmail.com>

Author: xing-yang

tags/tag-infrastructure/initiatives/storage-landscape/v3/overview.md

@@ -0,0 +1,50 @@
+# CNCF Storage Landscape Whitepaper V3
+
+**Authors (in alphabetical order):**
+
+Alex Chircop, Clinton Kitson, Jakub Scholz, Luis Pabón, Nick Connolly, Patrick McFadin, Quinton Hoole, Raffaele Spazzoli, Sugu Sougoumarne, Tom Prohofsky, Xiang Li, Xing Yang
+
+## 1 Scope of this document
+
+This is version 3 of the CNCF Storage Landscape Whitepaper. It updates and supersedes the [V2 whitepaper](https://bit.ly/cncf-storage-whitepaperV2), continuing to offer clear information on terminology, usage patterns and classes of technology as defined by the goals of the document, and incorporating feedback and understood production use gathered since the earlier versions.
+
+Non-goals continue to be tackled incrementally across versions, on the basis of community feedback, specifically in light of understood production use, and comparisons w.r.t. primary properties.
+
+### 1.1 Goals
+
+1. **Clarify the terminology** currently in use in the storage space, and the relationships between the various terms. Essentially a taxonomy of the storage landscape.
+2. This includes anything reasonably within scope of "storage", including block stores, key value stores, databases, object stores, volumes, file systems etc.
+3. Provide some general information as to **how these things are currently being used in production** in public or private cloud environments.
+4. **Compare and contrast** the various technology areas w.r.t. the primary properties of availability, scalability, consistency, durability, performance, API, etc.
+
+### 1.2 Non-goals
+
+1. Define what's in-scope and out of scope for the CNCF.
+2. Provide any recommendations regarding preferred storage approaches or solutions.
+
+## 2 Introduction and document layout
+
+Multiple options were considered when defining how to present the many storage systems and services in the landscape for the document.
+
+In order to simplify the consumption of information in a complex landscape, the document has been structured as follows:
+
+- Definition of the **attributes** of a storage system such that an end-user can understand the appropriate capabilities that might be required by an application or architectural pattern
+- Definition of the **layers** in a storage solution (or service) with a focus on **terminology** and how they impact the defined **attributes** - covering the container, orchestrator, transport, topology, virtual/physical, data protection, data services, the non-volatile layers, and **disaster recovery**.
+- Definition of the **data access interfaces** in terms of **volume** (including block, file system and shared file system) and **application API** (including object, KV, database, and **streaming/messaging**) as high level groupings
+- Separate sections with further detail on **Block** Storage, **File systems**, **Object Storage**, **Key Value** Stores, **Databases**, and **Streaming and Messaging**.
+- Definition of the **management interfaces** needed to orchestrate the storage layers to facilitate composability, dynamic provisioning and self service management.
+
+## Sections
+
+- [Attributes of a Storage Interface or System](topics/storage-attributes.md)
+- [Storage Stack / Layers](topics/storage-stack-layers.md)
+- [Disaster Recovery](topics/disaster-recovery.md)
+- [Data Access Interface](topics/data-access-interfaces.md)
+- [Block Stores](topics/block-stores.md)
+- [File Systems](topics/file-systems.md)
+- [Object Stores](topics/object-stores.md)
+- [Key-Value Stores](topics/key-value-stores.md)
+- [Databases](topics/databases.md)
+- [Streaming and Messaging](topics/streaming-and-messaging.md)
+- [Orchestration and Management Interfaces](topics/orchestration-and-management.md)
+- [Appendix](topics/appendix.md)

tags/tag-infrastructure/initiatives/storage-landscape/v3/topics/appendix.md

@@ -0,0 +1,68 @@
+# 13 Appendix
+
+## 13.1 Document History
+
+Initially, the document was structured based on classes of storage type which are categorised by the way the storage is consumed e.g. block, file or object. This did not provide a useful way to compare and contrast their attributes and how they are utilised in production as most storage systems have many layers and are formed of multiple components. While the data access interface (like block or file) might affect how the data is consumed and how it might failover between nodes, it does not effectively define attributes such as data protection, consistency, or durability.
+
+As a further complication, many commonly used systems are layered storage systems where, for example, a filesystem may be built on an object store (e.g. CephFS), or a block store may be built on a distributed filesystem (e.g. gluster block storage). This meant that the way the storage is accessed did not usefully define the attributes that an application cared about (such as the durability, data protection or some of the performance characteristics of the overall system), as those attributes are defined at other layers in the stack.
+
+### Version 3 (August 2026)
+
+Version 3 builds on the [V2 whitepaper](https://bit.ly/cncf-storage-whitepaperV2) with the following changes:
+
+- Added a new [Streaming and Messaging](streaming-and-messaging.md) section, covering messaging/streaming APIs and protocols, availability and durability, performance and scalability, and storage.
+- Added a paragraph on performance with a reference to the CNCF Performance and Benchmarking whitepaper.
+- Added a new [Disaster Recovery](disaster-recovery.md) section, covering the main disaster recovery design archetypes and their RTO/RPO trade-offs.
+- Updated the [Orchestration and Management Interfaces](orchestration-and-management.md) section, including updates to the Container Storage Interface (CSI) and Container Object Storage Interface (COSI) sections.
+- Refreshed the [Attributes of a Storage Interface or System](storage-attributes.md) section, adding Observability, Elasticity, and Instantiation & Deployment.
+- Updated the [Cloud Native Databases](databases.md) section.
+
+## 13.2 Consensus Protocols
+
+[Consensus protocols](https://en.wikipedia.org/wiki/Consensus_\(computer_science\)) provide reliable agreement among a group of potentially faulty distributed processes on a single data value or a replicated log. They are commonly used to decide whether to commit a data change transaction, for leader election, state machine replication, load balancing, clock synchronization and others in distributed systems. The two most popular (families of) consensus algorithms are Multi-Paxos and Raft, both of which have been formally proven correct (for practical uses, with some caveats). Both rely on a single elected leader, and (typically) agreement by a strict majority of participants (e.g. for 5 participants, at least 3 must explicitly agree). Raft is considered simpler to understand and implement than Multi-Paxos. Other ad-hoc attempts at consensus algorithms are notoriously prone to edge case failures.
+
+### 13.2.1 Paxos
+
+[Paxos](https://en.wikipedia.org/wiki/Paxos_\(computer_science\)) is arguably the oldest [formally studied](https://lamport.azurewebsites.net/pubs/lamport-paxos.pdf) family of consensus algorithms. It is considered highly robust when implemented properly, but [challenging to implement correctly for practical uses](https://static.googleusercontent.com/media/research.google.com/en//archive/paxos_made_live.pdf).
+
+### 13.2.2 Raft
+
+[Raft](https://en.wikipedia.org/wiki/Raft_\(computer_science\)) was developed about a decade after Paxos, to [address the issues mentioned above](https://web.stanford.edu/~ouster/cgi-bin/papers/raft-atc14). It has become widely used, and forms the basis of, amongst others, the popular [etcd](https://github.com/etcd-io/etcd) cloud-native key-value store, and [Consul](https://www.consul.io/docs/internals/consensus.html) distributed service mesh.
+
+### 13.2.3 Two-phase Commit ("2PC")
+
+[2PC](https://en.wikipedia.org/wiki/Two-phase_commit_protocol) is a specialized form of consensus protocol used for coordination between participants in a distributed atomic transaction to decide on whether to commit or abort (roll back) the transaction. 2PC is not resilient to all possible failures, and in some cases, outside (e.g. human) intervention is needed to remedy failures. Also, it is a blocking protocol. All participants block between sending in their vote (see below), and receiving the outcome of the transaction from the co-ordinator. If the co-ordinator fails permanently, participants may block indefinitely, without outside intervention. In normal, non-failure cases, the protocol consists of two phases, whence it derives its name:
+
+1. The commit-request phase (or voting phase), in which a coordinator requests all participants to take the necessary steps for either committing or aborting the transaction and to vote, either "Yes" (on success), or "No" (on failure)
+2. The commit phase, in which case the coordinator decides whether to commit (if all participants have voted "Yes") or abort, and notifies all participants accordingly.
+
+### 13.2.4 Three-phase Commit ("3PC")
+
+[3PC](https://en.wikipedia.org/wiki/Three-phase_commit_protocol) adds an additional phase to the 2PC protocol to address the indefinite blocking issue mentioned above. But 3PC still cannot recover from network segmentation, and due to the additional phase, requires more network round-trips, resulting in higher transaction latency.
+
+## 13.3 Consistency, Coherence and Isolation
+
+The above three terms are commonly used in various different contexts to mean different things in the fields of data stores and distributed systems. Without going into detail here, suffice to say that, consistency in particular, is a widely misunderstood term, so it's worth thinking twice before assuming that you understand exactly what's meant by a particular use of the term. For example, [ACID](https://en.wikipedia.org/wiki/ACID_\(computer_science\)#Characteristics) (Atomicity, Consistency, Isolation, Durability) properties and the [CAP Theorem](https://en.wikipedia.org/wiki/CAP_theorem) (concerning Consistency, Availability and Partition-tolerance) are both widely used terms, and many people assume that they understand what these terms mean. But considerably fewer people realise that "Consistency" means quite different things in those two contexts. For further details, [Wikipedia](https://en.wikipedia.org/wiki/Consistency_\(database_systems\)#As_an_ACID_guarantee) and [Irene Zhang's musing](https://irenezhang.net/research/consistency.html) provide good starting points.
+
+### 13.3.1 ACID
+
+With the above caveats, for data storage systems, Atomicity, Consistency, Isolation and Durability are generally considered to mean:
+
+1. Atomicity: a guarantee that each transaction across multiple data items is treated as a single "unit", which either succeeds completely, or fails completely, even in the case of various failures including machine crashes and network errors.
+2. Consistency: Usually understood to mean guarantees about whether a transaction started in the future can necessarily see the effects of all transactions committed in the past. Also sometimes understood to be a guarantee that a transaction can only bring the data from one valid state to another, while maintaining invariants (for example that stock count cannot be less than zero, or that two customers with the same id number cannot exist).
+3. Isolation: guarantees that concurrent execution of transactions leaves the database in the same state that would have been obtained if the transactions were executed sequentially, in some order.
+4. Durability: guarantees that once a transaction has been committed, it will remain committed even in the case of a system failure (e.g., power outage or crash). This usually means that completed transactions (or their effects) are recorded in non-volatile memory.
+
+### 13.3.2 The CAP Theorem
+
+[The CAP Theorem](https://en.wikipedia.org/wiki/CAP_theorem) states that it is impossible for a distributed data store to simultaneously provide more than two out of the following three guarantees:
+
+1. Consistency: Every read receives the most recent write or an error
+2. Availability: Every request receives a response that is not an error
+3. Partition tolerance: The system continues to operate despite an arbitrary number of messages being dropped (or delayed) by the network between nodes
+
+In the absence of network failure both availability and consistency can be satisfied. CAP is frequently misunderstood to mean that one has to choose to abandon one of the three guarantees at all times. In fact, the choice is really between consistency and availability only when a network partition or failure happens; at all other times, no trade-off has to be made.
+
+Database systems designed with traditional ACID guarantees in mind such as RDBMS choose consistency over availability, whereas systems designed around the BASE philosophy, common in the NoSQL movement for example, choose availability over consistency.
+
+The PACELC theorem builds on CAP by stating that even in the absence of partitioning, another trade-off between latency and consistency occurs.

tags/tag-infrastructure/initiatives/storage-landscape/v3/topics/block-stores.md

@@ -0,0 +1,29 @@
+# 6 Block Stores
+
+Block stores are a persistence target where data is stored in blocks in local, remote, or distributed locations. The blocks are typically numerically addressed using a method called [Logical Block Addressing](https://en.wikipedia.org/wiki/Logical_block_addressing) (LBA) and accessed as a client through a [device](https://en.wikipedia.org/wiki/Device_file#BLOCKDEV) interface provided by a Kernel. The location (local/remote/distributed) is determined by the physical persistence location of the blocks and serves as a method to group and categorize different stores.
+
+It is possible to transparently augment or enhance numerous characteristics of block stores such as availability, scalability, consistency, durability, and performance by adding additional software-based storage layers (ie. RAID) along with physical devices, networking, and nodes. Please refer to the Capacity, Availability, and Partition-tolerance (CAP) theorem overview in the [appendix](appendix.md) for more details.
+
+Virtualization adds another perspective which is important to consider. Operating systems may or may not be aware of the type of block store being used. Virtual machines and machine instances are likely not storing any blocks locally but completely leveraging remote or distributed block stores. In this case, instances provide virtualized hardware that store data remotely and emulate the connectivity and behavior of local physical storage devices. This storage would not be considered a local block store due to non-locality of stored data.
+
+Most applications do not directly store data in block format, but instead interface with file systems supported by block devices (ie. application -> local EXT4 filesystem -> local block device -> local/remote/distributed block store). See the [File Systems](file-systems.md) section for more details.
+
+*The following categories include examples solely with the intent of providing context to the category being described. Examples are intended to be widely known to the readers.*
+
+## 6.1 Local Block Stores
+
+Local block stores are built on [Direct Attached Storage](https://wikivisually.com/wiki/Direct_attached_storage) (DAS) where data is persisted locally on hardware devices. Since all data is stored locally, the scale is limited to the local resource capabilities. The availability of the data is a major consideration when applications are interacting directly with local block stores. [Logical volume management](https://en.wikipedia.org/wiki/Logical_volume_management) (LVM) and similar techniques can be used to augment and concatenate the capabilities that discrete hardware devices provide. These stores tend to be focused on specific use cases where latency is critical or to support other storage services.
+
+Generally accepted example terms, platforms, and protocols: ATA, IDE, logical volumes, LVM, physical volumes, physical storage devices, RAID, SCSI, volume groups
+
+## 6.2 Remote Block Stores
+
+Remote block stores provide storage attached by a network where data is persisted remotely across a network. This is different from local because there is a separation of application from storage. Generally, this has the ability to increase capacity and performance. The availability is also increased since high availability design patterns can be implemented. Without detailed information and assurances and intentional design, service levels are likely to be driven by this category.
+
+Generally accepted example terms, platforms, and protocols: NVMe over Fabrics, AWS EBS, FC, FCoE, iSCSI, SAN
+
+## 6.3 Distributed Block Stores
+
+Distributed block stores are similar to remote block stores but data is persisted across many nodes, possibly in conjunction with the local node, and clients are able to rely on many nodes to provide redundancy and horizontal scalability. When compared with local and remote block stores, distributed block stores require additional control and data access layers to manage data distribution (and often also replication). This added complexity can provide improved scalability, availability, and durability.
+
+Generally accepted example terms, platforms, and protocols: Ceph, DRBD, OpenEBS, Longhorn, hyper-converged