docs: organize and expand fundamentals section

Author: d-sys-admin

wiki/fundamentals/index.md

@@ -1,27 +1,159 @@
+# distributed systems fundamentals
+
 ## what are distributed systems?
 
-**distributed systems** are networks of independent computers (nodes) communicating through message passing, collaboratively delivering unified services or achieving shared objectives. each node maintains its own memory, processing capability, and local storage, working concurrently and autonomously. nodes interact through standardized protocols and interfaces, allowing the system to function effectively across geographic distances, network delays, and varied hardware or software platforms. the primary advantages of distributed systems include scalability, fault tolerance, resilience, and efficient resource utilization, ensuring consistent performance even if individual nodes fail.
+**distributed systems** are networks of independent computers (nodes) communicating through message passing, collaboratively delivering unified services or achieving shared objectives. each node maintains its own memory, processing capability, and local storage, working concurrently and autonomously. nodes interact through standardized protocols and interfaces, allowing the system to function effectively across geographic distances, network delays, and varied hardware or software platforms.
+
+the primary advantages of distributed systems include:
+
+- **scalability**: ability to handle growing workloads by adding more resources
+- **fault tolerance**: continuing operation despite component failures
+- **resilience**: recovering from failures automatically
+- **efficient resource utilization**: optimizing use of computing resources across the network
+
+these systems range from small local networks to globe-spanning cloud infrastructures, powering everything from web applications to financial systems and massive data processing pipelines.
+
+## the cap theorem
+
+distributed systems embody three essential properties described by the **cap theorem**, formulated by eric brewer in 1998:
+
+- **consistency**: ensuring all nodes have a synchronized and accurate view of data. when a write operation completes, all subsequent read operations should reflect that write.
+- **availability**: providing reliable access and responses to user requests. every request to a non-failing node must receive a response, without guaranteeing it contains the most recent write.
+- **partition tolerance**: maintaining operation despite network partitions or node failures. the system continues functioning even when network communication between some nodes is unreliable.
+
+the cap theorem dictates that no system can simultaneously achieve all three properties at full strength. system architects must strategically balance these properties based on specific application requirements:
+
+- **cp systems** (consistency + partition tolerance): prioritize data consistency at the potential cost of availability during partitions. examples include traditional banking systems and distributed databases like google spanner.
+- **ap systems** (availability + partition tolerance): favor availability over strict consistency. examples include nosql databases like amazon dynamodb and cassandra.
+- **ca systems** (consistency + availability): optimize for both properties but cannot handle network partitions effectively. these systems are theoretical in distributed environments, as partition tolerance is generally required.
+
+## distributed system architectures
+
+distributed systems employ diverse architectural patterns to address varying use cases:
+
+### client-server architecture
+
+centralizes resource management with dedicated servers responding to client requests. this architecture is ideal for predictable workloads and clear separation of concerns.
+
+**characteristics**:
+
+- clear separation between service providers (servers) and consumers (clients)
+- centralized resource management
+- relatively simple to implement and understand
+
+**examples**: traditional web applications, email services, file servers
+
+### peer-to-peer (p2p) architecture
+
+distributes responsibilities evenly among equivalent nodes, improving resilience and scalability. each node can act as both client and server.
+
+**characteristics**:
+
+- no centralized control
+- high resilience to node failures
+- excellent scalability
+- complex coordination requirements
+
+**examples**: bittorrent, blockchain networks, distributed file systems
+
+### microservices architecture
+
+decomposes applications into loosely coupled, independent services, streamlining development and deployment. each service handles a specific function and can be developed, deployed, and scaled independently.
+
+**characteristics**:
 
-fundamentally, distributed systems embody three essential properties described by the **cap theorem**:
+- service independence
+- technology diversity
+- focused development teams
+- complex orchestration
 
-- **consistency**: ensuring all nodes have a synchronized and accurate view of data.
-- **availability**: providing reliable access and responses to user requests.
-- **partition tolerance**: maintaining operation despite network partitions or node failures.
+**examples**: netflix, amazon, uber applications
 
-since the cap theorem dictates that no system can simultaneously achieve all three, architects must strategically balance these properties based on specific application requirements.
+### event-driven architecture
 
-distributed systems employ diverse architectural patterns to address varying use cases. for instance, a **client-server architecture** centralizes resource management with dedicated servers responding to client requests, ideal for predictable workloads. in contrast, **peer-to-peer architectures** distribute responsibilities evenly among equivalent nodes, improving resilience and scalability. **microservices** structures decompose applications into loosely coupled, independent services, streamlining development and deployment. **event-driven architectures** utilize asynchronous communication via events or messages, enhancing flexibility and responsiveness. finally, **service-oriented architectures (soa)** encapsulate functionalities into reusable, interoperable services with standardized interfaces.
+utilizes asynchronous communication via events or messages, enhancing flexibility and responsiveness. components react to events rather than direct calls.
+
+**characteristics**:
+
+- loose coupling between components
+- asynchronous processing
+- enhanced scalability
+- complex debugging and testing
+
+**examples**: iot systems, real-time analytics platforms, financial trading systems
+
+### service-oriented architecture (soa)
+
+encapsulates functionalities into reusable, interoperable services with standardized interfaces. this approach emphasizes service reusability and composition.
+
+**characteristics**:
+
+- business-aligned services
+- standardized interfaces
+- service reusability
+- enterprise service bus (often)
+
+**examples**: enterprise integration systems, banking platforms
+
+## core components of distributed systems
 
 typical distributed system components include:
 
-| component                  | role                                                  | example technologies      |
-|----------------------------|-------------------------------------------------------|---------------------------|
-| **load balancer**          | evenly distribute client requests                     | aws elb, nginx            |
-| **message queue**          | asynchronous message handling between services        | apache kafka, aws sqs     |
-| **database (relational)**  | structured, consistent data storage                   | postgres, mysql           |
-| **database (nosql)**       | flexible, scalable data storage                       | mongodb, dynamodb         |
-| **cache**                  | store frequently accessed data, reducing latency      | redis, memcached          |
-| **orchestration platform** | automate service deployment, scaling, and management  | kubernetes, aws ecs       |
-| **consensus algorithm**    | achieve agreement on shared state across nodes        | paxos, raft               |
-
-designing a robust distributed system demands consideration of latency, throughput, network efficiency, and reliability through meticulous fault detection and recovery strategies. architects must thoughtfully select appropriate consistency models (strong vs. eventual consistency), devise effective data replication and sharding schemes, and implement rigorous security measures—including authentication, authorization, and encryption—to guard against unauthorized access and ensure data integrity.
+| component                  | role                                                                                                                                 | example technologies                               |
+| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------- |
+| **load balancer**          | evenly distribute client requests across servers to optimize resource utilization, maximize throughput, and ensure high availability | aws elb, nginx, haproxy, f5                        |
+| **message queue**          | enable asynchronous communication between services, providing buffering, decoupling, and reliable message delivery                   | apache kafka, rabbitmq, aws sqs, azure service bus |
+| **database (relational)**  | store structured data with acid properties, supporting complex queries and transactions                                              | postgresql, mysql, oracle, sql server              |
+| **database (nosql)**       | provide flexible, schema-less data storage optimized for specific data models and high scalability                                   | mongodb, cassandra, dynamodb, couchbase            |
+| **cache**                  | store frequently accessed data in memory to reduce latency and database load                                                         | redis, memcached, hazelcast                        |
+| **orchestration platform** | automate deployment, scaling, and management of containerized services                                                               | kubernetes, docker swarm, aws ecs, nomad           |
+| **service discovery**      | enable services to find and communicate with each other dynamically                                                                  | consul, etcd, zookeeper                            |
+| **api gateway**            | provide a unified entry point for clients, handling cross-cutting concerns like authentication and rate limiting                     | kong, amazon api gateway, apigee                   |
+| **consensus algorithm**    | achieve agreement on shared state across distributed nodes                                                                           | paxos, raft, zab                                   |
+| **distributed tracing**    | track and visualize request flows across multiple services for debugging and monitoring                                              | jaeger, zipkin, aws x-ray                          |
+
+## key design considerations
+
+designing robust distributed systems requires addressing several critical concerns:
+
+### performance optimization
+
+- **latency**: minimize response time through caching, cdns, and optimized data access patterns
+- **throughput**: maximize system capacity through horizontal scaling and efficient resource utilization
+- **network efficiency**: reduce bandwidth consumption with compression, batching, and protocol optimization
+
+### consistency models
+
+- **strong consistency**: all nodes see the same data at the same time (e.g., linearizability)
+- **eventual consistency**: system will become consistent given enough time without updates
+- **causal consistency**: operations that are causally related appear in the same order to all nodes
+- **session consistency**: client operations in a session are consistent with their own operations
+
+### data management
+
+- **replication strategies**: synchronous vs. asynchronous, active-active vs. active-passive
+- **sharding approaches**: range-based, hash-based, directory-based partitioning
+- **data synchronization**: conflict detection and resolution mechanisms
+
+### fault tolerance and recovery
+
+- **failure detection**: heartbeats, gossip protocols, and health checks
+- **redundancy**: multiple instances, geographic distribution, and standby systems
+- **graceful degradation**: maintaining core functionality during partial system failures
+- **self-healing mechanisms**: automated recovery and repair procedures
+
+### security considerations
+
+- **authentication and authorization**: verifying identity and controlling access rights
+- **encryption**: protecting data at rest and in transit
+- **network segmentation**: limiting attack surfaces through isolation
+- **audit logging**: recording security-relevant events for compliance and forensics
+
+### monitoring and observability
+
+- **metrics collection**: gathering performance and health indicators
+- **distributed tracing**: following requests across service boundaries
+- **log aggregation**: centralizing and analyzing system logs
+- **alerting**: detecting and notifying about critical conditions
+
+creating effective distributed systems requires balancing these considerations against business requirements, available resources, and organizational constraints. the optimal design varies significantly based on specific use cases, scale requirements, and reliability needs.