Skip to content

Latest commit

 

History

History
206 lines (129 loc) · 4.31 KB

PLAN.md

File metadata and controls

206 lines (129 loc) · 4.31 KB

📘 CloudSync Development & Testing Plan

A detailed blueprint for building CloudSync in independent stages, ensuring that every component is testable even if others are incomplete.

🔷 1. Core Metadata Module (Master)

Build

  • In-memory metadata store
  • File + chunk metadata structures
  • Append-only operation log
  • Checkpoint writer

Test (No other services needed)

  • Pure Go unit tests using go test
  • Create file, chunk, verify metadata integrity
  • Test log replay and checkpoint reload

Dependencies

  • None

🔷 2. Agent Registration API (Master)

Build

  • /RegisterAgent gRPC
  • JWT issuing & validation
  • PostgreSQL writes (agent identity)

Test (Without real agents)

  • gRPC test client in Go
  • Use sqlmock or a local Docker PostgreSQL for DB tests

Dependencies

  • Metadata module
  • PostgreSQL

🔷 3. Heartbeat Handling (Master)

Build

  • Store agent health
  • Track last-seen timestamp
  • Offline detection logic

Test (Without agent CLI)

  • Simulate heartbeat RPC requests
  • Validate online/offline state transitions

Dependencies

  • Agent registration
  • Metadata module

🔷 4. Agent CLI + Heartbeat Sender

Build

  • register command
  • start command
  • Heartbeat loop

Test (Without real master)

  • Create a FakeMaster gRPC server returning static responses
  • Run CLI against FakeMaster

Dependencies

  • gRPC contracts

🔷 5. File Chunking Library

Build

  • Split file into 64MB chunks
  • Compute SHA-256 checksums

Test (Standalone)

  • Pure Go tests
  • Validate chunk size, checksum correctness, boundary cases

Dependencies

  • None

🔷 6. Upload Workflow (Master + Client)

Build

  • InitiateUpload
  • UploadChunk
  • FinalizeUpload
  • Chunk placement logic

Test (Without real agents storing data)

  • Create FakeAgent that pretends to store chunks
  • Master interacts with FakeAgent
  • Full upload pipeline works end-to-end

Dependencies

  • Chunking library
  • Metadata module
  • Heartbeats

🔷 7. Replication & Fault Tolerance

Build

  • Detect under-replicated chunks
  • Re-replication scheduler
  • Checksum verification & corruption detection

Test (Before real replication)

  • FakeAgent that says chunk is missing
  • Test replication queue logic
  • Test offline agent triggering re-replication

Dependencies

  • Heartbeats
  • Upload workflow

🔷 8. End-to-End Simulation (Docker)

Build

  • Docker Compose for: master + PostgreSQL + N agents
  • Upload → distribute chunks → kill agent → re-replicate

Test

  • Full system simulation
  • Load test with 10+, 50+, 100 agents

Dependencies

  • Entire system built

📂 Project Build Order Summary

  1. ✅ Metadata Store
  2. ✅ gRPC API Basics
  3. ✅ Agent Registration + Auth
  4. ✅ Heartbeats
  5. 🔜 Chunking
  6. 🔜 Upload Workflow
  7. 🔜 Replication Logic
  8. 🔜 Docker Simulation

🧪 Testing Matrix

Component Test Type Requires Real Agents? Requires DB?
Metadata Store Unit ❌ ❌
Registration gRPC Test Client ❌ ✔️
Heartbeats Simulated RPC ❌ ❌
Agent CLI Fake Master ❌ ❌
Chunking Unit ❌ ❌
Upload Workflow Fake Agents ❌ ❌
Replication Fake Agents ❌ ❌
End-to-end Docker ✔️ ✔️

🎯 Goal

This plan ensures:

  • Every part can be tested independently
  • No waiting for entire system to be built
  • Each stage gives confidence before integration
  • Development becomes predictable, modular, and scalable

Let me know if you want a FOLDER_STRUCTURE.md or full skeleton code for each stage!