docs(core): align documentation with encapsulated monorepo structure (#257)

Author: victoriacheng15

AGENTS.md

@@ -8,25 +8,20 @@ Agents must distinguish between the two primary orchestration tiers to avoid "ci
 
 ### 🌌 Hybrid Orchestration Layers
 
-- **Host Tier (Systemd)**: Reserved for hardware-level telemetry, security gates, and GitOps reconciliation. Reliability here is critical for cluster recovery. Core logic is extracted into `pkg/` libraries to ensure reusability and consistency across different execution triggers (CLI, API, and future AI tools).
+- **Host Tier (Systemd)**: Reserved for hardware-level telemetry, security gates, and GitOps reconciliation. Reliability here is critical for cluster recovery. Core logic is strictly encapsulated in `internal/` to ensure reusability and enforce project boundaries.
 - **Cluster Tier (K3s)**: Handles scalable data services (Postgres, Loki, Prometheus, Grafana, Tempo, MinIO). Orchestrated via **OpenTofu (IaC)** in `tofu/`.
 
-### 📦 Distribution Pattern
+### 🏗️ Directory Map (Consolidated Monorepo)
 
-To maintain a clean repository and ensure operational stability, all compiled binaries must be output to the root `dist/` directory. Systemd unit files and automated scripts should reference artifacts from this location.
-
-### 🏗️ Directory Map
-
-- **`pkg/`**: Shared Go modules (DB, brain, env, logger, metrics, secrets, telemetry). Maintain stable interfaces.
-- **`services/`**: Standalone binaries and operational entry points.
-  - **`services/proxy/`**: The \"Central Nervous System.\" API gateway and GitOps webhook listener.
-  - **`services/system-metrics/`**: Host hardware telemetry collector.
-  - **`services/second-brain/`**: Knowledge ingestion pipeline.
-- **`dist/`**: Production artifacts. Centralized directory for all compiled binaries.
+- **`cmd/`**: Minimal entry points for services. Focuses on configuration and orchestration.
+  - **`cmd/web/`**: Static site generator entry point.
+  - **`cmd/proxy/`**: API Gateway and GitOps webhook listener entry point.
+  - **`cmd/collectors/`**: Host telemetry collection daemon entry point.
+  - **`cmd/ingestion/`**: Daily data sync and second brain integration entry point.
+- **`internal/`**: Private Implementation Layer. Enforces Go's internal package visibility rules.
 - **`k3s/`**: Kubernetes manifests and Helm values for the data platform.
 - **`makefiles/`**: Modular logic for the root automation layer.
 - **`systemd/`**: Host-tier unit files for production service management.
-- **`web/`**: Static site generator for public-facing portfolio.
 - **`scripts/`**: Operational utilities (Traffic gen, ADR creation, Tailscale gate).
 - **`docs/`**: Institutional memory. ADRs, Architecture, Incidents, and Notes.
 
@@ -40,28 +35,22 @@ The project uses a unified automation layer. **Always prefer `make` commands** a
 | :--- | :--- | :--- |
 | **Governance** | `make adr` | Creates a new Architecture Decision Record. |
 | **IaC** | `tofu plan` / `apply` | Manages K3s data services and infrastructure state. |
-| **Quality** | `make lint` | Lints markdown and configuration files (`lint-configs`). |
+| **Quality** | `make lint` | Lints markdown and configuration files. |
 | **Go Dev** | `make go-test` | Runs full test suite across the monorepo. |
 | **Security** | `make go-vuln-scan` | Executes `govulncheck` for dependency auditing. |
-| **K3s Ops** | `make kube-lint` | Validates K8s manifests for security violations. |
-| **Host Ops** | `make reload-services` | Safely reloads host-tier systemd units. |
+| **K3s Ops** | `make build-collectors` | Builds and imports the collectors Docker image into K3s. |
+| **Host Ops** | `make proxy-build` | Builds proxy server to `bin/` and restarts the service. |
 
 ## 3. Engineering Standards
 
 ### 🐹 Go (Backend)
 
-- **Library-First**: Move core domain logic to `pkg/` before implementing the service entry point. Services should be thin wrappers around library capabilities.
-- **Environment Loading**: Always use `pkg/env` for standardized `.env` discovery. Do not use `godotenv` directly in services.
-- **Dependency Management**: Delegate driver registration (e.g., `lib/pq`) to `pkg/db` to avoid redundant blank imports in services.
-- **Failure Modes**: Never swallow errors. Use explicit wrapping: `fmt.Errorf(\"context: %w\", err)`.
-- **Observability**: Every service must emit JSON-formatted logs to `stdout` using `pkg/logger`.
-- **Telemetry**: All instrumentation must be handled through the centralized `pkg/telemetry` library.
-- **Testing**: Table-driven tests are the standard. Run `make go-cov` to verify coverage. Maintain a minimum of 80% coverage for `pkg/` libraries.
-
-### 🎨 HTML/CSS (Frontend)
-
-- **Zero Frameworks**: Use native HTML5 and CSS3 only.
-- **Styling**: Leverage CSS variables in `:root` for dark-theme consistency.
+- **Thin Main**: Entry points in `cmd/` must be minimal. Move all core domain logic to `internal/`.
+- **Internal-First**: Shared libraries reside in `internal/` to prevent external logic leakage.
+- **Environment Loading**: Always use `internal/env` for standardized `.env` discovery.
+- **Observability**: Every service must emit structured JSON logs using `internal/telemetry`.
+- **Telemetry**: All instrumentation must be handled through the centralized `internal/telemetry` library.
+- **Testing**: Table-driven tests are the standard. Run `make go-cov` to verify coverage.
 
 ### 📝 Institutional Memory (Documentation)
 
@@ -71,15 +60,6 @@ The project uses a unified automation layer. **Always prefer `make` commands** a
 
 ## 4. Operational Excellence & Safety
 
-- **Secrets**: NEVER commit secrets. Use `.env` for local dev and OpenBao for production secrets.
+- **Secrets**: NEVER commit secrets. Use `.env` for local dev and OpenBao for production.
 - **GitOps**: Host-tier changes are applied via `gitops_sync.sh` (triggered by Proxy webhooks).
-- **Observability**: Any new service must be integrated into the telemetry pipeline (Logs to Loki, Metrics to Postgres/Prometheus).
 - **Security**: All Kubernetes manifests must pass `kube-lint`. All Go code must pass `go-vuln-scan`.
-
-## 5. Failure Mode Analysis (FMA)
-
-Before proposing a change, agents should ask:
-
-1. "Does this create a circular dependency between the host and the cluster?"
-2. "How will this be debugged in production if the network is down?"
-3. "Is this change recorded in an ADR to preserve the 'Why'?"

README.md

@@ -12,11 +12,12 @@ Built using Go and orchestrated on Kubernetes (K3s), the platform unifies system
 
 This project highlights significant accomplishments in building a modern observability and platform engineering solution:
 
-* **Full OpenTelemetry (LMT) Implementation:** Achieved end-to-end observability with a unified OTel Collector, Tempo (Traces), Prometheus (Metrics), Loki (Logs), and Go SDK for instrumentation. Includes Service Graphs, synthetic transaction monitoring, and comprehensive host-level telemetry.
+* **Unified Go Monorepo:** Consolidated fragmented modules into a single root module, eliminating 17 `replace` directives and standardizing dependency management across all services.
+* **Encapsulated Architecture:** Transitioned to an `internal/` and `cmd/` layout, enforcing Go's package visibility rules and adopting the "Thin Main" pattern for better testability and system integrity.
+* **Full OpenTelemetry (LMT) Implementation:** Achieved end-to-end observability with a unified OTel Collector, Tempo (Traces), Prometheus (Metrics), Loki (Logs), and Go SDK for instrumentation.
 * **GitOps Reconciliation Engine:** Implemented a secure, templated GitOps reconciliation engine for automated state enforcement via webhooks, scaled to support multi-tenant synchronization.
 * **Kubernetes Migration & Cloud-Native Operations:** All core observability stack components (Loki, Grafana, Tempo, Prometheus, Postgres) are running natively in Kubernetes with persistent storage.
-* **Library-First Architecture:** Structural transition into `pkg/` and `services/` layout, decoupling core business logic into transport-agnostic modules for improved reusability and testability.
-* **Centralized Secrets Management:** Transitioned to OpenBao for secure secrets storage and retrieval, replacing insecure static configurations.
+* **Centralized Secrets Management:** Integrated OpenBao for secure, dynamic credential retrieval across all services, replacing insecure static configurations.
 * **Hybrid Cloud Architecture (Store-and-Forward Bridge):** Designed and implemented a secure bridge for ingesting external telemetry without exposing local ports, ensuring reliable data flow from diverse sources.
 * **Reproducible Local Development:** Ensures consistent and reproducible developer environments via `shell.nix` and `docker-compose`.
 * **Formalized Decision-Making & Incident Response:** Established an Architectural Decision Record (ADR) process and an Incident Response/RCA framework for structured decision-making and operational excellence.
@@ -72,7 +73,7 @@ flowchart TB
                 Tailscale[Tailscale]
             end
 
-            GoApps["Go Services (Proxy, Reading Sync, Second Brain)"]
+            GoApps["Go Services (Proxy, Ingestion)"]
             Collectors["Collectors (Host Metrics & Tailscale)"]
         end
 
@@ -128,7 +129,7 @@ This guide will help you set up and run the `observability-hub` locally using **
 
 Ensure you have the following installed on your system:
 
-* [Go](https://go.dev/doc/install) (version 1.25 or newer)
+* [Go](https://go.dev/doc/install)
 * [K3s](https://k3s.io/) (Lightweight Kubernetes)
 * [Helm](https://helm.sh/)
 * `make` (GNU Make)
@@ -174,13 +175,10 @@ Build and initialize the automation and telemetry collectors on the host:
 ```bash
 # Build Go binaries
 make proxy-build
-make reading-build
+make ingestion-build
 
 # Install and start Systemd services (requires sudo)
 make install-services
-
-# Run Second Brain sync manually
-make brain-sync
 ```
 
 ### 3. Verification
@@ -189,7 +187,6 @@ Once the stack is running, you can verify the end-to-end telemetry flow:
 
 * **Cluster Health:** Access Grafana at `http://localhost:30000` (NodePort).
 * **Service Logs:** Check logs for host components via Grafana Loki.
-* **Knowledge Sync:** Manually trigger a Second Brain ingestion with `make brain-sync`.
 
 ### 4. Managing the Cluster
 

docs/architecture/README.md

@@ -26,6 +26,7 @@ Deep dives into the logic and implementation of specific system components.
 
 - **[Collectors Service](./services/collectors.md)**: Host-level telemetry collector for metrics and Tailscale status.
 - **[Proxy Service](./services/proxy.md)**: The API Gateway and GitOps listener.
-- **[Reading Sync](./services/reading-sync.md)**: The automated MongoDB to Postgres ETL pipeline.
-- **[Second Brain](./services/second-brain.md)**: Knowledge ingestion from GitHub into PostgreSQL.
+- **[Ingestion Service](./services/ingestion.md)**: Unified data orchestration engine (Reading Analytics + Second Brain).
 - **[Tailscale Gate](./services/tailscale-gate.md)**: Logic for the automated funnel gatekeeper.
+
+

docs/architecture/core-concepts/automation.md

@@ -16,15 +16,14 @@ The system consists of several main service families, each with a `.service` uni
 | :--- | :--- | :--- | :--- |
 | **`tailscale-gate`** | `simple` | Continuous | **Security**: Monitors Proxy health and toggles Tailscale Funnel access. |
 | **`proxy`** | `simple` | Continuous | **API Gateway**: Core listener for data pipelines and GitOps webhooks. |
-| **`gitops-sync`** | `oneshot` | **Webhook** | **Reconciliation**: Triggered by Proxy to pull latest code and apply changes. |
-| **`reading-sync`** | `oneshot` | Twice Daily (00:00, 12:00) | **Data Pipeline Trigger**: Calls Proxy API to sync MongoDB data to Postgres. |
+| **`ingestion`** | `oneshot` | Daily (00:00) | **Data Ingestion**: Unified engine for Reading Analytics and Second Brain sync. |
 
 ## Operational Excellence
 
 Our systemd configurations employ several production-grade patterns:
 
 - **Security Gating**: The `tailscale-gate` service implements a loop that ensures the public entry point (Funnel) is automatically closed if the underlying `proxy` service stops, preventing "dead" endpoints from being exposed.
-- **Persistence (`Persistent=true`)**: Used in `reading-sync`. If the host is powered off during the scheduled time, systemd will trigger the service immediately upon the next boot.
+- **Persistence (`Persistent=true`)**: Used in `ingestion`. If the host is powered off during the scheduled time, systemd will trigger the service immediately upon the next boot.
 - **Unified Observability**: All units emit logs, metrics, and traces, which are captured, enriched, and forwarded by the host-level OpenTelemetry Collector.
 
 ## Architectural Patterns

docs/architecture/core-concepts/observability.md

@@ -75,7 +75,7 @@ The platform aggregates infrastructure metrics through Prometheus scraping, appl
 Distributed tracing is powered by OpenTelemetry for correlation and performance profiling across high-throughput pipelines.
 
 - **Collection Pipeline**:
-  - **Instrumentation**: Services use the **OpenTelemetry SDK** to generate spans in OTLP format. We follow a **Pure Wrapper** philosophy where shared libraries (`pkg/db`) provide standardized infrastructure spans (e.g., `db.postgres.record_metric`), while services own the root spans (`job.*` or `handler.*`).
+  - **Instrumentation**: Services use the **OpenTelemetry SDK** to generate spans in OTLP format. We follow a **Pure Wrapper** philosophy where shared libraries (`internal/db`) provide standardized infrastructure spans (e.g., `db.postgres.record_metric`), while services own the root spans (`job.*` or `handler.*`).
   - **Ingestion**: Spans are sent to the **OpenTelemetry Collector** via gRPC (NodePort `30317`) or HTTP (NodePort `30318`), which batches and exports them to **Grafana Tempo**.
   - **Processing**: Tempo analyzes raw spans to generate derived **Service Graphs** and **Span Metrics**, which are pushed to Prometheus via `remote_write` for operational correlation.
 - **Persistence**:

docs/architecture/infrastructure/deployment.md

@@ -25,16 +25,14 @@ Managed via **OpenTofu (IaC)** in `tofu/`.
 | Component | Role | Details |
 | :--- | :--- | :--- |
 | **Proxy Service** | API Gateway | Handles webhooks, GitOps triggers, and Data Pipelines. |
-| **Metrics Collector** | Telemetry Agent | Collects host hardware statistics (CPU, RAM, Disk). |
-| **Second Brain** | Knowledge Ingest | Ingests atomic journal entries from GitHub Issues. |
+| **Ingestion Service** | Data Orchestrator | Unified engine for syncing Reading Analytics and Second Brain knowledge. |
 
 ### 🛠️ Automation & Security (Native Script)
 
 | Component | Role | Details |
 | :--- | :--- | :--- |
 | **OpenBao** | Secret Store | Centralized, encrypted management for sensitive config. |
 | **Tailscale Gate** | Security Agent | Manages public funnel access based on service health. |
-| **Reading Sync** | Data Pipeline | Timer-triggered task to sync cloud data to local storage. |
 
 ## Data Flow: Unified Observability
 

docs/architecture/services/ingestion.md

@@ -0,0 +1,62 @@
+# Ingestion Service Architecture
+
+The Ingestion Service (`cmd/ingestion/`) is a unified data orchestration engine responsible for synchronizing external data sources into the platform's local analytical store (PostgreSQL). It operates as a periodic task runner managed by a Systemd timer.
+
+## Component Details
+
+### Task Overview
+
+The service follows a **Task-Oriented Design**, where specific data synchronization logics are encapsulated into independent, testable tasks managed by a centralized engine.
+
+| Task | Source | Purpose |
+| :--- | :--- | :--- |
+| `reading` | MongoDB Atlas | **Reading Analytics**: Syncs article metadata and engagement metrics from cloud to local store. |
+| `brain` | GitHub Issues | **Second Brain**: Ingests journal entries, performs atomization, and calculates token counts. |
+
+### Logic Details
+
+#### Reading Analytics Task (`reading`)
+
+Synchronizes Cloud-based MongoDB data with the local PostgreSQL environment.
+
+1.  **Fetch**: Retrieves unprocessed documents from MongoDB Atlas in configurable batches.
+2.  **Transform**: Maps MongoDB BSON/JSON metadata to the structured PostgreSQL `reading_analytics` schema.
+3.  **Persist**: Executes UPSERT operations in PostgreSQL to ensure data consistency.
+4.  **Acknowledge**: Marks documents as "processed" in MongoDB to prevent duplicate ingestion.
+
+#### Second Brain Task (`brain`)
+
+Transforms GitHub-based journaling entries into atomic, searchable database records.
+
+1.  **Check**: Queries PostgreSQL for the most recent entry date to determine the sync delta.
+2.  **Ingest**: Fetches new journal entries from the configured GitHub repository via the GitHub API.
+3.  **Atomize**: Decomposes long-form markdown logs into granular "thought atoms," including metadata like tags and categories.
+4.  **Quantify**: Calculates token counts for each atom to support future LLM-based analytical workloads.
+
+## Distributed Tracing
+
+The Ingestion Service is instrumented with the **OpenTelemetry SDK** to provide visibility into the data pipeline performance and task status.
+
+### Configuration
+
+The service initializes a global TracerProvider during startup, controlled by environment variables:
+
+| Variable | Description |
+| :--- | :--- |
+| `OTEL_EXPORTER_OTLP_ENDPOINT` | The gRPC endpoint of OpenTelemetry (e.g., `localhost:30317`). |
+| `OTEL_SERVICE_NAME` | The service identifier used in traces (defaults to `ingestion`). |
+
+### Trace Coverage
+
+Spans are created for the entire job lifecycle:
+
+-   **Job Lifecycle**: Root span `job.ingestion` tracks the overall synchronization run.
+-   **Task Execution**: Child spans (`task.reading`, `task.brain`) provide granular visibility into individual task performance.
+-   **API/DB Operations**: Sub-spans for GitHub API requests, MongoDB fetches, and PostgreSQL transactions.
+
+Traces are exported to the central **OpenTelemetry Collector** via gRPC and stored in **Grafana Tempo**.
+
+### Instrumentation Strategy
+
+1.  **Task Engine Wrapper**: The service uses a centralized `RunTask` engine that automatically wraps every registered task in a named OpenTelemetry span, capturing task-specific attributes and success/failure status.
+2.  **Manual Spans**: High-latency operations (external API calls and complex database syncs) are manually instrumented to provide precise timing and error context for pipeline optimization.

docs/architecture/services/proxy.md

@@ -1,6 +1,6 @@
 # Proxy Service Architecture
 
-The Proxy Service (`services/proxy/`) is a custom Go application that acts as the API gateway, Data Pipeline engine, and **GitOps automation trigger** for the platform. It runs as a native host process managed by Systemd.
+The Proxy Service (`cmd/proxy/`) is a custom Go application that acts as the API gateway, Data Pipeline engine, and **GitOps automation trigger** for the platform. It runs as a native host process managed by Systemd.
 
 ## Component Details