update readme

Author: ivansurif

README.md

@@ -2,6 +2,51 @@
 
 AI-driven extractor log analysis. Deploys CDF Functions and a data model that segments extractor logs into per-run chunks and generates structured root cause analyses using Atlas AI.
 
+## System Overview
+
+Argus AI is an end-to-end log monitoring and analysis system built on Cognite Data Fusion (CDF). It turns raw extractor log files into structured, AI-generated root-cause analyses surfaced through a web dashboard. The full system has three layers:
+
+### Layer 1 — Data Ingestion: Cognite File Extractor
+
+The starting point is an on-premises server where an extractor is running. A **Cognite File Extractor** is installed on the same VM. It watches a designated folder on the filesystem and continuously uploads log files to CDF as CDM `CogniteFile` nodes. No processing happens at this stage — the raw log file simply lands in CDF as a file resource.
+
+### Layer 2 — Data Modeling & Processing (this repository)
+
+This repository is packaged as a **CDF Toolkit library module** (`dp:argus_ai`) that clients deploy into their own CDF Toolkit project. It provisions a data model and two CDF Functions that process the uploaded log file:
+
+1. **`argus_ai_log_chopper`** reads only new lines from the source log file (tracking state in CDF RAW), segments them into discrete extractor runs based on start/end markers, and uploads each run as a separate `ArgusAILogChunkFile` node — a file containing just that run's log lines, together with metadata (timestamps, error/warning counts, pipeline ID, PI host).
+
+2. **`argus_ai_log_analyzer`** processes unanalyzed chunks for a given pipeline. Clean runs (zero errors/warnings) receive a `no_issues` result directly. For all others, the chunk is sent to an **Atlas AI agent**, and the structured JSON response is validated against allowed enum values fetched from the DMS container schema, then written as an `ArgusAILogAnalysis` node.
+
+### Layer 3 — Monitoring UI
+
+A Dune application queries the `ArgusAILogAnalysis` nodes from CDF and presents them as a health timeline and detected-issues list, with severity badges, category and root cause labels, and an embedded AI chat for troubleshooting guidance.
+
+### End-to-End Flow
+
+```
+Extractor VM
+  └── Cognite File Extractor
+        └── uploads log file ──► CDF (CogniteFile node)
+                                          │
+                                argus_ai_log_chopper
+                                (segments into per-run chunks)
+                                          │
+                                ArgusAILogChunkFile nodes
+                                (one file per extractor run)
+                                          │
+                                argus_ai_log_analyzer
+                                (Atlas AI → structured root cause)
+                                          │
+                                ArgusAILogAnalysis nodes
+                                (CogniteActivity with severity,
+                                 category, root cause code, …)
+                                          │
+                                ArgusAI Web Dashboard
+                                (health timeline, issue list,
+                                 AI-assisted troubleshooting)
+```
+
 ## What gets deployed
 
 | Resource | Description |