Make Area and Capability description fields required instead of optional.

Add file references to all dataclass sections in PIPELINE_SCHEMAS.md documentation.

Author: kohankhaki

src/schemas/PIPELINE_SCHEMAS.md

@@ -209,6 +209,8 @@ All dataclasses used across pipeline stages are defined below. Stage implementat
 
 ### PipelineMetadata
 
+**File:** [`metadata_schemas.py`](metadata_schemas.py)
+
 All pipeline outputs include a `metadata` object (represented by the `PipelineMetadata` dataclass) that provides pipeline execution context and traceability.
 
 **Required Fields:**
@@ -228,6 +230,8 @@ All pipeline outputs include a `metadata` object (represented by the `PipelineMe
 
 ### Experiment
 
+**File:** [`experiment_schemas.py`](experiment_schemas.py)
+
 **Fields:**
 - `experiment_id`: String (required, experiment identifier)
 - `domain`: String (required, human-readable domain name)
@@ -237,18 +241,22 @@ All pipeline outputs include a `metadata` object (represented by the `PipelineMe
 
 ### Domain
 
+**File:** [`domain_schemas.py`](domain_schemas.py)
+
 **Fields:**
 - `name`: String (required, human-readable domain name)
 - `domain_id`: String (required)
 - `description`: String (optional, domain description)
 
 ### Area
 
+**File:** [`area_schemas.py`](area_schemas.py)
+
 **Fields:**
 - `name`: String (required, human-readable area name)
 - `area_id`: String (required)
-- `description`: String (optional, area description)
-- `domain`: Optional[Domain] (optional, Domain dataclass object)
+- `domain`: Domain (required, Domain dataclass object)
+- `description`: String (required, area description)
 - `generation_metadata`: Dict (optional, nested dictionary containing process-specific information)
   - This field can contain any generation-specific data (e.g., generation method, parameters, intermediate steps)
   - Structure is flexible and depends on the generation method
@@ -257,11 +265,13 @@ All pipeline outputs include a `metadata` object (represented by the `PipelineMe
 
 ### Capability
 
+**File:** [`capability_schemas.py`](capability_schemas.py)
+
 **Fields:**
 - `name`: String (required, capability name)
 - `capability_id`: String (required)
-- `description`: String (optional, capability description)
-- `area`: Optional[Area] (optional, Area dataclass object)
+- `area`: Area (required, Area dataclass object)
+- `description`: String (required, capability description)
 - `generation_metadata`: Dict (optional, nested dictionary containing process-specific information)
   - This field can contain any generation-specific data (e.g., generation method, parameters, intermediate steps)
   - Structure is flexible and depends on the generation method
@@ -270,40 +280,46 @@ All pipeline outputs include a `metadata` object (represented by the `PipelineMe
 
 ### Task
 
+**File:** [`task_schemas.py`](task_schemas.py)
+
 **Fields:**
 - `task_id`: String (required, unique within capability)
 - `task`: String (required, the task/problem text)
-- `capability`: Optional[Capability] (optional, Capability dataclass object)
+- `capability`: Capability (required, Capability dataclass object)
 
 **Note:** When serialized to JSON, the `capability` object is flattened to `capability` (string), `capability_id` (string), `area` (string), `area_id` (string), `domain` (string), and `domain_id` (string) fields.
 
 ### TaskSolution
 
+**File:** [`solution_schemas.py`](solution_schemas.py)
+
 **Fields:**
 - `task_id`: String (required)
 - `task`: String (required, the task/problem text from Stage 3)
 - `solution`: String (required, the final solution)
 - `reasoning`: String (required, explanation of the solution)
+- `task_obj`: Task (required, Task dataclass object with full hierarchy)
 - `numerical_answer`: String (optional, JSON string with numerical results)
 - `generation_metadata`: Dict (optional, nested dictionary containing process-specific information)
   - This field can contain any generation-specific data (e.g., debate rounds, agent interactions, pipeline type)
   - Structure is flexible and depends on the generation method (agentic, single-agent, etc.)
-- `task_obj`: Optional[Task] (optional, Task dataclass object with full hierarchy)
 
 **Note:** When serialized to JSON, the `task_obj` object is flattened to `capability` (string), `capability_id` (string), `area` (string), `area_id` (string), `domain` (string), and `domain_id` (string) fields.
 
 ### ValidationResult
 
+**File:** [`validation_schemas.py`](validation_schemas.py)
+
 **Fields:**
 - `task_id`: String (required)
 - `task`: String (required, the task/problem text from Stage 3)
 - `verification`: Boolean (required, overall validation status - whether the solution is verified/valid)
 - `feedback`: String (required, detailed feedback on the validation)
+- `task_obj`: Task (required, Task dataclass object with full hierarchy)
 - `score`: Float (optional, validation score, typically 0.0 to 1.0)
 - `generation_metadata`: Dict (optional, nested dictionary containing process-specific information)
   - This field can contain any validation-specific data (e.g., validation method, criteria details, error details)
   - Structure is flexible and depends on the validation method
-- `task_obj`: Optional[Task] (optional, Task dataclass object with full hierarchy)
 
 **Note:** When serialized to JSON, the `task_obj` object is flattened to `capability` (string), `capability_id` (string), `area` (string), `area_id` (string), `domain` (string), and `domain_id` (string) fields.
 
@@ -333,7 +349,7 @@ This stage creates two files:
 #### Output 1: `experiment.json`
 
 **Stage Output:** Experiment dataclass + PipelineMetadata
-**Save Function:** `save_experiment(experiment: Experiment, metadata: PipelineMetadata, output_path: Path)`
+**Save Function:** `save_experiment(experiment: Experiment, metadata: PipelineMetadata, output_path: Path)` (see [`io_utils.py`](io_utils.py))
 
 **File Path:** `<output_dir>/<experiment_id>/experiment.json`
 
@@ -364,7 +380,7 @@ This stage creates two files:
 #### Output 2: `domain.json`
 
 **Stage Output:** Domain dataclass object + PipelineMetadata
-**Save Function:** `save_domain(domain: Domain, metadata: PipelineMetadata, output_path: Path)`
+**Save Function:** `save_domain(domain: Domain, metadata: PipelineMetadata, output_path: Path)` (see [`io_utils.py`](io_utils.py))
 
 **File Path:** `<output_dir>/<experiment_id>/domain/domain.json`
 
@@ -402,7 +418,7 @@ This stage creates two files:
 ### Output: `areas.json`
 
 **Stage Output:** List[Area] dataclasses + PipelineMetadata
-**Save Function:** `save_areas(areas: List[Area], metadata: PipelineMetadata, output_path: Path)`
+**Save Function:** `save_areas(areas: List[Area], metadata: PipelineMetadata, output_path: Path)` (see [`io_utils.py`](io_utils.py))
 
 **File Path:** `<output_dir>/<experiment_id>/areas/<tag>/areas.json`
 ```json
@@ -447,7 +463,7 @@ This stage creates two files:
 ### Output: `capabilities.json` (one per area)
 
 **Stage Output:** List[Capability] dataclasses + PipelineMetadata
-**Save Function:** `save_capabilities(capabilities: List[Capability], metadata: PipelineMetadata, output_path: Path)`
+**Save Function:** `save_capabilities(capabilities: List[Capability], metadata: PipelineMetadata, output_path: Path)` (see [`io_utils.py`](io_utils.py))
 
 **File Path:** `<output_dir>/<experiment_id>/capabilities/<cap_tag>/<area_id>/capabilities.json`
 
@@ -495,7 +511,7 @@ This stage creates two files:
 ### Output: `tasks.json` (one per capability)
 
 **Stage Output:** List[Task] dataclasses + PipelineMetadata
-**Save Function:** `save_tasks(tasks: List[Task], metadata: PipelineMetadata, output_path: Path)`
+**Save Function:** `save_tasks(tasks: List[Task], metadata: PipelineMetadata, output_path: Path)` (see [`io_utils.py`](io_utils.py))
 
 **File Path:** `<output_dir>/<experiment_id>/tasks/<task_tag>/<area_id>/<capability_id>/tasks.json`
 

src/schemas/area_schemas.py

@@ -1,7 +1,7 @@
 """Schemas for area generation stage (Stage 1).
 
 Defines Area dataclass for domain area. Areas are high-level categories
-within a domain (e.g., "Budgeting" within "Personal Finance").
+within a domain.
 """
 
 from dataclasses import dataclass, field
@@ -16,39 +16,35 @@ class Area:
 
     name: str
     area_id: str
-    description: Optional[str] = None
-    domain: Optional[Domain] = None
+    domain: Domain
+    description: str
     generation_metadata: Optional[Dict] = field(default_factory=dict)
 
     def to_dict(self):
         """Convert to dictionary."""
         result = {
             "name": self.name,
             "area_id": self.area_id,
+            "domain": self.domain.name,
+            "domain_id": self.domain.domain_id,
+            "description": self.description,
         }
-        if self.domain is not None:
-            result["domain"] = self.domain.name
-            result["domain_id"] = self.domain.domain_id
-        if self.description is not None:
-            result["description"] = self.description
         if self.generation_metadata:
             result["generation_metadata"] = self.generation_metadata
         return result
 
     @classmethod
     def from_dict(cls, data: dict):
         """Create from dictionary."""
-        domain = None
-        if "domain" in data and "domain_id" in data:
-            domain = Domain(
-                name=data["domain"],
-                domain_id=data["domain_id"],
-                description=None,
-            )
+        domain = Domain(
+            name=data["domain"],
+            domain_id=data["domain_id"],
+            description=data.get("domain_description"),
+        )
         return cls(
             name=data["name"],
             area_id=data["area_id"],
-            description=data.get("description"),
             domain=domain,
+            description=data["description"],
             generation_metadata=data.get("generation_metadata", {}),
         )

src/schemas/capability_schemas.py

@@ -1,7 +1,7 @@
 """Schemas for capability generation stage (Stage 2).
 
 Defines Capability dataclass for capability within an area. Capabilities
-are specific skills or abilities (e.g., "Budget Creation" within "Budgeting" area).
+are specific skills or abilities.
 """
 
 from dataclasses import dataclass, field
@@ -17,50 +17,44 @@ class Capability:
 
     name: str
     capability_id: str
-    description: Optional[str] = None
-    area: Optional[Area] = None
+    area: Area
+    description: str
     generation_metadata: Optional[Dict] = field(default_factory=dict)
 
     def to_dict(self):
         """Convert to dictionary."""
         result = {
             "name": self.name,
             "capability_id": self.capability_id,
+            "area": self.area.name,
+            "area_id": self.area.area_id,
+            "area_description": self.area.description,
+            "domain": self.area.domain.name,
+            "domain_id": self.area.domain.domain_id,
+            "description": self.description,
         }
-        if self.area is not None:
-            result["area"] = self.area.name
-            result["area_id"] = self.area.area_id
-            if self.area.domain is not None:
-                result["domain"] = self.area.domain.name
-                result["domain_id"] = self.area.domain.domain_id
-        if self.description is not None:
-            result["description"] = self.description
         if self.generation_metadata:
             result["generation_metadata"] = self.generation_metadata
         return result
 
     @classmethod
     def from_dict(cls, data: dict):
         """Create from dictionary."""
-        area = None
-        if "area" in data and "area_id" in data:
-            domain = None
-            if "domain" in data and "domain_id" in data:
-                domain = Domain(
-                    name=data["domain"],
-                    domain_id=data["domain_id"],
-                    description=None,
-                )
-            area = Area(
-                name=data["area"],
-                area_id=data["area_id"],
-                description=None,
-                domain=domain,
-            )
+        domain = Domain(
+            name=data["domain"],
+            domain_id=data["domain_id"],
+            description=data.get("domain_description"),
+        )
+        area = Area(
+            name=data["area"],
+            area_id=data["area_id"],
+            domain=domain,
+            description=data["area_description"],
+        )
         return cls(
             name=data["name"],
             capability_id=data["capability_id"],
-            description=data.get("description"),
             area=area,
+            description=data["description"],
             generation_metadata=data.get("generation_metadata", {}),
         )

src/schemas/solution_schemas.py

@@ -21,9 +21,9 @@ class TaskSolution:
     task: str
     solution: str
     reasoning: str
+    task_obj: Task
     numerical_answer: Optional[str] = None
     generation_metadata: Optional[Dict] = field(default_factory=dict)
-    task_obj: Optional[Task] = None  # Full task object with hierarchy
 
     def to_dict(self):
         """Convert to dictionary."""
@@ -32,16 +32,15 @@ def to_dict(self):
             "task": self.task,
             "solution": self.solution,
             "reasoning": self.reasoning,
+            "capability_id": self.task_obj.capability.capability_id,
+            "capability": self.task_obj.capability.name,
+            "capability_description": self.task_obj.capability.description,
+            "area": self.task_obj.capability.area.name,
+            "area_id": self.task_obj.capability.area.area_id,
+            "area_description": self.task_obj.capability.area.description,
+            "domain": self.task_obj.capability.area.domain.name,
+            "domain_id": self.task_obj.capability.area.domain.domain_id,
         }
-        if self.task_obj is not None and self.task_obj.capability is not None:
-            result["capability_id"] = self.task_obj.capability.capability_id
-            result["capability"] = self.task_obj.capability.name
-            if self.task_obj.capability.area is not None:
-                result["area"] = self.task_obj.capability.area.name
-                result["area_id"] = self.task_obj.capability.area.area_id
-                if self.task_obj.capability.area.domain is not None:
-                    result["domain"] = self.task_obj.capability.area.domain.name
-                    result["domain_id"] = self.task_obj.capability.area.domain.domain_id
         if self.numerical_answer is not None:
             result["numerical_answer"] = self.numerical_answer
         if self.generation_metadata:
@@ -51,40 +50,34 @@ def to_dict(self):
     @classmethod
     def from_dict(cls, data: dict):
         """Create from dictionary."""
-        task_obj = None
-        if "capability" in data and "capability_id" in data:
-            area = None
-            if "area" in data and "area_id" in data:
-                domain = None
-                if "domain" in data and "domain_id" in data:
-                    domain = Domain(
-                        name=data["domain"],
-                        domain_id=data["domain_id"],
-                        description=None,
-                    )
-                area = Area(
-                    name=data["area"],
-                    area_id=data["area_id"],
-                    description=None,
-                    domain=domain,
-                )
-            capability = Capability(
-                name=data["capability"],
-                capability_id=data["capability_id"],
-                description=None,
-                area=area,
-            )
-            task_obj = Task(
-                task_id=data["task_id"],
-                task=data["task"],
-                capability=capability,
-            )
+        domain = Domain(
+            name=data["domain"],
+            domain_id=data["domain_id"],
+            description=data.get("domain_description"),
+        )
+        area = Area(
+            name=data["area"],
+            area_id=data["area_id"],
+            domain=domain,
+            description=data["area_description"],
+        )
+        capability = Capability(
+            name=data["capability"],
+            capability_id=data["capability_id"],
+            area=area,
+            description=data["capability_description"],
+        )
+        task_obj = Task(
+            task_id=data["task_id"],
+            task=data["task"],
+            capability=capability,
+        )
         return cls(
             task_id=data["task_id"],
             task=data["task"],
             solution=data["solution"],
             reasoning=data["reasoning"],
+            task_obj=task_obj,
             numerical_answer=data.get("numerical_answer"),
             generation_metadata=data.get("generation_metadata", {}),
-            task_obj=task_obj,
         )