Hyperion: AI-Powered Instructor Checklist for Exercise Quality & Alignment

[HawKhiem]

Status: Draft
Related: #11940 (Problem statement refinement), #11594 (Problem statement generation)

Summary

Introduce an AI-powered Instructor Checklist that analyzes programming exercise problem statements to surface inferred learning goals, suggested difficulty, and quality issues—enabling instructors to verify constructive alignment before publishing.

Motivation (Why)

• Constructive alignment gap: Instructors write problem statements without immediate feedback on whether the task actually exercises the intended learning goals at the intended difficulty.
• Quality assurance: Statement issues (ambiguity, missing constraints, unclear success criteria) are often only discovered after students struggle.
• Actionable AI assistance: Current AI features (generate/refine) help produce content, but don't help instructors verify what they've written.

Proposal (Approach)

• Add an "Analyze" action in the exercise editor that triggers AI analysis of the current problem statement.
• Display results in a Checklist Panel with three sections:
  1. Inferred Learning Goals — skills the AI detects, with taxonomy levels (REMEMBER→CREATE) and confidence
  2. Difficulty Assessment — suggested EASY/MEDIUM/HARD with reasoning, compared to declared difficulty
  3. Quality Issues — clarity, coherence, completeness problems with severity and suggested fixes
• Instructor reviews the checklist against their intentions and can trigger refinement based on findings.

Architecture

flowchart TB
    subgraph Client
        Editor[Problem Statement Editor]
        Panel[Checklist Panel]
        Editor -->|"onClick: Analyze"| Panel
    end

    subgraph Server
        Resource[HyperionProblemStatementResource]
        Service[HyperionChecklistService]
        Resource --> Service
    end

    subgraph AI["AI Analysis (concurrent)"]
        P1[Learning Goals Prompt]
        P2[Difficulty Prompt]
        P3[Quality Prompt]
    end

    Panel -->|"POST /checklist-analysis"| Resource
    Service --> P1 & P2 & P3
    P1 & P2 & P3 -->|"structured output"| Service
    Service -->|"ChecklistAnalysisResponse"| Panel

    Panel -->|"displays"| Goals[Inferred Goals]
    Panel -->|"displays"| Diff[Difficulty Assessment]
    Panel -->|"displays"| Quality[Quality Issues]

Loading

UX Flow

sequenceDiagram
    participant I as Instructor
    participant UI as Editor
    participant API as Hyperion API
    participant AI as AI Model

    I->>UI: Clicks "Analyze Checklist"
    UI->>API: POST /checklist-analysis
    API->>AI: Run concurrent prompts
    AI-->>API: Goals, difficulty, quality
    API-->>UI: ChecklistAnalysisResponse
    UI->>I: Display checklist panel
    I->>UI: Reviews vs intended goals
    I->>UI: Clicks "Refine" on issue
    UI->>API: POST /refine with context

Loading

API Design

Endpoint

POST /api/hyperion/programming-exercises/{exerciseId}/checklist-analysis

Response schema

ChecklistAnalysisResponse {
    inferredLearningGoals: LearningGoalItem[]
    suggestedDifficulty: DifficultyAssessment
    qualityIssues: QualityIssue[]
}

LearningGoalItem {
    skill: string
    taxonomyLevel: 'REMEMBER' | 'UNDERSTAND' | 'APPLY' | 'ANALYZE' | 'EVALUATE' | 'CREATE'
    confidence: number  // 0.0–1.0
    explanation: string
}

DifficultyAssessment {
    suggested: 'EASY' | 'MEDIUM' | 'HARD'
    reasoning: string
    matchesDeclared: boolean
}

QualityIssue {
    category: 'CLARITY' | 'COHERENCE' | 'COMPLETENESS'
    severity: 'INFO' | 'WARNING' | 'ERROR'
    description: string
    location?: { startLine: number, endLine: number }
    suggestedFix?: string
}

Out of Scope (Future)

• Lecture/week alignment checking
• Cross-exercise issues detection
• Persistent checklist item states (approve/dismiss)
• Incorporate consistency check into the checklist

Open Questions

1. Competency matching: Should inferred learning goals auto-suggest matching existing course competencies from the Atlas module?
2. Threshold display: Should we show confidence scores to instructors, or just high-confidence inferences?
3. Telemetry: Store analysis runs in ai_metadata for Langfuse/OTel correlation?

---

Feedback welcome.

[sawys777]

That's a good idea here, since it's already covering 3 types of analysis, do you think it would be better if we also include the consistency check as part of the analysis, so instructor doesn't need to perform two actions separately

[HawKhiem]

Yes, my goal was to also incorporate it into this system. Currently the consistency check is only available after the exercise has been created, whereas the instructor's checklist should be available on the creation page as well. So I want to discuss this further and perhaps incorporate the consistency check as a follow up