Skip to content

feat: add emotion timeline to /process_video#20

Open
Aarchi-07 wants to merge 1 commit intoruxailab:mainfrom
Aarchi-07:feature/emotion-timeline
Open

feat: add emotion timeline to /process_video#20
Aarchi-07 wants to merge 1 commit intoruxailab:mainfrom
Aarchi-07:feature/emotion-timeline

Conversation

@Aarchi-07
Copy link
Copy Markdown

Context

Related: sentiment-analysis-api#14 | RUXAILAB#1227

The /process_video endpoint currently returns only a global average of emotions across the entire video. This makes it impossible for the frontend to render timelines or visualize how emotions shift over time.

After investigating, the 7-class emotion labels (Angry, Disgusted, etc.) referenced in sentiment-analysis-api#14 actually originate from this repo, not the audio sentiment API (which produces POS/NEG/NEU). The timeline feature belongs here.

Changes

The /process_video response now includes a timeline array alongside the existing emotions object:

{
  "emotions": { "Angry": 15.38, "Neutral": 53.85, ... },
  "timeline": [
    { "id": 1, "starting_time": "00:00", "ending_time": "00:10", "emotion": "NEUTRAL", "value": 87.5 },
    { "id": 2, "starting_time": "00:10", "ending_time": "00:20", "emotion": "SAD", "value": 72.3 }
  ]
}

What was added

  • TimelineEntry and EmotionAnalysisResponse schemas in emotion_schema.py
  • Two timeline modes in emotion_analysis_imp.py:
    • Fixed intervals (default, interval_s=10): groups frames into time windows, picks dominant emotion per window
    • Dynamic segmentation (interval_s=0): creates a new segment whenever the dominant emotion changes
  • value uses actual model prediction confidence (softmax output), averaged across the dominant emotion's frames in each window
  • interval_s optional request body param in the route (defaults to 10)
  • 18 unit tests covering both timeline modes, edge cases, and confidence calculation

Changes

File Change
schemas/emotion_schema.py Added TimelineEntry, EmotionAnalysisResponse
services/emotion_analysis/emotion_analysis_imp.py Added timeline helpers + per-frame timestamp/confidence tracking
services/emotion_analysis/emotion_analysis_service.py Updated interface signature
routes/video_routes.py Added interval_s param, returns full response via model_dump()
tests/test_timeline.py 18 unit tests (mocks heavy deps, tests pure logic)

Backward compatibility

The emotions key is preserved with the exact same structure. The timeline key is purely additive. Existing frontend code that reads res.data.emotions will continue to work unchanged.

@Aarchi-07 Aarchi-07 marked this pull request as draft March 19, 2026 18:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant