feat: dual coding — combine verbal and visual information for stronger encoding #77
Description
Summary
Implement dual coding theory (Paivio) in Engram's learning flow — present concepts using both verbal and visual representations simultaneously to create stronger, more retrievable memory traces.
Background
Dual coding theory shows that information encoded through both verbal and visual channels creates two independent memory traces, making recall significantly more robust. Currently Engram's quiz items are primarily text-based. Adding visual representations alongside verbal explanations would deepen encoding.
How it could work in Engram
During quiz sessions
- Show a visual alongside the verbal quiz item — e.g., a diagram, concept map snippet, or icon-based representation
- For relationships: visualize the connection in the knowledge graph while the verbal explanation is shown
- Claude could generate visual descriptions or simple diagrams at extraction time (e.g., ASCII diagrams, Mermaid syntax, or structured visual metadata)
On the knowledge graph
- Nodes could carry visual associations (icons, color-coded categories, small thumbnails)
- When reviewing a concept, zoom into its local neighborhood in the graph — the spatial layout itself becomes a visual encoding
- Animate relationship paths to show how concepts connect visually
During ingestion
- Extract both textual summaries and visual/structural representations from source material
- Flag concepts that are inherently visual (architecture diagrams, data flows, state machines) for richer visual treatment
Research
Paivio's Dual Coding Theory (1971, 1986)
Allan Paivio's Dual Coding Theory proposes that the human mind processes information along two functionally independent but interacting channels: a verbal system (linguistic information) and a nonverbal/imagery system (visual, spatial, and sensory information).
Key experimental findings:
- Dual encoding advantage: When information is encoded in both systems simultaneously, recall probability increases because retrieval can succeed via either code.
- Selective interference effect: Two simultaneous verbal tasks degrade performance, but pairing a verbal task with a visual task avoids interference — direct evidence for two distinct systems.
- Picture superiority effect: Images are recalled better in free recall; words are recalled better in sequential order — demonstrating distinct strengths of each channel.
Paivio, A. (1971). Imagery and Verbal Processes. Holt, Rinehart & Winston.
Paivio, A. (1986). Mental Representations: A Dual Coding Approach. Oxford University Press.
The Concreteness Effect
Concrete concepts (e.g., "banana") are dual-coded in both systems; abstract concepts (e.g., "belief") are primarily single-coded verbally. This leads to robust memory differences: free recall of concrete nouns (M=11.05) was more than double that of abstract words (M=4.70), t(19)=10.75, p<.001.
Relevance to Engram: The knowledge graph visualization may serve as a "concreteness bridge" — making abstract relationships visible and spatial, giving abstract knowledge a visual code it would otherwise lack.
Clark & Paivio (1991) — DCT Applied to Education
Landmark review extending DCT into a comprehensive educational framework. The theory predicts that instructional materials combining verbal explanations with visual representations will produce superior learning compared to either channel alone.
Clark, J. M., & Paivio, A. (1991). Dual coding theory and education. Educational Psychology Review, 3(3), 149-210.
Mayer's Multimedia Learning Principles
Richard Mayer built directly on DCT to produce some of the strongest effect sizes in educational psychology:
| Principle | Description | Effect Size |
|---|---|---|
| Multimedia | Words + pictures > words alone | d = 1.35 – 1.67 |
| Modality | Narration + graphics > text + graphics | d = 0.72 – 1.00 |
| Spatial contiguity | Integrated text/image > separated | g = 0.74 |
| Signaling | Highlighted cues > no cues | d = 0.52 |
The multimedia principle is the foundational finding: people learn better from words + pictures than from words alone, supported in 13 out of 13 tests.
Mayer, R. E. (2009). Multimedia Learning (2nd ed.). Cambridge University Press.
Noetel et al. (2022) — Meta-Meta-Analysis
The most comprehensive overview to date: a meta-meta-analysis of 29 systematic reviews covering 1,189 studies and 78,177 participants confirmed that multimedia design principles produce significant, positive effects on learning. 11 design principles demonstrated significant positive meta-analytic effects. The largest benefits came from captioning, temporal/spatial contiguity, and signaling.
Noetel, M., et al. (2022). Multimedia design for learning: An overview of reviews with meta-meta-analysis. Review of Educational Research, 92(1).
Concept Maps as Visual Encoding
Schroeder, Nesbit, Angangco, & Adesope (2018): Meta-analysis of 142 effect sizes (n=11,814). Overall moderate, statistically significant effect: g = 0.58 (p<.001). Constructing concept maps (g=0.72) produced greater benefit than studying pre-made maps (g=0.43). Effective across both STEM and non-STEM domains.
Schroeder, N. L., Nesbit, J. C., Angangco, C. J., & Adesope, O. O. (2018). Studying and constructing concept maps: A meta-analysis. Educational Psychology Review, 30, 431-455.
Cognitive Load Constraints for Graph Visualization
Research on node-link diagram scalability reveals important design constraints:
- Significant difficulty finding paths with >50 nodes (high-density) or >100 nodes (low-density)
- Diagrams arranged according to Gestalt principles produce higher learning gains
- Creating a meaningful reading order is critical to avoid disorientation
Relevance to Engram: This validates the current approach of scoping the visible graph (session filtering during ingestion, collection filtering on the dashboard) rather than showing the entire wiki graph at once.
Dual Coding + Spaced Repetition Combined
Dual-channel processing can improve recall by up to 65% compared to single-channel methods. Spaced repetition alone produces 10-50% better recall versus massed practice. The combination is multiplicative: the knowledge graph provides persistent spatial/visual encoding, while the quiz system provides verbal spaced repetition. Each review reinforces both codes — verbal through Q&A, visual through graph interaction (seeing a concept "light up" from grey to green). DCT predicts this dual encoding should produce measurably better retention than either alone.
Summary of Effect Sizes
| Finding | Effect Size | Source |
|---|---|---|
| Words + pictures vs. words alone | d = 1.35 – 1.67 | Mayer (2001, 2009) |
| Narration + graphics vs. text + graphics | d = 0.72 – 1.00 | Ginns (2005) |
| Spatial contiguity | g = 0.74 | Noetel et al. (2022) |
| Constructing concept maps vs. control | g = 0.72 | Schroeder et al. (2018) |
| Studying concept maps vs. control | g = 0.43 | Schroeder et al. (2018) |
| Dual-channel vs. single-channel recall | up to +65% | Multiple reviews |
Design Considerations
- Visual representations should complement, not replace, verbal content
- Keep visuals simple and consistent — cognitive load matters
- Could integrate with the existing
ForceDirectedGraphWidgetas the primary visual encoding tool - AI-generated visuals (diagrams, concept sketches) vs. source-extracted visuals vs. graph-based visuals
Related
- feat: collaborative interrogation (how/why deepening) + Ebbinghaus forgetting curve visualization #76 — Collaborative interrogation (verbal deepening pairs with visual deepening)
- feat: video-synchronized knowledge graph highlighting with relationship explanations #74 — Video-synced highlighting (video is inherently dual-coded)
- feat: semantic linking across sources — podcasts, books, and offline discovery #75 — Cross-source semantic linking (visual similarity across sources)