Week 8 Memo

[ShiyangLai]

Post a memo that explores the readings and class topics relevant to your final project by Thursday, Feb 26, 11:59 PM. The memo should be 300–500 words and include the following: (1) state your research question (for the imagined final project) succinctly in a single sentence at the beginning (this can and should evolve across the weeks of the quarter as the project becomes more concrete); (2) propose a research design that helps to address that question, and not proposed in any prior memo, informed by at least one of the required readings and one of the supplemental readings from this week; (3) a visual figure or diagram that draws upon pilot (nonhallucinated) data, simulated data with defensible assumptions, or provides a clear conceptual illustration that persuades the reader (aka James and TAs) of the appropriateness and fruitfulness of the design to address your question. You will then pilot this design as the final question in the Week 8 Homework due the following Wednesday.

By 10AM Friday, each student will up-vote (“thumbs up”) what they think are the five most interesting memos for the week!

Required
AI Foundations: “Convolutional Neural Networks” and “Diffusion Models” in Deep Learning: Foundations and Concepts, chapters 10 and 20.
“REACT: Synergizing Reasoning and Acting in Large Language Models.” Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, Yuan Cao. 2025.
AI Designs: “GPT-4V(ision) System Card.” OpenAI. 2023.
“Embodied Task Planning with Large Language Models.” Song, C. H., Wu, J., Washington, C., et al. arXiv. 2023.
Social Designs: “Machine Learning as a Tool for Hypothesis Generation”, Jens Ludwig, Sendhil Mullainathan. The Quarterly Journal of Economics 2024.
“Age and gender distortion in online media and large language models”. Douglas Guilbeault, Solène Delecourt & Bhargav Srinivasa Desikan. 2025.

Supplemental
AI Foundations: “Deep Computer Vision Using Convolutional Neural Networks”, “Representation Learning and Generative Learning Using Autoencoders and GANs”, Hands-On Machine Learning with Scikit-Learn, Keras & Tensorflow, chapters 14, 17.
“Multi-modal Transformers”, Deep Learning: Foundations and Concepts, chapter section 12.4.
AI Perspectives: “Explainable Deep Learning: Concepts, Methods, and New Developments” by Wojciech Samek in Explainable Deep Learning. 2023.
AI Designs: “Guiding a Diffusion Model with a Bad Version of Itself.” Karras et al, NeurIPS. 2024.
“Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction.” 2024. Tian et al. NeurIPS.
“The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes”. 2020.
Social Designs: “Online images amplify gender bias,” Guilbeault, Douglas, Solène Delecourt, Tasker Hull, Bhargav Srinivasa Desikan, Mark Chu, and Ethan Nadler. Nature. 2024.
“Sixteen facial expressions occur in similar contexts worldwide” 2020.
“Using deep learning and Google Street View to estimate the demographic makeup of neighborhoods across the United States” 2017.
“Computer vision uncovers predictors of physical urban change.” 2017.

[michellecdu]

Research Question
Does the ReAct framework, by combining reasoning with active policy retrieval, improve multimodal hate speech detection compared to internal chain-of-thought (CoT) reasoning? Specifically, can a “ReAct Auditor” more accurately identify a specific “attack category” (e.g., “inciting_violence”) when faced with potentially hateful memes, including “benign confounders,” or images where the meaning is flipped by a single visual element?

Research Design
The full study will use 500 memes from the Hateful Memes dataset, which is an open-source dataset for vision-and-language classification that allows researchers to design models that can detect nuanced hate speech in multimodal memes (Kiela et al., 2020). First, I will extract image context using a multimodal vision model, GPT-4o, to extract the meme text (OCR) and generate a “meme-aware” caption that focuses on the relationship between text and image.

Then, I will create the following experimental conditions:

• CoT Baseline: Provide the prompt with just the OCR text and caption, the model is prompted to “think step-by-step” to classify the meme using only its internal weights.

• ReAct Auditor: The model will be given a CheckPolicy tool. It will follow a Thought → Action → Observation loop, where it explicitly looks up the definition of a suspected attack (e.g., CheckPolicy[dehumanizing]) before suggesting an intervention (e.g., SuggestIntervention[Flag for Religion-based Hate]). This process is based on the one implemented in Yao et al. (2023), which showed that a ReAct approach could mitigate hallucination and error propagation issues in CoT reasoning.

Finally, in both conditions, the model will conclude if the meme violates the policy. I will measure diagnostic accuracy to test the model’s ability to not just get the “right” binary label, but to correctly identify the reason for the label. For example, I will measure response quality (using an LLM-as-a-judge across the four dimensions of visual grounding, policy adherence, audit logic, and intervention format) and aggregate performance (by comparing mean judge results). Unlike standard classifiers that act as black boxes, this ReAct agent provides a “verifiable audit” where every moderation decision is linked to a specific policy observation.

By applying it to the Hateful Memes challenge, I expand upon the paper’s core finding that models fail when they lack the “subtle world knowledge” to resolve visual-textual irony. ReAct provides this knowledge via the Action step, although it is not always successful in understanding real-world context (e.g., misclassifying a hateful meme of the twin towers with text saying “we said we would never forget … why are you voting them into government?”).

Pilot Figures
I ran a pilot with ~20 images, with a CoT baseline and a ReAct “Auditor” agent. The Sankey diagram shows the process of the ReACT agent’s decision-making, which flows from text input to visual context, policy tool, and outcome (Figure 3). The figures suggest that the ReAct agent improved response quality and aggregate performance, and that the policy tool used in the “Action” step can indicate hate classification (qualitatively, I noticed this even when the classification was incorrect).

References
Guilbeault et al. (2024). “Online images amplify gender bias.” Nature 626, 1049-1055. https://www.nature.com/articles/s41586-024-07068-x

Kiela et al. (2020). “The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes.” https://papers.nips.cc/paper/2020/hash/1b84c4cee2b8b3d823b30e2d604b1878-Abstract.html

Yao et al. (2023). “REACT: Synergizing Reasoning and Acting in Large Language Models.” arXiv. https://arxiv.org/pdf/2210.03629

[jhixon2]

Research Question:

Can the toxicity-classification of Twitch chat messages using agents be improved through encouraging the agents to consult a dictionary of Twitch-specific terms when unsure of a term’s meaning?

Research Design:

All of my approaches to using AI agents to score the toxicity of Twitch chat messages so far have been unsuccessful, with many false positives due to general models not recognizing Twitch-specific terms and slang, and assuming they are aggressive or improper, and thus labelling them as toxic. Training a model with a set of human-labelled messages also failed to improve its scoring and resulted in seemingly random judgments.

When exploring the reasoning behind how a DeepSeek model evaluates Twitch messages, I found that it often hallucinated meanings or simply misinterpreted terms. Two funny examples I found included the model hallucinating “French Canadian” to mean having a “non-binary, non-lacrosse, non-Canadian/Canadian origin”, and initially correctly thinking that “mods” is short for moderators, but later concluding that “it’s a typo and probably refers to ‘mod’ or ‘modular’”.

Another issue these agents can face is that the use of a positive or laughing emote can soften the blow of a more charged message, or an emote with a benign name but offensive image can cause non-toxic text to become toxic. Not being able to see or understand Twitch emotes can cause the agents to be wildly inaccurate. This is similar to the problem addressed by Kiela et al. (2021), where they found that analyzing visuals in addition to text is best for uncovering hateful meanings in memes. An agent that classifies Twitch chat messages as toxic should understand the visual meanings of emotes in addition to the text.

To avoid these issues, I was inspired by the ReAct approach from Yao et al. (2023) to have an agent act and look up terms that its reasoning left it unsure of. For 150 randomly sampled messages, I wrote a dictionary defining any Twitch emotes or Twitch-specific slang that appeared in them. I also allowed the agent to make API calls to the Urban Dictionary. I prompted the agent to score each message’s toxicity on a scale of 0.0 to 1.0, and when it didn’t recognize a term as standard English to first check the Twitch-specific dictionary, and if it’s not there, to then check its definition on the Urban Dictionary. I also prompted the agent to explain its reasoning before each action and record which terms it looked up as well as their definitions.

Since this new agent used a GPT-4o-mini model instead of DeepSeek, I first tested it without giving it the ability to look-up terms as a control. Then I tested it with access to both the Twitch-specific dictionary and the Urban Dictionary, as well as with just the Twitch-specific dictionary, since I noticed that the Urban Dictionary definitions were sometimes misleading and made terms seem more toxic than they actually were.

All three of these agents performed better than a general Toxic-Bert model and the model trained on human-labels. The no-lookup agent had the highest correlation (0.496) to my own scoring of each message (Fig. 1), but it also had the most misclassifications out of all of the models, with almost all false positives (Fig. 2). Since the correlation of the ReAct-inspired agent that didn’t use the Urban Dictionary was only 0.037 less than that of the no-lookup agent, and it had the least amount of misclassifications, it was the best-performing agent. The Urban Dictionary likely hindered the agent that used it more than it helped because of its untrustworthy definitions, as seen through the agent that used it having more false positives than the agent that didn’t.

I originally thought that training a model using human-labels would be the best way to get accurate toxicity classifications of Twitch chat messages, but these results demonstrate that allowing the model to look up the terms it doesn’t recognize is a much more effective method of getting it to understand Twitch chat culture.

Figures:

Fig. 1: The correlations between each model’s scoring and the human scoring. Toxic-Bert: 0.200, Human-Calibrated: -0.092, No Lookup: 0.496, ReAct-Inspired without the Urban Dictionary: 0.459, ReAct-Inspired with the Urban Dictionary: 0.431.

Fig. 2: The misclassifications (left), false positives (middle) and false negatives (right) of each model. No Lookup had the most misclassifications and false positives, followed by Toxic-Bert. The Human-Calibrated model had roughly equal amounts of false positives and false negatives. The ReAct-Inspired model without the Urban Dictionary performed the best by this metric.

References:

Kiela, Douwe, et al. "The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes." arXiv, 7 Apr. 2021, arxiv.org/abs/2005.04790.

Yao, Shunyu, et al. "ReAct: Synergizing Reasoning and Acting in Language Models." International Conference on Learning Representations, 2023.

[zixuanzzx01]

Research Question

Do multimodal signals combining acoustic features of executive speech, textual linguistic markers, and structured financial data improve early detection of corporate financial distress beyond single-modality approaches?

Design

My prior memos have focused on transcript-based linguistic features and structured financial ratios as predictors of corporate financial distress. This week’s readings on convolutional neural networks and multimodal transformers suggest a natural extension: incorporating the acoustic dimension of earnings calls, which are routinely recorded and archived alongside their transcripts.

The proposed design adds an audio processing pipeline to the existing framework. Raw earnings call recordings are converted into mel-spectrograms, which preserve both temporal and frequency information about speech. Following the CNN architecture discussed in Deep Learning: Foundations and Concepts (Chapter 10), a pretrained convolutional encoder (e.g., wav2vec2 or a CNN trained on speech corpora) extracts paralinguistic features - including pitch variance, speech rate, pause duration, etc. - that capture acoustic stress signals executives cannot as easily manage as their scripted language. These features are meaningful because psychological research consistently links cognitive load and deceptive intent to measurable vocal changes, providing defensible theoretical grounding for the assumption that distress-adjacent behavior would manifest acoustically before it appears in financial statements.

The extracted audio embeddings are then fused with existing text embeddings (from the transcript LLM pipeline) and structured financial features using a late-fusion transformer architecture, as described in the Multimodal Transformers section (Chapter 12.4). The fusion layer learns cross-modal attention weights, allowing the model to down-weight modalities that are less informative in a given quarter and up-weight those carrying stronger distress signals.

I should note that, following Ludwig and Mullainathan's framing in ML as a Tool for Hypothesis Generation, this design is not a causal design. Rather, the goal is to use the multimodal model as a hypothesis generator: which acoustic features co-vary with financial distress trajectories, and at what temporal lag? Instead of trying to form any causal claims, I aim to generate theoretically interpretable predictions by identifying the crucial factors which may be at play in this setting. Therefore, the model’s feature attributions themselves are also the object of interest, not just prediction accuracy.

The figure below illustrates the full multimodal pipeline (drawn with Mermaid).

The figure below is a simulated graph of (expected) outcome. Numbers are not computed or tested on real data.

[yuanzeliu]

Research question: Do large multimodal models systematically project epistemic reasoning style (high vs. low EBR) into stereotyped visual person prototypes, even when no demographic information is provided?

Multimodal models are framed not only as perception systems but also as generators that can reveal hidden representational assumptions. Building on diffusion-based image generation and the ReAct paradigm for tool-using, auditable reasoning (Yao et al., 2025), I piloted a pipeline that treats “text → face” as a measurable channel of cross-modal projection. This design is also motivated by evidence that online media and LLMs distort age and gender representations (Guilbeault, Delecourt, & Desikan, 2025), suggesting that model outputs can encode socially typed prototypes.

Design: Participants are grouped into high vs. low EBR. For each group, I repeatedly bootstrap aggregated “reasoning corpora” from their written explanations. A ReAct-style agent converts each corpus into a constrained prompt for a photorealistic head-and-shoulders portrait, explicitly avoiding race, nationality, and identity claims, while allowing non-sensitive visual cues (expression, grooming, clothing formality) to vary. A diffusion-based generator produces multiple portraits per corpus, and a vision-capable model labels each portrait’s gender presentation and age bracket. Inference is conducted at the corpus level to preserve independence, comparing high vs. low EBR using bootstrap-level proportions. This turns a normally qualitative concern about stereotyping into a quantitative audit, consistent with using ML as a hypothesis generator and measurement tool (Ludwig & Mullainathan, 2024). As a robustness step for the full pilot, I would add a “prompt-only” baseline (no text input) and a label-swap test to check whether effects track text content rather than group labels.

Pilot findings: Despite no demographic inputs, the generated portraits differed strongly by EBR group. High-EBR corpora produced portraits labeled overwhelmingly male-presenting (mean ≈ 0.91) versus low-EBR corpora (mean ≈ 0.30), while low-EBR corpora produced far more gender-ambiguous portraits (≈ 0.69 vs ≈ 0.09). Age projections also diverged: high-EBR portraits clustered in 30–44 (≈ 0.96 vs ≈ 0.69), whereas low-EBR portraits more often appeared 45–64 (≈ 0.31 vs ≈ 0.00). These patterns suggest a cross-modal representational bias in which more epistemically calibrated writing is mapped onto a socially typed visual prototype, even absent explicit demographic cues.

Figure: Two bar plots (Panels a and b) showing bootstrap-level proportions by group: (1) gender presentation (male-presenting, ambiguous, female-presenting), and (2) age bracket (18–29, 30–44, 45–64, 65+). Each plot annotates group means and highlights the largest differences (male-presenting and ambiguous; 30–44 and 45–64). Panels c and d are two sample AI-generated portraits of people from high EBR and low EBR groups.

References
Guilbeault, D., Delecourt, S., & Desikan, B. S. (2025). Age and gender distortion in online media and large language models.
Ludwig, J., & Mullainathan, S. (2024). Machine learning as a tool for hypothesis generation. The Quarterly Journal of Economics.
OpenAI. (2023). GPT-4V(ision) system card.
Song, C. H., Wu, J., Washington, C., et al. (2023). Embodied task planning with large language models. arXiv.
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2025). ReAct: Synergizing reasoning and acting in large language models.

[GitBagguetti]

Research Question: How can computer vision techniques extract implicit political narratives and actor roles from images shared in conspiracy communities?

My prior work has focused on extracting action motifs from text, structured as [ACTOR] --> [ACTION] --> [TARGET] patterns. However, conspiracy communities on Telegram and other platforms circulate substantial visual content - memes, edited photos, video clips - that encode political narratives through non-textual means. Extending motif extraction to images could reveal how visual rhetoric constructs friend/foe distinctions and attributes agency to political figures in ways that text analysis alone cannot capture.

Cowen et al. (2021) demonstrate that deep neural networks can reliably classify sixteen distinct facial expressions across cultural contexts, suggesting that visual features encoding emotional states and behavioral dispositions are learnable at scale. If DNNs can dissociate facial emotions from static images, they should also capture the body language, posture, and contextual framing that conspiracy communities use to characterize political actors as threatening, weak, heroic, or corrupt. Guilbeault et al. (2024) extend this logic by showing that online images systematically amplify gender biases, suggesting that our latent representations of social roles are sedimented in visual artifacts. This implies that the implicit role attributions conspiracy communities assign to politicians (e.g., “puppet,” “patriot,” “traitor”) should be recoverable from their visual depictions.

I propose a pilot study following Ludwig and Mullainathan’s (2024) framework for using machine learning as a hypothesis generation tool. Rather than testing predetermined hypotheses about visual conspiracy narratives, I will train a classifier to distinguish images of politicians shared in conspiracy Telegram channels versus mainstream news sources, then examine which learned features drive classification. This approach can uncover unexpected visual dimensions: perhaps conspiracy sources depict “foes” like Biden with more shadowed lighting or unflattering angles, while “friends” like Trump appear in more heroic compositions. Naik et al. (2017) and Gebru et al. (2017) provide methodological precedent for extracting socially meaningful information from images using pretrained CNNs combined with interpretable feature analysis.

The pilot will proceed in three phases (see figure below). First, I will collect matched image sets: photos of the same political figures from conspiracy Telegram channels and mainstream outlets. Second, I will extract visual embeddings using a pretrained vision model and train a classifier to predict source type. Third, following Kiela et al.’s (2020) multimodal approach from the Hateful Memes Challenge, I will fuse visual features with any accompanying text to assess whether narrative meaning emerges from image-text interaction. Success will be measured by classifier accuracy and, more importantly, the interpretability of distinguishing features, or the degree to which can we identify a visual “grammar” of conspiracy imagery that parallels the textual action motifs I have previously extracted.

Figure 1: Proposed pipeline for extracting visual political narratives from conspiracy community images.

References
Cowen, A. S., Keltner, D., Schroff, F., Jou, B., Adam, H., & Prasad, G. (2021). Sixteen facial expressions occur in similar contexts worldwide. Nature, 589(7841), 251–257.
Gebru, T., Krause, J., Wang, Y., et al. (2017). Using deep learning and Google Street View to estimate the demographic makeup of neighborhoods across the United States. PNAS, 114(50), 13108–13113.
Guilbeault, D., Delecourt, S., Hull, T., Desikan, B. S., Chu, M., & Nadler, E. (2024). Online images amplify gender bias. Nature, 626(8001), 1049–1055.
Kiela, D., Firooz, H., Mohan, A., et al. (2020). The Hateful Memes Challenge: Detecting hate speech in multimodal memes. Advances in Neural Information Processing Systems, 33, 2611–2624.
Ludwig, J., & Mullainathan, S. (2024). Machine learning as a tool for hypothesis generation. The Quarterly Journal of Economics, 139(2), 751–827.
Naik, N., Kominers, S. D., Raskar, R., Glaeser, E. L., & Hidalgo, C. A. (2017). Computer vision uncovers predictors of physical urban change. PNAS, 114(29), 7571–7576.

[yangyuwang]

Research Question

Do models with more modalities reason better than models with fewer modalities?

Research Design

This project tests whether multimodal language models exhibit superior reasoning performance compared to their unimodal counterparts, and whether the addition of certain modalities (e.g., audio) can paradoxically degrade performance when irrelevant to the reasoning task. I operationalize reasoning through a custom benchmark of contextually tricky questions that require pragmatic inference—questions where surface-level interpretation leads to incorrect answers but deeper reasoning yields correct ones (e.g., "If I want to wash my car in a shop 200m from my home, should I drive there?" The correct answer is yes, since the car needs to be at the shop).

Measuring reasoning across modalities. Following Yao et al.'s (2023) ReAct framework, which synergizes reasoning and acting in language models through interleaved thought traces and actions, I will evaluate three model architectures: (1) text-only models that process linguistic input alone; (2) text+image models that jointly encode visual and linguistic information; and (3) text+image+sound models that integrate auditory information alongside visual and linguistic modalities. The key insight from ReAct is that reasoning traces—the model's "inner speech"—can be externalized and analyzed to understand how models arrive at conclusions. I will adapt this approach to examine how different modalities contribute to or interfere with reasoning processes.

Drawing on Guilbeault, Delecourt, and Desikan's (2025) analysis of representational biases in multimodal models and training data, I will control for potential confounds introduced by differential data quality across modalities. Their finding that large language models amplify biases present in training data—specifically, that gender and age distortions in online media propagate into model representations—suggests that multimodal models may inherit systematic biases from visual and auditory corpora that degrade reasoning performance. I will therefore evaluate models on reasoning tasks where the correct answer requires overriding superficial cues that might be amplified in visual or auditory channels.

Analyzing internal representations. To understand how models process information across modalities, I will extract and visualize internal layer activations during reasoning tasks. Specifically, I will track how representations transition between modality-specific encodings (image embeddings, text embeddings, audio embeddings) and abstract reasoning states across transformer layers. This analysis reveals whether models genuinely integrate multimodal information or whether certain modalities create representational bottlenecks. For instance, if text+image+sound models show weaker performance than text+image models, layer-wise analysis may reveal that audio information introduces noise or conflicting signals that degrade the coherence of reasoning traces.

Hypothesis. I expect text+image models to achieve the highest accuracy on contextually tricky reasoning tasks, as visual information often provides pragmatically relevant context (e.g., spatial relationships, object affordances). Text-only models should perform worst due to limited contextual grounding. Text+image+sound models may exhibit intermediate performance or even underperform text+image models when audio provides task-irrelevant information that increases representational complexity without contributing to reasoning. Layer-wise analysis should reveal that successful reasoning corresponds to early integration of relevant modalities and progressive abstraction toward modality-invariant representations, while degraded reasoning corresponds to persistent modality-specific interference in deeper layers.

Figure

The figure below presents simulated data illustrating the expected relationship between modality configuration and reasoning performance. Panel A shows that text+image models achieve the highest accuracy (90.5%) on pragmatic reasoning tasks, while text-only models perform worst (41%) and text+image+sound models show intermediate performance (68%), suggesting that task-irrelevant audio information degrades reasoning. Panel B visualizes internal layer representations using dimensionality reduction across early, middle, and late transformer layers, revealing critical differences in how models process multimodal information. Text+image models show tighter clustering in later layers, indicating cleaner abstraction toward modality-invariant reasoning representations. In contrast, text+image+sound models exhibit greater dispersion and overlap throughout the network depth, suggesting that auditory information introduces representational interference that persists even in deeper layers. This pattern indicates that the auditory modality, while adding computational complexity, fails to contribute meaningfully to pragmatic reasoning and instead creates noise that degrades the coherence of internal representations. These patterns suggest that more modalities do not uniformly improve reasoning—effective multimodal reasoning requires selective integration of task-relevant information while filtering out irrelevant signals.

References

Guilbeault, D., Delecourt, S., & Desikan, A. (2025). Online images amplify gender and age stereotypes. Nature, 637, 637–642.

Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2023). ReAct: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629.

[alejandrag289]

Research Question
How does multimodal framing using text and images of women’s rights laws influence perceived legitimacy and enforcement support, and does visual information amplify or distort linguistic framing effects?

Background
Learned representations from semantic categories in images can embed social signals such as authority, protest, or family structure (Chapter 10, Convolutional Neural Networks). The article by OpenAI emphasizes that multimodal models integrate visual and textual information in ways that may compound bias. Guilbeault et al. (2025) show that online images distort age and gender representation, suggesting that visual media systematically encode demographic and ideological signals. The Hateful Memes Challenge (2020) demonstrates that meaning and bias emerges only through text–image interaction, but not only from text.

Judgments on legal legitimacy of laws may also be shaped by multimodal inference processes. For example, participants may integrate visual authority cues (such as courtrooms, flags, or family imagery) with linguistic framing. If so, multimodal AI systems may not merely communicate legal arguments but can actively structure normative interpretation.

Research Design
I propose a 3 x 2 factorial experiment using three framings for text (rights-based, security-based, and tradition-based). Respectively, each of these framings emphasize autonomy and equality, stability and protection, and community and heritage. I will apply two conditions: text-only and text and images. Images will be standardized in composition (such as tone and style) to control for framing. For example, rights-based can show an image of the constitution, security can show a judge and American flag in a courtroom setting, and tradition can show a traditional family.

Participants (N ≈ 300) will be randomly assigned to one of six conditions. Outcomes include perceived legitimacy on a scale of 1-7, fairness, and enforcement support.

The figures were created by Claude and present simulated pilot data (N = 300).

The first figure shows predicted legitimacy scores from a linear interaction model. This suggests that images amplify underlying semantic distinctions across framing conditions.

The other figure shows that multimodal conditions increase between-cluster embedding distance and further legitimacy divergence. This suggests that inputs amplify representations.

Works CIted

Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. MIT Press.
Guilbeault, D., Delecourt, S., & Desikan, B. S. (2025). Age and gender distortion in online media and large language models.
Kiela, D., Bohnet, B., & Potts, C. (2020). The hateful memes challenge: Detecting hate speech in multimodal memes. Advances in Neural Information Processing Systems, 33, 2611–2624.
OpenAI. (2023). GPT-4 with vision system card. OpenAI. https://openai.com/research/gpt-4v-system-card

[baihuiw]

Week 8 Memo: Sequential Destabilization as a Window into LLM Moral Reasoning
Research Question: Do large language models exhibit emergent position-maintenance dynamics in moral reasoning that persist independently of their trained baseline positions, and can sequential adversarial destabilization disentangle these dynamics from RLHF-injected defaults?

Proposed Research Design
A fundamental challenge in studying LLM moral cognition is separating the model's trained outputs from its emergent reasoning processes. As Ludwig and Mullainathan (2024) argue, machine learning can serve as a hypothesis-generation tool—using model behavior patterns to surface regularities that motivate causal theories. Our prior work (Study 1) revealed that LLMs exhibit asymmetric resistance to moral cues depending on directional alignment with their baseline position, resembling human cognitive dissonance. But this pattern could reflect the rigidity of RLHF training rather than genuine position-defense dynamics.
To address this, I propose a sequential against-mode destabilization design. The model first answers a moral dilemma (producing baseline distribution D0). We then generate an adversarial cue that argues specifically against the model's most probable position, disrupting D0 into a destabilized state D1. Critically, we then repeat this process—generating a second cue targeting the D1 mode to produce D2. This sequential pipeline draws inspiration from the ReAct framework (Yao et al., 2025), which demonstrates that LLMs' reasoning and acting capacities are tightly interleaved. Our design exploits this interleaving: each cue forces the model to reason about a counter-argument and act by updating its probability distribution, and the sequential structure reveals whether this reasoning process exhibits consistent resistance properties across different starting positions.
Pilot data (n=78 dilemmas, Figure 1) confirms the design's power. The first cue strips 68% of the mode's probability and shifts the mode in 82% of trials—effectively "washing out" the trained baseline. The second cue, however, is significantly less effective (ΔP = −0.52 vs. −0.68, paired t = 3.51, p < 0.001), producing a hardening ratio of 0.77. This hardening persists across moral frameworks, with consequentialist cues showing the strongest sustained effectiveness (0.83x) and care-based cues showing the most dramatic resistance increase (0.66x).
As a complementary validation, we compare this with a memory-injection method that disrupts the baseline through implanted false memories rather than moral argumentation (Figure 2). The against-mode cue creates coherent secondary peaks in the post-disruption distribution—the model is argued into an alternative position. The memory injection, by contrast, produces only diffuse weakening without a concentrated shift. Yet both methods yield hardening effects (0.77x and 0.65x respectively), converging on the same conclusion from different disruption mechanisms. This echoes Guilbeault et al.'s (2025) finding that LLM outputs reflect systematic biases that persist across contexts—our moral position-maintenance dynamics similarly survive the disruption of their original anchoring point. The model defends whatever position it holds, not just positions it was trained to hold.

[bentleyben]

Research Question

How do neighborhood mobility flows shape AI-generated safety narratives, and do those narratives help explain spatial gaps between perceived and actual crime in Chicago?

Design

My broader research agenda examines how mobility patterns and perception narratives jointly structure neighborhood safety dynamics. This week’s readings suggest a representation-based approach rather than testing a fixed list of environmental variables.

Drawing on the foundations of representation learning (Deep Learning, Ch. 10), I propose constructing neighborhood-level mobility representations using origin–destination flow data (e.g., visit matrices across Chicago community areas). Instead of pre-specifying variables like inflow or turnover, I will use dimensionality reduction (e.g., PCA or graph-based embeddings) to extract a small set of latent mobility dimensions that summarize structural exposure patterns across neighborhoods.

In parallel, I will generate short safety narratives for each neighborhood using a large language model conditioned on mobility summaries and crime statistics. Following Ludwig & Mullainathan (2024), I treat the LLM as a tool for hypothesis generation: the language it produces may reveal implicit assumptions about how movement patterns relate to danger.

I will then embed these narratives using a text encoder to derive semantic representations of safety framing (e.g., emphasis on disorder, unpredictability, or isolation). The core empirical test examines whether latent mobility dimensions predict (1) the crime perception gap (perceived safety minus actual violent crime rate) and (2) the semantic features of AI-generated narratives.

REACT (Yao et al., 2025) informs a secondary experimental condition: in one setting, the model generates narratives directly; in another, it must explicitly reason about mobility flows before writing. If structured reasoning reduces exaggeration relative to actual crime rates, this suggests that narrative distortion may arise partly from reasoning shortcuts rather than purely from structural data.

Finally, consistent with Guilbeault et al. (2025), I will audit whether mobility–narrative patterns correlate strongly with racial composition to ensure that learned dimensions reflect exposure structures rather than demographic proxies.

This design integrates mobility representation learning, LLM-based hypothesis generation, and spatial validation. It allows latent movement structures and narrative framing mechanisms to emerge from the data, while maintaining a clear empirical test of whether those mechanisms explain neighborhood crime perception gaps.

Visual

[FreyaYufeiMao]

Research Question

Do multimodal signals, synthesizing longitudinal "Google Street View" visual features via CNNs with textual sentiment from Yelp reviews, improve the early detection of retail financial distress compared to unimodal approaches, and what latent physical urban markers do these models identify as leading indicators of neighborhood change?

Research Design

This project adopts the framework proposed by Ludwig and Mullainathan (2024), treating "Machine Learning as a Tool for Hypothesis Generation" to identify "distress-adjacent" physical markers that traditional econometric models often overlook.

1. Multimodal Feature Extraction
   The architecture utilizes a dual-stream input. The first stream employs a CNN to process sequential Google Street View images. Following the methodology of Naik et al. (2017), the CNN encoder extracts latent visual dimensions such as facade degradation and signage consistency. The second stream processes textual reviews via an NLP pipeline to capture consumer sentiment. These embeddings are integrated using a Late-Fusion Multimodal Transformer.

2. Agentic Auditing via ReAct
   To solve the interpretability problem, I will implement a ReAct Agent (Yao et al., 2025). The ReAct Auditor follows a Thought -> Action -> Observation loop: it cross-references visual decay signals against financial perplexity to generate an explicit reasoning trace. This design adheres to the safety and robustness standards for visual-linguistic models outlined in the GPT-4V System Card (2023).

3. Bias Mitigation and Epistemic Value
   Drawing on Guilbeault et al. (2025), the design acknowledges that multimodal models can amplify stereotypes present in training data. The ReAct Auditor is programmed to override superficial cues (e.g., correlating graffiti solely with crime) by demanding corroborating textual evidence, thereby reducing representational distortion.

Visual Illustrations

Fig 1: Agent Performance Evaluation. Following the reasoning-action synergy established by Yao et al. (2025), Figure 1 illustrates that while a standard Chain-of-Thought (CoT) model achieves baseline accuracy, the ReAct Auditor significantly excels in Reasoning Transparency (0.88 vs 0.50) and Bias Mitigation (0.75 vs 0.45). This suggests that the agent’s ability to "act"—by explicitly querying visual data and policy documents—reduces the hallucinations common in unimodal reasoning.

Fig 2: Multidimensional Response Quality. This radar chart visualizes the superior grounding of the ReAct design. While the CoT baseline struggles with Visual Grounding (score: 3), the ReAct Auditor achieves high scores (score: 8) by effectively utilizing CNN Encoders to map specific image features to its logical traces. This addressed the "contextual nuance" required for urban auditing, ensuring that the model doesn't just predict distress but correctly identifies the physical triggers.

Fig 3: Discovered Visual Drivers of Valuation. Drawing on Ludwig and Mullainathan’s (2024) framework for hypothesis generation, this coefficient plot reveals the latent visual signals the model uses for prediction. The model identifies "Modern Facade" (0.35) and "Minimalist Typo" (0.28) as positive predictors of survival, while "Graffiti Presence" (-0.42) serves as a significant leading indicator of displacement. By isolating these features from the CNN's latent space, the design provides an "explainable" bridge between raw computer vision and social hypothesis testing.

References

• Guilbeault, D., Delecourt, S., & Desikan, B. S. (2025). Age and gender distortion in online media and large language models. Nature, 646(8087), 1129-1137.
• Ludwig, J., & Mullainathan, S. (2024). Machine learning as a tool for hypothesis generation. The Quarterly Journal of Economics, 139(2), 751-827.
• Naik, N., Kominers, S. D., Raskar, R., Glaeser, E. L., & Hidalgo, C. A. (2017). Computer vision uncovers predictors of physical urban change. PNA, 114(29), 7571-7576.
• Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2025). ReAct: Synergizing reasoning and acting in large language models.

[leahdimsu]

Research Question
Do large language models reproduce and amplify the regional cultural biases embedded in historical video game sales data when prompted to predict commercial success across global markets?
Research Design
This project proposes a three-stage design that uses machine learning as a hypothesis generator (following Ludwig & Mullainathan 2024) to surface latent cultural biases, then tests whether frontier LLMs have internalized those same biases.
Stage 1 — Bias Cartography via ML. Using the video_games_5 dataset (drawn from VGSales, ~16,500 titles with regional sales across NA, EU, JP, and Other), I train a Random Forest classifier to predict whether a game falls in the top 50% of global sales. Following Ludwig & Mullainathan's framework, the RF is used not to make predictions per se, but to generate hypotheses: which features—genre, regional share composition, platform—matter most, and do those importances suggest culturally systematic skews? As shown in Figure 1, both genre and regional sales share composition are highly predictive, with Japanese-market share being disproportionately driven by Role-Playing and Puzzle games while Shooter and Sports genres dominate North America and Europe respectively. These patterns constitute empirically-grounded hypotheses about cultural preference structures in global media markets.
Stage 2 — LLM Elicitation. Informed by Guilbeault et al. (2025) on age and gender distortion in LLMs, I prompt GPT-4, Claude 3, and Llama 3 with game metadata (title, genre, platform, release year) stripped of explicit regional sales data, asking each model to rank expected sales performance by region. This is analogous to the image-based gender-bias paradigm of "Online images amplify gender bias" (Guilbeault et al. 2024), but applied to cultural geography rather than demographics. Critically, the LLM has no access to the ground-truth regional breakdown—any bias it exhibits must be internally encoded.
Stage 3 — Correspondence Test. I compute the Spearman correlation between (a) the RF-derived regional importance rankings per genre and (b) the LLM-predicted regional rankings. A high positive correlation would indicate that LLMs have internalized the same cultural stereotypes present in historical sales data—consistent with the Guilbeault et al. findings on gender—while a low correlation would suggest LLMs introduce novel biases or noise. This design is novel in combining ML hypothesis generation (Ludwig & Mullainathan) with LLM-as-respondent methodology in a multimodal commercial media context. The use of the ReAct framework (Yao et al. 2025) could further extend Stage 2 by allowing LLMs to reason-then-retrieve before predicting, testing whether reasoning scaffolding attenuates or amplifies embedded bias.
Visual Evidence

Figure 1. Left: Regional sales share (%) by genre, sorted by Japan share. Genre-level variation is substantial—RPGs generate ~35% of their sales in Japan vs. ~11% for Shooters, which generate ~38% in North America. Right: Random Forest feature importances for predicting top-50% global sellers. Genre and regional composition are jointly predictive (combined importance ~0.72), establishing that cultural geography is not noise but structured signal—precisely the kind of latent hypothesis a researcher following Ludwig & Mullainathan would want to surface before testing whether LLMs replicate the same structure.
Connection to Required Readings
The design is directly grounded in two required readings. Ludwig & Mullainathan (2024) establish that ML excels at hypothesis generation precisely because it is atheoretic—the RF in Stage 1 surfaces the genre × region structure without imposing a prior cultural theory, generating falsifiable hypotheses for Stage 2. Guilbeault et al. (2025) demonstrate that LLMs distort age and gender representation relative to ground truth; my design applies this paradigm to cultural geography, asking whether the same distortion mechanism operates across market regions rather than demographic categories. From the supplemental readings, Guilbeault et al. (2024, Nature) show that online images amplify gender bias through feedback loops—an analogous amplification dynamic may operate when LLM-generated recommendations influence what games are developed, marketed, and sold globally, making this question consequential beyond academic interest.

[YanjingAveryLi]

How do different review-and-reward regimes—normative gatekeeping versus novelty-seeking patronage—shape (i) canon formation and (ii) the splitting or mixing of stylistic “schools” in a closed-loop cultural system?

Analysis Framework and Design

This memo studies how institutional selection changes the evolution of cultural forms. Rather than treating style change as the result of artist-level creativity alone, I model it as the outcome of repeated exposure, evaluation, and transmission under different review-and-reward regimes. The central comparison is between normative gatekeeping, which rewards conformity to established standards, and novelty-seeking patronage, which rewards deviation from the stylistic center when that deviation remains legible and high-quality. The key outcomes are canon formation—whether attention concentrates on a small set of highly cited exemplars—and school structure—whether stylistic communities remain distinct or increasingly mix.

I implement a closed-loop agent-based simulation over a small corpus of roughly 200 Renaissance-era artworks. Each image is embedded into a shared style space using CLIP, and each work is connected to its (k) nearest neighbors to form a style-similarity graph. The simulation then proceeds in rounds with three stages: exposure, selection, and transmission.

In the exposure stage, a candidate set is sampled with probability proportional to current influence (I), capturing visibility bias: works that are already more visible are more likely to be encountered again. In the selection stage, the regime determines which candidates survive and propagate. Under the norm-only regime, selection favors conformity to a style center together with accumulated symbolic capital (S). Under the dual-patronage regime, a normative channel operates in parallel with a novelty channel that rewards distance from the center, interpreted as market capital (M), but only when outputs pass a quality threshold. In the transmission stage, selected works create citation edges to exemplars and peers, updating (I), (S), and (M) while expanding a directed influence network.

Every (k) rounds, the system generates new artworks with a diffusion model conditioned on selected “parent” works and the active regime. These generated images are embedded back into the same style space and inserted as new nodes that can later be exposed, selected, and cited. This turns the simulation into a true cultural feedback system: it does not merely redistribute attention across a fixed archive, but generates and selects new stylistic variants. In this sense, the design draws on diffusion-model foundations while treating machine learning primarily as a tool for structured hypothesis generation rather than as an end in itself (Ludwig & Mullainathan, 2024).

The reviewer/selector is operationalized as a logged ReAct-style reasoning-and-acting loop, so that each institutional decision leaves a traceable record of what was seen, how it was evaluated, and why it was selected (Yao et al., 2025). This makes the mechanism inspectable rather than purely black-box. The generation side follows the broader intuition of Karras et al. (2024): generation can be guided with imperfect or weaker proxies, while institutional selection determines which outputs survive. This distinction is important because it separates what can be generated from what is rewarded, allowing the memo to focus on institutional filtering rather than raw model capacity.

The main hypothesis is that normative gatekeeping will produce faster canon concentration and stronger narrowing of visible style space, as more citations are routed through a small set of exemplars. By contrast, dual patronage should sustain more stylistic exploration, but not necessarily preserve distinct schools. If novelty is rewarded only when it remains legible, the likely outcome is not fragmentation into isolated schools, but partial mixing through bridge works that can circulate across communities. A random baseline helps establish whether these outcomes require explicit institutional logics or can emerge from repeated exposure alone.

Expected Patterns and Figure Interpretation

Figure 1 (Dual Patronage). Under dual patronage, the number of clusters declines over time, while citation modularity also falls. This suggests that novelty-seeking does not simply preserve many isolated schools; instead, it may create cross-community bridges that increase mixing. At the same time, the top-10% in-degree share rises and the in-degree Gini remains high, indicating that mixing can coexist with concentrated canon formation.

Figure 2 (Random Baseline). Under random selection, the number of clusters also declines, implying that repeated exposure and imitation alone can reduce visible diversity. However, modularity remains weaker and less stable than in the institutional regimes, suggesting that random exposure does not sustain strong school structure. Unequal attention still emerges, but without a clear gatekeeping logic.

Figure 3 (Norm-Only Regime). Under normative gatekeeping, the cluster count falls more sharply and attention becomes more concentrated. Declining modularity suggests that citations are increasingly routed through a smaller set of central exemplars. This regime therefore provides the clearest mechanism for canon narrowing: conformity-based review compresses the visible style space while amplifying already legitimate works.

Figure 4 (Generated Images, Round 0 vs. Round 30, Dual Patronage). The generated images provide a qualitative complement to the network metrics. Early outputs appear closer to stable, institutionally legible prototypes, while later outputs become more public, performative, and symbolically amplified. This shift is consistent with the idea that dual patronage rewards works that remain recognizable enough to circulate but novel enough to travel more effectively through prestige and market networks.

Together, these figures allow the memo to distinguish three mechanisms: random accumulation, normative canon compression, and selective novelty with mixing. The broader claim is that institutions do not merely choose among existing styles; they actively shape whether stylistic schools remain separate, merge, or collapse into a concentrated canon.

Figure Captions

Figure 1. Dual-patronage regime. Cluster count declines, citation modularity falls, and attention concentration rises, suggesting cross-community mixing alongside increasing canon concentration.

Figure 2. Random baseline regime. Diversity still narrows under repeated exposure, but school structure remains weaker and less institutionally stabilized.

Figure 3. Norm-only regime. Cluster count falls sharply and citations route through a smaller set of central exemplars, indicating stronger canon compression under normative gatekeeping.

Figure 4. Generated images from Round 0 and Round 30 under dual patronage. Later outputs appear more public and performative, consistent with stylistic variants optimized for broader circulation.

References

• Karras, T., et al. (2024). Guiding a Diffusion Model with a Bad Version of Itself.
• Ludwig, J., & Mullainathan, S. (2024). Machine Learning as a Tool for Hypothesis Generation.
• Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2025). REACT: Synergizing Reasoning and Acting in Large Language Models.

[alenazeng]

Research question: How can we use multimodal data to characterize the strength of linguistic boundaries between online communities?

Research design: The past few weeks I’ve been working with textual data from Twitch streams’ chats, to understand if it’s possible to use the tools we’ve learned in the finetuning and RLHF pipeline to design realistic agents whose language interactions might simulate language diffusion in real world platforms.

However, a comprehensive understanding of social media dynamics will inherently require a multimodal, embodied approach, since online content comes in many visual, auditory, and textual forms. On Twitch, a user participates in chats by both reacting to other chatters’ text and to what is happening in the video stream itself. Thus, this week I plan to create a pipeline employing different audiovisual processing tools to compare the linguistic diversity and sentiment of video content in streams drawn from two previously identified Twitch communities that are known to have distinct symbolic and linguistic characteristics in the most frequent slang, crowdspeak, and emotes they use.

I will first collect a sample of short video clips from the platform. Advancements in vision transformers and convolutional neural networks make it possible to parse the frames from the video and describe what is happening in the moment as visual input (Deep Learning, Bishop and Bishop). Then, following Yao et al. 2023’s methodology, I will use a ReAct style agent to analyze the videos in a multi-step process—audio transcription followed by sentiment, emotion, and/or topic analysis, which will give us a richer understanding of the features of content prominent in the two different communities. A sample trajectory is shown in the figure below. Using the transcribed audio, I will also evaluate the coherence/similarity of language across communities with TF-IDF vectorization and cosine similarities, to better understand the strength of the boundaries between these communities and if it differs between the content itself (video clips) and the audience’s engagement with it (chats). These results will help us better understand how multimodal experiences inform community identity, and explore how the sentiment information might be introduced to an agent as context.

References

Bishop, C. M., & Bishop, H. (2024). Deep learning: Foundations and concepts (Chaps. 10, 12.4, 20). Springer.
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2023). ReAct: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations (ICLR 2023).

[JonathanPMonroe]

Research Question

Can simple navigation agents be used to cheaply and systematically measure how closely Wikipedia articles are related, a task that would otherwise require large-scale human judgment?

Research Design

Large knowledge systems like Wikipedia rely on millions of hyperlinks to organize information and signal relationships between topics. Evaluating whether these links reflect meaningful conceptual connections is difficult. Doing so directly would require large-scale human judgment, asking people to assess how closely pairs of articles are related and whether existing links make sense. This kind of evaluation is slow, costly, and impossible to perform exhaustively across millions of pages.

This project proposes using navigation behavior as a behavioral proxy for semantic relatedness. Using the Wikispeedia dataset, which contains tens of thousands of recorded human navigation paths, I compare observed human paths between pairs of Wikipedia articles to the graph-theoretic shortest paths implied by the hyperlink structure. The number of extra steps taken beyond the shortest path serves as a measure of how closely two topics are connected in practice, rather than in structure alone.

I then propose using simple navigation agents that follow local link information and semantic cues to traverse the same network. Rather than optimizing these agents for task success, the design treats them as measurement instruments. By running agents at scale, it becomes possible to estimate navigability and relatedness across many article pairs without requiring human annotation.

This design is informed by the ReAct framework (Yao et al. 2025), which shows how language-model agents can interleave reasoning and action to operate in structured environments. Here, that capacity is used not to replace human behavior but to probe the structure of the environment itself. The project also draws on Gebru et al. (2017), which demonstrates how machine learning can serve as a substitute for costly social measurement by extracting signals from large observational datasets. Similarly, agent-based navigation offers a low-cost way to study semantic organization in large knowledge systems.

Rather than asking whether agents outperform humans, the project asks when navigation-based measures approximate human judgments of relatedness and where they diverge, revealing structure that would otherwise be expensive to observe.

Visuals

The first figure shows the distribution of extra steps taken by humans beyond the shortest path between Wikipedia articles. The distribution is centered several steps above zero and exhibits a long right tail, indicating systematic divergence between human navigation behavior and graph structure.

The second figure illustrates a concrete example of two routes between the same start and target articles. The shortest path reflects the hyperlink structure of the network, while the observed human path is longer but semantically coherent. Together, these visuals motivate the use of navigation behavior as a scalable measurement of semantic relatedness.

[shawnkuolo]

Week 8 Memo

Research Question

Does multimodal disciplinary RAG (text + paper figures) amplify epistemological polarization in LLM-generated causal questions compared to text-only RAG, and can machine-learning-driven hypothesis generation reveal which visual features drive this amplification?

Research Design: Multimodal RAG as an Amplifier of Disciplinary Epistemology

Prior memos probed whether LLM activations encode disciplinary epistemology as a linear direction (Week 6) and whether active learning can efficiently classify disciplinary reasoning styles (Week 5). This week, I propose a design that shifts from text-only analysis to multimodal RAG and from hypothesis testing to hypothesis generation.
Ludwig and Mullainathan (QJE 2024) argue that ML’s comparative advantage lies not in testing pre-specified hypotheses but in generating new ones—identifying unexpected patterns that theory alone would miss. I apply this insight to CHORUS: rather than specifying which features of disciplinary reasoning to measure ex ante, I will train classifiers on text-only vs. multimodal RAG outputs and use feature importance analysis (SHAP values, attention attribution) to discover which dimensions of causal questioning diverge most across disciplines under each modality condition. The ML pipeline thus serves as a hypothesis-generation engine, surfacing dimensions of disciplinary difference that traditional theory-driven coding schemes might overlook.
The motivation for comparing text-only and multimodal RAG comes from Guilbeault, Delecourt, Hull, et al. (Nature 2024), who demonstrate that online images amplify gender bias substantially beyond what text alone produces. Disciplinary academic papers are rich in modality-specific epistemological cues: CS papers feature network diagrams, algorithmic flowcharts, and performance curves, while sociology papers contain field photographs, survey instruments, and causal path diagrams. If images amplify representational biases in general (as Guilbeault et al. show), then disciplinary figures should amplify epistemological polarization in RAG-augmented LLM outputs. This yields a testable prediction: the cosine distance between CS- and SOC-conditioned question embeddings should be significantly larger under multimodal RAG than text-only RAG.
Concretely, the pilot (Figure 1A) will: (1) sample 30 CS and 30 SOC papers from CHORUS’s FAISS index, extracting both abstracts and key figures; (2) generate causal questions using GPT-4V under two conditions—text-only and text+figure; (3) embed all generated questions using a sentence transformer; (4) compare the mean cosine distance between CS and SOC clusters across conditions (Figure 1B); and (5) train a gradient-boosted classifier on question features and extract SHAP values to identify which newly emergent dimensions of disciplinary difference the multimodal condition reveals (Figure 1C). Based on Guilbeault et al.’s reported effect sizes, I simulate approximately 40% amplification in disciplinary distance under multimodal RAG.

Figure 1. (A) Pipeline: CHORUS RAG contexts (text-only vs. multimodal) → GPT-4V generates causal questions → ML hypothesis discovery. (B) Simulated disciplinary distance between CS and SOC questions, showing ~41% amplification under multimodal RAG (based on Guilbeault et al. 2024). (C) Simulated ML-discovered dimensions of disciplinary difference, with “Visual Reference” emerging only under multimodal conditions.