Questions for James Evans' Talk on "Scientific Misinterpretation in Policy-making"

[zixuanzzx01]

Questions for James Evans' Talk on "Scientific Misinterpretation in Policy-making"

Abstract
Evidence-based policymaking depends on the accurate interpretation of scientific research, yet partisan polarization has intensified concerns about the distortion of knowledge in pursuit of ideological agendas. Drawing on a large corpus of U.S. policy documents and the scientific articles they cite, we use a language-model approach to assess whether policy actors faithfully represent scientific findings. We find that think tanks are nearly twice as likely to misinterpret science as governments or intergovernmental organizations. These misinterpretations are strategically aligned with think tanks' policy positions, disproportionately target high-impact journals (impact factor ≥ 20), and are subsequently more likely to be cited by government documents. These dynamics show the structure of information laundering - the strategic diffusion of misinterpretations through long chains of indirect citation that embed distorted science into policy discourse. Replication with an embedding-based analysis of a larger sample confirms these patterns at scale. We show how the current Trump administration has "cut out the middleman", stopped citing think-tanks, and begun distorting science at think-tank levels for policy support, creating new complications of legitimacy. The findings highlight the distinctive role of partisan think tanks in shaping how misrepresented science circulates and gains legitimacy in evidence-based governance.

[jstanleyi]

If policy actors selectively misinterpret high-impact scientific research to advance ideological goals, can policymaking still meaningfully be described as “evidence-based”?

[dario-marino]

Can you do a breakdown of the think tanks and their characteristics to see the more and less accurate?

[ben-knell]

It might be interesting to use ML to classify most cited think tanks by how inaccurate they are over time and how their citations have changed as their accuracy has. If you find that more biased results leads to more government organizations citing it that could be useful confirmation.

[abikzer]

If info laundering happens through long chains of citations, should we be more skeptical of indirect citations in policy documents? I mean, how far back should accountability actually go?

[jwrampino]

Have you considered taking public observational data (i.e. public talks or panels) or structured interviews on how think tank researchers express or rationalize their own research process? In the same way people often filter their religious views through their politics, or vice versa, I would be curious how researchers narratively explain or justify these misrepresentations. It could be that they have not thoroughly read the studies they cite and have never been challenged on points of fact, or it may be they have read and legitimately read what they want out of it, or it may be that they purposefully curate a narrative that fits their worldview. While each is plausible given the data, it could very well be that the individual or institution-level answer to this varies.

[yifand1023]

Thanks for sharing! Where do you draw the boundary between policy translation and scientific misinterpretation in your measurement framework?

[mikezheng2001]

Professor Evans, your finding that the current Trump administration has 'cut out the middleman' to distort science directly is fascinating. Do you think this direct distortion carries the same level of 'legitimacy' or persuasive power in public discourse as the traditional method of 'laundering' information through ostensibly independent think tanks? Why it works: It engages directly with the most provocative and recent finding in the abstract, opening the floor for him to discuss the psychology and politics of credibility.

[shengyishu1]

Why are think tanks nearly twice as likely as governments to misrepresent scientific findings?

[yilinx-10]

Do we observe disproportionate levels of scientific misinterpretation in different policy areas? Which issue areas do we see more distortions/misinterpretations? Or no difference?

[larissaxcn]

How exactly does that “information laundering” work in practice, and why are think tanks so much more likely than governments to misinterpret research in the first place?

[bkehoechicago]

Is there a feasible solution to the selective misinterpretation of scientific studies? It seems clear that the average person has no interest in reading the studies directly, and so if a couple of think tanks misinterpret the science in different directions according to their biases what mechanism could be implemented to let people consistently know the truth?

[JonathanPMonroe]

I would be interested in whether the misrepresentation of policy information by thinktanks can be, at least partly, explained by the saliency of an issue. For example, think tanks may be more inclined to report on a policy position if the issue is particularly popular or well-known. We may see that, with less well-known policies and issues, think tanks may be subsequently less inclined to misrepresent the policy. There are a few hypotheticals that could support this trend. First, think tanks may feel pressured to report quicker on more salient, or popular, issues, leading to a rushed, misrepresented report of an issue. Second, think tanks may be pressured to consider their political affiliation, and are therefore incentivized to report their findings through a partisan lens, instilling bias into a process that should be grounded in theory and evidence. Either way, integrating issue saliency into your analysis as an additional dimension of inference may result in a more nuanced outlook on why think tanks are more likely to misrepresent information during the policy-creation process.

[Abenezer2]

When your model flags a policy passage as a ‘misinterpretation,’ how do you validate that it’s genuinely distorting the science rather than making a defensible normative leap? Concretely, what’s your strongest evidence that your misinterpretation score tracks truthfulness and not just difference in framing?

[YouLi0312]

How do you distinguish between genuine interpretive disagreement and strategic misinterpretation in your language-model approach?

[AkinEfeEser]

Thanks for sharing.
My question would be on what significance does the distortion you mentioned has an accumulated effect on the foresight of our predictions,it seems especially relevant for topics such as climate change predictions.