How to prevent LLM agents from taking part in jspsych experiments?

[trig11]

Hi,

is there any way to functionally prevent LLM agents (such as the ChatGPT agent feature) from taking part in jspsych-based experiments? I just instructed ChatGPT to take part in my own study, and while it is extremely slow working through the instructions, it managed to do so. It also did not struggle with html-keyboard-response inputs.

Has anybody else tried what is effective in keeping such agents from taking part and can provide any hints?

Thanks!

[jodeleeuw]

Hi @trig11,

Can you share some more details here about how the LLM interacted with the experiment? Did you point it at a link to the experiment? If you could share an example that could be helpful for thinking about this.

[trig11]

Hi Josh,

My prompt was the following (originally in German):

I have created an online study that I would like to conduct using Prolific. To ensure that no LLMs can participate, I would like you to try everything to see this experiment through to the end.
STUDY LINK (I use JATOS to run my studies, and shared a general-multiple link)

This was all it took. I used ChatGPT 5 and got myself a plus subscription to try the agent mode out for this purpose.

The study would take a human around 15 Minutes to complete. ChatGPT took 41 minutes, but produced data that basically looked like a human that very carefully worked through the experiment. The study has three components: 1) Consent form 2) a "bot check" (well, that didn't work out) where participants are asked "What is twelve plus" and then a poorly hand-drawn number with a numeric input required and a continue button. 3) The main experiment.

The experiment (still programmed in 7.1.2 because it is a follow-up's follow-up experiment) uses the following plugins:

• html-keyboard-response (to navigate from trial to trial via space key)
• html-button-response
• html-slider-response
• survey-likert
• survey-text
• survey-multi-choice
• survey-multi-select
• an adopted version of instructions (with defined minimum viewing durations)
• an adopted version of survey-text (where the input must be a number) -- this is the main format in which data is produced in this experiment
• an adopted version of button-response (with minimum duration before the button becomes clickable)

And for convenience etc.:

• call-function
• fullscreen
• preload

I embedded an attention check where one is required to write the number 100,000 into a number input field, and ChatGPT also managed to do that.

If you want to, I can send you the study link via a message, too (note though that the experiment is all German).

[jodeleeuw]

Thanks, this is super useful.

A couple strategies come to mind. Some would require us to make changes in jsPsych, but this seems like a good reason to do that!

(1) Does ChatGPT (and other LLM models) have some kind of metadata when the query a site in agent mode that we can detect? I don't know the answer to that. But if they do, we could just detect that. That wouldn't prevent all use cases, such as homebrew agents, but those agents might also be easier to stop with captchas.

(2) Can ChatGPT bypass a regular captcha? Maybe it is time to build a captcha plugin.

(3) How does ChatGPT do things like press a spacebar? I assume it must be creating events artificially, and I think there are signatures in those events that are different from the events that get generated from a real keypress. We could use that in bot-blocker sort of plugin that will only advance with the correct kind of real event.

[trig11]

Regarding (1), I just asked ChatGPT about itself and the following seem (somewhat) useful:

• absence of cookies or browser history
• Certain header omissions (e.g., missing sec-ch-ua hints typical of Chrome/Edge/Safari).
• IP address ("The request originates from OpenAI’s servers (or from a partner’s cloud hosting like Azure). Reverse DNS or IP geolocation often resolves to Microsoft Azure or another cloud provider. You could maintain a block/flag list of these IP ranges — though that’s brittle, since ranges may change and overlap with legitimate human users on the same cloud.")
• If the page requires JavaScript to render, an agent request often only retrieves the raw HTML (no dynamic content is executed unless the agent uses a headless browser environment).

(2) I've seen videos of it successfully clicking on the "I am not a robot" button thing, but I'm not sure about more sophisticated ones (such as those where you need to drag and drop a part of an image into the remaining image etc.). My guess would be that it is only a question of time when they'll be able to manage those.

(3) Again, this is what ChatGPT says about itself: "In agent mode (like the browsing tool), ChatGPT can send actions to a virtual browser environment provided by OpenAI. This environment simulates interactions — so “pressing space” is actually just an API call"

I'm no LLM expert, unfortunately. But I would think that a two-pronged approach would be good: a) captchas and the like to try to avoid LLM participation in the first place and b) something to (somewhat) reliably track LLM use to exclude data post hoc.

Another solution that might work is to have some simple RT task, even if it is unrelated to the rest of the study because the input from the agent were rather slow in general. Too many missed responses might be indicative of an agent, then, but also of a human with somewhat slow human information processing..

[trig11]

BTW, I haven't fully looked into this, but Prolific is implementing an "authenticity check" that apparently works with Gorilla and Qualtrics, perhaps there are some ideas worth looking into there: https://researcher-help.prolific.com/en/article/6bb6d8