Question: Can the Haystack have variations?

[BradKML]

Since most of the needle in a "haystack" test injects a line into a pre-defined book text (that can be part of the original dataset), it can be hypothesized that the LLM simply is "smelling" for something that does not fit the context.
So, is it possible to create a "haystack" that is a mix of multiple articles, or just a list of one-liners such that it cannot guess?

[gkamradt]

Hey! You are able to specify the needle, the question, and the background context, so you can choose whatever you want

We did this to enable others to do their own context

[BradKML]

What would you consider "fair" conditions for ad-hoc generation of haystack vs needle? Would there be tools to help with the randomized construction of the haystack (and maybe averaged performance of multiple tests)?

• Word length of the needle (Single sentence vs single paragraph)
• The size of the haystack (relative to the needle, singular book or single whole anthology)
• The variance of the haystack (single source vs multiple source shuffled into a collection)

Bonus question: can this be used to evaluate FOSS models as well (esp. those without OpenAI APIs)? Would Ollama or similar do the job?

[gkamradt]

All those questions you asked are great research questions and I haven't seen anyone dig into them rigorously yet

Yep you can definitely test out other models