Suggestion: add WFGY (TXT based long horizon stress test + failure map)

[onestardao]

Hi, and thanks for putting this list together. It is a very useful overview of the AI evaluation ecosystem.

I would like to ask whether a project called WFGY could fit somewhere in this repo, probably under the robustness / stress-testing side of evaluation.

Short description:

• WFGY 3.0 · Singularity Demo is a pure TXT pack used as a long horizon “tension crash test” for LLMs. Models read one S-class problem after another and we check when their reasoning quietly drifts or collapses.
• WFGY ProblemMap is a diagnostic map of common AI system failure modes (for example RAG issues, vector store mistakes, deployment order problems). It is used to label what went wrong when a model starts behaving strangely during those stress tests.

Everything is open source on GitHub (MIT license, 1k+ stars) and fully transparent. There is no hidden code and no external calls.

If this looks in scope for awesome-ai-eval, I am happy to prepare a small PR that adds a single line in the appropriate section. If it is out of scope, that is also totally fine; I mainly wanted to ask before sending a PR.

Thanks again for curating this list.

[abflh-beep]

Hi

[onestardao]

Hi, thanks for the reply 🙏

Just to add a bit more context: right now I feel there is not really a clear “standard” eval for long-horizon / long-context reasoning collapse. Most benchmarks are short questions or one-shot tasks. WFGY 3.0 is more like a TXT-based stress test: the model reads one S-class problem after another, and we watch when the reasoning starts to drift or break.

The ProblemMap part is then used to label what kind of failure mode it is (RAG issue, deployment order issue, hallucination under stress, etc.), so it is half benchmark, half diagnostic map for real systems. Everything is open source and can be run with any GPT-4-class model.

If this still sounds in-scope for awesome-ai-eval, I am happy to open a very small PR and add one line in whatever section you think is best (maybe under robustness / long-horizon / methodology). If you feel it is out of scope, that is also totally fine, I just wanted to check with you first.

Thanks again for curating this list, it is very helpful for people like me who work on debugging side of LLMs.

if you think its okay then may I open PR ?

[onestardao]

Hi, just checking in to see if the team has any preference or direction for this item.
Happy to prepare a PR if it fits the roadmap.

[onestardao]

Submitted a PR to add WFGY 3.0 (Singularity Demo) as an eval stress-test resource. Thanks.