Replication and "Riffing"

[BradKML]

This spawned off of this post a few month back under AI-Scientist with @clxyder and I am thinkful he reminded me how important it is to do replications. SakanaAI/AI-Scientist#166 (comment)

There are a few ways of doing "riffs": (borrowed from health nuts https://www.exfatloss.com/p/potato-riff-study-the-quicksort-of)

1. Throw in new benchmarks or goals, or maybe ones adjacent to what the model is trained on, but are "forgotten"
2. Just benchmark novel parts of the model (yes, it's an HRM reference Tiered LLM Systems like Titan and HRM #180)
3. Mixing model architectures together (not enough people talk about MLP-Mixer?) https://github.com/tanishqkumar/beyond-nanogpt

[BradKML]

A good tests case would be the Axelrod experiments, it's not AI architecture but hits similar, and at a smaller scale Axelrod-Python/Axelrod#1476