Tiered LLM Systems like Titan and HRM

[BradKML]

There are multiple systems out there that go beyond adding mixture-of-experts and Mamba to Transformers. HRM and Titan seemed to be a good template for what is to come, and they are apparently good enough for Zebra Puzzles and Sudoku variants. Yet there are not enough text-based tasks for these models, so it would be fun to explore in that general direction. https://github.com/lucidrains/titans-pytorch https://github.com/sapientinc/HRM

Some have noted tht HRM has data leaks in training, that would be a bit concerning sapientinc/HRM#12 (comment)

[BradKML]

There are a few things

• Speculation on multi-domain training Research Discussion: Multi-Domain Extensions for Creative Problem Solving sapientinc/HRM#33
• Reasoning-Gym as example task dataset Add Reasoning-Gym Experiments sapientinc/HRM#5
• HRM modification with embedding (DeepSeek-esque?) Extending HRM spatial reasoning to language by replacing integer cell state (representing color indices) with language model embedding vectors (representing semantic material) and expanding the training set sapientinc/HRM#23

[BradKML]

Other repos trying their hands on Titans

• Hacking transformers into Titan https://github.com/fabienfrfr/tptt
• MCP for Titan https://github.com/henryhawke/mcp-titan
• Implementations https://github.com/lucidrains/titans-pytorch https://github.com/rajveer43/titan_transformer
• "Platform" https://github.com/ai-in-pm/Titans---Learning-to-Memorize-at-Test-Time
• For any tool, someday someone will work finance into it https://github.com/todorkolev/titans-market-data