Are MCTS/TTC/Strawberry even that good?

[BradKML]

It is kind of the weird thought how OpenHands got beaten by TTC-centric solutions, and not even sure how fast they can counter that in SWE-Bench performance. Would it requre fine-tune, or just a LoRA? codestoryai/aide#1063

[SmartManoj]

The latest version of OpenHands may produce better results than that. Did you run any SWE bench instances?

[BradKML]

Haven't even tried cus it takes a lot more compute, would do it if I can tho

[SmartManoj]

Even for a single instance?

[SmartManoj]

Modified for running in Kaggle

Result using Nemotron;

[SmartManoj]

Would you reproduce this bug in sympy/sympy#21537?

Context: SWE-bench/SWE-bench#273 (comment)

[BradKML]

How about you run this first, since SWE-Bench is considered a "gold standard" now especially by academia, which means only OpenHands/Kevin (or the underlying LLM) can change, not SWE-Bench. SWE-bench/SWE-bench#273 (comment)

On why delete "unit test" but not the others

• https://www.mrlacey.com/2023/12/never-delete-tests.html
• https://riad.blog/2020/07/21/deleting-tests-is-a-best-practice/#comment-1300
• https://fs.blog/chestertons-fence/

[BradKML]

Regarding the 7 layers of testing

Test Type	Tied concept	Trait	Example
Manual tests or GUI tests	Vision, Values, and Mission Statement	User centric and includes exploratory work	Chatbot-enabled code agents
Smoke tests or UX tests or E2E tests or Service Test	Principles, Tenants, and Commitments	Should be minimal for critical paths	Most of the CI/CD ecosystem
Monitoring tests or API tests or Workflow tests	Policy, Process, and Expectations	Orchestration related since the system is live	LLM reporting tools for AirFlow and Docker
Integration tests or Contract tests	Standards, Mandates, Safeguards	Sensitive to live databases and external APIs	MetaGPT's SOP architecture
Acceptance tests or Component tests	Procedures and Implementation	Mock-ups or In-memory data for singular tasks	MetaGPT's SOP architecture
Unit tests	Guidelines and Best Practices	Overdone and based on Single Responsibility Principle (SRP)	pairing Mutahunter or CoverUp test generator with Pytest/Jest
Static Code Analysis	Baseline and Uniform Work	Testing Trophy recommendation, displaces classical unit tests	MyPy/TS type checks, Black/Prettier code formatting, SCC for complexity

Reference on testing:

• https://www.hibri.net/2019/06/10/the_testing_pyramid/
• https://diego-pacheco.blogspot.com/2020/02/going-faster-with-testing.html
• https://dev.to/borysshulyak/high-confidence-testing-levels-1n1m
• https://x.com/khalilstemmler/status/1371239905637056515
• https://blog.bluflametech.com/blog/test-software/

Reference on their complement:

• https://www.7sec.com/blog/develop-policies-for-an-all-round-approach-to-information-security/
• https://cloudadvisor.medium.com/principles-policies-standards-guidelines-and-procedures-whats-the-difference-6e7a406c279
• https://www.innovatia.net/blog/documentation-hierarchy-why-is-it-important

[BradKML]

For reference regarding o1-adjacent finetuning

• https://github.com/AIDC-AI/Marco-o1
• https://github.com/NovaSky-AI/SkyThought
• https://github.com/SimpleBerry/LLaMA-O1
• https://github.com/sunnynexus/Search-o1
• https://github.com/llSourcell/O1-nano
• https://github.com/PromtEngineer/Agent-0

[BradKML]

In hindsight, I should have mentioned RL and the way Dr. GRPO and others trying to fix DeepSeek