readme

Author: ZhuohanX

readme.md

@@ -22,8 +22,9 @@ Financial Reasoning* (EMNLP 2025 submission)
 
 ## 📚 Key Features
 
-- **54 topics** across **12 financial domains**
+- **58 topics** across **12 financial domains**
 - **5 symbolic templates per topic** (2 easy, 2 intermediate, 1 advanced)
+- **2,900 machine-verifiable instances** (58 topics × 5 templates × 10 seeds)
 - **Executable Python traces** for step-level answer verification
 - **ChainEval**, a custom metric for evaluating both final answers and intermediate steps
 
@@ -42,9 +43,11 @@ This example shows a symbolic template for Compound Interest:
 
 ```
 finchain/
+├── assets/               # Logos, taxonomy visuals, and supporting figures
+├── chaineval/            # ChainEval metric implementation & aggregation
 ├── data/
-│   └── templates/        # Symbolic prompt templates for 54 financial topics
-├── eval/                 # ChainEval evaluation scripts (coming soon)         
+│   └── templates/        # Symbolic prompt templates for 58 financial topics
+├── website/              # Static site & leaderboard assets
 └── README.md
 ```
 
@@ -55,7 +58,7 @@ Each instance includes:
 
 ## 🧭 Taxonomy of Domains and Topics
 
-FinChain covers 54 financial topics across 12 domains:
+FinChain covers 58 financial topics across 12 domains:
 
 <p align="center">
   <img src="assets/taxonomy.png" width="3000"/>
@@ -87,15 +90,17 @@ This allows precise tracking of where models hallucinate, skip, or miscalculate.
 
 ## 📈 Benchmarking Results
 
-We evaluate **30 models**, including:
-- GPT-4.1, GPT-4o-mini, LLaMA 3.3 70B
-- Qwen3, DeepSeek-R1, Mixtral, Mathstral
-- Fin-tuned models: Fino1, FinR1, WiroAI Finance Qwen
+We evaluate **26 models** spanning four categories:
+- **Frontier proprietary**: GPT-5/4.1 (+ mini variants), Claude Sonnet 4.5/4/3.7, Gemini 2.5 Pro/Flash, DeepSeek V3.x/R1, Grok 4 Heavy/Fast
+- **Finance-specific**: Fino1, FinR1, DianJin-R1, WiroAI Finance-LLaMA, WiroAI Finance-Qwen
+- **Math-enhanced**: WizardMath, MetaMath, Mathstral, Qwen2.5-Math
+- **General-purpose open**: LLaMA 3.1, Qwen 2.5/3
 
 **Findings:**
-- Larger models outperform smaller financial-tuned models
-- Even top models struggle on advanced templates and multi-hop symbolic chains
-- FinChain reveals reasoning gaps not captured by standard accuracy metrics
+- Frontier models lead ChainEval yet still struggle on advanced, compositional templates
+- Finance-tuned and math-enhanced 7B models (FinR1, Mathstral) approach frontier performance under ChainEval
+- Domain-wise analysis shows frontier systems remain balanced, while fine-tuned models excel in their target areas (e.g., FinR1 in reporting/risk, Mathstral in quantitative domains)
+- Accuracy drops across all model families from basic to advanced templates, highlighting persistent gaps in symbolic financial reasoning
 
 ## 🚀 Quick Start
 
@@ -109,11 +114,25 @@ Explore templates:
 ls data/templates/
 ```
 
-Evaluate predictions (scripts coming soon):
+Generate sample problems (each template script exposes a `main()` helper):
 ```bash
-python eval/eval_chain.py --pred path/to/your_outputs.jsonl
+python data/templates/investment_analysis/npv.py
 ```
 
+Evaluate model predictions with ChainEval:
+```bash
+python chaineval/evaluate_predictions.py --input path/to/your_outputs.jsonl --output evals/
+```
+
+Aggregate metrics across domains, subtopics, and difficulty levels:
+```bash
+python chaineval/aggregate.py
+```
+
+## 📘 Documentation
+
+- Detailed methodology, data pipeline, and evaluation discussion are available in the accompanying paper (`paper.pdf`).
+
 ## 💬 Feedback & Contributions
 
 **FinChain is an ongoing project**, and we’re continuously working to expand its coverage, refine evaluation metrics, and improve data quality. We **welcome feedback, suggestions, and community contributions**—whether it's about financial domains we missed, new evaluation ideas, or improving symbolic template diversity. If you're interested in collaborating or contributing, feel free to open an issue or contact us directly.
@@ -126,7 +145,7 @@ If you find **FinChain** useful in your research, please consider citing our pap
 
 @article{xie2025finchain,
   title={FinChain: A Symbolic Benchmark for Verifiable Chain-of-Thought Financial Reasoning},
-  author={Xie, Zhuohan and Sahnan, Dhruv and Banerjee, Debopriyo and Georgiev, Georgi and Thareja, Rushil and Madmoun, Hachem and Su, Jinyan and Singh, Aaryamonvikram and Wang, Yuxia and Xing, Rui and Koto, Fajri and Li, Haonan and Koychev, Ivan and Chakraborty, Tanmoy and Lahlou, Salem and Stoyanov, Veselin and Nakov, Preslav},
+  author={Xie, Zhuohan and Orel, Daniil and Thareja, Rushil and Sahnan, Dhruv and Madmoun, Hachem and Zhang, Fan and Banerjee, Debopriyo and Georgiev, Georgi and Peng, Xueqing and Qian, Lingfei and Huang, Jimin and Su, Jinyan and Singh, Aaryamonvikram and Xing, Rui and Elbadry, Rania and Xu, Chen and Li, Haonan and Koto, Fajri and Koychev, Ivan and Chakraborty, Tanmoy and Wang, Yuxia and Lahlou, Salem and Stoyanov, Veselin and Ananiadou, Sophia and Nakov, Preslav},
   journal={arXiv preprint arXiv:2506.02515},
   year={2025}
 }
@@ -139,12 +158,13 @@ If you find **FinChain** useful in your research, please consider citing our pap
 
 FinChain is developed by:
 
-Zhuohan Xie, Dhruv Sahnan, Debopriyo Banerjee, Georgi Georgiev,  
-Rushil Thareja, Hachem Madmoun, Jinyan Su, Aaryamonvikram Singh,  
-Yuxia Wang, Rui Xing, Fajri Koto, Haonan Li, Ivan Koychev,  
-Tanmoy Chakraborty, Salem Lahlou, Veselin Stoyanov, Preslav Nakov
+Zhuohan Xie, Daniil Orel, Rushil Thareja, Dhruv Sahnan, Hachem Madmoun,  
+Fan Zhang, Debopriyo Banerjee, Georgi Georgiev, Xueqing Peng, Lingfei Qian,  
+Jimin Huang, Jinyan Su, Aaryamonvikram Singh, Rui Xing, Rania Elbadry,  
+Chen Xu, Haonan Li, Fajri Koto, Ivan Koychev, Tanmoy Chakraborty,  
+Yuxia Wang, Salem Lahlou, Veselin Stoyanov, Sophia Ananiadou, Preslav Nakov
 
-Affiliations: MBZUAI, Sofia University, Quantsquare, Cornell University, IIT Delhi
+Affiliations: MBZUAI, Syllogia, The University of Tokyo, Sofia University "St. Kliment Ohridski", The Fin AI, Cornell University, The University of Melbourne, IIT Delhi, INSAIT, The University of Manchester
 
 For questions or collaborations, contact: **zhuohan.xie@mbzuai.ac.ae**