arXiv-2024/06-A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges

[BrambleXu]

Summary:

本调查的目的是提供LLMs在金融应用中的当前状态的全面概述，突出进展、前景和挑战。调查旨在检查金融应用中的特定LLMs，分析其架构、预训练方法和定制。它还旨在分析数据集和基准，提供有价值的资源集合。此外，调查解决了将LLMs应用于金融的独特挑战，如前瞻性偏差、法律问题、数据污染和可解释性，并探索潜在的解决方案和未来研究方向。最终目标是促进LLMs在金融领域的采用和进一步发展，为创新解决方案和增强决策过程铺平道路。

Resource:

• pdf
• [code](
• [paper-with-code](

Paper information:

• Author:
• Dataset:
• keywords:

Notes:

金融関連のLLAMAモデル

Llama-series: Llama [26], an LLM introduced in 2023, offers flexibility with model sizes ranging from 7B to 65B parameters. Trained on publicly available datasets for transparency, Llama outperforms larger models, including GPT-3, on most benchmarks despite its smaller size. Its financial variants, which include FinMA [27], Fin-Llama[28], Cornucopia – Chinese [29], Instruct-FinGPT [30] and InvestLM [31], provide specialized capabilities for various financial tasks. Among them, InvestLM, based on LLaMA-65B and a diverse investment-related dataset, offers investment recommendations comparable to cutting-edge commercial models. Llama 2 [32], which was released later, included various enhancements over Llama, including a 40% larger pretraining corpus, a doubled context length, and the adoption of grouped-query attention for improved inference scalability. It has financial variants such as FinGPT [33], FinLlama [34], and GreedLlama [35]. Particularly, FinGPT is an open-source model that focuses on providing accessible and transparent resources for developing financial LLMs. Despite having relatively small training data compared to BloombergGPT, FinGPT claims to offer a more accessible, flexible, and cost-effective solution for financial language modeling. In April 2024, Meta introduced Llama 3 [36], featuring 8B and 70B parameter models that showcase state-of-the-art performance and improved reasoning capabilities, marking them as the most capable openly available LLMs to date. The LLM community is visibly excited, and we expect more Llama 3 variants for financial LLM models to emerge soon.

In addition to the models mentioned above, there are also other financial domain-specific LLMs such as FinTral [37], driven from Mistral 7B [38]; SilverSight [39], based on the Qwen 1.5-7B chat model [40]; DISC-FinLLM [41], used Baichuan-13B [42] as the backbone; CFLLM [43], based on InternLM-7B [44]; FinVIS-GPT [45] which is a multimodal LLM for financial chart analysis, based on LLaVA [46]. These domain-specific LLMs utilize vast financial datasets and advanced training techniques to provide more accurate and context-aware financial analysis than general-domain models. As research in this area continues to progress, we expect the development of even more sophisticated financial LLMs that could transform various sectors of the financial industry, including investment strategies, risk management, forecasting, and customer service. However, it's crucial to acknowledge the limitations and potential biases of these models and to employ them thoughtfully alongside human expertise and judgment.

In financial text analysis, summarizing and extracting key information from documents is crucial for quickly understanding and processing important data within lengthy and complex texts [103].

• WeaverBird: Empowering Financial Decision-Making with Large Language Model, Knowledge Base, and Search Engine

Financial Relation Construction (KG関連)

• Constructing financial relationships, particularly through the use of knowledge graphs, represents a powerful methodology for organizing and making sense of the extracted entities and their interrelations from extensive and complex financial datasets [104].

  • 104 M. van Zwam, A. Khalili, J. Jessurun, S. Oberoi, M. Beerepoot, S. Fernandez, J. Bijman, A. Easton, and I. Karatas, "Knowledge graphs for financial services: The path to unlock new insights from your data," 2020. [Online]. Available: https: //www2.deloitte.com/content/dam/Deloitte/de/ Documents/operations/knowledge-graphs-pov.pdf

• Knowledge graphs consist of interconnected descriptive structures about entities (objects, events, people, etc.), the attributes of those entities, and the relationships that link them together. This framework offers a structured way of representing relationships within data and enables sophisticated analyses to be derived from them [105], [106].

  • 105 Jiang, X., Xu, C., Shen, Y., Sun, X., Tang, L., Wang, S., … Guo, J. (n.d.). On the Evolution of Knowledge Graphs: A Survey and Perspective.
  • 106 “Large Language Models and Knowledge Graphs: Opportunities and Challenges.” 2023, August.

• Recent advancements in LLMs have led researchers to explore the potential of using information extracted by LLMs to construct and analyze knowledge graphs in the financial sector [108], [109], [110]. Notably, Trajanoska et al. [108] generate a knowledge graph by leveraging LLMs to extract structured Environmental, Social, and Governance (ESG) information from sustainability reports, using a format of triples consisting of node-edge-node, to enable deeper analysis and understanding of corporate sustainability practices. Similarly, Cheng et al. [111] develop a Semantic-Entity Interaction Module. This module combines a language model with a Conditional Random Field (CRF) layer to comprehend the interaction between entities and their semantic contexts in texts. It automatically constructs financial knowledge graphs from brokerage research reports without the need for explicit financial knowledge or extensive manual rules.

  • 108 (1), MilenaTrajanoska, et al. Enhancing Knowledge Graph Construction Using Large Language Models. May 2023.
  • 109 K. Ouyang, Y. Liu, S. Li, R. Bao, K. Harimoto, and X. Sun, "Modal-adaptive knowledge-enhanced graph-based financial prediction from monetary pol- icy conference calls with LLM," arXiv preprint arXiv:2403.16055, 2024.
  • 110 Wang, X., Sun, Y., Chen, C., & Cui, J. (2022). A Relation Extraction Model Based on BERT Model in the Financial Regulation Field. 2022 2nd International Conference on Computer Science, Electronic Information Engineering and Intelligent Control Technology (CEI), 496–501. https://doi.org/10.1109/cei57409.2022.9950087
  • 111 Z. Cheng, L. Wu, T. Lukasiewicz, E. Sallinger, and G. Gottlob, "Democratizing financial knowledge graph construction by mining massive brokerage re- search reports." in EDBT/ICDT Workshops, 2022.

• Moreover, financial research analysts often face challenges in identifying critical documents, key entities, and important events during their research on complex financial subjects. Mackie and Dalton [112] tackle these issues by developing automated methods to create detailed, queryspecific knowledge graphs from documents and entities.

  • 112 Mackie, Iain, and Jeffrey Dalton. Query-Specific Knowledge Graphs for Complex Finance Topics. Nov. 2022.

• As illustrated above, knowledge graphs have demonstrated their utility in information retrieval. A special case within this domain is the translation of Natural Language (NL) into Graph Query Language (GQL).

  • 「NLP->GQLへの変換」この方向は考慮しない

• Knowledge graphs can also be used to significantly enhance question-answering systems. Wang et al. [114] introduce an innovative Knowledge Graph Prompting (KGP) for multi-document question answering (MD-QA). Their approach constructs a knowledge graph from multiple documents, highlighting semantic or lexical relationships between passages or document structures. An LLM-based graph traversal agent then uses this knowledge graph to gather contextually relevant information, thereby enhancing the LLM's accuracy in answering questions

  • Wang, Yu, et al. "Knowledge graph prompting for multi-document question answering." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 17. 2024.（この論文は研究課題に使える）

• Another beneficial aspect of knowledge graphs is their 9ability to be enriched over time through the use of LLMs. Li [115] presents FinDKG, a dynamic knowledge graph with LLMs used in financial domain.

  • 時系列は考慮しない

• There exist other financial relation extraction studies using LLMs, though not necessarily for knowledge graph construction [116], [117], [118], [119].

  • LLMと関係ないが、KG構築の関連論文。必要なら参考する

Textual Classification

• This classification task can be further categorized into several sub-tasks, such as industry/company classification and document/topic classification. By effectively classifying and organizing this information, businesses and researchers can extract valuable insights and make informed decisions
  • サービス選択なしのケースは分類必要が、まず選択なしの質問の割合を確認したほうが良い
  • 
• 業種分類（＠山岸さん）
  • Company or industry classification involves grouping companies into distinct categories based on shared characteristics such as business activities and market performance, with the aim of creating coherent and differentiated groups. Identifying similar company profiles is a fundamental task in finance, with applications spanning investment portfolio construction, securities pricing, and financial risk attribution. Traditionally, financial analysts have relied on industry classification systems, such as the Global Industry Classification System (GICS), the Standard Industrial Classification (SIC), the North American Industry Classification System (NAICS), and the Fama French (FF) model, to identify companies with similar profiles [123]. However, these systems do not provide a means to rank companies based on their degree of similarity and require time-consuming, effortintensive manual analysis and data processing by domain experts [123].
  • Recently, a team at BlackRock [124] explores a novel approach to company classification using LLMs. They investigated the use of pre-trained and fine-tuned LLMs to generate company embeddings based on business descriptions from SEC filings. Their study aimed to assess the embeddings' ability to reproduce GICS classifications, benchmark LLM performance on various downstream financial tasks, and examine the impact of factors such as pre-training objective, fine-tuning, and model size on embedding quality. The results showed that LLM-generated embeddings, particularly those from fine-tuned SentenceBERT models, could accurately reproduce GICS sector and industry classifications and outperform them on tasks like identifying similar companies based on return correlations and explaining cross-sectional equity returns.
    • 124 Vamvourellis, D., Tóth, M., Mehta, D., & Pasquali, S. (n.d.). Company Similarity using Large Language Models.
  • Interestingly, knowledge graphs can also be used to enrich industry classification and improve the performance of domain-specific text classification tasks. Wang et al. [125] propose a novel Knowledge Graph Enriched BERT (KGEB) model that integrates external knowledge from a local knowledge graph with word representations. They demonstrated the effectiveness of their approach by constructing a large dataset based on companies listed on the Chinese National Equities Exchange and Quotations (NEEQ) and showing that the KGEB model outperforms competitive baselines, including graph convolutional network, Logistic Regression, TextCNN, BERT, and K-BERT, achieving an accuracy of 91.98% and an F1 score of 90.89%.
    • 125 Wang, S., Pan, Y., Xu, Z., Hu, B., & Wang, X. (2021). Enriching BERT With Knowledge Graph Embedding For Industry Classification. In Communications in Computer and Information Science,Neural Information Processing (pp. 709–717). https://doi.org/10.1007/978-3-030-92310-5_82
• Document or topic classification
  • This task involves categorizing financial documents or texts, such as news articles [126], [127] or company filings [128], [129], into predefined topics or themes. Alias et al. [130] propose a novel approach that utilizes the FinBERT model to extract and categorize relevant topics of Key Audit Matters (KAM) from the annual reports of publicly listed companies in Bursa Malaysia. Similarly, Burke et al. [131] fine-tune the FinBERT model to classify accounting topics within three unlabelled financial disclosures, including custom notes to the financial statements, the Management's Discussion and Analysis section, and the risk factor section.
• Sentiment Analysis
  • CSと関係がない
• Financial Time Series Analysis
  • CSと関係がない
• Forecasting
  • CSと関係がない
• Anomaly Detection
  • CSと関係がない
• Financial Reasoning
  • to support strategic financial planning, generate investment recommendations, provide advisory services, and assist in financial decisionmaking.
  • Financial planning involves setting financial goals, assessing current financial situations, and devising strategies to achieve those goals. This process includes analyzing income, expenses, investments, and risk management to create a comprehensive plan for long-term financial stability and growth.
    • ユーザーのデータが取得できたら。CSログイン後の機能として
    • corporate planning:
      • LLMs can be utilized to support various aspects of financial planning. For instance, LLMs can analyze market trends and competitor data to help organizations develop business strategies. Nguyen and Tulabandhula [225] examine the use of generative AI models, such as GPT-4 and other transformer-based models, for business strategy development.
      • Moreover, LLMs can streamline financial planning processes, as demonstrated by Ludwig and Bennetts [226]. By integrating ChatGPT into financial planning practices, they illustrate how financial planners could leverage this AI model to enhance client communication and provide immediate, semi-personalized responses to common financial concerns, such as preparing for economic recessions. They also highlight ChatGPT's role in client education and its ability to simplify complex financial concepts for better understanding. Despite these benefits, the authors emphasize the need for human oversight to ensure the accuracy and quality of the advice provided, addressing potential limitations of the models.
    • personal financial planning
      • LLMs can help individuals create customized strategies for long-term financial well-being.
      • A recent study by Lakkaraju et al. [227] evaluates the performance of LLM-based chatbots, ChatGPT and Bard, in providing personal financial advice. The study covers various aspects of personal finance, including decisions related to bank accounts, credit cards, and certificates of deposits (CDs).
      • Additionally, LLMs can optimize budgeting strategies by incorporating AI-driven recommendations into individual and household financial models. de Zarz`a et al. [228] present an optimization framework for individual budget allocation to maximize savings and extend this approach to household finances, addressing the complexities of multiple incomes and shared expenses. In high-net-worth contexts, LLMs can also be used to simulate various tax scenarios, identify optimal tax strategies, and provide proactive advice based on changing tax law to minimize tax liabilities and maximize financial growth [229].
        • 229 Tax関連　D. Fava, "The future of tax planning: Leveraging generative AI in high-net-worth contexts: Artifical intelligence could help planers optimize their HNW clients' complex tax planning needs." Journal of Finan- cial Planning, no. 10, 2023.
• Recommendation
  • LLM in Investment Advisory
  • Impact on Investment Strategies
  • Regulatory and Ethical Considerations
• Support Decision-making
  • Financial Auditing and Regulatory Compliance
  • Fraud Detection and Risk Management
• Real-time Reasoning
  • Chatbots and Virtual Assistants
  • Question-Answering
    • Recent studies have focused on enhancing the numerical reasoning capabilities of these systems, enabling them to handle multi-step calculations and extract relevant information from various data sources（数字の計算問題は必要のとき参考する）
    • Moreover, Xue et al. [103] propose a cutting-edge dialogue system designed specifically for the finance sector, named WeaverBird. It leverages a LLM with GPT architecture fine-tuned on extensive financial corpora. This enables WeaverBird to understand and provide informed responses to complex financial queries, such as investment strategies during inflation. The system's performance is further enhanced by integrating a local knowledge base and search engine, allowing it to retrieve relevant information and generate responses conditioned on web search results, complete with proper source references for enhanced credibility. Comparative evaluations across a broad spectrum of financial question-answering tasks demonstrate WeaverBird's superior performance compared to other models, positioning it as a powerful tool for financial dialogue and decision support.（このシステム強そう）
• Agent-based Modeling（基本はトレードに関するもの）
  • Trading and Investments
  • Simulating Markets and Economic Activities
  • Automated Financial Processes
  • Multi-agent Systems

Datasets

• • Financial PhraseBank (FPB) [302]: This is a dataset consisting of financial phrases annotated with sentiment labels. It is widely used for sentiment analysis in financial contexts due to its detailed and domainspecific annotations.
• • Financial Question Answering and Opinion Mining (FiQA) [303]: This dataset focuses on aspect-based sentiment analysis and opinion-based question answering. It includes financial news headlines and microblogs, annotated for sentiment and aspect categories. The dataset is designed to challenge models with tasks that require fine-grained sentiment and opinion extraction from financial texts.
• • FinQA [304]: A dataset designed for numerical reasoning over financial data. FinQA includes questions that require understanding and manipulating numerical information from financial reports. It emphasizes the need for models to perform complex reasoning tasks involving financial metrics and calculations.
• Other datasets such as ECTSum [305], FiNER [306],FinRED [307], REFinD [117], FinSBD [308] and CFLUE[309] contribute to various specific financial NLP tasks