LARGE LANGUAGE MODELS IN FINANCIAL ANALYSIS: OPPORTUNITIES AND LIMITATIONS FOR INVESTMENT DECISION-MAKING

Abstract

The LLMs are becoming widely popular for financial analysis, yet there is limited existing literature that addresses their reliability to coexist between high performance with language and high level of numeric hallucination. Furthermore, limited studies have taken a comprehensive look at these capabilities using non-US data, or whether the LLM stock justifications are factually accurate based on data provided. To fill these gaps, four datasets are used in this study comprising 100 BSE listed companies in India and created a 5-part evaluation process: Financial statements, news, peer multiples, and historical prices. This pipeline evaluates the accuracy of numeric reasoning against ground truth, news sentiment extraction, quality of the Buy/Hold/Sell recommendation using a novel explanation faithfulness metric, economic value of the pipeline through a back test and robustness to repeated queries and cognitive-bias framing. Pilot testing indicated a differentiated reliability pattern, as sentiment extraction was executed flawlessly, while numeric calculations were characterized by large drops in accuracy, explanation faithfulness depended on the type of recommendation, and outputs were sensitive to framing and repeated runs, and were being generated on low cost reproducibility with 100% free, open-weight tooling. In conclusion this study provides an open, re-usable evaluation framework, a new faithfulness metric for investment recommendations and for the first time the joint evaluation of numeric reliability, reasoning faithfulness, and economic value for stock market data pertaining to India.