Getty Images
- ChatGPT wouldn't be able to pass all levels of the exam required to get become a CFA.
- A study by JPMorgan researchers and university academics found ChatGPT made a lot of "knowledge-based errors."
- GPT-4 did better than ChatGPT and could pass the first two levels of the exam with enough prompting.
Wall Street analysts worried that AI is coming for their jobs can breathe easy—for now.
According to a study, ChatGPT couldn't pass the chartered financial analyst exam—a benchmark qualification for many finance professionals across Wall Street.
In research conducted by a team of JPMorgan AI researchers and university academics, ChatGPT and GPT-4—both AI large language models created by OpenAI—were asked CFA-style questions to test whether they are capable of complex financial reasoning.
"Based on estimated pass rates and average self-reported scores, we concluded that ChatGPT would likely not be able to pass the CFA level I and level II under all tested settings," the researchers wrote.
The CFA is a rigorous three-part test, designed gauge knowledge of securities analysis and portfolio management. It takes professionals multiple years to complete all three levels of the exam, with test-takers often pouring hundreds of hour into prep time. Once certified, it's considered a mark of distinction and can reward recipients often with more opportunities throughout their careers.
And many who take the exam fail. According to the CFA institute, only 38% of test-takers who took the level one exam in February 2023 passed. 52% passed the level two exam in May. The most recent level three test in August had a pass rate of 47%.
ChatGPT joins the ranks of those who couldn't cut it, according to the study. When asked mock level one questions that involved numerous multiple-choice answers, ChatGPT made many "knowledge-based errors," which the researchers defined as "errors where the model lacks critical knowledge required to answer the question." GPT-4 made the same kinds of knowledge-based mistakes, but they were not as bad as ChatGPT.
Level one is typically a harder test than level two, but the models did worse on the second level, which requires more analysis and interpretation of information on topics like equity investments and financial reporting.
Overall, GPT-4 did better than ChatGPT on both levels — in fact, it did well enough to pass the first two levels of the CFA exam if it was prompted enough.
"GPT-4 would have a decent chance of passing the CFA Level I and Level II if prompted with [few shot] and/or [chain of thought scenarios]," the researchers wrote.
To be sure, natural language processing, or the ability of a software to understand and manipulate human language, has already been used in several sectors of finance, like in sentiment analysis or in summarizing reports.
But with the advancement of AI large language models like ChatGPT, which made a huge splash early this year with its human-like responses, the question of AI replacing human finance pros has been a topic of much discussion on Wall Street.
Luckily for now, that scenario doesn't appear close to becoming reality.