Thought of the week – ChatGPT is learning to forecast
ChatGPT has created concerns that it may replace some white-collar jobs and significantly disrupt some business models. Just take a look at the share price of Chegg, the online homework support company. They found out the hard way that students can now do their homework for free using chatGPT. No need to pay for the Chegg platform anymore. But it’s not just helping kids with their homework. If you are an economist or an equity analyst, you may want to think about how you can justify your job when chatGPT comes for you.
A colleague from my company’s sales team sent me two papers describing experiments to teach chatGPT to forecast the Fed and share prices. No doubt he had good intentions, but being as paranoid as I am, I can’t help thinking he might have sent these papers to my bosses too, to argue that we don’t need so many research analysts after all…
Be that as it may, our jobs as research analysts and economists seem safe for now because chatGPT’s ability to forecast is still rudimentary.
The first experiment looked at the communication of the Federal Reserve’s Open Market Committee after monetary policy decisions. Three humans classified each sentence on a scale from dovish to hawkish and then the experimenters used different language learning models to classify these sentences as well. Here is a visual overview of the classification done by different models compared to the human classification. It is immediately clear how much closer chatGPT is to the human assessment of Fed statements than other models.
So, AI is getting better at identifying the nuances of human language. How can this be used to forecast? The obvious next step is to let chatGPT make forecasts based on its assessment of Fed announcements and see how good these predictions are vs. those of economists.
As an economist, I am glad nobody has done that so far and long may it stay that way. But Alejandro Lopez-Lira and Yuehua Tang have done something similar for stock market news. They asked chatGPT to pretend it is a financial analyst and rate whether a given news item about a stock is positive, negative or uncertain. Then they created a score of the day’s news flow as the average score from all the chatGPT ratings. Based on these scores, the researchers then created portfolios of stocks with mostly positive news and mostly negative news and tracked the performance of these portfolios if one bought the stocks with positive news and held them for a day or if one shorted the stocks with negative news and held them for a day.
For GPT-1 and GPT-2, the classification by chatGPT was not accurate enough to create any performance with these simple forecasts. But look at what happened when they asked GPT-3 and GPT-4 to do the same exercise. In this chart, the black line is an equal-weighted portfolio of all the stocks with any news (good, bad or uncertain), while the green line shows an equal-weighted portfolio of stocks with good news and the red line and equal-weighted portfolio that shorts all the stocks with bad news. The blue line is a long-short portfolio that goes long the good news stocks and short the bad news stocks.
The tests look very promising, but we have to keep in mind that the chart above shows returns for a daily strategy without transaction costs. I presume transaction costs will be so large as to kill any outperformance observed above. But we are only at the beginning and the advances in chatGPT from version 1 to version 4 are large. Who knows how long it will take chatGPT to become good enough to identify longer-lasting trends in share prices that allow it to compete with the forecasts of human analysts? My guess is that our jobs in finance will become a lot more competitive in the future.
Thought of the Day features investment-related and economics-related musings that don’t necessarily have anything to do with current markets. They are designed to take a step back and think about the world a little bit differently. Feel free to share these thoughts with your colleagues whenever you find them interesting. If you have colleagues who would like to receive this publication please ask them to send an email to [email protected]. This publication is free for everyone
