Since its unveiling in November of last year, ChatGPT has set a new standard for what machines are capable of achieving. It has accomplished impressive feats, including passing the US Medical Licensing Exam and a Wharton MBA Exam. However, it appears that accounting is an area in which humans still outperform ChatGPT.
The researchers published a study in the American Accounting Association journal to evaluate ChatGPT’s performance in answering accounting exam questions using the GPT-4 model.
To carry out the study, 327 co-authors from 186 institutions across 14 countries collaborated to provide a total of 25,181 classroom accounting exam questions. Additionally, undergraduate students were recruited to supply 2,268 textbook test bank questions to the ChatGPT.
In a press release, David Wood, a professor of accounting at Brigham Young University (BYU) and the lead author of the study, noted that there were concerns about students potentially cheating when ChatGPT was first introduced. However, he believes that the focus should be on exploring the ways in which this technology can enhance the learning process for students and the teaching process for faculty. Wood further noted that the study’s findings were enlightening.
Brigham Young University (BYU) reported that while ChatGPT’s performance on the accounting exam questions was noteworthy, students outperformed the chatbot. On average, students scored 76.7%, while ChatGPT scored 47.7%.
The study found that ChatGPT scored higher than the student average in 11.3% of the questions, excelling in the areas of accounting information systems (AIS) and auditing. However, the AI chatbot’s performance was weaker in tax, financial, and managerial assessments. The researchers suggested that ChatGPT may have struggled with the mathematical processes required for these types of questions, leading to its lower performance.
The study also found that ChatGPT performed better on true or false questions and multiple-choice questions than on short-answer questions. However, the chatbot struggled with higher-order questions, and in some cases, it provided authoritative written responses that were incorrect.