The Pre-trained Transformer (ChatGPT) is an NLP model that can produce writing that is eerily close to human level. A large language model (LLM) that has been taught to predict word sequences based on context is the helpful device in question. Tested and proven to be successful on the US Medical Licencing Exam, ChatGPT is now available for use.
The researchers at the Feinstein Institutes wanted to see if ChatGPT (versions 3 and 4) might be used as a reliable predictor of success on the American Board of Internal Medicine (ABIM) Gastroenterology board exam, so they conducted this new study.
The American College of Gastroenterology (ACG) Self-Assessment Tests for 2022 and 2021 were completed using ChatGPT-3 and 4, respectively. Both versions of it used the same set of questions as input. To pass the test, you needed 70% or above.
Each ACG exam consists of 300 timed, multiple-choice questions. Each question and answer was manually copied and pasted from versions 3 and 4 of ChatGPT. Four hundred and fifty questions were answered by it (145 were not included because of the need for images). Chat GPT-3 got 296/ 455 questions (65.1% right) on the first exam, and Chat GPT-4 got 284/ 355 (62.4%) right.
“ChatGPT has sparked enthusiasm,” said Andrew C. Yacht, MD, senior vice president of academic affairs and chief academic officer at Northwell Health. “But with that enthusiasm comes scepticism around the accuracy and validity of AI’s current role in health care and education.”
The study implies that even if the software is only considered as a possible teaching tool, it will eventually receive medical specialty certification.
ChatGPT can not think critically
According to the paper’s principal author, Arvind Trindade, MD, associate professor at the Feinstein Institutes’ Institute of Heath System Science: “Recently, there has been a lot of emphasis on the usage of AI across multiple businesses. There hasn’t been enough study of this potentially game-changing resource for medical education. Our findings suggest that ChatGPT is not ready for use in medical education in gastroenterology and requires further development before it can be introduced into the healthcare setting.
ChatGPT does not have any preexisting familiarity with the situation or topic at hand. It may have received a failing grade because of its reliance on dubious, out-of-date, or non-medical sources; additional study is required before it can be utilised consistently.
To read our article about “US health official sounds alarm on child social media use” click here.
