Evaluating patient risk from ChatGPT in allergy care
Published online March 27, 2025
Artificial intelligence (AI) tools like ChatGPT are increasingly used by the public to seek medical advice. While these tools offer broad accessibility and natural-sounding responses, they are not certified medical devices and may provide incomplete or incorrect information. In allergology, where inaccurate recommendations can result in severe or even life-threatening outcomes such as anaphylaxis, assessing the safety of AI-driven advice is especially critical.
In a recent study published in The Journal of Allergy and Clinical Immunology: In Practice, Mathes et al. systematically evaluated ChatGPT’s performance in answering 120 allergology-related questions from dermatology, pediatrics, and pulmonology. The study was conducted by an interdisciplinary team of allergologists from major allergological centers in Germany. The selected questions reflected typical concerns encountered in routine allergy care.
Each response generated by ChatGPT was assessed for medical accuracy, completeness, readability, and its resemblance to human communication. To evaluate the clinical consequences of misleading information, the authors introduced the Allergological Error Impact Assessment (AEIA), a novel classification system to identify and rate the potential risk for patients associated with incorrect chatbot responses. ChatGPT’s answers were graded as minor, major, or health-threatening critical errors.
Although ChatGPT performed well overall in terms of accuracy and responses were strongly human-like, more than half of the answers contained errors, with critical errors occurring in 5% of all responses. Critical errors included potentially severely harmful advice, such as broadly recommending hypoallergenic formulas for children with confirmed cow’s milk allergy, a recommendation that, without proper clarification, could lead to life-threatening anaphylaxis in certain patients. Additionally, most answers were written at a high academic reading level, making them difficult to fully understand for many patients. The authors conclude that while ChatGPT shows promise, it is not reliable for unsupervised use in clinical allergy contexts, and expert guidance remains essential. The AEIA offers a practical tool for future patient safety evaluations of AI in allergology.
The Journal of Allergy and Clinical Immunology: In Practice is an official journal of the AAAAI, focusing on practical information for the practicing clinician.
Full Article