Quality and Readability of AI-Generated Information on Bipolar Disorder: A Cross-Sectional Content Analysis

Abstract

Background: Bipolar disorder is a clinically sensitive and diagnostically complex condition in which unclear or incomplete psychoeducational information may contribute to misunderstanding of symptoms, delayed helpseeking, and unsafe interpretation of treatment options. Large language models are increasingly used as on-demand sources of mental health information, yet comparative evidence on the quality and readability of AIgenerated information about bipolar disorder remains limited. Methods: This cross-sectional content analysis evaluated 180 responses generated by ChatGPT, Gemini, and DeepSeek to 20 bipolar disorderrelated questions derived from Google Trends. Each question was asked in three independent new sessions for each model. Information quality was assessed using the 20-item EQIP instrument, and readability was evaluated using Flesch-Kincaid Grade Level, Flesch Reading Ease, and word count. To address the non-independence of repeated responses nested within prompts, a linear mixed-effects model was used with AI model and question category as fixed effects and question ID as a random intercept. Results: In the mixed-effects analysis, AI model significantly predicted EQIP scores. Compared with ChatGPT, Gemini and DeepSeek generated higher EQIP scores, with DeepSeek showing the largest estimated difference. Question category also contributed to information quality, although category-level pairwise comparisons did not remain significant after Bonferroni adjustment. Higher EQIP scores were moderately associated with longer responses and more favorable readability indices. Inter-rater analyses showed moderate absolute agreement for total EQIP scores and variable item-level agreement. Conclusions: Within the specific models, access conditions, prompts, date, and settings tested in this study, AI-generated bipolar disorder information differed across models in EQIP-rated quality and readability. These findings should be interpreted as content-quality findings rather than evidence of clinical accuracy, safety, or patient benefit. AI-generated psychoeducation should therefore be treated as a supplementary information source requiring expert review rather than a replacement for clinician-guided education.