001049992 001__ 1049992
001049992 005__ 20251219202234.0
001049992 0247_ $$2doi$$a10.2196/79868
001049992 0247_ $$2datacite_doi$$a10.34734/FZJ-2025-05706
001049992 037__ $$aFZJ-2025-05706
001049992 082__ $$a610
001049992 1001_ $$0P:(DE-HGF)0$$aBöke, Annkathrin$$b0$$eCorresponding author
001049992 245__ $$aObserver-Independent Assessment of Content Overlap in Mental Health Questionnaires: Large Language Model–Based Study
001049992 260__ $$aToronto, Ont.$$bJMIR Publications$$c2025
001049992 3367_ $$2DRIVER$$aarticle
001049992 3367_ $$2DataCite$$aOutput Types/Journal article
001049992 3367_ $$0PUB:(DE-HGF)16$$2PUB:(DE-HGF)$$aJournal Article$$bjournal$$mjournal$$s1766153088_25560
001049992 3367_ $$2BibTeX$$aARTICLE
001049992 3367_ $$2ORCID$$aJOURNAL_ARTICLE
001049992 3367_ $$00$$2EndNote$$aJournal Article
001049992 520__ $$aBackground: Mental disorders are frequently evaluated using questionnaires, which have been developed over the past decades for the assessment of different conditions. Despite the rigorous validation of these tools, high levels of content divergence have been reported for questionnaires measuring the same construct of psychopathology. Previous studies that examined the content overlap required manual symptom labeling, which is observer-dependent and time-consuming.Objective: In this study, we used large language models (LLMs) to analyze content overlap of mental health questionnaires in an observer-independent way and compare our results with clinical expertise.Methods: We analyzed questionnaires from a range of mental health conditions, including adult depression (n=7), childhood depression (n=15), clinical high risk for psychosis (CHR-P; n=11), mania (n=7), obsessive-compulsive disorder (n=7), and sleep disorder (n=12). Two different LLM-based approaches were tested. First, we used sentence Bidirectional Encoder Representations from Transformers (sBERT) to derive numerical representations (embeddings) for each questionnaire item, which were then clustered using k-means to group semantically similar symptoms. Second, questionnaire items were prompted to a Generative Pretrained Transformer to identify underlying symptom clusters. Clustering results were compared to a manual categorization by experts using the adjusted rand index. Further, we assessed the content overlap within each diagnostic domain based on LLM-derived clusters.Results: We observed varying degrees of similarity between expert-based and LLM-based clustering across diagnostic domains. Overall, agreement between experts was higher than between experts and LLMs. Among the 2 LLM approaches, GPT showed greater alignment with expert ratings than sBERT, ranging from weak to strong similarity depending on the diagnostic domain. Using GPT-based clustering of questionnaire items to assess the content overlap within each diagnostic domain revealed a weak (CHR-P: 0.344) to moderate (adult depression: 0.574; childhood depression: 0.433; mania: 0.419; obsessive-compulsive disorder [OCD]: 0.450; sleep disorder: 0.445) content overlap of questionnaires. Compared to the studies that manually investigated content overlap among these scales, the results of this study exhibited variations, though these were not substantial.Conclusions: These findings demonstrate the feasibility of using LLMs to objectively assess content overlap in diagnostic questionnaires. Notably, the GPT-based approach showed particular promise in aligning with expert-derived symptom structures.Keywords: GPT; content overlap; large language models; questionnaires; sBERT; scales; sentence Bidirectional Encoder Representations from Transformers; symptom overlap.
001049992 536__ $$0G:(DE-HGF)POF4-5251$$a5251 - Multilevel Brain Organization and Variability (POF4-525)$$cPOF4-525$$fPOF IV$$x0
001049992 588__ $$aDataset connected to CrossRef, Journals: juser.fz-juelich.de
001049992 7001_ $$00009-0000-4709-5180$$aHacker, Hannah$$b1
001049992 7001_ $$00009-0009-4353-2641$$aChakraborty, Millennia$$b2
001049992 7001_ $$00000-0002-5070-2137$$aBaumeister-Lingens, Luise$$b3
001049992 7001_ $$00000-0002-6306-3346$$aVöckel, Jasper$$b4
001049992 7001_ $$00000-0003-1009-9625$$aKoenig, Julian$$b5
001049992 7001_ $$0P:(DE-Juel1)177935$$aVogel, David HV$$b6
001049992 7001_ $$00000-0001-5573-1212$$aLichtenstein, Theresa Katharina$$b7
001049992 7001_ $$0P:(DE-Juel1)176404$$aVogeley, Kai$$b8
001049992 7001_ $$00000-0002-8218-0425$$aKambeitz-Ilankovic, Lana$$b9
001049992 7001_ $$aKambeitz, Joseph$$b10
001049992 773__ $$0PERI:(DE-600)3161670-7$$a10.2196/79868$$gVol. 4, p. e79868 - e79868$$pe79868 - e79868$$tJMIR AI$$v4$$x2817-1705$$y2025
001049992 8564_ $$uhttps://juser.fz-juelich.de/record/1049992/files/PDF.pdf$$yOpenAccess
001049992 909CO $$ooai:juser.fz-juelich.de:1049992$$popenaire$$popen_access$$pVDB$$pdriver$$pdnbdelivery
001049992 9101_ $$0I:(DE-588b)5008462-8$$6P:(DE-Juel1)176404$$aForschungszentrum Jülich$$b8$$kFZJ
001049992 9131_ $$0G:(DE-HGF)POF4-525$$1G:(DE-HGF)POF4-520$$2G:(DE-HGF)POF4-500$$3G:(DE-HGF)POF4$$4G:(DE-HGF)POF$$9G:(DE-HGF)POF4-5251$$aDE-HGF$$bKey Technologies$$lNatural, Artificial and Cognitive Information Processing$$vDecoding Brain Organization and Dysfunction$$x0
001049992 9141_ $$y2025
001049992 915__ $$0LIC:(DE-HGF)CCBY4$$2HGFVOC$$aCreative Commons Attribution CC BY 4.0
001049992 915__ $$0StatID:(DE-HGF)0501$$2StatID$$aDBCoverage$$bDOAJ Seal$$d2023-06-08T15:24:59Z
001049992 915__ $$0StatID:(DE-HGF)0500$$2StatID$$aDBCoverage$$bDOAJ$$d2023-06-08T15:24:59Z
001049992 915__ $$0StatID:(DE-HGF)0510$$2StatID$$aOpenAccess
001049992 915__ $$0StatID:(DE-HGF)0030$$2StatID$$aPeer Review$$bDOAJ : Anonymous peer review$$d2023-06-08T15:24:59Z
001049992 915__ $$0StatID:(DE-HGF)0561$$2StatID$$aArticle Processing Charges$$d2025-01-02
001049992 915__ $$0StatID:(DE-HGF)0300$$2StatID$$aDBCoverage$$bMedline$$d2025-01-02
001049992 915__ $$0StatID:(DE-HGF)0700$$2StatID$$aFees$$d2025-01-02
001049992 920__ $$lyes
001049992 9201_ $$0I:(DE-Juel1)INM-3-20090406$$kINM-3$$lKognitive Neurowissenschaften$$x0
001049992 980__ $$ajournal
001049992 980__ $$aVDB
001049992 980__ $$aUNRESTRICTED
001049992 980__ $$aI:(DE-Juel1)INM-3-20090406
001049992 9801_ $$aFullTexts