001021988 001__ 1021988
001021988 005__ 20240226075418.0
001021988 0247_ $$2doi$$a10.48550/ARXIV.2311.14079
001021988 0247_ $$2datacite_doi$$a10.34734/FZJ-2024-01127
001021988 037__ $$aFZJ-2024-01127
001021988 1001_ $$0P:(DE-Juel1)192490$$aYu, Jinyang$$b0$$eCorresponding author
001021988 245__ $$aEmpirical Comparison between Cross-Validation and Mutation-Validation in Model Selection
001021988 260__ $$barXiv$$c2023
001021988 3367_ $$0PUB:(DE-HGF)25$$2PUB:(DE-HGF)$$aPreprint$$bpreprint$$mpreprint$$s1706624922_2473
001021988 3367_ $$2ORCID$$aWORKING_PAPER
001021988 3367_ $$028$$2EndNote$$aElectronic Article
001021988 3367_ $$2DRIVER$$apreprint
001021988 3367_ $$2BibTeX$$aARTICLE
001021988 3367_ $$2DataCite$$aOutput Types/Working Paper
001021988 520__ $$aMutation validation (MV) is a recently proposed approach for model selection, garnering significant interest due to its unique characteristics and potential benefits compared to the widely used cross-validation (CV) method. In this study, we empirically compared MV and $k$-fold CV using benchmark and real-world datasets. By employing Bayesian tests, we compared generalization estimates yielding three posterior probabilities: practical equivalence, CV superiority, and MV superiority. We also evaluated the differences in the capacity of the selected models and computational efficiency. We found that both MV and CV select models with practically equivalent generalization performance across various machine learning algorithms and the majority of benchmark datasets. MV exhibited advantages in terms of selecting simpler models and lower computational costs. However, in some cases MV selected overly simplistic models leading to underfitting and showed instability in hyperparameter selection. These limitations of MV became more evident in the evaluation of a real-world neuroscientific task of predicting sex at birth using brain functional connectivity.
001021988 536__ $$0G:(DE-HGF)POF4-5254$$a5254 - Neuroscientific Data Analytics and AI (POF4-525)$$cPOF4-525$$fPOF IV$$x0
001021988 588__ $$aDataset connected to DataCite
001021988 650_7 $$2Other$$aMachine Learning (cs.LG)
001021988 650_7 $$2Other$$aMachine Learning (stat.ML)
001021988 650_7 $$2Other$$aFOS: Computer and information sciences
001021988 7001_ $$0P:(DE-Juel1)184874$$aHamdan, Sami$$b1$$ufzj
001021988 7001_ $$0P:(DE-Juel1)190306$$aSasse, Leonard$$b2$$ufzj
001021988 7001_ $$0P:(DE-Juel1)151166$$aMorrison, Abigail$$b3$$ufzj
001021988 7001_ $$0P:(DE-Juel1)172843$$aPatil, Kaustubh R.$$b4$$ufzj
001021988 773__ $$a10.48550/ARXIV.2311.14079
001021988 8564_ $$uhttps://juser.fz-juelich.de/record/1021988/files/preprint.pdf$$yOpenAccess
001021988 8564_ $$uhttps://juser.fz-juelich.de/record/1021988/files/preprint.gif?subformat=icon$$xicon$$yOpenAccess
001021988 8564_ $$uhttps://juser.fz-juelich.de/record/1021988/files/preprint.jpg?subformat=icon-1440$$xicon-1440$$yOpenAccess
001021988 8564_ $$uhttps://juser.fz-juelich.de/record/1021988/files/preprint.jpg?subformat=icon-180$$xicon-180$$yOpenAccess
001021988 8564_ $$uhttps://juser.fz-juelich.de/record/1021988/files/preprint.jpg?subformat=icon-640$$xicon-640$$yOpenAccess
001021988 909CO $$ooai:juser.fz-juelich.de:1021988$$pdnbdelivery$$pdriver$$pVDB$$popen_access$$popenaire
001021988 9101_ $$0I:(DE-588b)5008462-8$$6P:(DE-Juel1)184874$$aForschungszentrum Jülich$$b1$$kFZJ
001021988 9101_ $$0I:(DE-588b)5008462-8$$6P:(DE-Juel1)190306$$aForschungszentrum Jülich$$b2$$kFZJ
001021988 9101_ $$0I:(DE-588b)5008462-8$$6P:(DE-Juel1)151166$$aForschungszentrum Jülich$$b3$$kFZJ
001021988 9101_ $$0I:(DE-588b)5008462-8$$6P:(DE-Juel1)172843$$aForschungszentrum Jülich$$b4$$kFZJ
001021988 9131_ $$0G:(DE-HGF)POF4-525$$1G:(DE-HGF)POF4-520$$2G:(DE-HGF)POF4-500$$3G:(DE-HGF)POF4$$4G:(DE-HGF)POF$$9G:(DE-HGF)POF4-5254$$aDE-HGF$$bKey Technologies$$lNatural, Artificial and Cognitive Information Processing$$vDecoding Brain Organization and Dysfunction$$x0
001021988 9141_ $$y2023
001021988 915__ $$0StatID:(DE-HGF)0510$$2StatID$$aOpenAccess
001021988 920__ $$lyes
001021988 9201_ $$0I:(DE-Juel1)INM-7-20090406$$kINM-7$$lGehirn & Verhalten$$x0
001021988 980__ $$apreprint
001021988 980__ $$aVDB
001021988 980__ $$aUNRESTRICTED
001021988 980__ $$aI:(DE-Juel1)INM-7-20090406
001021988 9801_ $$aFullTexts