000843816 001__ 843816
000843816 005__ 20210129232655.0
000843816 0247_ $$2doi$$a10.1186/s13007-018-0278-7
000843816 0247_ $$2Handle$$a2128/17579
000843816 0247_ $$2pmid$$apmid:29449872
000843816 0247_ $$2WOS$$aWOS:000425015500001
000843816 0247_ $$2altmetric$$aaltmetric:32947011
000843816 037__ $$aFZJ-2018-01356
000843816 041__ $$aEnglish
000843816 082__ $$a580
000843816 1001_ $$0P:(DE-HGF)0$$aGiuffrida, M. Valerio$$b0
000843816 245__ $$aCitizen crowds and experts: observer variability in image-based plant phenotyping
000843816 260__ $$aLondon$$bBioMed Central$$c2018
000843816 3367_ $$2DRIVER$$aarticle
000843816 3367_ $$2DataCite$$aOutput Types/Journal article
000843816 3367_ $$0PUB:(DE-HGF)16$$2PUB:(DE-HGF)$$aJournal Article$$bjournal$$mjournal$$s1548351573_3169
000843816 3367_ $$2BibTeX$$aARTICLE
000843816 3367_ $$2ORCID$$aJOURNAL_ARTICLE
000843816 3367_ $$00$$2EndNote$$aJournal Article
000843816 520__ $$aBackgroundImage-based plant phenotyping has become a powerful tool in unravelling genotype–environment interactions. The utilization of image analysis and machine learning have become paramount in extracting data stemming from phenotyping experiments. Yet we rely on observer (a human expert) input to perform the phenotyping process. We assume such input to be a ‘gold-standard’ and use it to evaluate software and algorithms and to train learning-based algorithms. However, we should consider whether any variability among experienced and non-experienced (including plain citizens) observers exists. Here we design a study that measures such variability in an annotation task of an integer-quantifiable phenotype: the leaf count.ResultsWe compare several experienced and non-experienced observers in annotating leaf counts in images of Arabidopsis Thaliana to measure intra- and inter-observer variability in a controlled study using specially designed annotation tools but also citizens using a distributed citizen-powered web-based platform. In the controlled study observers counted leaves by looking at top-view images, which were taken with low and high resolution optics. We assessed whether the utilization of tools specifically designed for this task can help to reduce such variability. We found that the presence of tools helps to reduce intra-observer variability, and that although intra- and inter-observer variability is present it does not have any effect on longitudinal leaf count trend statistical assessments. We compared the variability of citizen provided annotations (from the web-based platform) and found that plain citizens can provide statistically accurate leaf counts. We also compared a recent machine-learning based leaf counting algorithm and found that while close in performance it is still not within inter-observer variability.ConclusionsWhile expertise of the observer plays a role, if sufficient statistical power is present, a collection of non-experienced users and even citizens can be included in image-based phenotyping annotation tasks as long they are suitably designed. We hope with these findings that we can re-evaluate the expectations that we have from automated algorithms: as long as they perform within observer variability they can be considered a suitable alternative. In addition, we hope to invigorate an interest in introducing suitably designed tasks on citizen powered platforms not only to obtain useful information (for research) but to help engage the public in this societal important problem.
000843816 536__ $$0G:(DE-HGF)POF3-582$$a582 - Plant Science (POF3-582)$$cPOF3-582$$fPOF III$$x0
000843816 536__ $$0G:(DE-HGF)POF3-583$$a583 - Innovative Synergisms (POF3-583)$$cPOF3-583$$fPOF III$$x1
000843816 588__ $$aDataset connected to CrossRef
000843816 7001_ $$0P:(DE-HGF)0$$aChen, Feng$$b1
000843816 7001_ $$0P:(DE-Juel1)129394$$aScharr, Hanno$$b2$$ufzj
000843816 7001_ $$0P:(DE-HGF)0$$aTsaftaris, Sotirios A.$$b3$$eCorresponding author
000843816 773__ $$0PERI:(DE-600)2203723-8$$a10.1186/s13007-018-0278-7$$gVol. 14, no. 1, p. 12$$n1$$p12$$tPlant methods$$v14$$x1746-4811$$y2018
000843816 8564_ $$uhttps://plantmethods.biomedcentral.com/articles/10.1186/s13007-018-0278-7
000843816 8564_ $$uhttps://juser.fz-juelich.de/record/843816/files/s13007-018-0278-7.pdf$$yOpenAccess
000843816 8564_ $$uhttps://juser.fz-juelich.de/record/843816/files/s13007-018-0278-7.gif?subformat=icon$$xicon$$yOpenAccess
000843816 8564_ $$uhttps://juser.fz-juelich.de/record/843816/files/s13007-018-0278-7.jpg?subformat=icon-1440$$xicon-1440$$yOpenAccess
000843816 8564_ $$uhttps://juser.fz-juelich.de/record/843816/files/s13007-018-0278-7.jpg?subformat=icon-180$$xicon-180$$yOpenAccess
000843816 8564_ $$uhttps://juser.fz-juelich.de/record/843816/files/s13007-018-0278-7.jpg?subformat=icon-640$$xicon-640$$yOpenAccess
000843816 8564_ $$uhttps://juser.fz-juelich.de/record/843816/files/s13007-018-0278-7.pdf?subformat=pdfa$$xpdfa$$yOpenAccess
000843816 909CO $$ooai:juser.fz-juelich.de:843816$$pdnbdelivery$$pdriver$$pVDB$$popen_access$$popenaire
000843816 9101_ $$0I:(DE-588b)5008462-8$$6P:(DE-Juel1)129394$$aForschungszentrum Jülich$$b2$$kFZJ
000843816 9131_ $$0G:(DE-HGF)POF3-582$$1G:(DE-HGF)POF3-580$$2G:(DE-HGF)POF3-500$$3G:(DE-HGF)POF3$$4G:(DE-HGF)POF$$aDE-HGF$$bKey Technologies$$lKey Technologies for the Bioeconomy$$vPlant Science$$x0
000843816 9131_ $$0G:(DE-HGF)POF3-583$$1G:(DE-HGF)POF3-580$$2G:(DE-HGF)POF3-500$$3G:(DE-HGF)POF3$$4G:(DE-HGF)POF$$aDE-HGF$$bKey Technologies$$lKey Technologies for the Bioeconomy$$vInnovative Synergisms$$x1
000843816 9141_ $$y2018
000843816 915__ $$0StatID:(DE-HGF)0200$$2StatID$$aDBCoverage$$bSCOPUS
000843816 915__ $$0StatID:(DE-HGF)1030$$2StatID$$aDBCoverage$$bCurrent Contents - Life Sciences
000843816 915__ $$0LIC:(DE-HGF)CCBY4$$2HGFVOC$$aCreative Commons Attribution CC BY 4.0
000843816 915__ $$0StatID:(DE-HGF)0600$$2StatID$$aDBCoverage$$bEbsco Academic Search
000843816 915__ $$0StatID:(DE-HGF)0100$$2StatID$$aJCR$$bPLANT METHODS : 2015
000843816 915__ $$0StatID:(DE-HGF)0501$$2StatID$$aDBCoverage$$bDOAJ Seal
000843816 915__ $$0StatID:(DE-HGF)0500$$2StatID$$aDBCoverage$$bDOAJ
000843816 915__ $$0StatID:(DE-HGF)0111$$2StatID$$aWoS$$bScience Citation Index Expanded
000843816 915__ $$0StatID:(DE-HGF)0150$$2StatID$$aDBCoverage$$bWeb of Science Core Collection
000843816 915__ $$0StatID:(DE-HGF)9900$$2StatID$$aIF < 5
000843816 915__ $$0StatID:(DE-HGF)0510$$2StatID$$aOpenAccess
000843816 915__ $$0StatID:(DE-HGF)0030$$2StatID$$aPeer Review$$bASC
000843816 915__ $$0StatID:(DE-HGF)1060$$2StatID$$aDBCoverage$$bCurrent Contents - Agriculture, Biology and Environmental Sciences
000843816 915__ $$0StatID:(DE-HGF)0310$$2StatID$$aDBCoverage$$bNCBI Molecular Biology Database
000843816 915__ $$0StatID:(DE-HGF)1050$$2StatID$$aDBCoverage$$bBIOSIS Previews
000843816 915__ $$0StatID:(DE-HGF)0300$$2StatID$$aDBCoverage$$bMedline
000843816 915__ $$0StatID:(DE-HGF)0199$$2StatID$$aDBCoverage$$bThomson Reuters Master Journal List
000843816 920__ $$lyes
000843816 9201_ $$0I:(DE-Juel1)IBG-2-20101118$$kIBG-2$$lPflanzenwissenschaften$$x0
000843816 980__ $$ajournal
000843816 980__ $$aVDB
000843816 980__ $$aI:(DE-Juel1)IBG-2-20101118
000843816 980__ $$aUNRESTRICTED
000843816 9801_ $$aFullTexts