001043528 001__ 1043528
001043528 005__ 20250716202229.0
001043528 037__ $$aFZJ-2025-02904
001043528 041__ $$aEnglish
001043528 1001_ $$0P:(DE-Juel1)177675$$aUpschulte, Eric$$b0$$eCorresponding author$$ufzj
001043528 1112_ $$aHelmholtz AI Conference 2025$$cKarlsruhe$$d2025-06-03 - 2025-06-05$$gHAICON25$$wGermany
001043528 245__ $$aConText Transformer: Text-guided Instance Segmentation in Scientific Imaging
001043528 260__ $$c2025
001043528 3367_ $$033$$2EndNote$$aConference Paper
001043528 3367_ $$2BibTeX$$aINPROCEEDINGS
001043528 3367_ $$2DRIVER$$aconferenceObject
001043528 3367_ $$2ORCID$$aCONFERENCE_POSTER
001043528 3367_ $$2DataCite$$aOutput Types/Conference Poster
001043528 3367_ $$0PUB:(DE-HGF)24$$2PUB:(DE-HGF)$$aPoster$$bposter$$mposter$$s1752680519_22200$$xAfter Call
001043528 520__ $$aScientific imaging gives rise to a multitude of different segmentation tasks, many of which involve manually annotated datasets. We have collected a large number of such heterogeneous datasets, comprising over 10 million instance annotations, and demonstrate that in a multi-task setting, segmentation models at this scale cannot be effectively trained using solely image-based supervised learning. A major reason is that images from the same domain may be used to address different research questions, with varying annotation procedures and styles. For example, images of biological tissues may be evaluated for nuclei or cell bodies, despite using the same image modality. To overcome these challenges, we propose using simple text-based task descriptions to provide models with the necessary context for solving a given objective. We introduce the ConText Transformer, which implements a dual-stream architecture, processing and fusing both image and text data. Based on the provided textual descriptions, the model learns to adapt its internal feature representations to effectively switch between segmenting different classes and annotation styles observed in the datasets. These descriptions can range from simple class names (e.g., “white blood cells”)—prompting the model to only segment the referenced class—to more nuanced formulations such as toggling the use of overlapping segmentations in model predictions or segmenting a nucleus, even in the absence of cytoplasm or membrane, as is common in datasets like TissueNet but omitted in Cellpose. Since interpreting these descriptions is part of the model training, it is also possible to define dedicated terms abbreviating very complex descriptions. ConText Transformer is designed for compatibility. It can be used with existing segmentation frameworks, including the Contour Proposal Network (CPN) or Mask R-CNN. Our experiments on over 10 million instance annotations show that ConText Transformer models achieve competitive segmentation performance and outperform specialized models in several benchmarks; confirming that a single, unified model can effectively handle a wide spectrum of segmentation tasks; and eventually may replace specialist models in scientific image segmentation
001043528 536__ $$0G:(DE-HGF)POF4-5254$$a5254 - Neuroscientific Data Analytics and AI (POF4-525)$$cPOF4-525$$fPOF IV$$x0
001043528 536__ $$0G:(GEPRIS)313856816$$aDFG project G:(GEPRIS)313856816 - SPP 2041: Computational Connectomics (313856816)$$c313856816$$x1
001043528 536__ $$0G:(EU-Grant)101147319$$aEBRAINS 2.0 - EBRAINS 2.0: A Research Infrastructure to Advance Neuroscience and Brain Health (101147319)$$c101147319$$fHORIZON-INFRA-2022-SERV-B-01$$x2
001043528 536__ $$0G:(DE-HGF)InterLabs-0015$$aHIBALL - Helmholtz International BigBrain Analytics and Learning Laboratory (HIBALL) (InterLabs-0015)$$cInterLabs-0015$$x3
001043528 536__ $$0G:(DE-Juel-1)E.40401.62$$aHelmholtz AI - Helmholtz Artificial Intelligence  Coordination Unit – Local Unit FZJ (E.40401.62)$$cE.40401.62$$x4
001043528 7001_ $$0P:(DE-Juel1)131631$$aAmunts, Katrin$$b1$$ufzj
001043528 7001_ $$0P:(DE-Juel1)165746$$aDickscheid, Timo$$b2$$ufzj
001043528 909CO $$ooai:juser.fz-juelich.de:1043528$$popenaire$$pVDB$$pec_fundedresources
001043528 9101_ $$0I:(DE-588b)5008462-8$$6P:(DE-Juel1)177675$$aForschungszentrum Jülich$$b0$$kFZJ
001043528 9101_ $$0I:(DE-588b)5008462-8$$6P:(DE-Juel1)131631$$aForschungszentrum Jülich$$b1$$kFZJ
001043528 9101_ $$0I:(DE-588b)5008462-8$$6P:(DE-Juel1)165746$$aForschungszentrum Jülich$$b2$$kFZJ
001043528 9131_ $$0G:(DE-HGF)POF4-525$$1G:(DE-HGF)POF4-520$$2G:(DE-HGF)POF4-500$$3G:(DE-HGF)POF4$$4G:(DE-HGF)POF$$9G:(DE-HGF)POF4-5254$$aDE-HGF$$bKey Technologies$$lNatural, Artificial and Cognitive Information Processing$$vDecoding Brain Organization and Dysfunction$$x0
001043528 9141_ $$y2025
001043528 920__ $$lyes
001043528 9201_ $$0I:(DE-Juel1)INM-1-20090406$$kINM-1$$lStrukturelle und funktionelle Organisation des Gehirns$$x0
001043528 980__ $$aposter
001043528 980__ $$aVDB
001043528 980__ $$aI:(DE-Juel1)INM-1-20090406
001043528 980__ $$aUNRESTRICTED