Gast ::
| Hauptseite > Publikationsdatenbank > Rescuing Easy Samples in Self-Supervised Pretraining |
| Hauptseite > Publikationsdatenbank > Rescuing Easy Samples in Self-Supervised Pretraining |
| Contribution to a conference proceedings/Contribution to a book | FZJ-2025-04879 |
; ;
2025
SCITEPRESS - Science and Technology Publications
ISBN: 978-989-758-728-3
This record in other databases:
Please use a persistent id in citations: doi:10.5220/0013167900003912
Abstract: Many recent self-supervised pretraining methods use augmented versions of the same image as samples for their learning schemes. We observe that ’easy’ samples, i.e. samples being too similar to each other after augmentation, have only limited value as learning signal. We therefore propose to rescue easy samples and make them harder. To do so, we select the top k easiest samples using cosine similarity, strongly augment them, forward-pass them through the model, calculate cosine similarity of the output as loss, and add it to the original loss in a weighted fashion. This method can be adopted to all contrastive or other augmented-pair based learning methods, whether they involve negative pairs or not, as it changes handling of easy positives, only. This simple but effective approach introduces greater variability into such self-supervised pretraining processes, significantly increasing the performance on various downstream tasks as observed in our experiments. We pretrain models of di fferent sizes, i.e. ResNet-50, ViT-S, ViT-B, or ViT-L, using ImageNet with SimCLR, MoCo v3, or DINOv2 training schemes. Here, e.g., we consistently find to improve results for ImageNet top-1 accuracy with a linear classifier establishing new SOTA for this task.
|
The record appears in these collections: |
PDF
Volltext