Exploring Simple Siamese Representation Learning

What is the problem with Siamese Networks? Problem: Collapsing solutions

Cause: Attributed to a lack of repulsive component in the optimization objective January 14, 2022 Computer Aided Medical Procedures Slide 4 Training progress<br>

Proposed solutions January 14, 2022 Computer Aided Medical Procedures Slide 5 Input image(s) Network architecture Pre/post representation Distance Different augmentations Different architectures
Different ways to share the weights W/ & w/o prediction or projection
W/ & w/o stop gradient Different distances<br>

Exploring Simple Siamese Representation Learning Related Work<br>

Contrastive Learning Based on: Attraction between positive sample pairs and the repulse of negative pairs.

Loss function example: Contrastive Loss Function (2005) [9]







Other popular contrastive learning loss functions: Noise Contrastive Estimation(2010), Triple Loss (2015), InfoNCE Loss (2018) January 14, 2022 Computer Aided Medical Procedures Slide 7<br>

Contrastive Learning: SimCLR (Simple framework for Contrastive Learning of visual Reprs.) [2] January 14, 2022 Computer Aided Medical Procedures Slide 8<br>

Clustering Based on: alternation of
Clustering the representation
Learning to predict the cluster assignments

Important fact: Not defined negative samples → Cluster centres play as negative prototypes January 14, 2022 Computer Aided Medical Procedures Slide 9<br>

Clustering: SwAV (SWapping Assignments between multiple Views of the same image) [4] January 14, 2022 Computer Aided Medical Procedures Slide 10 ResNet-50 based encoder Image augmentations Representations (projected into unit sphere) Online with Sinkhorn-Knopp alg.<br>

BYOL (Bootstrap Your Own Latent) [5] January 14, 2022 Computer Aided Medical Procedures Slide 11 ResNet-50 based encoders with Exponential Moving Average) Image augmentations Representations MSE between normalized predictions and target projections<br>

Exploring Simple Siamese Representation Learning Method<br>

SimSiam (Simple Siamese) [1] January 14, 2022 Computer Aided Medical Procedures Slide 13 Image augmentations ResNet-50 + Three-layers Projection MLP head encoder Two-layers MLP (bottleneck structure 2048-512-2048) Predictor output compared to encoder output with stop-grad<br>

SimSiam (Simple Siamese) [1] Loss: January 14, 2022 Computer Aided Medical Procedures Slide 14<br>

Baseline configuration Optimization:
SGD with cosine decay and schedule learning rate
Base learning rate: 0.05
Momentum: 0.9
Weight decay: 0.0001
Batch size: 256

Architecture aspects:
Encoder:
ResNet-50 + Projection MLP head (3 layers)
BN in each FC layer
Hidden FC of dim = 2.048.
Predictor:
Two layers MLP
BN in its hidden FC layer
Bottleneck structure: 2048-512-2048 January 14, 2022 Computer Aided Medical Procedures Slide 15<br>

Optimization problem:






Solution: Alternating algorithm An implementation of the Expectation-Maximization (EM)? January 14, 2022 Computer Aided Medical Procedures Slide 16 Network Augmentation<br>

SimSiam’s architecture vs. other options January 14, 2022 Computer Aided Medical Procedures Slide 17<br>

Exploring Simple Siamese Representation Learning Experiments and Results<br>

Why SimSiam’s solutions do not collapse? January 14, 2022 Computer Aided Medical Procedures Slide 19<br>

Why SimSiam’s solutions do not collapse? January 14, 2022 Computer Aided Medical Procedures Slide 20<br>

Why SimSiam’s solutions do not collapse? January 14, 2022 Computer Aided Medical Procedures Slide 21<br>

Comparison – ImageNet Classification January 14, 2022 Computer Aided Medical Procedures Slide 22<br>

Comparison – Transfer Learning January 14, 2022 Computer Aided Medical Procedures Slide 23<br>

Exploring Simple Siamese Representation Learning Conclusions & Student’s Review<br>

Conclusions New simple Siamese method
Model on par or improving previous complex models
Good model for learning representations for transfer learning January 14, 2022 Computer Aided Medical Procedures Slide 25<br>

Strengths Achieved simple state-of-the-art model
Achieved common architecture on which to make improvements January 14, 2022 Computer Aided Medical Procedures Slide 26 Weaknesses Collapsing solution prevention only shown through empirical observations
Comparison carried out only with ImageNet<br>

Suggestion for Future Work Application of the acquired improvement in NLP (embeddings)
Theoretical proof of collapsing solution prevention January 14, 2022 Computer Aided Medical Procedures Slide 27 Application to medical data Application of the model in medical applications in which data representations are required, e.g., obtaining image representations to compare while searching for similar ones in an image bank.<br>

Exploring Simple Siamese Representation Learning References<br>

[1] Chen, X., & He, K. (2021). Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 15750-15758).
 
[2] Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020). A simple framework for contrastive learning of visual representations. In International conference on machine learning (pp. 1597-1607). PMLR.
 
[3] He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020). Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 9729-9738).
 
[4] Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., & Joulin, A. (2020). Unsupervised learning of visual features by contrasting cluster assignments. arXiv:2006.09882.
 
[5] Grill, J. B., Strub, F., Altché, F., Tallec, C., Richemond, P. H., Buchatskaya, E., ... & Valko, M. (2020). Bootstrap your own latent: A new approach to self-supervised learning. arXiv:2006.07733.
 
[6] He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770-778).
 
[7] You, Y., Gitman, I., & Ginsburg, B. (2017). Large batch training of convolutional networks. arXiv:1708.03888.
 
[8] Deng, J., Dong, W., Socher, R., Li, L. J., Li, K., & Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition (pp. 248-255).

[9] Chopra, S., Hadsell, R., & LeCun, Y. (2005, June). Learning a similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) (Vol. 1, pp. 539-546). IEEE. January 14, 2022 Computer Aided Medical Procedures Slide 29<br>