Dual Graph: A Graph based method for reasoning

slide1. Dual Graph: A Graph based method for reasoning about label noise Chair for Computer Aided Medical Procedures & Augmented Reality

Deep Learning for Medical Application Master Seminar HaiYang Zhang, XiMing Xing, Liang Liu Beijing University of Posts and Telecommunications School of Computer Science Student: Mohmmad Kashif Akhtar
Tutor: Mahsa Ghorbani<br>
slide2. Outline Introduction
Motivation and Challenges
Related Work
Methodology
Experiments and Results
Ablation study
Conclusion
Student’s Review January 20, 2022 Computer Aided Medical Procedures Slide 2<br>
slide3. Motivation and Challenges Dataset used to train neural networks is either crowd-sourced [1] or downloaded via web crawlers [2]

Both these techniques are unreliable and introduces error in labels

Large neural networks requires large datasets , downloading and cleaning large dataset is a hectic work

Neural networks in earlier training epochs try to learn correct labels instances mapping but as the number of epochs increases, they starts memorizing noisy labels instances mapping

This deteriorates performance of neural networks and results in poor performance on test dataset [3].

[1]. Yan Yan, Romer Rosales, Glenn Fung, Ramanathan Subramanian, and Jennifer Dy. Learning from multiple annotators with varying expertise. Machine Learning, 95(3):291–327, 2014.
[2] Yunchao Gong, Qifa Ke, Michael Isard, and Svetlana Lazebnik. A multi-view embedding space for modelling internet images, tags, and their semantics. International journal of computer vision, 106(2):210–233, 2014.
[3] Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J. Gordon. An empirical study of example forgetting during deep neural network learning. In International Conference on Learning Representations (ICLR), 2019. January 20, 2022 Computer Aided Medical Procedures Slide 3<br>
slide4. Related Works Label transition matrix(T) estimation: heuristically estimates probability of a label transitioning into other possible noisy labels. e.g. F-correction [4]. Disadvantage: hard to estimate T for large number of classes.
Sample selection: uses two networks, first network selects small loss instances and feeds it into second network to train it and vice versa. e.g. Co-teaching [5] & Co-teaching+ [6]. Disadvantage: removes too many instances or keeping wrongly labelled instances
Correcting noisy labels to true labels: update model parameters and labels alternatively. Uses prototypes to represent labels deep feature distribution and then check similarity between data sample and protype to find correct label. e.g. Joint Optimization [7] & PENCIL [8]. Disadvantage: unreliable as sometimes changes true labels to some other labels
Graph-based approaches: uses graphs to deal with label noise correction task. e.g. GCN label noise cleaner [9], Face Graph [10]. Disadvantage: either requires clean subsets or limited to specific domain.
[5] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in neural information processing systems (NIPS), pages 8527– 8537, 2018.
[6] Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor W Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? In International Conference on Machine Learning (ICML), 2019.
[7] Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization framework for learning with noisy labels. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5552–5560, 2018.
[8] Kun Yi and Jianxin Wu. Probabilistic end-to-end noise correction for learning with noisy labels. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7017–7025, 2019.
[9] Jia-Xing Zhong, Nannan Li, Weijie Kong, Shan Liu, Thomas H Li, and Ge Li. Graph convolutional label noise cleaner: Train a plug-and-play action classifier for anomaly detection. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1237–1246, 2019.
[10] Yaobin Zhang, Weihong Deng, Mei Wang, Jiani Hu, Xian Li, Dongyue Zhao, and Dongchao Wen. Globallocal gcn: Large-scale label noise cleansing for face recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020. January 20, 2022 Computer Aided Medical Procedures Slide 4<br>
slide5. Methodology Uses graph neural networks to correct label noise

Captures structural relationship among labels at two levels, instance and distribution level

Train two graph neural networks iteratively in alternating fashion until convergence or ‘K’ iterations.

Instance graph captures instance level relation and distribution graph captures distribution level relation

Distribution graph is used to refine the instance graph in order to learn more discriminative features. January 20, 2022 Computer Aided Medical Procedures Slide 5<br>
slide6. Proposed Model January 20, 2022 Computer Aided Medical Procedures Slide 6 Fig 1: Dual Graph<br>
slide7. Algorithm January 20, 2022 Computer Aided Medical Procedures Slide 7<br>
slide8. Initialize both Graphs Creates two undirected acyclic graphs, with number of nodes equals to number of samples in a batch ‘p’.

Initialize instance graph nodes by passing images through a convolutional neural network to get the node embeddings
(3)

Distribution graph nodes are initialized by a ‘p’ dimension vector where j-th index of i-th vector is 1 if both samples have same labels otherwise 0, and 1/p if label is unknown to have unifrorm probability

(6) January 20, 2022 Computer Aided Medical Procedures Slide 8<br>
slide9. Instance similarity update Instance similarity is the edge weight in instance graph which shows how similar two nodes are in instance graph

Computes dissimilarity vector in Euclidean space which shows how dissimilar two nodes ‘i’ and ‘j’ are

This dissimilarity vector is passed through a neural network to get a edge weight showing the probability of how similar two nodes are

The neural network consists of two Conv-ReLU-BatchNorm blocks, followed by one dropout and one sigmoid activation layer

(4) January 20, 2022 Computer Aided Medical Procedures Slide 9<br>
slide10. Distribution feature generation Distribution graph integrates instance level relation from instance graph to generate distribution features

Passes instance graph edge weights and distribution graph node features through an aggregation network, consisting of fully connected followed by a leaky relu.


(7)

Each node of distribution graph is ‘p’ dimension vector where ‘j-th’ entry of ‘i-th’ node represents the relation between samples ‘i’ and ‘j’ January 20, 2022 Computer Aided Medical Procedures Slide 10<br>
slide11. Distribution similarity update Edge weights of distribution graph is calculated

It represents the similarity between distribution features of each sample

Edge weights of distribution graph are calculated in similar way as that of instance graph, by calculating dissimilarity vector and then passing through a similar neural network

(8) January 20, 2022 Computer Aided Medical Procedures Slide 11<br>
slide12. Rebuild instance graph Uses the distribution relation that was captured during the construction of distribution graph to reconstruct instance graph nodes at the end of each iteration

Uses distribution relation to reweight samples in order to remove abnormal noise points from the perspective of distribution

The reconstructed instance graph nodes captures the distribution relations by aggregating all node vectors with distribution graph edge weights

The aggregation network consists of Convolution block followed by leaky relu and batch norm layers

(10) January 20, 2022 Computer Aided Medical Procedures Slide 12<br>
slide13. Loss function Class prediction of a sample is done by aggregating all its neighbours one-hot encoding weighted by their instance similarity edge weights

Instance graph loss is the cross entropy loss between the above prediction and ground truth label

Distribution graph loss is the cross entropy between prediction, made by distribution features, and the ground truth

Total loss is the weighted sum of these two losses summed over all iterations January 20, 2022 Computer Aided Medical Procedures Slide 13<br>
slide14. Experiments Used CIFAR-10, CIFAR-100 and Clothing 1M datasets

In CIFAR datasets introduced two types of noises, symmetric [11] and asymmetric [12] noise

Experimented with Clothing 1M datasets in 3 ways:
Used only noisy labels dataset
Verification labels to initialize the graph, so that it has stable initial topology
Both noisy dataset and 50 K clean labels used to first train model on noisy dataset and then fine-tuned using clean dataset
Used Resnet-12 to train CIFAR datasets and Resnet-34 pretrained on image-net dataset to train Clothing 1M dataset

[11] Brendan van Rooyen, Aditya Krishna Menon, and Robert C. Williamson. Learning with symmetric label noise: The importance of being unhinged. In International Conference on Neural Information Processing Systems (NIPS), page 1018, 2015.
[12] Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1944–1952, 2017. January 20, 2022 Computer Aided Medical Procedures Slide 14<br>
slide15. Results on CIFAR datasets January 20, 2022 Computer Aided Medical Procedures Slide 15<br>
slide16. Observation on CIFAR results At symmetry 20%, which is a natural error rate, all models performed well but dual graph performed significantly better

At symmetry 50% and asymmetry 40% Co-teaching begins to fail but Dual-Graph performs significantly better

At symmetry 80% is the hardest, still Dual-Graph achieved best result

Dual-Graph beats other models in asymmetric-flipping also by a significant margin

CIFAR 100 overall accuracy is less when compared to performance on CIFAR 10 since 100 classes but still Dual-Graph performed the best compared to other models January 20, 2022 Computer Aided Medical Procedures Slide 16<br>
slide17. Result on Clothing 1M dataset First seven are evaluation using only noisy dataset

8 to 10 uses verification labels

11 to 12 uses clean subset to fine tune the model

Dual graph still performs best in all cases January 20, 2022 Slide 17 Computer Aided Medical Procedures<br>
slide18. Ablation study January 20, 2022 Computer Aided Medical Procedures Slide 18<br>
slide19. Conclusion January 20, 2022 Computer Aided Medical Procedures Slide 19 Captures structural relationship among labels at two levels: instance and distribution level, using graph neural network.

Since distribution level relation is robust to noise, this network propagates distribution level relation as a supervised signal to refine instance level relation.

Uses iterative optimization technique to create distribution graph from instance graph and then uses this distribution graph to update instance graph.

This end-to-end training technique counteract noisy labels while generating reliable predictions.<br>
slide20. Student’s Review Strengths
Regularization techniques like L2 regularization, dropout & early stopping don’t guarantee optimization as they prevents the network from reducing loss
But this technique doesn’t prevent network from reducing loss, instead it reweights each instance loss contribution by their similarity edge weights
Not domain specific, can be trained on any type of images to reduce label noise
Doesn’t require a clean subset for pretraining, which could be hectic task.

Weaknesses
Didn’t provide any code to verify approach
Beside Resnet no other model used
Method wasn’t tested on any other modality besides images, text embeddings could have been used to initialize nodes vector to correct text noisy labels. January 20, 2022 Computer Aided Medical Procedures Slide 20<br>
slide21. Thank you!<br>