Quick Introduction of Batch Normalization Hung-yi
Description: Quick Introduction of Batch Normalization Hung-yi Lee 李宏毅 1 Changing Landscape 1 1, 2 small small smooth small small 2 Changing Landscape 1 1, 2 100, 200 small large large smooth steep same range large large 3 Feature Normalization
Related Topics
Download Presentation
"Quick Introduction of Batch Normalization Hung-yi" is the property of its rightful owner. Permission is granted to download and print the materials on this website for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Presentation Transcript
slide1. Quick Introduction of Batch Normalization Hung-yi Lee 李宏毅 1<br>
slide2. Changing Landscape 1 1, 2 …… small small smooth small small 2<br>
slide3. Changing Landscape 1 1, 2 …… 100, 200 …… small large large smooth steep same
range large large 3<br>
slide4. Feature Normalization …… …… The means of all dims are 0, and the variances are all 1 In general, feature normalization makes gradient descent converge faster. 4<br>
slide5. Sigmoid …… …… …… Sigmoid Sigmoid Feature Normalization Also need normalization Different dims have different ranges. Also difficult to optimize Considering Deep Learning 5<br>
slide6. Considering Deep Learning 6<br>
slide7. Sigmoid Sigmoid Sigmoid This is a large network! Considering Deep Learning Consider a batch Batch Normalization 7<br>
slide8. Batch normalization 8<br>
slide9. Batch normalization – Testing We do not always have batch at testing stage. …… ……<br>
slide10. Batch normalization Original paper: https://arxiv.org/abs/1502.03167 10<br>
slide11. Internal Covariate Shift? How Does Batch Normalization Help Optimization? https://arxiv.org/abs/1805.11604 11 …… …… update Experimental results do not support the above idea.<br>
slide12. Internal Covariate Shift? 12 How Does Batch Normalization Help Optimization? https://arxiv.org/abs/1805.11604 Experimental results (and theoretically analysis) support batch normalization change the landscape of error surface. penicillin<br>
slide13. To learn more …… Batch Renormalization
https://arxiv.org/abs/1702.03275
Layer Normalization
https://arxiv.org/abs/1607.06450
Instance Normalization
https://arxiv.org/abs/1607.08022
Group Normalization
https://arxiv.org/abs/1803.08494
Weight Normalization
https://arxiv.org/abs/1602.07868
Spectrum Normalization
https://arxiv.org/abs/1705.10941 13<br>
slide14. 14<br>
slide2. Changing Landscape 1 1, 2 …… small small smooth small small 2<br>
slide3. Changing Landscape 1 1, 2 …… 100, 200 …… small large large smooth steep same
range large large 3<br>
slide4. Feature Normalization …… …… The means of all dims are 0, and the variances are all 1 In general, feature normalization makes gradient descent converge faster. 4<br>
slide5. Sigmoid …… …… …… Sigmoid Sigmoid Feature Normalization Also need normalization Different dims have different ranges. Also difficult to optimize Considering Deep Learning 5<br>
slide6. Considering Deep Learning 6<br>
slide7. Sigmoid Sigmoid Sigmoid This is a large network! Considering Deep Learning Consider a batch Batch Normalization 7<br>
slide8. Batch normalization 8<br>
slide9. Batch normalization – Testing We do not always have batch at testing stage. …… ……<br>
slide10. Batch normalization Original paper: https://arxiv.org/abs/1502.03167 10<br>
slide11. Internal Covariate Shift? How Does Batch Normalization Help Optimization? https://arxiv.org/abs/1805.11604 11 …… …… update Experimental results do not support the above idea.<br>
slide12. Internal Covariate Shift? 12 How Does Batch Normalization Help Optimization? https://arxiv.org/abs/1805.11604 Experimental results (and theoretically analysis) support batch normalization change the landscape of error surface. penicillin<br>
slide13. To learn more …… Batch Renormalization
https://arxiv.org/abs/1702.03275
Layer Normalization
https://arxiv.org/abs/1607.06450
Instance Normalization
https://arxiv.org/abs/1607.08022
Group Normalization
https://arxiv.org/abs/1803.08494
Weight Normalization
https://arxiv.org/abs/1602.07868
Spectrum Normalization
https://arxiv.org/abs/1705.10941 13<br>
slide14. 14<br>