Quick Introduction of Batch Normalization Hung-yi

natalia-silvester

Published 2026-09-07 · 14 slides · 0 views

↓ Download

Quick Introduction of Batch Normalization Hung-yi

Quick Introduction of Batch Normalization Hung-yi - slide 1 of 14

Quick Introduction of Batch Normalization Hung-yi - slide 2 of 14

Quick Introduction of Batch Normalization Hung-yi - slide 3 of 14

Quick Introduction of Batch Normalization Hung-yi - slide 4 of 14

Quick Introduction of Batch Normalization Hung-yi - slide 5 of 14

Quick Introduction of Batch Normalization Hung-yi - slide 6 of 14

Quick Introduction of Batch Normalization Hung-yi - slide 7 of 14

Quick Introduction of Batch Normalization Hung-yi - slide 8 of 14

Quick Introduction of Batch Normalization Hung-yi - slide 9 of 14

Quick Introduction of Batch Normalization Hung-yi - slide 10 of 14

Quick Introduction of Batch Normalization Hung-yi - slide 11 of 14

Quick Introduction of Batch Normalization Hung-yi - slide 12 of 14

Quick Introduction of Batch Normalization Hung-yi - slide 13 of 14

Quick Introduction of Batch Normalization Hung-yi - slide 14 of 14

Description

Quick Introduction of Batch Normalization Hung-yi Lee 李宏毅 1 Changing Landscape 1 1, 2 small small smooth small small 2 Changing Landscape 1 1, 2 100, 200 small large large smooth steep same range large large 3 Feature Normalization

Download this presentation From Below

"Quick Introduction of Batch Normalization Hung-yi" is the property of its rightful owner. Permission is granted to download and print the materials on this website for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.

Presentation Transcript

Quick Introduction of Batch Normalization Hung-yi Lee 李宏毅 1

Changing Landscape 1 1, 2 …… small small smooth small small 2

Changing Landscape 1 1, 2 …… 100, 200 …… small large large smooth steep same
range large large 3

Feature Normalization …… …… The means of all dims are 0, and the variances are all 1 In general, feature normalization makes gradient descent converge faster. 4

Sigmoid …… …… …… Sigmoid Sigmoid Feature Normalization Also need normalization Different dims have different ranges. Also difficult to optimize Considering Deep Learning 5

Considering Deep Learning 6

Sigmoid Sigmoid Sigmoid This is a large network! Considering Deep Learning Consider a batch Batch Normalization 7

Batch normalization 8

Batch normalization – Testing We do not always have batch at testing stage. …… ……

Batch normalization Original paper: https://arxiv.org/abs/1502.03167 10

Internal Covariate Shift? How Does Batch Normalization Help Optimization? https://arxiv.org/abs/1805.11604 11 …… …… update Experimental results do not support the above idea.

Internal Covariate Shift? 12 How Does Batch Normalization Help Optimization? https://arxiv.org/abs/1805.11604 Experimental results (and theoretically analysis) support batch normalization change the landscape of error surface. penicillin

To learn more …… Batch Renormalization
https://arxiv.org/abs/1702.03275
Layer Normalization
https://arxiv.org/abs/1607.06450
Instance Normalization
https://arxiv.org/abs/1607.08022
Group Normalization
https://arxiv.org/abs/1803.08494
Weight Normalization
https://arxiv.org/abs/1602.07868
Spectrum Normalization
https://arxiv.org/abs/1705.10941 13

14

Quick Introduction of Batch Normalization Hung-yi

Related Topics

Share

Embed code

Download this presentation From Below

Presentation Transcript