Puspita Majumdar Mayank Vatsa Richa Singh Indraprastha Institute of Information Technology Delhi IIITD India Saheb Chhabra Richa Singh Mayank Vatsa Puspita Majumdar ID: 729622
Download Presentation The PPT/PDF document "Data Fine-tuning Saheb Chhabra," is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Data Fine-tuning
Saheb Chhabra, Puspita Majumdar, Mayank Vatsa, Richa SinghIndraprastha Institute of Information Technology Delhi (IIIT-D), India
Saheb Chhabra
Richa
Singh
Mayank
Vatsa
Puspita
MajumdarSlide2
MOTIVATION: COTS2
MONEYCOTSCommercial off-the-shelf (COTS )Black-BoxCOTS RequirementsUpgradation with timeRetraining of the systemNote: Images are taken from Internet.Slide3
MOTIVATION: COTS Performance
3
Input
Expectation
Reality
Decision
COTS
Note: Images are taken from Internet
.Slide4
MOTIVATION4
Don’t know the modelCan’t access the model
Don’t know the parameters
Note: Images are taken from Internet
.Slide5
What do we
have?5DatasetModel AccessModel HyperparametersModel TrainingOutput of Model
Note: Images are taken from Internet
.Slide6
What do we
have?6DatasetModel AccessModel HyperparametersModel TrainingOutput of Model
Note: Images are taken from Internet
.
Can we enhance the performance of a black-box system?Slide7
Black Box ModelBlack Box Model
7 Weight
Bias
Input DataSlide8
Black Box ModelBlack Box Model
8 Weight
BiasInput DataSlide9
Data Fine-tuningData Fine-tuning (DFT)
9
DFTSlide10
Data Fine-tuningData Fine-tuning (DFT)
10
DFT
Pre-trained model’s
decision boundaryDFT
X-axis
Y-axis
Class 1
Class 2
(a)
X-axis
Y-axis
Class 1
Class 2
(b)Slide11
Model Fine-tuningModel Fine-tuning
11
MFTSlide12
Model Fine-tuning vs Data Fine-tuning12
X-axis
Y-axis
Class 1
Class 2
X-axis
Class 1
Class 2
Pre-trained model’s
decision boundary
Fine-tuned model’s
decision boundary
X-axis
Y-axis
Class 1
Class 2
Data
Fine-tuning
Model
Fine-tuning
(a)
(b)
(c)
Y-axis
MFT
Model Fine-tuning
DFT
Data Fine-tuningSlide13
Literature: Adversarial Perturbation13
[1]. Xie, Cihang, et al. "Mitigating adversarial effects through randomization." arXiv preprint arXiv:1711.01991 (2017).[2]. S. Chhabra, R. Singh, M. Vatsa, G. Gupta, Anonymizing k Facial Attributes via Adversarial Perturbations, IJCAI , 2018,Slide14
Data Fine-tuning: ChallengesLearn a single perturbation for a given datasetThe visual appearance of the image should be preserved
after performing data fine-tuning.14Slide15
Optimization15
Original Training Set
Perturbed Training Set
Perturbation
Set of Attributes
True Labels Number of Images
Transform image in
range of 0 to 1
Model Input
Output scores
Enforces the outputs scores
towards true labelsSlide16
Block Diagram16
Attribute Prediction
Minimize Loss
True Labels
Optimize over variable
(a)
(b)Slide17
Illustration of Data Fine-tuning17
Dataset:
Class 2
Class 1
(a)
X-axis
Y-axis
Input Image
Space
Class 2
Class 1
(b)
X-axis
Y-axis
Output Class
Scores
Training on
Dataset
Attribute
Prediction
Model
Class 1
Class 2
Dataset:
(c)
X-axis
Input Image
Space
Class 1
Class 2
(d)
X-axis
Output Class
Scores
Class 1
Class 2
(f)
X-axis
Output Class
Scores
Class 1
Class 2
(e)
X-axis
Input Image
Space
Fine tuned
Dataset:
Pre-trained
Attribute
Prediction
Model
Pre-trained
Attribute
Prediction
Model
Add
Perturbation
Data fine-tuning
Y-axis
Y-axis
Y-axis
Y-axis
(
W
+b)
(
W
X
+b
)
(
W
Z
+b
)
Slide18
Experiments and ResultsTwo experiments are performed for Facial attribute classification
Black Box Data Fine-tuning: Intra DatasetBlack Box Data Fine-tuning: Inter DatasetThe proposed algorithm is evaluated on three datasets: CelebA, LFW, and MUCT18[1] . Liu, Ziwei, et al. "Deep learning face attributes in the wild." Proceedings of the IEEE International Conference on Computer Vision. 2015.[2] . Huang, Gary B., et al. "Labeled faces in the wild: A database for studying face recognition in unconstrained environments." Workshop on faces in'Real-Life'Images:
detection
, alignment, and recognition
. 2008.[3] . Milborrow, Stephen, John Morkel, and Fred Nicolls. "The MUCT landmarked face database." Pattern Recognition Association of South Africa 201.0 (2010).Slide19
Black Box Data Fine-tuning: Intra Dataset19
LFWSmilingBushy EyebrowsPale SkinBeforeAfterBeforeAfterBeforeAfter76.1882.4268.3469.94
72.83
74.81
CelebA
SmilingAttractiveWearing LipstickBeforeAfterBeforeAfterBeforeAfter67.8271.30
70.4870.5480.9581.29Before DFTAfter DFTLFW87.9491.17CelebA82.13
83.08MUCT91.6794.31Table: Classification accuracy (%) of before and after Data Fine-tuning for ‘Gender’ attribute.Table: Classification accuracy (%) before and after Data Fine-tuning for three attributes.Classification accuracy improved by 1% to 3% for all three datasets
Overall increase in the classification accuracies of all the attributes for both the datasetsSlide20
Black Box Data Fine-tuning: Intra Dataset20
ScoreProbability DistributionFigure: Smiling attribute score distribution pertaining to before and after Data Fine-tuning on LFW datasetSmiling Score distributionOverlapping region among both the classes reduced after Data Fine-tuningSlide21
Black Box Data Fine-tuning: Intra Dataset21
Misclassified Before DFTCorrectly ClassifiedBefore DFTSmiling
Smiling
Not Smiling
Not Smiling
Bushy Eyebrows
Bushy EyebrowsNot Bushy EyebrowsNot Bushy Eyebrows
Pale SkinPale SkinNot Pale SkinNot Pale Skin
Smiling Attribute
Bushy Eyebrows Attribute
Pale Skin AttributeSlide22
Black Box Data Fine-tuning: Inter Dataset22
Dataset used to train the modelMUCTLFWCelebABeforeAfterBeforeAfterBeforeAfter
Dataset
MUCT
--
57.8483.6580.2792.84LFW63.0980.45--56.0186.33CelebA49.14
74.7367.5376.59--Table: Classification accuracy (%) for ‘Gender’ attribute.Pre-trained Model trained on CelebA
LFWSmilingBushy EyebrowsPale SkinBeforeAfterBeforeAfterBeforeAfter55.2978.6145.40
68.91
56.62
84.21
Pre-trained Model trained on LFW
CelebA
Smiling
Attractive
Wearing Lipstick
Before
After
Before
After
Before
After
49.07
66.97
49.71
66.60
60.25
77.15
Table: Classification accuracy (%) for other attributes
Significant Improvement after
Data Fine-tuning
Classification accuracy increases by atleast 12% and upto 30 after Data Fine-tuningSlide23
Black Box Data Fine-tuning: Inter Dataset23
False Positive RateTrue Positive RateDataset: LFW
Model: CelebA
Dataset: CelebAModel: LFWSlide24
Black Box Data Fine-tuning: Inter Dataset24
Before
Data Fine-tuning
After
Data Fine-tuning
ScoreProbability DistributionDataset: LFW Model trained on: CelebA:Dataset: CelebA Model trained on: LFW:
Figure: Score Distributions pertaining to before and after Data Fine-tuningSlide25
Model Fine-tuning vs Data Fine-tuning25Slide26
Summary26
Proposed a novel concept, Data Fine-tuning for enhancing the performance of black-box models.Experiments are performed on CelebA, LFW, and MUCT databases.The proposed concept, Data Fine-tuning uses adversarial perturbation to learn a single noise for a given dataset.Slide27
Thank you
27