PaperReviewer Matching Dina Elreedy Supervised by Prof Sanmay Das Agenda Problem Definition and Motivation System Design Datasets Experiments Results Extension Problem Definition and ID: 549014
Download Presentation The PPT/PDF document "October Rotation" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
October Rotation Paper-Reviewer Matching
Dina
Elreedy
Supervised by: Prof.
Sanmay
DasSlide2
Agenda
Problem Definition and Motivation
System Design
Datasets
Experiments Results
ExtensionSlide3
Problem Definition and Motivation
Large Conferences receive hundreds (or thousands) of papers and have hundreds of reviewers!
Constraints
Reviewer Load
Matching Quality
Challenging Task
!!
Automatic
Reviewer-Paper Matching systems try to
maximize
overall reviewers’ preferences through the
assignment.Slide4
Problem Formulation
[1] L.
Charlin
and R. S.
Zemel
, “The
toronto
paper matching system: an automated paper-reviewer assignment system,” in International Conference on Machine Learning (ICML), 2013.
Given: Matrix of paper-reviewer preferences AGoal: Find a matching Y satisfying constraints and maximizing total affinity.Challenge: Input preferences matrix A is very sparse!
We have followed same structure of
Toronto Paper matching system
[1
]
, which is widely used in large AI conferences (
NIPS, ICML, UAI, AISTATS
,..
etc
).Slide5
System Design
Prediction
Module (BPMF
)
Optimization Module
Given Preferences
A
Filled
Preferences
A’
Matching
Y Slide6
Prediction Module
Collaborative Filtering is successfully used in Recommender Systems.
We have used Bayesian Probabilistic Matrix Factorization(BPMF), the public available implementation of [2].
Matrix Factorization
U
: Matrix of Papers’ Latent Variables V: Matrix of Reviewers’ Latent Variables BPMF assumes Gaussian prior distribution for U and V.[2] R. Salakhutdinov and A.
Mnih
, “Bayesian probabilistic matrix factorization using
markov
chain
monte
carlo
,”inProceedingsofthe25thinternationalconferenceonMachinelearning. ACM,2008,pp.880–887.
A=U
TVSlide7
Bayesian PMF
A=R in this figure
Gibbs Sampling is used to estimate Posterior Probabilities of U
and V.Slide8
System Design
Prediction
Module (BPMF
)
Optimization Module
Given Preferences
A
Filled
Preferences
A’
Matching
Y Slide9
Optimization Module
Objective Function
Maximize Affinity of Matching.
Considered Constraints
Number
of
reviewers per
p
aper Number of Papers per reviewer
Can be solved using Linear Programming using Taylor Formulation
3
3
.
C
. J. Taylor, “On the optimal assignment of conference papers to reviewers,” 2008. Slide10
Datasets
NIPS
2006 Paper-Reviewer Dataset
148
papers submitted to NIPS 2006
and
364
reviewers.Reviewers’ preferences range from 0 to
3Data Sparsity (Percentage of Known Ratings)=393/(148*365)= 0.0073Netflix Movie Rating Dataset We have used a small portion of Netflix Dataset as it is a very large dataset (6000 movies and 3500 users)that makes optimization intractable. We have only considered the problem of 300 users and 500 movies.Data Sparsity= 4035/(500*300)= 0.0269Slide11
Experiments and Results
For
each dataset, we evaluate both
prediction
accuracy(RMSE)
and
matching
quality (Affinity score).
NIPS DatasetNetflix DatasetSlide12
Results (cont.)
NIPS Dataset
Netflix DatasetSlide13
ExtensionDevelop Active Learning Strategies for ratings elicitation to enhance matching quality.
We aim at selecting the most useful pairs for the matching processing (November Rotation).Slide14
Thank you