Francois de Sorbier Hiroyuki Shiino Hideo Saito Introduction Overview of our system Violin extraction and 3D registration Virtual advising Conclusion Outline Violin Pedagogy for Finger and Bow Placement using Augmented Reality F de Sorbier ID: 594006
Download Presentation The PPT/PDF document "Violin Pedagogy for Finger and Bow Place..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Violin Pedagogy for Finger and Bow Placement using Augmented Reality
Francois de
Sorbier
Hiroyuki
Shiino
Hideo
SaitoSlide2
Introduction
Overview of our system
Violin extraction and 3D registrationVirtual advisingConclusion
Outline
Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier
2Slide3
Violin is a beautiful instrument…… but
one of the most complicatedNo fret on the fingerboardNo help for the position the bow on stringsMotivation
Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier
3
I
ntroductionSlide4
Music JacketVibro
-tactile feedback
Guide the bowing armGuitar playing supportTracking with markerGuide with virtual hand
Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier
4
Previous works
Y.
Motokawa
, H. Saito.
“Support
system for guitar playing using augmented reality
display”.
In Proceedings of the 5th IEEE and ACM
ISMAR
,
243-244,
2006
van der Linden, J.,
Schoonderwaldt
, E. and Bird, J.
“Good Vibrations: Guiding Body Movements with Vibrotactile Feedback”. Proceedings of the Third International Workshop on Physicality, 13-18, 2009
I
ntroductionSlide5
Overlay virtual information on the violinVirtual frets
Guides for the bow and fingers
Sound analysisViolin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier
5
Our objective
I
ntroductionSlide6
No intrusive deviceNo marker
Real time feedback
Marker-free violin trackingusing a RGBD cameraViolin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier
6
Our constraints
I
ntroductionSlide7
Tracking of the violinFeedback displayed on the screen
Constant view of the violin
Virtual informationViolin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier
7
Overview of our system
Overview of our systemSlide8
Features detection and extraction
Many occlusions caused by the player
The surface has a poor textureThe material is highly specular
Difficult to use features in this context
Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier
8
Marker less tracking
Overview of our systemSlide9
Use Kinect for tracking the violinDepth values for the pose estimation
Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier
9
Depth-based pose estimation
Overview of our systemSlide10
Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de
Sorbier
10Workflow
Overview of our system
Color
Depth
Segmentation
V
iolin detection
Violin extraction
Database
Registration
Virtual information
displayedSlide11
Detect the brown color in the imageRemove noise
Many parts are missing
OcclusionsSpecular materialStrings and fingerboardNot enough for tracking
Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier
11
Color segmentation
Violin extraction and 3D registrationSlide12
Get the 3D points from extracted colorCompute the corresponding plane equation
Optimized with RANSAC
Align a 3D volume to the planeTypical dimensions of a violinKeep the 3D points in itViolin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier
12
Extended segmentation
Violin extraction and 3D registrationSlide13
Iterative Closest Point algorithm
Compare input points with a model
Slow if too many pointsInaccurate if not enoughOur proposed solutionIncrease the number of modelsReduce the number of points per modelFast retrieval with a plane equation comparison
Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier
13
3D registration with ICP
Violin extraction and 3D registrationSlide14
Offline phase
25 models
Compare the plane equationsStore candidate if the difference is big enoughStore also the plane equationViolin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier
14
Creating the database
Violin extraction and 3D registrationSlide15
Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier
15
Result of the tracking
Violin extraction and 3D registrationSlide16
Real time (21 milliseconds)Pose also estimated using markers
Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier
16
Evaluation of the tracking
Violin extraction and 3D registration
Rx(
deg
)
Ry
(
deg
)
Rz
(
deg
)
T(mm)
Minimum error
0.12
0.25
0.20
0.22
Maximum error
13.29
8.27
7.89
32.1
Average error
3.07
2.69
2.78
7.20Slide17
Use the pose estimationLocation manually defined during the capture of the models
Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier
17
Display the frets
Virtual advisingSlide18
Emphasize the string to be pressedDisplay a red dot at the junction of the string and the fret
Define where the finger has to press the string
Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier18
Fingering advices
Virtual advisingSlide19
Suggest the violinist to play a given noteDisplay the fret/string to be pressed
Analyze the sound obtained
If fingering is considered correctAdvice about the position of the bow given the difference of pitch (OK / LOW / HIGH)Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier
19
Other advices
Virtual advisingSlide20
Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier
20
Results
Virtual advisingSlide21
Performed by confirmed playerBowing is correct
Compute the difference of pitch on each fret with the expected one
Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier
21
Evaluation of the results
Fret number
1
2
3
4
5
6
7
8
9
average
Difference of pitch
11.1
14.1
12.0
12.4
13.4
15.8
12.8
13.9
19.2
13.8
Virtual advisingSlide22
Original marker-free method for virtual advising on a violinMethod based on several pre-computed models
Real-time
AccurateDisplay virtual guides on the fingerboardAnalyze the note played for further advicesViolin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier
22
Conclusions
Virtual advisingSlide23
Perform a user based analysisValidate or improve our approach
Use different kind of display
See-through HMDSpatial augmented realityApply to other similar instrumentsJapanese shamisen
Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier
23
Future worksSlide24
Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier
24
Questions
Thank you for your attention