/
Violin Pedagogy for Finger and Bow Placement using Augmente Violin Pedagogy for Finger and Bow Placement using Augmente

Violin Pedagogy for Finger and Bow Placement using Augmente - PowerPoint Presentation

marina-yarberry
marina-yarberry . @marina-yarberry
Follow
380 views
Uploaded On 2016-04-19

Violin Pedagogy for Finger and Bow Placement using Augmente - PPT Presentation

Francois de Sorbier Hiroyuki Shiino Hideo Saito Introduction Overview of our system Violin extraction and 3D registration Virtual advising Conclusion Outline Violin Pedagogy for Finger and Bow Placement using Augmented Reality F de Sorbier ID: 283771

bow violin finger augmented violin bow augmented finger reality pedagogy placement sorbier virtual extraction registration advising system tracking overview

Share:

Link:

Embed:

Download Presentation from below link

Download Presentation The PPT/PDF document "Violin Pedagogy for Finger and Bow Place..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.


Presentation Transcript

Slide1

Violin Pedagogy for Finger and Bow Placement using Augmented Reality

Francois de

Sorbier

Hiroyuki

Shiino

Hideo

SaitoSlide2

Introduction

Overview of our system

Violin extraction and 3D registrationVirtual advisingConclusion

Outline

Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier

2Slide3

Violin is a beautiful instrument…… but

one of the most complicatedNo fret on the fingerboardNo help for the position the bow on stringsMotivation

Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier

3

I

ntroductionSlide4

Music JacketVibro

-tactile feedback

Guide the bowing armGuitar playing supportTracking with markerGuide with virtual hand

Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier

4

Previous works

Y.

Motokawa

, H. Saito.

“Support

system for guitar playing using augmented reality

display”.

In Proceedings of the 5th IEEE and ACM

ISMAR

,

243-244,

2006

van der Linden, J.,

Schoonderwaldt

, E. and Bird, J.

“Good Vibrations: Guiding Body Movements with Vibrotactile Feedback”. Proceedings of the Third International Workshop on Physicality, 13-18, 2009

I

ntroductionSlide5

Overlay virtual information on the violinVirtual frets

Guides for the bow and fingers

Sound analysisViolin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier

5

Our objective

I

ntroductionSlide6

No intrusive deviceNo marker

Real time feedback

Marker-free violin trackingusing a RGBD cameraViolin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier

6

Our constraints

I

ntroductionSlide7

Tracking of the violinFeedback displayed on the screen

Constant view of the violin

Virtual informationViolin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier

7

Overview of our system

Overview of our systemSlide8

Features detection and extraction

Many occlusions caused by the player

The surface has a poor textureThe material is highly specular

Difficult to use features in this context

Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier

8

Marker less tracking

Overview of our systemSlide9

Use Kinect for tracking the violinDepth values for the pose estimation

Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier

9

Depth-based pose estimation

Overview of our systemSlide10

Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de

Sorbier

10Workflow

Overview of our system

Color

Depth

Segmentation

V

iolin detection

Violin extraction

Database

Registration

Virtual information

displayedSlide11

Detect the brown color in the imageRemove noise

Many parts are missing

OcclusionsSpecular materialStrings and fingerboardNot enough for tracking

Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier

11

Color segmentation

Violin extraction and 3D registrationSlide12

Get the 3D points from extracted colorCompute the corresponding plane equation

Optimized with RANSAC

Align a 3D volume to the planeTypical dimensions of a violinKeep the 3D points in itViolin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier

12

Extended segmentation

Violin extraction and 3D registrationSlide13

Iterative Closest Point algorithm

Compare input points with a model

Slow if too many pointsInaccurate if not enoughOur proposed solutionIncrease the number of modelsReduce the number of points per modelFast retrieval with a plane equation comparison

Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier

13

3D registration with ICP

Violin extraction and 3D registrationSlide14

Offline phase

25 models

Compare the plane equationsStore candidate if the difference is big enoughStore also the plane equationViolin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier

14

Creating the database

Violin extraction and 3D registrationSlide15

Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier

15

Result of the tracking

Violin extraction and 3D registrationSlide16

Real time (21 milliseconds)Pose also estimated using markers

Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier

16

Evaluation of the tracking

Violin extraction and 3D registration

Rx(

deg

)

Ry

(

deg

)

Rz

(

deg

)

T(mm)

Minimum error

0.12

0.25

0.20

0.22

Maximum error

13.29

8.27

7.89

32.1

Average error

3.07

2.69

2.78

7.20Slide17

Use the pose estimationLocation manually defined during the capture of the models

Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier

17

Display the frets

Virtual advisingSlide18

Emphasize the string to be pressedDisplay a red dot at the junction of the string and the fret

Define where the finger has to press the string

Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier18

Fingering advices

Virtual advisingSlide19

Suggest the violinist to play a given noteDisplay the fret/string to be pressed

Analyze the sound obtained

If fingering is considered correctAdvice about the position of the bow given the difference of pitch (OK / LOW / HIGH)Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier

19

Other advices

Virtual advisingSlide20

Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier

20

Results

Virtual advisingSlide21

Performed by confirmed playerBowing is correct

Compute the difference of pitch on each fret with the expected one

Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier

21

Evaluation of the results

Fret number

1

2

3

4

5

6

7

8

9

average

Difference of pitch

11.1

14.1

12.0

12.4

13.4

15.8

12.8

13.9

19.2

13.8

Virtual advisingSlide22

Original marker-free method for virtual advising on a violinMethod based on several pre-computed models

Real-time

AccurateDisplay virtual guides on the fingerboardAnalyze the note played for further advicesViolin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier

22

Conclusions

Virtual advisingSlide23

Perform a user based analysisValidate or improve our approach

Use different kind of display

See-through HMDSpatial augmented realityApply to other similar instrumentsJapanese shamisen

Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier

23

Future worksSlide24

Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier

24

Questions

Thank you for your attention