Dinghuang Ji Facial Animation And Skinning - PowerPoint Presentation

1. Dinghuang JiFacial Animation And Skinning

2. OutlineDefinition of the ProblemHistory HighlightsInterested TopicsData driven facial animationMesh deformationPhysical based facial animation

3. Definition of the Problem(Computer) facial animation is primarily an area of computer graphics that encapsulates models and techniques for generating and animating images of the human head and face[7]. There are some related topics that I will not cover. 3D Face modelling requires knowledge of anthropometry and assistance of range scanner or structured light [14][15]. Skin and hair rendering could enhance animation effects[16][17]. Performance driven facial animation supplies intuitive interface[18][19].

4. Definition of the ProblemThis is a well defined problem, but it is difficult to solve.We are intimately aware of how human faces should look and sensitive to subtleties in the form and motion.Face expression and head movement work together to convey meaning.Expressions are further affected by the emotions.Facial anatomy is both structurally and physically complex: there are many layers of different kinds of material (skin, fat, muscle, bones).

5. MilestonesIn 1970’s, Parke, F. I. build the first 3D parametric facial model enable generate specific facial animation with few parameters. For example, rotation of the jaw or direction of eye gaze[2].In 1970’s, Paul Ekman et. Al. made a standard to encode movements of individual facial muscles.[4]In the 1980’s, Keith Waters developed a muscle based facial model with physically based skin tissue.[5]

6. Parke’s Facial Model“Tony de Peltrie” (1985) marked the first time computer facial animation played an important role in telling a storyWater’s Facial ModelThe Pixar short “Tin Toy” (1988) was the first computer-animated film to win an Oscar. The childʼs face is animated using a Waters-style model.

7. Facial Action Coding System[4]FACS is a description of the movements of the facial muscles and jaw/tongue derived from an analysis of facial anatomy, which includes forty four basic action units.Combinations of independent action units generate facial expressions

8. Facial Action Coding SystemFor its simplicity, FACS is widely utilized with muscle or simulated muscle based approaches. Limitations:AUs are purely local patterns while actual facial motion is rarely completely localizedFACS offers spatial motion descriptions but not temporal components.

9. Muscles of the Face

10. Uncanny ValleyReal PersonSimilarityFamiliarity

11. Interested TopicsData DrivenLinear Blend shapeNon-Linear Blend shapeTemporal-spatial analysisInterface designMesh DeformationFFD3D MorphingExpression CloningDeformation GradientPhysical modellingSpring-MassVector MuscleVibration modelSkin fabrication

12. What’s blend shape?Blend shape is a set of geometrical models :With vertices correspondenceWith same topology Blend shape InterpolationThe weights wk are manipulated by the animator in the form of sliders (with one slider for each weight) or automatically determined by algorithms. vj is the position of jth vertex and Bkj is the positon of jth vertex for kth blend shapes. NBS is the number of blend shapes.It continues to be used in projects such as the Stuart Little, Star Wars, and Lord of the Rings and was adopted in many commercial animation software packages such as Maya and 3D Studio Max.

13. Sidebar: weightsThe weights need to be large than 0 and less than 1.W = 1W = -1OutlierInlier, 0<wi <1

14. Data representationMocap DataWhere is the 0th frame captured from Vicon, which contains NM vertices. Usually we use the 0th frame as a reference. Each is a 3 dimensional position vector. Blend shape generationCapturing the actor’s performanceCompute the boundary of convex setObtain through sparse sampling on facial model

15. Reconstruction Error MinimizationWe aim to minimize the reconstructed error with blend shape and weights. Because head moves in the capture process, it’s necessary to remove the movement (rotation) and (translation). And operator remove rotation and translation from jth frame [10] .

16. NumericsWe hope to optimize both weights and blend shapes. The optimized blend shapes should not be far away from initial value. is the initial value for ith blend shape , is lagrangian multipliers .

17. Demo

18. Pros and ConsThis method is very easy to implement and can generate visually good facial animation.However, the linear blend model can’t generate very natural animation (following some curve) and for some asymmetric expression, it can’t handle very well.

19. Interested TopicsData DrivenLinear Blend shapeNon-Linear Blend shapeTemporal-spatial analysisInterface designMesh DeformationFFD3D MorphingExpression CloningDeformation GradientPhysical modellingSpring-MassVector MuscleVibration modelSkin fabrication

20. Nonlinear Blend shapeIn order to generate more natural animation, we could use nonlinear blend shapes, experiments show cubic polynomial model would be a good choice[11]. Where is actually linear blend shape . and have equal dimension with

21. NumericsWe need to optimize both nonlinear blend shapes and weights, the initial values of and are 0.

22. Demo

23. Pros and ConsThis method could achieve more natural expression and smaller reconstruction error.But because the optimization is a nonlinear problem, it’s more time consuming.

24. Interested TopicsData DrivenLinear Blend shapeNon-Linear Blend shapeTemporal-spatial analysisInterface designMesh DeformationFFD3D MorphingExpression CloningDeformation GradientPhysical modellingSpring-MassVector MuscleVibration modelSkin fabrication

25. Free Form DeformationFree-Form deformation (FFD) is a technique for manipulating any shape in a free-form manner.To compute deformation with FFD, we need to Generate surrounding grid for objectCompute the homogeneous coordinates using Bernstein polynomial equationAt last, the x,y,z surfaces are mapped to a Bezier surface.

26. Pros and ConsThis method is easy to implement and running very fast.But the deformation control is not intuitive, we can’t obtain much details we expected.

27. Interested TopicsData DrivenLinear Blend shapeNon-Linear Blend shapeTemporal-spatial analysisInterface designMesh DeformationFFD3D MorphingExpression CloningDeformation GradientPhysical modellingSpring-MassVector MuscleVibration modelSkin fabrication

28. Expression CloningTransfer vertex motion vectors from a source face model to a target model [8]1. Determine surface points Correspondence2. Transfer motion vectors

29. Dense surface correspondencesRadial Basis Functions (RBF)Roughly project vertices in the source model onto the target model.

30. Dense surface correspondences

31. Dense surface correspondencesCylindrical Projections

32. Transfer motion vectorsBarycentric coordinates

33. Transfer motion vectorsDirection Adjustment is the rotation matrix between original local space and world space is the rotation matrix between world space and deformed local space

34. Transfer motion vectorsMagnitude Adjustment

35. Expression CloningAssume the motion vector of vertex on source model is , the motion vector of vertex on deformed model is

36. Pros and ConsThis method is intuitive and easy to implement.But the method is not graceful in formulation.

37. Interested TopicsData DrivenLinear Blend shapeNon-Linear Blend shapeTemporal-spatial analysisInterface designMesh DeformationFFD3D MorphingExpression CloningDeformation GradientPhysical modellingSpring-MassVector MuscleVibration modelSkin fabrication

38. Deformation TransferTransfer transformations from source polygons to target polygons [9]

39. Deformation GradientQ + dv4v3v2v1 is vertex of original triangle and is vertex of deformed triangleOriginalDeformed?

40. Deformation TransferAfter build correspondence between original and deformed model, we need to minimize the difference between non-translational components of the source and target deformations. Here are deformation gradients like Q we just mentioned. is a correspondence of face. The constraint is added to keep mesh from broken. Is all neighbor faces of

41. Deformation TransferCorrespondence are several vertex correspondence assigned manually Closest valid points Keep close to IdentitySmoothness

42. LimitationsPerhaps the most conspicuous limitation of our technique is the requirement of gross similarity between the source and target meshes.

43. Interested TopicsData DrivenLinear Blend shapeNon-Linear Blend shapeTemporal-spatial analysisInterface designMesh DeformationFFD3D MorphingExpression CloningDeformation GradientPhysical modellingSpring-MassVector MuscleVibration modelSkin fabrication

44. Physical based Skin modelingBased on FACS system, Platt et. Al propose a spring mesh system to simulate bone, muscle fiber and skin. Forces are applied to elastic meshes through muscle arcs generate various facial expressions. [6]Waters proposed a vector muscle model with delineated deformation field which models the action of muscles upon skin. A muscle definition includes the vector field direction, an origin, and an insertion point . The field extent is defined by cosine functions and fall off factors that produce a cone shape when visualized as a height field. Waters animates human emotions such as anger, fear, surprise, disgust, joy, and happiness using vector based linear muscles utilizing the FACS. [1]

45. Interested TopicsData DrivenLinear Blend shapeNon-Linear Blend shapeTemporal-spatial analysisInterface designMesh DeformationFFD3D MorphingExpression CloningDeformation GradientPhysical modellingSpring-MassVector MuscleVibration modelSkin fabrication

46. Dynamic Response Textures [12]The linear elastodynamic equation for a finite element model, describes the displacements of N nodes within a volume.The displacement field u is expanded in a modal displacement basiswhere denotes the model's modal matrix, whose ith column represents the ith model shape, are the corresponding modal amplitudes. Combine the two formulations we get

47. Dynamic Response TexturesIf we make the common assumption of proportional (Rayleigh) damping, then the system of ODEs are completely decoupled by the modal transformationThe system of decoupled ordinary differential equations may be written aswhere the undamped natural frequency of vibration and and the dimensionless modal damping factor are

48. Pros and ConsThis method can be used to simulate geometrically complex, volumetric, physically-based, dynamic deformation models with negligible main CPU costs by exploiting commodity graphics hardware.It’s a generalized method, for facial animation application we need to carefully adjusting parameters.

49. Interested TopicsData DrivenLinear Blend shapeNon-Linear Blend shapeTemporal-spatial analysisInterface designMesh DeformationFFD3D MorphingExpression CloningDeformation GradientPhysical modellingSpring-MassVector MuscleVibration modelSkin fabrication

50. Physical Face Cloning [13]Let X and x denote smooth functions describing the position of the skin in its undeformed and deformed state, respectively. The deformation of the skin at each point can be characterized by the deformation gradient FWe denote the corresponding energy density function by

51. Physical Face CloningThe strain energy density of the compressible neo-Hookean material is given aswhere and , are shear and bulk moduli, which are related to the familiar Young’s modulus E and Poisson’s ratio as

52. Physical Face CloningWe discretize in space using tetrahedral finite elements. The superscript e is used to identify quantities pertaining to a given tetrahedral element. For example, let and denote position vectors for a given element in its undeformed and deformed state, respectively.Where Na are piece-wise linear basis functions associated with the nodes of the mesh.

53. Physical Face Cloningthe discrete elastic energy of a deformed element can be expressed aswhere is the element’s parameter domain and its undeformed volume.The deformed configuration of the skin is then determined as the minimum of the total energy

54. Pros and ConsThis method combines mesh deformation and physical model and novel optimization scheme that determines the shape of the synthetic skin as well as the actuation parameters that provide the best match to the target expressions.It’s more time consuming.

55. Performance driven facial animationTo supply a simple and intuitive controlling method in making facial animation.Facial feature localizationGaze estimationMotion parameterizationexpressionsHead motionFeature localization

56. Speech driven facial animationVisual speech animation can be regarded as visual motions of the face (especially the mouth part) when humans are speaking.Lip motionViseme or DataexpressionHead motiondrive

57. Resources in Facial animation Software and Device :Facegen, could generate a large variety of model and blend shapesFaceworx, vertices texture map generation and modellingPoser, DIY blend shapes 3ds Max, model editing and animation rendering (vertex plugin)Maya, blend shape pluginFaceRobot, facial rigging and animationVicon IQ, Blade motion capture and editing Vicon , high frame rate optical marker trackerStructured light, high precision model generation3D scanner, high precision model generation

58. References [1] Z. Deng and J. Noh. Computer Facial Animation: A Survey. DATA-DRIVEN 3D FACIAL ANIMATION 2007, 1-28, DOI: 10.1007/978-1-84628-907-1_1[2] F. Parke. Computer generated animation of faces. In Proc. ACM Nat’l Conf., volume 1, pages 451–457, 1972.[3] F.I. Parke and K. Waters. Computer Facial Animation. 1996.[4] P. Ekman and W.V. Friesen. Facial Action Coding System. Consulting Psychologists Press, 1978.[5] K. Waters. A muscle model for animating three-dimensional facial expression.In SIGGRAPH Proceedings, volume 21, pages 17–24, 1987.[6] S. Platt and N. Badler. Animating facial expression. computer graphics. Com-puter Graphics, 15(3):245–252, 1981.[7] http://en.wikipedia.org/wiki/Computer_facial_animation

59. References[8] J. Y. Noh and U. Neumann. Expression cloning. Proc. of ACM SIGGRAPH’01,pages 277–288, 2001.[9] R. W. Sumner and J. Popovi´c. Deformation transfer for triangle meshes. ACMTrans. Graph., 23(3):399–405, 2004.[10] X. Liu T. Mao S. Xia Y. Yong Z.Wang. Facial animation by optimized blendshapes from motion capture data. Computer Animation and Virtual Worlds 2008. Volume 19 Issue 3-4, September 2008 pp 235-245 [11] X. Liu S. Xia Y. Fan Z.Wang. Exploring non-linear relationship of blendshape facial animation. Computer Graphics Forum, doi: 10.1111/j.1467-8659.2011.01852.x, 2011[12] D.L. James, D.K. Pai. DyRT: dynamic response textures for real time deformation simulation with graphics hardware. ACM Transactions on Graphics (TOG) - Proceedings of ACM SIGGRAPH 2002. Volume 21 Issue 3, July 2002 pp 582 - 585 [13] B. Bickel, P. Kaufmann, M. Skouras, B. Thomaszewski, D. Bradley, T. Beeler, P. Jackson, S. Marschner, W. Matusik, M. Gross. Physical Face Cloning. ACM Transactions on Graphics (Proc. SIGGRAPH 2012), vol. 31, no. 3, August 2012. 

60. References[14] T. BEELER, F. HAHN, D. BRADLEY, B. BICKEL, P. BEARDSLEY, C. GOTSMAN, R. W. SUMNER, M. GROSS. High-Quality Passive Facial Performance Capture Using Anchor Frames. ACM Transactions on Graphics (Proceedings of SIGGRAPH). 2011. (Vancouver, Canada).[15] Hao Li, Bart Adams, Leonidas J. Guibas, Mark Pauly . ROBUST SINGLE-VIEW GEOMETRY AND MOTION RECONSTRUCTION. ACM Transactions on Graphics, Proceedings of the 2nd ACM SIGGRAPH Conference and Exhibition in Asia 2009, 12/2009[16] Craig Donner and Henrik Wann Jensen. A Spectral BSSRDF for Shading Human Skin. EGSR 2006[17] Iman Sadeghi, Heather Pritchett, Henrik Wann Jensen, Rasmus Tamstorf. An Artist Friendly Hair Shading System. ACM Transactions on Graphics 29(4), SIGGRAPH 2010.[18] Manfred Lau, Ying-Qing Xu and Harry Shum. Interactive Manipulation of 3D Facial Expressions Using Facial Priors . ACM Transactions on Graphics (Presented at SIGGRAPH 2010), 29(1): Article No. 3.[19] Jing Xiao and Jessica Hodgins. Vision-based Control of 3D Facial Animation. ACM SIGGRAPH/Eurographics Symposium on Computer Animation 2003.

61. Thank youQuestions are appreciated.