G3: Morphing Example, Blendshapes

1. Get the tools working to turn a 2.5D depth map into a 3D face. Modify the depth map in some way using any tool and produce a new face. The data is stored in a 16 bit png. I know you can load this in MATLAB, and I presume many other image editing programs, to make a simple modification. 

• Turn in a short 5 second screen capture of your rotating your own modified face. Show both the before and after.
• Tool is compiled as Windows .exe. Apologies to those on Linux and Mac. The start of the file name is the date/version. I only tested the older one. I was given the newer one as fixing some bug in the landmark .yml file.
  • 20181008_b3dexport&depthtoobj.zip

2. Do one of the following 2A or 2B

2A: Implement view morphing between two faces captured with the scanner. You will need to morph both the color texture and the depth. I recommend finding a tool that supports view morphing as opposed to writing new code, but if you find a good library that might work.

• Turn in a video showing the texture morphing back and forth. This is all 2D and you should be able to produce the frames and then make a movie out of them.
• Turn in a video showing the 3D face morphing back and forth. Produce all the 3D 'frames'. Render each frame. Then build a movie of the frames.
• (If you cant get morphing working, then at least get a sequence of modified faces which you then play back the frames. So make the head stretch or whatever, produce a sequence of frames, render them, and make a video.)

2B: Explore a tool that supports morphing, or blendshapes, or putting 3D shapes in correspondence. Learn what it is capable of and what sort of input and output it requires. Try it on some data from our scanner

• Turn in: A link to a short presentation of the capabilities, either as a screen capture movie or a online google presentation. Be ready to explain the tool to the class using your presentation and answer questions. Probably 5 minute presentation is about right.
• Examples (but dont feel limited, there are lots of tools):
  • https://canvas.ucsc.edu/courses/16371/pages/links-i-was-clicking-on-about-blendshapes
  • The above page has the links I was clicking on in class
    • Wrap Tool for aligning 3D models like our scans
      • Available in Files tab, there are a set of scanned expressions to try out.
    • Apple Animoji blendshape code examples for iPhone and Unity
    • Downloadable data from Polywink that is supposed to be usable in Unity somehow
• Blendshape data
  • Look in the Files tab. I have added some that AriShapiro generated.

3. Read about blendshapes: 3 papers + something you find

• Tutorial Paper:Practice and Theory of Blendshape Facial Models, Lewis, Anyo, Rhee, Zhang, Pighin, Deng, Eurographics 2014 Links to an external site..
• Classic: A Morphable Model For The Synthesis of 3D Faces, Blanz, Vetter. SIGGRAPH99. Links to an external site.
  • Video [link] Links to an external site.
• Retargetting: Performance Driven Facial Animation using Blendshape Interpolation, Chuang, Bregler, Stanford CS-TR-2002-02, 2002. Links to an external site.
• A paper you find yourself about image morphing

Turn in a reading log with a short paragraph on each paper that explains what the main research contribution of that work is. Also a short proposal of a new research idea that could be a future paper if it worked out. 1 page of text total is sufficient.