Computer Graphics

3D and Computational Photography

Prof. James Davis

Tue/Thur 11:40am-1:15pm

CMPS260 Computer Graphics is going to focus on 3D and Computational Photography in the Winter 2021 session. 3D cameras are integral to some of the hottest tech trends, for example AR/VR systems and self-driving cars. Your mobile phone takes pictures rivaling an expensive pro camera in large part because of the computation that happens whenever you take a picture.

In addition to lectures you will read research papers in this area, some classic, some recent.

Piazza Link: https://piazza.com/class/kjc5ekwna0w3z6

TA: Fahim Hasan Khan (fkhan4@ucsc.edu)

TA Office hour: Mon 3:30 - 4:30 pm (Zoom link)

Learning Objectives:

• Understanding of a sampling of classic 3D acquisition methods .e.g. Time of Flight, Stereo, Structured Light, Photogrammetry / Structure from Motion
• Understanding of a sampling of classic Computational Photography methods .e.g. Flash no flash, Panoramas, Lightfields, RTI
• A broad but shallow survey of what is hot currently (papers on these topics at SIGGRAPH in the last few years)

Homework/grading:

• Read (up to) 4 papers a week (for exposure to research ideas) (20%) (about 2-5hrs/week)
  • Write short responses to each paper 
• Participation (20%) (about 3hrs/week)
  • You have to come to class
  • You have to participate in paper discussions 
• Assignments (60%) (about 0-10hrs/week, highly variable depending on assignments)
  • Some assignments build on others, some are completely different
  • Some assignments require coding, some do not

Topics: (not complete and in sorted order, just to provide examples)

• Camera: sensors, bayer mosaic, assorted pixels, how cameras work, flash no-flash
• Image: RSME vs perceptual error metrics; color spaces YUV vs RGB, JPG vs RAW
• Light ray: Lightfields, Coded aperture, Ptyography
• Stereo3D: Stereo, Structured Light, ToF
• ML: PCA, dim reduction, tex. synth, vector quant, CGAN, Image Analogies
• RTI: PTM, Photometric Stereo
• Classic Application: Panoramas (10 years ago)
• Classic Application: Deblurring/denoising: Photo stacking vs Deconvolution (now)
• Classic Application: Background blur, refocusing (still works poorly)
• The above are classic topics, we will also have lots of recent papers

Class time schedule 

• Jan 5 - Intro lecture and course description
• Jan 7 - How a camera works high level
• Jan 12 - Look at results from A1. Introduce A2.
• Jan 14 - Camera sensors
• Jan 19 - Sensor Demosaicing & Noise.
• Jan 21 -  Range Scanning Pipeline
• Jan 26 - Panoramic Mosaics, (supplement: how to ICP)
• Jan 28 - Meshes, About Assignment 2
• Feb 2 - Light Fields
• Feb 4 - Time of Flight
• Feb 9 - Results of A2, Introduce A3
• Feb 11 - Passive stereo, triangulation
• Feb 16 - Triangulation with active lighting
• Feb 18 - Photometric stereo
• Feb 23 - Relighting
• Feb 25 - Discuss progress on A3

Directions for reading responses and how class discussion will work:

• Reading Response and Discussion

Readings:
(Note response due the day before discussion)

• Discuss Jan 12
  • Deconvolution  -  Deblurring  (has  video) (hascode)
    • Rob Fergus, Barun Singh, Aaron Hertzmann, Sam T. Roweis, and William T. Freeman. 2006. Removing camera shake from a single photograph. In ACM SIGGRAPH 2006 Papers (SIGGRAPH '06).
  • Burst Imaging for Low Light - alternate to Deconvolution 
    • Jacob Telleen, Anne Sullivan, Jerry Yee, Prabath Gunawardane, Oliver Wang, Ian Collins, James Davis, Synthetic Shutter Speed Imaging, Computer Graphics Forum 26(3), Eurographics 2007
• Discuss Jan 14
  • None (we ran out of time, the prior papers discussed on Jan 14)
• Discuss Jan 19
  • Coded aperture (builds on deconvolution approach) (introduces coded aperture) (has code)
    • Anat Levin, Rob Fergus, Frédo Durand, and William T. Freeman. 2007. Image and depth from a conventional camera with a coded aperture. In ACM SIGGRAPH 2007
  • Night-sight (builds on burst imaging approach) (example modern feature)
    • Orly Liba, Kiran Murthy, Yun-Ta Tsai, Tim Brooks, Tianfan Xue, Nikhil Karnad, Qiurui He, Jonathan T. Barron, Dillon Sharlet, Ryan Geiss, Samuel W. Hasinoff, Yael Pritch, Marc Levoy, “Handheld mobile photography in very low light.” ACM Transactions on Graphics (TOG) 38, no. 6 (2019): 1-16.
    • https://ai.googleblog.com/2018/11/night-sight-seeing-in-dark-on-pixel.html
• Discuss Jan 21
  • Quanta photography (SIGGRAPH 2020 paper so new I haven't read it either) - Relates to burst imaging - I think  introduces  single  photon  imaging
    • http://wisionlab.cs.wisc.edu/project/quanta-burst-photography/
• Discuss Jan 26
  • Panorama generation (Panorama is one of the first widely available computational photography successes)
    • Recognizing Panoramas, Brown and Lowe, ICCV 2003.
• Discuss Jan 28
  • Synthetic depth of field / Portrait Mode
    • Here is a paper from google (Skim for figures and any ideas not in the blog post below, try to get the main ideas from minimal in depth reading)
      • http://graphics.stanford.edu/papers/portrait/wadhwa-portrait-sig18.pdf
    • Google  blog  posts  on  this  topic  (shorter  read  these  )
      • https://ai.googleblog.com/2014/04/lens-blur-in-new-google-camera-app.html
        
      • https://ai.googleblog.com/2017/10/portrait-mode-on-pixel-2-and-pixel-2-xl.html
• Discuss Feb 2
  • None - Oops, didnt post in time
• Discuss Feb 4
  • Classic on image translation (read and write discussion)
    • A. Hertzmann, C. Jacobs, N. Oliver, B. Curless, D. Salesin., Image Analogies, SIGGRAPH 2001 Conference Proceedings
    • https://mrl.cs.nyu.edu/projects/image-analogies/
  • Modern ML based image translation with GAN (do not write a discussion report)

• • • Only need to read intro, figures, conclusion, ok to skip math and detailed implementation
    • https://phillipi.github.io/pix2pix/
    • Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A. Efros, Image-to-Image Translation with Conditional Adversarial Networks, CVPR, 2017
• Discuss Feb 9
  • Skip reading because we are going to talk about what we found with Assignment 2, and if possible introduce A3, so I am not certain there is time for discussion.
• Discuss Feb 11
  • A high quality face scanning system using passive stereo 
    • Thabo Beeler, Bernd Bickel, Paul Beardsley, Bob Sumner, and Markus Gross. 2010. High-quality single-shot capture of facial geometry. ACM Trans. Graph. 29, 4, Article 40 (July 2010), 9 pages.
  • An early paper that proposes using a prior model for 3D faces
    • A Morphable Model for the Synthesis of 3D Faces, Volker Blanz and Thomas Vetter, SIGGRAPH'99 Conference Proceedings
• Discuss Feb 16
  • (The paper you are trying to get code to run. Different for everyone. See Piazza post)
• Discuss Feb 18
  • Paper (from UCSC students) under review analyzing how good data needs to be to get a 3D Face
  • This is a short paper. I think its directly related to our HW assignment, so the discussion will be useful. 
  • Use your ucsc.edu login to get access in this google drive link
  • https://drive.google.com/file/d/1w41kabYAc2HAfK37bMdxsAqRAb_KdMZk/view?usp=sharing
• Discuss Feb 23
  • Lightstage 1
  • Acquiring the reflectance field of a human face (paper and video)
    • http://www.pauldebevec.com/Research/LS/
  • Lightstages 2,3,4,5,6 (just watch videos for context of follow on research)
  • https://vgl.ict.usc.edu/Data/LightStage/
• Discuss Feb 25
  • Skip reading because we are going to discuss A3.
• Discuss Mar 2
  • Oops.. too much thinking about HW3, forgot to post.
• Discuss Mar 4
  • Skip reading since we are discussing HW3 in class.
• Discuss Mar 9
  • Transient imaging
  • http://giga.cps.unizar.es/~diegog/ficheros/pdf_papers/femto.pdf
  • Ramesh trillion fps TED talk
• Discuss Mar 11 
  • Class over - no paper discussion

• Classic paper that introduces spatial multiplexing
  • Shree K. Nayar and Srinivasa G. Narasimhan. 2002. Assorted Pixels: Multi-sampled Imaging with Structural Models. In Proceedings of the 7th European Conference on Computer Vision-(ECCV '02)
• Burst photography for super resolution/lowlight
  • http://www.hdrplusdata.org/
  • Supplemental: https://google.github.io/night-sight/
  • https://ai.googleblog.com/2018/11/night-sight-seeing-in-dark-on-pixel.html
  • https://sites.google.com/view/handheld-super-res/
• Needed for HW/Relighting
  • Tom Malzbender, Dan Gelb, and Hans Wolters. 2001. Polynomial texture maps. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques (SIGGRAPH '01).
  • Pick a paper from Debevec on lightstage, or maybe some videos, to give background on the area
  • Deep Reflectance Fields - SIGGGRAPH19
    • https://gvv.mpi-inf.mpg.de/projects/DeepReflectanceFields/DRF.pdf
• Relationship between stereo and structured light, but need those first
  • James Davis, Ravi Ramamoothi, Szymon Rusinkiewicz. Spacetime Stereo : A Unifying Framework for Depth from Triangulation, IEEE Comp. Soc. Conf. on Computer Vision and Pattern Recognition (CVPR), 2003.
• Classic  on  representing  data  as  vector  and  using  PCA
  • Matthew Turk, Alex Pentland. Eigenfaces for Recognition. Journal of Cognitive Neuroscience. v3.n1, 1991.
• Easy to understand classic on image translation (precursor to GAN)
  • A. Hertzmann, C. Jacobs, N. Oliver, B. Curless, D. Salesin., Image Analogies, SIGGRAPH 2001 Conference Proceedings
• Also related to making up images  (texture  synthesis)
  • Alexei A. Efros and William T. Freeman, Image Quilting for Texture Synthesis and Transfer'', Proceedings of SIGGRAPH '01, Los Angeles, California, August, 2001.
• Original paper on lightfields - fundamental data type
  • Marc Levoy, Pat Hanrahan, Light Field Rendering, SIGGRAPH '96.
• Panoramas and image stitching (mine, is there a better one?)
  • Davis, J., "Mosaics of Scenes with Moving Objects", IEEE Comp. Soc. Conf. on Computer Vision and Pattern Recognition (CVPR98), 1998.
• Classic paper that introduces spatial multiplexing
  • Shree K. Nayar and Srinivasa G. Narasimhan. 2002. Assorted Pixels: Multi-sampled Imaging with Structural Models. In Proceedings of the 7th European Conference on Computer Vision-(ECCV '02)
• Transient imaging  (I wish there was a good intro paper. The first Seeing-Around-Corner paper is not a good intro. The first below ties to ToF, the second may be a better intro to the idea. Neither has a good video, so maybe the TED talk as intro?)
  • http://www.cs.ubc.ca/labs/imager/tr/2013/TransientPMD/
  • http://giga.cps.unizar.es/~diegog/ficheros/pdf_papers/femto.pdf
  • Ramesh trillion fps TED talk
• SIGGRAPH 2020 paper
  • https://studios.disneyresearch.com/2020/08/14/single-shot-high-quality-facial-geometry-and-skin-appearance-capture/
• SIGGRAPH 2020 paper
  • https://facebookresearch.github.io/one_shot_3d_photography/
• SIGGRAPH 2020 paper
  • https://roxanneluo.github.io/Consistent-Video-Depth-Estimation/
• Synthetic depth of field/Background blur
  • From google, but is there a better one or video? 
  • http://graphics.stanford.edu/papers/portrait/wadhwa-portrait-sig18.pdf
  • Background blur / Portrait mode
    • https://ai.googleblog.com/2014/04/lens-blur-in-new-google-camera-app.html
      
    • https://ai.googleblog.com/2017/10/portrait-mode-on-pixel-2-and-pixel-2-xl.html