Spring 2017 CSCI 5980/8980
Multiview 3D Geometry in Computer Vision

Tue/Thr 4:00pm-5:15pm @ Rapson Hall 43


Multiple cameras are continually capturing our daily events involving social and physical interactions in a form of first person camera (e.g., google glass), cellphone camera, and surveillance camera. Multiview geometry is a core branch in computer vision that studies the 3D spatial relationship between cameras and scenes. This technology is used to localize and plan robots, reconstruct a city, e.g. Rome, from internet photos, and understand human behaviors using body-worn cameras. In this course, we will focus on 1) fundamentals of projective camera geometry; 2) implementation of 3D reconstruction algorithm; and 3) research paper review. The desired outcome of the course is for each student to have his/her own 3D reconstruction algorithm called "structure from motion''. This will cover core mathematics of camera multiview geometry including perspective projection, epipolar geometry, point triangulation, camera resectioning, and bundle adjustment. This geometric concept will be then, in parallel, implemented to directly apply to domain specific research such as robot localization.


Instructor: Hyun Soo Park (hspark at umn dot edu)
Syllabus: pdf
Moodle: https://ay16.moodle.umn.edu/course/view.php?id=13616
Office hour: Tue/Thr 2:00pm-4:00pm (Keller Hall 5-225E)
Prerequisite: Linear Algebra
Textbook: Not required but the following book will be frequently referred:
                   "Multiple View Geometry in Computer Vision", R. Hartley and A. Zisserman


Single view Camera model
Camera projection matrix
Projective line
Single view metrology
Image transformation
Estimation I (Linear algebra)
Camera calibration
Rotation representation
Where am I? (vanishing lines)
Where am I? (pnp)
Multiview Epipolar geometry
Fundamental matrix
Triangulation and stereo
Feature and matching
Estimation II (robust modeling)
Estimation III (nonlinear optimization)
Bundle adjustment I (geometric error)
Bundle adjustment II (spare structure and analytic jacobian)
Bundle adjustment III (implementation)


The course will not directly follow the textbook but students may want to read the related book chapter for deeper understanding.

c: Paper committee

Date Topic Slide Book ch. Homework Paper pres.
Jan 17 Computer Vision Introduction pdf
Jan 18 Camera Model pdf Ch. 6
Jan 24 Camera Projection Matrix pdf Ch. 6, 7 HW #1 out
Jan 26 Projective Line I pdf Ch. 2
Jan 31 Projective Line II pdf Ch. 2
Feb 2 Where am I? (Vanishing Point) / Single View Metrology pdfCh. 8 HW #1 due
Feb 7 Image Transformation pdfCh. 2 HW #2 out
Feb 9 Linear Least Squares / Homography Computation pdf Ch. 2, A4, A5
Feb 14 Fun with Homography / Camera Calibration pdfCh. A4, A5, 4
Feb 16 Camera Calibration (checkerboard pattern)
MATLAB calibration toolbox
pdf HW #2 due
Feb 21 Where am I? (Homography) / Tour into Photo pdf HW #3 out
Feb 23 Rotation Representation pdf
Feb 28 Rotation Representation / Epipolar Geometry pdf Paper selection due
Mar 2 Rotation Representation / Epipolar Geometry pdf
Mar 7 Epipolar Geometry pdf
Mar 9 Fundamental Matrix Computation / Where am I? (Relative Pose) pdf HW #3 due
Mar 14 & 16 Spring Break
Mar 21 No class HW #4 out
Mar 23 Where am I? (Relative Pose) / RANSAC pdf
Mar 28 RANSAC pdf
Mar 30 Triangulation and Stereo pdf
Apr 4 Stereo and PnP pdf HW #4 due and HW #5 outTong Ke pdf
c: Cheng Peng
Apr 6 Recitation (Shan Su) pdf
Apr 11 PnP and Nonlinear Estimation pdf Cheng Peng pdf
c: Jiawei Mo
Apr 13 Nonlinear Estimation pdf HW #5 dueJingbin Yang pdf
c: Sanfer D'souza
Apr 18 Jacobian pdf HW #6 out, DataTien Do pdf
c: Aarti Sundararjan
Apr 20 Jacobian and Bundle Adjustment pdf Anushree Jagrawal pdf
c: Tong Ke
Apr 25 Bundle Adjustment pdf Jiawei Mo pdf
c: Tien Do
Apr 27 HW #6 review pdf Aarti Sundararjan pdf
c: Jingbin Yang
May 2 Tomasi-Kanade Factorization pdf Sanfer D'souza pdf
c: Anushree Jagrawal
May 4 HW #6 review and Future Computer Vision pdf HW #6 due


Example codes can be found here.

Paper reading

Format: 20 min presentation and 15+ min Q&A.
Presenter: defending the paper.
Committee: attacking the paper.

Recommended paper
Tomasi and Kanade, Shape and Motion from Image Streams under Orthography: a Factoriaztion Method, IJCV, 1992
Reid and Zisserman, Goal-directed Video Metrology, ECCV, 1996
Zhang, A Flexible New Technique for Camera Calibration, PAMI, 2000
Nister, An Efficient Solution to the Five-Point Relative Pose Problem, PAMI, 2004
Criminisi, Reid, and Zisserman, Single View Metrology, IJCV, 2000
Xiao and Furukawa, Reconstructing the World’s Museum, IJCV, 2014
Izadi et al., KinectFusion: Real-time 3D Reconstruction and Interaction Using a Moving Depth Camera, UIST, 2011


Due date Topic Desciption
Feb 2 Building a camera obscura and Dolly Zoom
Feb 16 Creating a 360 panoramic image
Mar 2 Virtual tour into your photos
Apr 4 Structure from Motion I (Fundamental matrix / RANSAC / Camera Pose)
Apr 18 Structure from Motion II (Triangulation / PnP)
May 4 Structure from Motion III (Bundle adjustment)

Grading policy

90%: 6 programming assignments (15% each)
10%: paper presentation
Late submission: 20% off from each extra late day
For 8980: scalable SfM implementation running +30 images

Scholastic Misconduct

Scholastic misconduct is broadly defined as "any act that violates the right of another student in academic work or that involves misrepresentation of your own work. Scholastic dishonesty includes, (but is not necessarily limited to): cheating on assignments or examinations; plagiarizing, which means misrepresenting as your own work any part of work done by another; submitting the same paper, or substantially similar papers, to meet the requirements of more than one course without the approval and consent of all instructors concerned; depriving another student of necessary course materials; or interfering with another student's work."