# Mon/Wed 4:00pm-5:15pm @ Ford Hall B15

## Description

Multiple cameras are continually capturing our daily events involving social and physical interactions in a form of first person camera (e.g., google glass), cellphone camera, and surveillance camera. Multiview geometry is a core branch in computer vision that studies the 3D spatial relationship between cameras and scenes. This technology is used to localize and plan robots, reconstruct a city, e.g. Rome, from internet photos, and understand human behaviors using body-worn cameras. In this course, we will focus on 1) fundamentals of projective camera geometry; 2) implementation of 3D reconstruction algorithm; and 3) research paper review. The desired outcome of the course is for each student to have his/her own 3D reconstruction algorithm called "structure from motion". This will cover core mathematics of camera multiview geometry including perspective projection, epipolar geometry, point triangulation, camera resectioning, and bundle adjustment. This geometric concept will be then, in parallel, implemented to directly apply to domain specific research such as robot localization. This course includes a final term project that uses the multicamera system to build a creative system such as Robotics/AR/VR/Vision/Graphics.

## Information

Instructor: Hyun Soo Park (hspark at umn.edu)
Syllabus: pdf
Moodle: https://ay17.moodle.umn.edu/course/view.php?id=11243
Office hour: Tue/Thr 5:00pm-6:00pm (Keller Hall 5-225E)
Prerequisite: Linear Algebra; MATLAB
Textbook: Not required but the following book will be frequently referred:
"Multiple View Geometry in Computer Vision", R. Hartley and A. Zisserman

I am working on writing a book based on this lecture:
Chapter 1 (Camera model)
Chapter 2 (Projective line)
Chapter 3 (Image transformation)

## Slide

 Lecture Supplementary Camera model Dolly zoom Camera projection matrix Affine camera model Lens distortion Projective line Point and line at infinity Localization using vanishing points Celestial navigation Single view metrology Image transform Image warping code Cylindrical panorama Linear estimation Least squares via vector derivative SVD on Mondrian Painting Camera calibration MATLAB calibration toolbox Physical focal point Where am I via homography? 3D Spatial Rotation Epipolar geometry (fundamental matrix) Relative pose estimation Local visual feature matching Robust estimation (RANSAC) 3D point triangulation Paper writing I (Introduction) Where am I? (Perspective-n-Point algorithm) Nonlinear estimation Point and camera refinement Sparse bundle adjustment Tomasi-Kanade factorization Final review Bundle adjustment homework review

## Topic

 Single view Camera model Camera projection matrix Projective line Single view metrology Image transformation Estimation I (Linear algebra) Camera calibration Rotation representation Where am I? (vanishing lines) Where am I? (pnp) Multiview Epipolar geometry Fundamental matrix Triangulation and stereo Feature and matching Estimation II (robust modeling) Estimation III (nonlinear optimization) Bundle adjustment I (geometric error) Bundle adjustment II (spare structure and analytic jacobian) Bundle adjustment III (implementation)

## MATLAB code

Example codes can be found here.

50%: 5 programming assignments (10% each)
20%: exam
5%: project proposal
10%: project related work
15%: project final
Late submission: 20% off from each extra late day

## Important dates

Guest lecture (Apr 18): Jaesik Park (Intel Labs)

## Project

A team of students will write/present a computer vision conference paper throughout this lecture. Each team can include up to two students, and the subject is open to diverse applications, e.g., vision, graphics, robotics, AR/VR, and deep learning, but must be related to 3D geometry. The multi-camera system at Shepherd Laboratory can be used for your data capture.

Most technical papers are composed of following sections: introduction, related work, method, data, results, and discussion (check out state-of-the-art vision papers).
+ The first milestone is teaming and potential idea discussion with the instructor by Feb 16.
+ Once approved, the project can be proposed in Mar 7 (Week 8). The project proposal includes the introduction write-up (1-1.5 pages) and data capture.
+ In the Week 12, the related work will be presented and submitted (1-1.5 pages)
+ In the Week 15, the final work will be presented and a full paper will be submitted (6-8 pages)
+ Project submission format (CVPR) can be found here

## Homework

 Building a camera obscura and dolly zoom Creating a 360 panoramic image Virtual tour into your photos Structure from motion I (fundamental matrix / RANSAC / camera pose / triangulation) Structure from motion II (bundle adjustment)

## Scholastic misconduct

Scholastic misconduct is broadly defined as "any act that violates the right of another student in academic work or that involves misrepresentation of your own work. Scholastic dishonesty includes, (but is not necessarily limited to): cheating on assignments or examinations; plagiarizing, which means misrepresenting as your own work any part of work done by another; submitting the same paper, or substantially similar papers, to meet the requirements of more than one course without the approval and consent of all instructors concerned; depriving another student of necessary course materials; or interfering with another student's work."