First cycle
degree courses
Second cycle
degree courses
Single cycle
degree courses
School of Engineering
ICT FOR INTERNET AND MULTIMEDIA
Course unit
3D AUGMENTED REALITY
INP6075460, A.A. 2017/18

Information concerning the students who enrolled in A.Y. 2017/18

Information on the course unit
Degree course Second cycle degree in
ICT FOR INTERNET AND MULTIMEDIA
IN2371, Degree course structure A.Y. 2017/18, A.Y. 2017/18
N0
bring this page
with you
Degree course track CYBERSYSTEMS [002PD]
Number of ECTS credits allocated 6.0
Type of assessment Mark
Course unit English denomination 3D AUGMENTED REALITY
Department of reference Department of Information Engineering
Mandatory attendance No
Language of instruction English
Branch PADOVA
Single Course unit The Course unit can be attended under the option Single Course unit attendance
Optional Course unit The Course unit can be chosen as Optional Course unit

Lecturers
Teacher in charge SIMONE MILANI ING-INF/03

Mutuating
Course unit code Course unit name Teacher in charge Degree course code
INP6075460 3D AUGMENTED REALITY SIMONE MILANI IN2371

ECTS: details
Type Scientific-Disciplinary Sector Credits allocated
Core courses ING-INF/03 Telecommunications 6.0

Mode of delivery (when and how)
Period First semester
Year 1st Year
Teaching method frontal

Organisation of didactics
Type of hours Credits Hours of
teaching
Hours of
Individual study
Shifts
Lecture 6.0 48 102.0 No turn

Calendar
Start of activities 25/09/2017
End of activities 19/01/2018

Syllabus
Prerequisites: Computer Vision class is recommended although not strictly necessary
Target skills and knowledge: The course offers a guided tour of the computer vision and computer graphics topics needed for current virtual and augmented reality applications.
The course rationale can be divided into three main parts:
a) description and modelling of imaging systems;
b) building a 3D model of reality starting from standard 2D images and/or depth sensors;
c) rendering real or virtual 3D models to standard images and 3D/AR devices.
Part a) has the objective of explaining the operation and the mathematical models of current imaging systems (e.g., video-cameras, Time of Flight systems like MS Kinect, and many more) in the language of computational photography. Part b) is focused on the reconstruction of a 3D model of static and dynamic scenes using standard images (with special focus on stereo and active stereo systems) and/or depth cameras. This part will also deal with the problem of image classification and scene understanding by means of machine learning algorithms. The objective of Part (c) is to introduce the rendering methods and their adaptation to the specific viewing devices. In this final part, the course will also introduce the problem of the interaction between real and virtual world (mixed reality, human-computer interfaces).

The course is structured in order to provide students with both theoretical and practical knowledge on the topics of virtual, augmented and mixed reality; moreover, it will give a clear sense for the deep interconnections between computer vision and computer graphics.

Students will also have the opportunity to develop and use some computer vision, graphics, and augmented reality algorithms by means of lab sessions.

Within its time limits, the course also aims to introduce the students to current computer vision and computer graphics tools such as OpenCV, Point CloudLibrary and OpenGL.
Examination methods: Written exam + report
Assessment criteria: The "3D augmented reality" class covers a wide range of topics mainly across Computer Vision and Computer Graphics since a wide panorama is a good asset to face the fast evolution of these fields.
Nevertheless the student evaluation will be focused on the concepts necessary for building and visualizing 3d models within typical augmented and virtual reality contexts.
Such topics will be clearly indicated during the course and in the course material.
Every efforts on the student part revealing personal involvement and special care will be recognized in terms of scores.
Course unit contents: a) From 3D scene to images via real imaging systems

1) Image formation and camera model

Perspective projection
Pin-hole camera
Thin lenses
Fish-eye lenses
Simplified and general camera model
Digital images

2) Computation of salient points and features

Harris e Stephens method
Scale Invariant Feature Transform (SIFT)
Salient points correspondences

3) Camera calibration

Basic notions of camera calibration

4) Homographies

Computation of the homography (DLT)
Homographics and object recognition
Applications of 2D augmented reality


b) From images to 3D scene model (or 3D reconstruction in Computer Vision)

1) Stereopsys: geometry

3D Triangulation
Epipolar geometry
Epipolar Rectification
Essential matrix and factorizzation
Motion and structure from calibrated homography


2) Stereopsys: Corrispondence

Local correspondence methods
Window correspondence methods
Accuracy-reliability trade-off
Occlusions
Other local methods
Global correspondence methods


3) 3D reconstruction from other sensors

IR Structured-light depth sensors (MS Kinect v.1)
Time-of-Flight depth sensors (MS Kinect v.2)
Active stereo sensors
Laser scanners

4) Non-calibrated reconstruction

Fundamental matrix and its computation
Projective reconstruction from 2 and N views
Method of Mendonca e Cipolla
Tomasi-Kanade factorization
Incremental reconstruction
Structure-from-Motion
Bundle adjustment

5) Optical flow

Motion field: computation of motion and structure
Optical flow: Lucas-Kanade method

6) Orientation methods

Quaternions
Orientation 2D-2D:
Orientatio 3D-3D: DLT and ICP methods
Orientation 3D-2D
Surface integration
Mesh semplification

7) Object detection and scene understanding

Image and 3D models features
Image/object classification strategies
Machine learning algorithms for object classification
Support Vector Machine for image/object classification
Deep Neural Networks for image/object classification
Human-computer interfaces (HCI)

c) From 3D models to images and beyond

1) 3D displays, VR visors, and augmented reality devices

2) Rendering

Rendering: projective geometry and convention
Ray tracing and ray casting
The radiance (or rendering) equation and its solution

3) Illumination

The radiance solution by local methods: Phong and Cook-Torrance models
Light types
The radiance solution by global methods
Ray tracing: Whitted method
Radiosity: radiosity equation in continuous and discrete form
Shading

4) Rasterization

The OpenGL pipeline
Planned learning activities and teaching methods: The course offers a guided tour of the computer vision and computer graphics concepts needed for current virtual and augmented reality applications. The main topics are:

• Image formation: mathematical models of cameras and Time of Flight systems.
* Depth sensors (Time-of-Flight and struyctured light cameras) and 3D acquisition devices
• Computational stereopsis: 3D scene structure derived from 2 or more images obtained from calibrated cameras
• Structure from motion: 3D scene structure derived from 1 or more calibrated moving cameras
• Un-calibrated 3D reconstruction: 3D scene structure derived from un-calibrated cameras.
• 3D registration: pairwise and global registration (or SLAM) of images and depth-maps into a point cloud
• 3D data integration and geometrical simplification: integration of overlapping point clouds into tessellated surfaces and their simplification
*Scene classification and object detection
*3D visualization displays and augmented reality devices
• Rendering methods: ray casting, ray tracing, radiosity and rasterization

The topics are treated by means of frontal lectures with computational examples based on MATLAL, Open CV and Open GL.
The appraisal is stimulated by lab sessions and a final project confronting the student with practical situations due to the concepts seen in class.
Additional notes about suggested reading: The course purposely hybridates two disciplines, computer vision and computer topics, in order to focus on 3D model construction and augmented-reality applications.

The study material is given by the class-notes made available before every class meeting.
The notes distill and condense various research papers and content coming from several textbooks, among which:

A. Fusiello, "Visione Computazionale", F. Angeli, Milano, 2013

Klette, Reinhard, Concise Computer Vision. Springer London: --, 2014.

Kenichi Kanatani, Yasuyuki Sugaya, Yasushi Kanazawa, Guide to 3D Vision Computation. --: Springer International Publishing, 2016.

C. M. Bishop, "Pattern recognition and machine learning", Springer, New York, 2006
Textbooks (and optional supplementary readings)
  • Fusiello, Andrea, Visione computazionaletecniche di ricostruzione tridimensionale. Milano: Angeli, 2013. Cerca nel catalogo
  • Klette, Reinhard, Concise Computer Vision. Springer London: --, 2014. Cerca nel catalogo
  • Bishop, Christopher M., Pattern recognition and machine learning. New York: Springer, --. Cerca nel catalogo
  • Forsyth, David; Ponce, Jean, Computer VisionA Modern ApproachDavid A. Forsyth, Jean Ponce. Boston: ©Pearson, 2012. Cerca nel catalogo
  • Szeliski, Richard, Computer visionalgorithms and applicationsRichard Szeliski. New York: Springer, 2011. Cerca nel catalogo
  • Kenichi Kanatani, Yasuyuki Sugaya, Yasushi Kanazawa, Guide to 3D Vision Computation. --: Springer International Publishing, 2016. Cerca nel catalogo