This article's factual accuracy may be compromised due to out-of-date information. Please help update this article to reflect recent events or newly available information. (October 2019)

3D reconstruction from multiple images is the creation of three-dimensional models from a set of images. It is the reverse process of obtaining 2D images from 3D scenes.

The essence of an image is a projection from a 3D scene onto a 2D plane, during which process the depth is lost. The 3D point corresponding to a specific image point is constrained to be on the line of sight. From a single image, it is impossible to determine which point on this line corresponds to the image point. If two images are available, then the position of a 3D point can be found as the intersection of the two projection rays. This process is referred to as triangulation. The key for this process is the relations between multiple views which convey the information that corresponding sets of points must contain some structure and that this structure is related to the poses and the calibration of the camera.

In recent decades, there is an important demand for 3D content for computer graphics, virtual reality and communication, triggering a change in emphasis for the requirements. Many existing systems for constructing 3D models are built around specialized hardware (e.g. stereo rigs) resulting in a high cost, which cannot satisfy the requirement of its new applications. This gap stimulates the use of digital imaging facilities (like a camera). An early method was proposed by Tomasi and Kanade.^[2] They used an affine factorization approach to extract 3D from images sequences. However, the assumption of orthographic projection is a significant limitation of this system.

Processing

The task of converting multiple 2D images into 3D model consists of a series of processing steps:

Camera calibration consists of intrinsic and extrinsic parameters, without which at some level no arrangement of algorithms can work. The dotted line between Calibration and Depth determination represents that the camera calibration is usually required for determining depth.

Depth determination serves as the most challenging part in the whole process, as it calculates the 3D component missing from any given image – depth. The correspondence problem, finding matches between two images so the position of the matched elements can then be triangulated in 3D space is the key issue here.

Once you have the multiple depth maps you have to combine them to create a final mesh by calculating depth and projecting out of the camera – registration. Camera calibration will be used to identify where the many meshes created by depth maps can be combined to develop a larger one, providing more than one view for observation.

By the stage of Material Application you have a complete 3D mesh, which may be the final goal, but usually you will want to apply the color from the original photographs to the mesh. This can range from projecting the images onto the mesh randomly, through approaches of combining the textures for super resolution and finally to segmenting the mesh by material, such as specular and diffuse properties.

Mathematical description of reconstruction

Given a group of 3D points viewed by N cameras with matrices ${\displaystyle \{P^{i}\}_{i=1\ldots N}}$, define ${\displaystyle m_{j}^{i}\simeq P^{i}w_{j}}$ to be the homogeneous coordinates of the projection of the ${\displaystyle j^{th}}$ point onto the ${\displaystyle i^{th}}$ camera. The reconstruction problem can be changed to: given the group of pixel coordinates ${\displaystyle \{m_{j}^{i}\}}$, find the corresponding set of camera matrices ${\displaystyle \{P^{i}\}}$ and the scene structure ${\displaystyle \{w_{j}\}}$ such that

${\displaystyle m_{j}^{i}\simeq P^{i}w_{j}}$ (1)

Generally, without further restrictions, we will obtain a projective reconstruction.^[4][5] If ${\displaystyle \{P^{i}\}}$ and ${\displaystyle \{w_{j}\}}$ satisfy (1), ${\displaystyle \{P^{i}T\}}$ and ${\displaystyle \{T^{-1}w_{j}\}}$ will satisfy (1) with any 4 × 4 nonsingular matrix T.

A projective reconstruction can be calculated by correspondence of points only without any a priori information.

Auto-calibration

In auto-calibration or self-calibration, camera motion and parameters are recovered first, using rigidity. Then structure can be readily calculated. Two methods implementing this idea are presented as follows:

Kruppa equations

With a minimum of three displacements, we can obtain the internal parameters of the camera using a system of polynomial equations due to Kruppa,^[6] which are derived from a geometric interpretation of the rigidity constraint.^[7][8]

The matrix ${\displaystyle K=AA^{\top }}$ is unknown in the Kruppa equations, named Kruppa coefficients matrix. With K and by the method of Cholesky factorization one can obtain the intrinsic parameters easily:

${\displaystyle K={\begin{bmatrix}k_{1}&k_{2}&k_{3}\\k_{2}&k_{4}&k_{5}\\k_{3}&k_{5}&1\\\end{bmatrix}}}$

Recently Hartley ^[9] proposed a simpler form. Let ${\displaystyle F}$ be written as ${\displaystyle F=DUV^{\top }}$, where

Then the Kruppa equations are rewritten (the derivation can be found in ^[9])

Mendonça and Cipolla

This method is based on the use of rigidity constraint. Design a cost function, which considers the intrinsic parameters as arguments and the fundamental matrices as parameters. ${\displaystyle {F}_{ij}}$ is defined as the fundamental matrix, ${\displaystyle {A}_{i}}$and ${\displaystyle {A}_{j}}$ as intrinsic parameters matrices.

Stratification

Recently, new methods based on the concept of stratification have been proposed.^[10] Starting from a projective structure, which can be calculated from correspondences only, upgrade this projective reconstruction to a Euclidean reconstruction, by making use of all the available constraints. With this idea the problem can be stratified into different sections: according to the amount of constraints available, it can be analyzed at a different level, projective, affine or Euclidean.

The stratification of 3D geometry

Usually, the world is perceived as a 3D Euclidean space. In some cases, it is not possible to use the full Euclidean structure of 3D space. The simplest being projective, then the affine geometry which forms the intermediate layers and finally Euclidean geometry. The concept of stratification is closely related to the series of transformations on geometric entities: in the projective stratum is a series of projective transformations (a homography), in the affine stratum is a series of affine transformations, and in Euclidean stratum is a series of Euclidean transformations.

Suppose that a fixed scene is captured by two or more perspective cameras and the correspondences between visible points in different images are already given. However, in practice, the matching is an essential and extremely challenging issue in computer vision. Here, we suppose that ${\displaystyle n}$ 3D points ${\displaystyle A_{i}}$ are observed by ${\displaystyle m}$ cameras with projection matrices ${\displaystyle P_{j},j=1,\ldots ,m.}$ Neither the positions of point nor the projection of camera are known. Only the projections ${\displaystyle a_{ij}}$ of the ${\displaystyle i^{th}}$ point in the ${\displaystyle j^{th}}$ image are known.

Projective reconstruction

Simple counting indicates we have ${\displaystyle 2nm}$ independent measurements and only ${\displaystyle 11m+3n}$ unknowns, so the problem is supposed to be soluble with enough points and images. The equations in homogeneous coordinates can be represented:

${\displaystyle a_{ij}\sim P_{j}A_{i}\qquad i=1,\ldots n,~~j=1,\ldots m}$ (2)

So we can apply a nonsingular 4 × 4 transformation H to projections $$Zdroj:https://en.wikipedia.org?pojem=3D_reconstruction_from_multiple_images
Text je dostupný za podmienok Creative Commons Attribution/Share-Alike License 3.0 Unported; prípadne za ďalších podmienok. Podrobnejšie informácie nájdete na stránke Podmienky použitia.