Activity 8: 3D surface reconstruction from structured light

Patterns tend to be modified when projected onto an object in such a way that the modification is related to the object surface topology. The modified patterns can then be analyzed to measure a particular topology of the object. This activity involves 3D shape reconstruction using the Structured Light Illumination (SLI), a non-contact and hence non-invasive measurement of the surface topology. SLI is the process of projecting a pattern of pixels often in the form of grids and in the case of this activity, vertical bars onto a scene. The way that these patterns deform when striking surfaces allow optical systems to calculate the depth and surface information of the objects in the scene.

Experiment

We assemble the basic SLI set-up consisting of a projector and a camera. The projector is used to project a light pattern, which can be a single stripe on a target surface. By measuring the distortion between the captured and reflected image, the depth information can be extracted. This technique can be useful for imaging and acquiring three-dimensional information.

We used the gray code representation as our pattern. In the gray code, the numbers are represented as binary patterns and the consecutive numbers differ by only one bit position as shown in Table 1. The gray code structured light patterns we set up in class are shown in Figure 1.

Table 1. Binary code and Gray code representation




Figure 1. Gray code structure light patterns for the different binary patterns projected onto a white background (IPL whiteboard).




Figure 2. Gray code structure light patterns projected on a pyramid

We determine the stripe ordinals by recombining the different stripe codes. The binarized images are multiplied by their corresponding bit weights and are added to form the image and background bit stacks. Note that each vertical line in the bit stack represents a unique number. Using the depth from disparity equation, we were able to determine the surface depth or height of our pyramid.

Figure 3. Reconstruction I. The pyramid object is reconstructed using the SLI technique.



Figure 4. Reconstruction II. The Matlab command, medfilt2 was used to lessen the other stray peaks present in the reconstructed figure.


Tips:

1. Make sure that the background and object binarized images are multiplied by their corresponding bit weights in the gray code. I did not obtain good results during the early parts of the activity because I inadvertently switched the two background images and their gray codes.


We were able to successfully reconstruct the surface of our pyramid using the structured light illumination technique. I give myself a grade 0f 10 for this activity.




References:

[1]. M. Soriano, " Activity 8: 3D reconstruction from structured light" Physics 305 Lectures, 2009.

For teaching concerns please visit: https://sites.google.com/site/alongjas/

Activity 7: Stereometry

Try this!

1. Hold your left thumb near you face, say around two feet, and look at a distant background (any object like a tree or building).

2. Close your left eye only and focus on a distant backdrop with your thumb still placed in front of you.

3. Open your left eye and then close your right eye. Look again at the distant backdrop with thumb still placed in front.

You will notice the difference in the position of your thumb relative to the distant backdrop.

Figure 1. Parallax effect*

This observation is what we call parallax effect. There is an apparent shift in the angle that occurs when a nearby object is seen against a distant backdrop from two different perspectives. The images that we see are essentially 3D objects projected onto a 2D frame. However, multiple 2D views allow the computation of depth information that serves to reconstruct a 3D object. In a way, each of our eyes acts as a single camera and the combined views of the left eye and the right eye can reconstruct 3D objects.

In this activity, we will reconstruct a 3D object captured using a digital camera by employing a technique called stereo imaging inspired by how our eyes perceive depth.

I. Theory

Figure 1 considers two identical cameras positioned such that the centers of the lenses are placed a traverse distance b apart. Let the object point P lie at the axial distance z. It is this z that we wish to recover. The image planes of each camera are at a distance f from the camera lens. In the image plane, P appears at a traverse distance x1 and x2 from the centers of the left and right cameras respectively.

Figure 2. Geometry for stereometry using two views


From similar triangles, we see that



Solving for z we find that




If done for several points on the object, we can reconstruct its 3D shape.


II. Experiment

Our 3D object was a box wrapped with graphing paper, which served as a guide in determining the points to be used for the reconstruction. Eight points were utilized in the experiment. Two shots were taken of the same camera settings with the second shot moved a distance d.


Figure 3. A shot of the 3D object, box, covered with a graphing paper.




Figure 4. A second shot using the same camera settings and displaced by a distance b.

Figure 5. Reconstructed image using the stereometry technique


We were able to successfully reconstruct our 3D object using the stereometry technique. Remember, as long as one have multiple views of a 3D object, one can obtain the surface of the image using a digital camera with a 2D view.

Tips:

1. In taking the pictures, make sure not to rotate the camera and carefully(and accurately) measure the distance, b.

I give myself a grade of 10 for this activity since I was able to render accurately a 3D reconstructed image of the sample images. I easily finished the activity because I already did a similar one during my AP 186 class.

*image taken from: http://http://spot.pcc.edu/.../lecture%201/parallax.jpg

Reference:

1. Maricor Soriano. Applied Physics Lectures: Stereometry. 2009

For teaching concerns please visit: https://sites.google.com/site/alongjas/

Activity 6: Camera Calibration

Camera calibration allows us to obtain the camera properties that relates the image coordinates to the real world coordinates. The process results in a possible recovery of the principal point (Xi, Yi) in the image plane from the object coordinates (Xo, Yo, Zo) and vice - versa. Note that the image points are space points in the image plane and are therefore in 2D while the object or world points are space points in the real world and are in 3D. Being able to correctly allows us to model the physical processes involved in the geometric aspects of image formation allows us to[2]:

1. Determine 3D scene structures
2. Develop stereo or multiple camera systems for range measurement
3. Identify and correct image distortions


In this activity, we utilize camera calibration techniques to two set-ups namely (i) using a folded checkerboard (image in 3D) and (ii) using a flat checkerboard (image in 2D). The former will be done manually while the latter will use the camera calibration toolbox available for Matlab users.


I. Using a folded checkerboard

Figure 1. An image of the folded Tsai board (in 3D) taken inside the Instrumentation Physics Laboratory.

The camera properties are stored in the matrix containing the a's and it can be solved by using the image coordinates (Xi, Yi) and object coordinates (Xo, Yo, Zo).

The equation can be simplified to:

where Q is the 2 x 11 matrix on the left hand side of the above equation and p are the image coordinates. Note that more than 11 sample points are needed in order for the calibration to be successful.

Figure 2. Reconstruction of the Tsai board with the 15 points used in the calibration.

Shown in Figure 2 are the points chosen(small filled rectangles in red) superimposed in the checkerboard. The green square in the center represents an edge. The corresponding image coordinates of the chosen points is presented in Table 1 below.

Table 1. Real world points (Xo, Yo, Zo) and corresponding Image points (Xi, Yi)

The concept of the activity seems easy but actually the implementation part can be very tricky. You need to carefully track the corresponding Xo, Yo and Zo values of the points you have selected. Manual calibration can be frustrating sometimes since once you make an error in clicking the chosen points, you have to repeat the whole procedure.

Tips:

1. Choose points in the different planes. In the start, I encountered an error which says that the outputs for the variable a are NaN. I realized that this happened because I chose all the points to be in the xy plane.

2. Do not choose the same point again. Errors will come out when you accidentally select a point that you have previously chosen.



II. Using the Matlab Camera Calibration Toolbox

The camera calibration toolbox for Matlab can easily be downloaded from this link[1]: http://www.vision.caltech.edu/bouguetj/calib_doc/

It provides a detailed outline and demonstration with a complete documentation on camera calibration. In this part of the activity, we used a total of 20 images of a planar checkerboard.

Figure 3. Image of the Tsai grid in 2D.

Figure 4. Sample reconstruction using the camera calibration toolbox in Matlab.

Figure 5. Reprojection error for the images
Figure 6. Extrinsic parameters in camera-centered view



Calibration results after optimization (with uncertainties):

Focal Length: fc = [ 910.33832 921.76778 ] [ 8.77607 10.10174 ]
Principal point: cc = [ 132.31855 -11.31912 ] [ 0.00000 0.00000 ]
Skew: alpha_c = [ 0.00000 ] [ 0.00000 ]
Distortion: kc = [ -0.08941 0.14265 -0.04032 -0.03135 0.00000 ]
Pixel error*: err = [ 1.25516 1.10094 ]

*The pixel error value we obtained is actually small considering that the size of our image is in Megabytes.

This activity may seem easy since everything is available in the Matlab toolbox. But actually, it was also quite difficult since the procedure to assign the endpoints is done manually. This leads to errors especially in images where the edges are not that resolved. Like the manual procedure, the camera calibration toolbox can also be frustrating since once you make an error in assigning the endpoints, you have to repeat the whole procedure.

I give myself a grade of 9.5 for this camera calibration activity.

References:

[1]. Camera Calibration Toolbox available at: http://www.vision.caltech.edu/bouguetj/calib_doc/.
[2]. M. Soriano. "A Geometric Model for 3D Imaging " Applied Physics Lecture Notes 2010.



For teaching concerns please visit: https://sites.google.com/site/alongjas/

Activity 5: Shape from texture using local spectral moments

Upon examining a picture of a curved 3D object made up of uniform patterns, we will notice that the size of the patterns and the spaces in between them seem to diminish as they approach the edges while the shape is somewhat distorted. It gives us an impression that the patterns differ in distance with respect to your line of vision. This disparity in the observed patterns is mainly due to the object being curved.

In this activity, I will outline how to recover the shape of the curved surface from texture information using a non-feature based solution. This technique assumes that the image texture variation is due to the surface curvature rather than to surface texture variation. Texture in this activity refers to the patterns present in the object.

Experiment

We were tasked to take a picture of a 3D object with repeating texture or pattern in its surface. I used a picture of a cylinder (3D object) with repeating black dot patterns of uniform size, shape and spacing on its surface that was used in Super and Bovik’s paper.

Figure 1. Sample image of a curved 3D surface (image size: 164 x 310)

The goal is to reconstruct the surface shape from the recovered tilt and slant using Super and Bovik's technique. A successful implementation of this activity will result in a 3D rendition of the object's surface.

The Gabor filter set we used was composed of radial frequencies ranging from 12 to 96 cycles/image and angular frequencies ranging from -70^o to +90^o in 20^o increments.

Gabor filter set:

Frequency = [12 17 24 34 48 69 96];

Orientation = [-70 -50 -30 -10 10 30 50 70 90]*pi/180;

Steps from Super and Bovik's paper:

1) Convolve the image with Gabor functions and their partial derivatives, and smooth the filter output amplitudes by convolving them with a Gaussian.

The sigma value in the Gaussian window can be varied but as a rule it should be at least 10% the size of the test image.

2) Compute the normalized (a, b, c) image moments from the filter outputs.

3) Optional (for textures with local inhomogeneity): smooth the moments.

4) Compute the canonical moments (M, m, 0) at each point using Equation 8.

5) Find the point x0 of minimum slant (point of minimum sqrt(Mm)). Assume this is a frontal point. Set M_s, ms to be M(xo), m(xo), respectively.

6) Compute sigma, tau at each point from M(x), m(x), q(x), and M_s, m_s using Equations 21- 23.

To reconstruct an estimate of the surface, we used shapeletsurf.m. It reconstructs an estimate of a surface from its surface normals by correlating the surface normals with those of available shapelet basis functions. The correlation results are summed to produce the reconstruction. The summation of shapelet basis functions results in an implicit integration of the surface while enforcing surface continuity.

Figure 2. Reconstructed shape from texture details.


Figure 2 shows the reconstructed shape of the sample cylinder image.

Tips:

1. Dr. Soriano suggested to use the axis equal command to make the size of the reconstructed image equal to the sample image. Unfortunately, when i used the command, the rendered image was very thin. I had to stick with Figure 2 as it looks a lot better than the former.

2. Remember when viewing the surface you should use axis ij so that the surface corresponds to the image slant and tilt axes.

I give myself a grade 0f 10 for this activity.

Reference:

[1]. Super B, Bovik A, Shape from texture using local spectral moments, IEEE Trans Pattern Analysis and Machine Intelligence, 17(4): 333-343,1995

For teaching concerns please visit: https://sites.google.com/site/alongjas/

Activity 4: High Dynamic range Imaging

I. Theory

While taking pictures using a digital camera outdoors under sunlight or in a place with a very bright artificial light source, have you noticed that after the shot is taken a very bright image is seen on the LCD screen? We sometimes refer to it as being saturated. This occurs mainly because of the limited dynamic range of the camera used. One has to choose the range of radiance values that are of interest and determine the exposure time suitably. Sunlit scenes and scenes with shiny materials and artificial light sources often have extreme differences in radiance values that are impossible to capture without either underexposing or saturating the film.

In this work, I will be outlining how to address this problem by using a technique called High Dynamic Range Imaging. The idea is simple. You take multiple pictures of the same scene under different exposure settings in order to recover the full dynamic range in such a scene. We then use the algorithm presented in the paper of Debevec and Malik, which fuses the multiple photographs into a single high dynamic range radiance map whose pixel values are proportional to the true radiance values in the scene.

This technique is very important as digitized photography is becoming increasingly popular and more widely used than its analog counterpart.

II. Experiment

We were fortunate that the Plasma Laboratory was very accommodating, allowing us to use their experimental set-up through which plasma sheets are created. The sample pictures were taken from an intensely bright plasma sheet under six (1/320 1/400 1/500 1/640 1/800 and 1/1000) different camera exposure values but with the same f-number. The f-number is inversely related to the size of the aperture, which can affect how irradiance is perceived by the sensor. We further observed that the camera is fixed (e.g. placed on a tripod) to avoid the need of aligning the images.

Figure 1. Plasma sheet images taken at 6 different exposure values: (a) 1/320 (b) 1/400 (c) 1/500 (d) 1/640 (e)1/800 and (f)1/1000

We picked 20 points in the image and plotted their ln (shutter speed) vs gray level value (Z's). The response function g(Z) of the camera was then solved using the steps outlined in Debevec and Malik. Therefore, given a set of pixel values obtained from twenty points in six images with different exposure times, this function returns the imaging system’s response function g as well as the log film irradiance values for the observed pixels.

Figure 2. System's image response g as a function of the observed pixels

Figure 3. Pixel values, Z, for different exposure times

Figure 4. Log film irradiance at diferent pixel values




I give myself a grade 0f 8.5 for this activity.

References:

[1] Maricor Soriano. Physics 305 Activity: High Dynamic Range Imaging 2009

[2] Paul Debevec and Jitendra Malik "Recovering High Dynamic Range Radiance Maps from photographs"

For teaching concerns please visit: https://sites.google.com/site/alongjas/