Computing/Hybrid Rendering

Hybrid Rendering

- Creating Realistic Models from Laser Data and Photographs

This page describes a system I built between 2002 and 2004 to investigate a new framework for building extremely realistic models of architecturally-based environments.

So far, there have been two main approaches to this problem:

Photogrammetry - tries to reconstruct environments from sets of photographs of the scene. Usually, a rough simple block model of the main objects in the scene is built. Then edges visible in the photographs are matched to edges in the blocks and this information is used to reconstruct the relative dimensions of the blocks and positions of the cameras. Images are then projected back onto the blocks to make the result look photorealistic (first image pair below). Where the underlying structure is inadequately detailed or very difficult to model (organic objects for example), the results can be poor (second image pair below)

Laser Scanning - collects millions of points around a sphere, each point telling you its distance away from the scanner (image pair below). It's also possible to map colour photos onto the laser data to give an extremely dense set of coloured points all at the correct position in 3D space. Gaps between points can be filled in (if the holes are small) to create very detailed structural representations of a scene. The problems here are fourfold: 1) areas behind the scanner line of sight appear as (often very large) holes; 2) scanned data sets can be very large while in architectural scenes a lot of the points are redundant (like thousands of points used to scan a wall, where a single plane would do) 3) It is possible to take multiple scans from different locations to fill in holes, but it's difficult to match up the overlapping areas between scans 4) matching colour photos to scans is cumbersome and often inaccurate.

My Hybrid Rendering software is built on the premise that a system which integrates aspects of photogrammetry and laser scanning can model the best parts of both while eliminating the associated problems. The object oriented approach allows various reader, writer and modifying objects to be plugged in and offers a very flexible way in which to interact with and process the different types of input. The implementation is written around VTK (freely available).

Integrating the model types requires that we first align them with each other. Because the scanned models contain lots of detailed areas, we have to first extract features which match across the model types. The best features to get from laser data are planes. To extract planes from a laser scan we read it in as a panoramic image whose pixels represent distance from the scanner by a shade of grey. these are called range images (below).

Next the range image is divided up into a grid containing typically 7x7 pixels called a cluster. The 3D values corresponding to the pixels in a cluster are computed and the plane which fits closest to the most points in each cluster is calculated (using a RANSAC approach). Regions of superclusters are then grown for neighbouring clusters of similar orientation and proximity in space. These superclusters represent the planar regions in the scanned scene. See below for the above image processed. The bright colours are planes.

The photogrammetric model contains all the details about where and how the original photographs used to construct the model were taken. We want to retain all these details, so we align the range data to the photogrammetric model. To do this, the planes extracted from the range image are aligned to the planes in the photogrammetric model. The Iterative Closest Point algorithm does this and the transformation computed is then used to align the other non-planar data in the range image. Because all the points in the range image planes are present in the photogrammetric planes, all these redundant range points can be discarded. This reduces the scanned data set by up to 80%. Aligning the range data to the photogrammetric model means it is straightforward to add other scans taken from other locations, provided there are four corresponding planes visible in both data set types. The figure below shows multiple scans aligned to a photogrammetric model.

With the data sets aligned, the model is exported in a RenderMan compliant format for rendering using the Blue-Moon Rendering Tools - a high quality rendering package. BMRT has no point-type primitive, so the laser points are exported as tiny spheres. This creates an overlapping surface where data is sufficiently dense. Holes are seen elsewhere, but the photogrammetric model is exported as a watertight set of polygons and this fills in any gaps in the laser data.

It is often the case that some spurious noisy laser scanned points have to be removed. Clipping functionality is built in to the system. Also, because of the simple nature of the photogrammetric model structure, local concavities in the scene can often be hidden. This is solved by snipping holes in the polygons to reveal the scene beyond. See figure below for an example.

The simple photogrammetric model geometry and the remaining scanned points after plane extraction -->

The two model types are aligned and holes are cut to reveal detail behind local concavities in the photogrammetric model. Note the detail hidden in the doorway recess in the left image is visible after a hole is cut in the covering wall. Also note how all the holes in the laser data are filled by the photogrammetric model, which gets detail added from the laser points. -->

Finally the original images are projected back onto the model using view-dependent texture mapping. Surface occlusions are solved for by ray tracing. Three example videos are shown below (6.9MB each). The top left image shows the photogrammetric block model which is the underlying structure in the top right video. The top right video is the photogrammetric model only with photographic texture mapping. Note how the foreground objects appear to change position suddenly when the view-dependent texture mapping chooses a different image. The bottom left video shows the combined simple photogrammetric model plus laser derived structure rendered with plastic texture under simple lights. The right hand video is the combined geometries from the previous video rendered with photographic texture mapping. Note how the foreground objects now appear to move round and obscure the background as would happen in real life. Despite some slight mismatch of textures, the overall impression is a more realistic representation. Blue areas in the output are areas where no photographic information is available to be textured (additional photos can be added to fix this). -->

Publications:

Watson, G. & Lamond, B. 2004. Towards a unified approach to 3D environment capture and rendering. Presented at ACM SIGGRAPH 2004.

Lamond, B. & Watson, G. 2004. Hybrid Rendering - A New Integration of Photogrammetry and Laser Scanning for Image Based Rendering. IEEE Theory and Practice of Computer Graphics 2004.

Presentations given at Theory & Practice of Computer Graphics 2004 in Bournemouth, UK and IPAB, Edinburgh, 2005 here