An Implementation of "Removing Photography Artifacts using Gradient Projection"

An Implementation of Removing Photography Artifacts using Gradient Projection

Overview

I chose to implement this paper for thefinal project of my computer vision class. The goal was to implement an algorithm that combines a photograph taken with a flash and one in ambient light in order to produce a higher quality resulting image. Flashes are useful to capture a scene under low-light conditions, but are notorious for producing undesirable artifacts, such as strong highlights due to reflection. Ambient images, on the other hand, may not fully capture the details of scene when it is not well-lit, or contain undesirable reflections. A shown in the example above, it is possible to combine the two images using a gradient projection method to obtain a reflection-free ambient image without a hot spot, and recover the reflection layer in the process.

The project is based on the work of Agrawal et al. 2005 "Removing photography artifacts using gradient projection and flash-exposure sampling".

Method

It is possible to compute and compare the gradients of the flash and ambient images at each pixel. At pixel where no artifacts (such as reflections) are present the gradients will lie in the same direction. On the other hand, where artifacts are present the gradients differ, which we take as noise perturbing the original image. The approach taken is to remove that noise by projecting the gradient of the image that contains the reflection onto the gradient of the image that does not. This will remove the component of the noise that is orthogonal to the original gradient vector. The parallel component of the noise, however, is undistinguishable.

Reconstructing the image intensities from the gradients

rotating the gradients of an image, and reconstructing it

Once the projected gradients are computed, it is possible to reconstruct the image intensities I from the gradients G by solving the following Poisson equation:

grad^2 I = div G

where G = grad I, and therefore div G = div grad I = grad^2 I.

It is possible to solve the Poisson equation using a 2D sine transform: apply the sine transform, solve the equation in Fourier space, and then invert the sine transform. The figure above shows the result of rotating the gradients of the image on the reconstructed image. Rotating the gradients by 90 degrees and reconstructing the image will always produce an image that is zero everywhere. This shows that the component of noise that is orthogonal to the desired gradient can be safely removed while preserving the visual information.

The first result is the reconstructed image. The color balance is slightly off and the image is a bit darker than the original. This is due to the imprecise computation of the gradients and the boundary pixels being zero. The authors also mention getting better result in YUV space instead of RGB, but I have not tried this. The next two images show the gradients rotated by pi/6, and pi/3. The image gradually becomes darker, and at pi/2 the image should be zero everywhere, as described in the paper. At pi, the gradients are reversed and as expected the image is the negative of the original.

Reflection Removal

Museum

This example combines an image taken with a flash in order to illuminate a close object, and an image of the same scene taken without a flash. The second image does not display a hot spot, but the object appears darker. We create a mask image that contains the location of the hot spot, and another mask that defines how similar in orientation the gradients of the two images are. We can then combine the gradients using those masks, and then solve the Poisson equation to find the result.

Observation deck

In this case, the image taken with a flash illuminates a person next to a window at night. In the flash image the person is well lit, but its reflection hides the scene outside the window. In the ambient image, the outside scene is visible. We combine the sum of the two images and the flash image using a underexposed map that detects the regions of the ambient image that are well lit.

Self Reflection

In this example (see top figure), we wish to remove the reflections of the ambient image by using the flash image. However the flash image shows a hot spot, that we can detect using a saturation map. Everywhere except at the location of the hot spot we project the gradient of the ambient image into the gradient of the flash image to remove the reflections. At the location of the hot spot, we just use the ambient image, assuming there are no reflections there. This technique cannot remove collocated artifacts. We can also recover the reflection by subtracting the resulting image gradients from the original gradients.