Image Analogies Implementation

This project, implemented for my CS224 class, was based on the Image Analogies Paper by Hertzmann et Al.. In this paper, the authors present a single algorithm that can solve many different image-based problems, such as Texture Synthesis (generating a larger picture from a sample texture), Super-resolution (inferring a high-resolution image from a low-resolution source), texture transfer (in which images are made to look like they were made out of a source texture), Artistic filters (in which various drawing and painting styles, including oil, pastel, and pen-and-ink rendering, are synthesized based on scanned real-world examples).

The algorithm works by using the relationship between a pair of images (A and A') to generate the second image B' of another pair (B and B'). Essentially "A is to A' as B is to B'." For this page's examples A and A' are the top images, and B and B' are the bottom images (B' is the generated image).

The process works at multiple scales: a Gaussian pyramid of the image is created (each level of the pyramid contains the image in the above level scaled by half). Starting at the lowest level (the smallest), pixels are generated in raster order. For each pixel a feature vector combining the pixel's neighbors in both the current level and the one below, for both A' and B' is created. That feature vector is then matched against all the feature vectors for each pixel in the A/B pair. The color of the closest matching feature is used for the current pixel, and the algorithm moves on to the next pixel. Coherence is maintained by preferring pixels that come from the same region of the image.

This paper also introduces "Texture-by-Numbers." A user has to first take any photograph (A') and generate a mask (A) of its different regions (it the example here, water, forest, grass...). By creating a new, imaginary mask (B), the user can then use the algorithm to recreate a new photograph (B') matching that mask !

Check out the paper's website for more examples.