MIT and Google develop system for real-time image retouching on smartphones

Researchers from the Massachusetts Institute of Technology’s (MIT) Computer Science and Artificial Intelligence Laboratory (CSAIL) and Google have developed a smartphone-based system for providing enhancements to photographs in real time. This means that users can apply professional grade image processing algorithms on images before even capturing the image. Users can incorporate the enhancements to the image while framing the photograph. [caption id=“attachment_3888177” align=“alignleft” width=“380”]  Image: MIT.[/caption] Photographers will no longer have to spend time balancing the colours or tuning the photograph after taking the image. The same system can also be used to accelerate image processing algorithms, and was tested on a new Google algorithm for producing high dynamic range images. The system was able to deliver results indistinguishable from conventional approaches, in about one eighth of the time that it normally takes. Jon Barron from Google says, “Using machine learning for computational photography is an exciting prospect but is limited by the severe computational and power constraints of mobile phones. This paper may provide us with a way to sidestep these issues and produce new, compelling, real-time photographic experiences without draining your battery or giving you a laggy viewfinder experience.” The system is based on machine learning algorithms that were trained on thousands of pairs of regular photos and professionally retouched versions of the same photos. Most of the image processing is performed on a low resolution version of the image. The machine learning system outputs a set of formulae for changing the colour values of the pixels on the low resolution image, and these formulae are applied to the high resolution version.

To make the changes on the pixels in the high resolution version that were not present in the low resolution version, a combination of the formulae is applied according to the distance of the pixels from those that were present in the low resolution version. The approach allowed the engineers to reduce the memory requirements to about a hundredth of the requirement of processing a full-resolution image.