Facebook wants to use AI to understand image content and transform the way we search

Facebook is applying computer vision and artificial intelligence (AI) technology that it developed for visually impaired people to its image search function.

Using its Lumos computer vision platform, Facebook will now enhance your image search results by actually “looking” at images and understanding the content within, reports TechCrunch .

As Facebook explains on its developer blog , search today is based on tags and text. Images need to have the right tags and captions or search will simply not work.

Lumos is a deep neural network, one implementation of AI that learns from experience. This system looks at images and then at the tags that people have given them. Considering that Facebook has over 1.7 billion users, there must be billions upon billions of images to learn from.

Facebook engineers were working on a project that developed ‘automatic alt text’ (AAT) for images. This text is a computer-generated description of an image and was designed to help visually impaired people. Early models would describe an image solely based on its content, without context. It could, for example, say that an image had a cat that was sitting or a light that was on, but that’s it.

Today, AAT can describe things like “people walking”, “people dancing”, “people riding horses”, etc., says Facebook.

In an example, Facebook asks us to assume that we’re looking for a person riding a horse. Lumos will first look for images of people and horses. It will then attempt to understand the relation between the horses and the people and “semantically cluster” images together. With sufficient training, Lumos will be able to determine the clusters that you’re looking for and in fact, give you images of people riding horses.

The system is also smart enough, apparently, to not give you dozens of images of the same thing taken from slightly different angles. All this time, Lumos will be adding ever more enhanced AAT to images.