Google I/O 2019: Live Caption lets you overlay text on any audio or video content

One of the interesting features announced here at the **Google I/O 2019 conference** has to be the Live Caption mode. Developed keeping in mind the deaf and hard of hearing community, the Live Caption feature can be useful for anyone who may not want to listen to audio, but read it. [caption id=“attachment_6590561” align=“alignnone” width=“1280”]  Live Caption in Android Q.[/caption] The Live Caption feature is expected to be **coming out on Android Q** whenever it has a public rollout, and will first be **seen on Pixel devices** with eventual rollout to other OEMs. Android’s Live Caption feature uses the Android Audio Framework APIs and on-device speech recognition models. Google CEO Sundar Pichai stressed a lot on using on-device machine learning rather than sending data to the cloud for processing. Live Caption is an example of that. The demos showcased involved live captions coming on the screen as soon as the words were spoken out. “We believe technology can help us be more inclusive, and AI is providing us with new tools to dramatically improve the experience for people with disabilities,” said Pichai.

Live Caption: How does it work?

When you are adjusting the volume, just below the volume rocker logo, you will see a keyboard icon. When that is showing without a cross across it, that means the Live Caption feature is active. Once it is active, any video or audio file playing on your phone will have a text overlay of what is being spoken. This feature does not work on songs. It works with videos which have people speaking, it works with TV shows as well as news programs. In fact, Live Caption even works on podcasts and audio-only speech. It is also expected to support video chat apps such as Google Duo and Skype. You are presented with a rectangular box which can be double tapped to increase in size, which will have the live transcription text visible. You can move around this rectangle anywhere on the screen. Since it uses on-device machine learning, Live Caption feature does not require you to be connected to the internet. In fact, at the Google I/O demo, it was shown that this feature even works in the airplane mode. Also, you don’t need to have the volume turned on. So long as there is audio detected, the Live Caption feature will work. You cannot copy this transcribed text, it will only be present when the audio is turned on. The idea behind this feature is to help those from the deaf and hard of hearing community to be able to read out content they cannot listen to. This opens up a whole world of content to those from this community. Apart from the deaf community, Live Caption can also help those who may not be able to hear audio clearly. For instance, when you are watching a video but are commuting on the train, the background noise may make it impossible for you to listen to the audio. In that case, the Live Caption works well. When listening to audio files, the Live Caption mode can be turned on even when you switch between apps. So long as the audio is turned on, the Live Caption will be visible. So far, from the demo it looks like it is only available in English. But it shouldn’t be that difficult for Google to bring it to other languages as well. Disclaimer: Tech2 was invited to Google I/O 2019 by Google India. All expenses related to travel and accommodation were borne by Google India.