Facebook is using an artificial intelligence system to automatically caption photos in an effort to increase the accessibility of its website and apps.
The feature called “automatic alternative text”, uses image recognition technology developed through machine-learning to identify the objects pictured.
It’s not perfect yet, with sample captions from Facebook, reading more like a laundry list than a descriptive overview: “Image may contain: two people, smiling, sunglasses, sky, outdoor, water” reads the caption on one picture of a couple taking a photograph on a beach, while “image may contain: pizza, food” is the caption for a picture of a tasty-looking pepperoni and olive pizza.
In March, Twitter announced its own attempt to solve the problem, rolling out a feature for its apps that lets users manually add captions to images they upload. Because people are better than machines at describing images (for now), the captions are likely to be more accurate and descriptive, but the burden of entering a caption for every image means that few actually have a description appended.
That’s not helped by the fact that Twitter’s feature is off by default, buried in the accessibility settings menu, alongside other settings aimed at people who need accessibility help, not those who want to provide the help.
Facebook’s approach could overcome those hurdles, but the company faces problems of its own. Inaccuracy is the big one: not only do machines still get things wrong, sometimes they get them horribly wrong. A similar system rolled out by Flickr last year managed to tag an image of Auschwitz with “sport” and a picture of a black man with “ape”.
Facebook hopes to dodge those bullets by only applying captions when the system is “80% sure”, and applying no caption at all in sensitive situations.