What is facial recognition?
Facial recognition technology has spread prodigiously. It’s there on Facebook, tagging photos from the class reunion, your cousin’s wedding and the office summer party. Google, Microsoft, Apple and others have built it into apps to compile albums of people who hang out together.
It verifies who you are at airports and is the latest biometric to unlock your mobile phone where facial recognition apps abound. Need to confirm your identity for a £1,000 bank transfer? Just look into the camera.
New applications crop up all the time. Want to know who’s at the door? A video doorbell with facial recognition will tell you, provided you’ve uploaded a photo of the person’s face. Other systems are used to spot missing persons and catch slackers who lie about the hours they spend in the office. Advertisers, of course, are in on the act. Thanks to facial recognition, billboards can now serve up ads based on an estimate of your sex, age and mood.
That sounds quite Big Brother. Is it a surveillance tool?
Sometimes, yes. China uses facial recognition for racial profiling and its tracking and control of the Uighur muslims has been roundly condemned as a shameful first for a government.Its cameras also spot and fine jaywalkers, verify students at school gates, and monitor their expressions in lessons to ensure they are paying attention.
Russia has embraced the technology too. In Moscow, video cameras scan the streets for “people of interest” and plans have been mooted to equip the police with glasses that work the same way.
There have been reports that Israel is using facial recognition for covert tracking of Palestinians deep inside the West Bank. Meanwhile in Britain, the Metropolitan and South Wales police forces have trialled facial recognition to find people in football and rugby crowds, on city streets, and at commemorations and music festivals. Taylor Swift even installed the tech at a gig in California to weed out stalkers.
Shops are increasingly installing the technology to deter and catch thieves. Next year, it will make its Olympic debut in Tokyo.
How did it get everywhere?
Advances in three technical fields have played a major part: big data, deep convolutional neural networks and powerful graphics processing units or GPUs.
Thanks to Flickr, Instagram, Facebook, Google and others, the internet holds billions of photos of people’s faces which have been scraped together into massive image datasets. They are used to train deep neural networks, a mainstay of modern artificial intelligence, to detect and recognise faces. The computational grunt work is often done on GPUs, the superfast chips that are dedicated to graphics processing. But there is more to it than flashy new technology. Over the past decade in particular, facial recognition systems have been deployed all over the place, and the data gathered from them has helped companies hone their technology.
How does it work?
First off, the computer has to learn what a face is. This can be done by training an algorithm, usually a deep neural network, on a vast number of photos that have faces in known positions. Each time the algorithm is presented with an image, it estimates where the face is. The network will be rubbish at first, like a drunk playing pin the tail on the donkey. But if this is done multiple times, the algorithm improves and eventually masters the art of spotting a face. This is the face detection step.
Next up is the recognition part. This is done in various ways, but it’s common to use a second neural network. It is fed a series of face pictures and learns – over many rounds – how best to tell one from another. Some algorithms explicitly map the face, measuring the distances between the eyes, nose and mouth and so on. Others map the face using more abstract features. Either way, the network outputs a vector for each face – a string of numbers that uniquely identifies the person among all the others in the training set.
In live deployments, the software goes to work on video footage in real time. The computer scans frames of video usually captured at crowd pinch points, such as entrances to football stadiums. It first detects the faces in a frame, then churns out vectors for each one. The face vectors are then checked against those for people on a watchlist. Any matches that clear a preset threshold are then ranked and displayed. In the British police trials a typical threshold for a match was 60%, but the bar can be set higher to reduce false positives.
Who has the technology?
Tech firms around the world are developing facial recognition but the US, Russia, China, Japan, Israel and Europe lead the way. Some nations have embraced the technology more readily than others.
China has millions of cameras connected to facial recognition software and Russia has declared hopes to expand its own surveillance networks. In Europe, as elsewhere, facial recognition has found its way into shops to spot thieves and into businesses to monitor staff and visitors, but live face recognition in public spaces is still mostly at trial stage.
In the US, the police typically use facial recognition to search CCTV footage for suspects rather than scanning live crowds. But it is becoming more pervasive. A 2016 report from Georgetown Law’s Center on Privacy and Technology found that half of all Americans are in police facial recognition databases, meaning that algorithms pick suspects from virtual line-ups of 117 million mostly law-abiding citizens.