ON THE 21ST FLOOR of a high-rise hotel in Cleveland, in a room full of political operatives, Microsoft’s Research Division was advertising a technology that could read each facial expression in a massive crowd, analyze the emotions, and report back in real time. “You could use this at a Trump rally,” a sales representative told me.

At both the Republican and Democratic conventions, Microsoft sponsored event spaces for the news outlet Politico. Politico, in turn, hosted a series of Microsoft-sponsored discussions about the use of data technology in political campaigns. And throughout Politico’s spaces in both Philadelphia and Cleveland, Microsoft advertised an array of products from “Microsoft Cognitive Services,” its artificial intelligence and cloud computing division.

At one exhibit, titled “Realtime Crowd Insights,” a small camera scanned the room, while a monitor displayed the captured image. Every five seconds, a new image would appear with data annotated for each face — an assigned serial number, gender, estimated age, and any emotions detected in the facial expression. When I approached, the machine labeled me “b2ff” and correctly identified me as a 23-year-old male.

It interpreted my facial expression as “neutral,” with a bit of “surprise.”


Photo: Alex Emmons

I asked Christina Pearson, a nearby Microsoft spokesperson, to confirm that the technology was meant to be used on a large crowd, like at a Trump rally. “Yes,” she confirmed. “Or it’s meant to be the Super Bowl, whatever you want.”“Realtime Crowd Insights” is an Application Programming Interface (API), or a software tool that connects web applications to Microsoft’s cloud computing services. Through Microsoft’s emotional analysis API — a component of Realtime Crowd Insights — applications send an image to Microsoft’s servers. Microsoft’s servers then analyze the faces and return emotional profiles for each one.

In a November blog post, Microsoft said that the emotional analysis could detect “anger, contempt, fear, disgust, happiness, neutral, sadness or surprise.”

Microsoft’s sales representatives told me that political campaigns could use the technology to measure the emotional impact of different talking points — and political scientists could use it to study crowd response at rallies.

But the use of facial analysis at political events is eerily reminiscent of George Orwell’s 1984, where the government monitors faces for any sign of dissatisfaction, or “facecrime.” In Orwell’s world, “to wear an improper expression on your face (to look incredulous when a victory was announced, for example) was itself a punishable offense.”

Microsoft’s Realtime Crowd Insights could potentially pick out the stern faces of dissenters, or angry faces of future protestors, all in a matter of seconds.

Donald Trump’s security personnel have already tried to pre-empt protests at rallies by kicking out people they thought likely to protest. At one rally in February, security asked 30 black students to leave before Trump started speaking. According to USA Today, the students had planned to sit in silent protest, but one 19-year-old student said, “We didn’t plan to do anything.”

In Politico’s suite in Cleveland, one passerby told me he was “slightly creeped out,” and another asked me why Microsoft was collecting their facial information. The machine also picked up on a small range of negative responses in the room, including “fear, contempt, and disgust.”


Photo: Alex Emmons

When I attended the “Realtime Crowd Insights” display in Philadelphia, I asked to speak with a spokesperson and was introduced to Kathryn Stack, a managing director with the public affairs firm Burson-Marsteller. I asked Stack whether the product could be used to identify protestors or dissidents at rallies or political events.“I think that would be a question for a futurist, not a technologist,” she responded.


FACIAL RECOGNITION TECHNOLOGY — the identification of faces by name — is already widely used in secret by law enforcement, sports stadiums, retail stores, and even churches, despite being of questionable legality. As early as 2002, facial recognition technology was used at the Super Bowl to cross-reference the 100,000 attendees to a database of the faces of known criminals. The technology is controversial enough that in 2013, Google tried to ban the use of facial recognition apps in its Google glass system.

But “Realtime Crowd Insights” is not true facial recognition — it could not identify me by name, only as “b2ff.” It did, however, store enough data on each face that it could continuously identify it with the same serial number, even hours later. The display demonstrated that capability by distinguishing between the number of total faces it had seen, and the number of unique serial numbers.

#GauriLankeshMurder" data-image-description="<div class="td-pb-row"><strong> What does it say when the people who trolled slain journalist Gauri Lankesh are followed on social media by the Prime Minister?</strong></p> <h1 class="header1"></h1> <p><strong>Right wing spews venom on social media after #GauriLankeshMurder</strong></div> <div class="td-pb-row"> <div class="td-pb-span8 td-main-content" role="main"> <div class="td-ss-main-content"> <div class="td-post-content"> <div class="mashsharetop"> by- Altnews</div> <p>“<em>After Burhan Wani Gauri lankesh also killed how sad</em>“, <a href="https://twitter.com/aashish81us/status/905112888234328064" target="_blank" rel="noopener">tweeted</a> Ashish Singh. He further retweeted a tweet which said, “<em>Hame chahiye azadi jihadiyose Jai Shree Ram,Jai Shree Ram</em>“. In another <a href="https://twitter.com/aashish81us/status/905102760978276352" target="_blank" rel="noopener">tweet</a>, Ashish Singh retweeted a Bangalore Times tweet about Gauri Lankesh’s death with the comment, “<em>Jaisi karni vaisi bharni</em>“. Ashish Singh’s twitter profile picture is a selfie with Maharashtra CM Devendra Fadnavis. Ashish is followed by PM Modi on Twitter. Other senior ministers in the Union Government like Ravi Shankar Prasad and Vijay Goel also follow his Twitter handle. His Twitter bio describes him as “<em>India272Pune , ( IIT Bombay,IIMP) – Political Strategist , Cartoonist I am Hindu and Team PM Modi</em>“. Ashish was one of the many who adhere to the right-wing ideology and started spewing venom after senior journalist Gauri Lankesh was shot dead at her home in Bangalore.</p> <div class="myadsense" align="center"></div> <p><img class="aligncenter size-large wp-image-5337" src="https://i0.wp.com/www.altnews.in/wp-content/uploads/2017/09/ashish-singh.jpg?resize=640%2C491" sizes="(max-width: 640px) 100vw, 640px" srcset="https://i0.wp.com/www.altnews.in/wp-content/uploads/2017/09/ashish-singh.jpg?resize=1024%2C785&amp;ssl=1 1024w, https://i0.wp.com/www.altnews.in/wp-content/uploads/2017/09/ashish-singh.jpg?resize=300%2C230&amp;ssl=1 300w, https://i0.wp.com/www.altnews.in/wp-content/uploads/2017/09/ashish-singh.jpg?resize=768%2C589&amp;ssl=1 768w, https://i0.wp.com/www.altnews.in/wp-content/uploads/2017/09/ashish-singh.jpg?resize=80%2C60&amp;ssl=1 80w, https://i0.wp.com/www.altnews.in/wp-content/uploads/2017/09/ashish-singh.jpg?resize=696%2C534&amp;ssl=1 696w, https://i0.wp.com/www.altnews.in/wp-content/uploads/2017/09/ashish-singh.jpg?resize=1068%2C819&amp;ssl=1 1068w, https://i0.wp.com/www.altnews.in/wp-content/uploads/2017/09/ashish-singh.jpg?resize=548%2C420&amp;ssl=1 548w, https://i0.wp.com/www.altnews.in/wp-content/uploads/2017/09/ashish-singh.jpg?w=1175&amp;ssl=1 1175w" alt="ashish-singh" width="640" height="491" /></p> <p>Another account that is followed by PM Modi and Union Minister Giriraj Singh tweeted, “<em>एक कुतिया कुत्ते की मौत क्या मरी सारे पिल्ले एक सुर में बिलबिला रहे है। 😜😜 अब ये कौन कह रहा है किसी शिष्य ने गुरु दक्षिणा में #GauriLankesh को वही दे दिया जिसकी शिक्षा वामपंथी अपने शिष्यों को देते है? (Translation: A bitch died a dog’s death and all of her litter is crying in the same voice)</em>”</p> <p><img class="aligncenter size-large wp-image-5347" src="https://i2.wp.com/www.altnews.in/wp-content/uploads/2017/09/Selection_06_09_2017_005.jpg?resize=640%2C665" sizes="(max-width: 640px) 100vw, 640px" srcset="https://i2.wp.com/www.altnews.in/wp-content/uploads/2017/09/Selection_06_09_2017_005.jpg?resize=985%2C1024&amp;ssl=1 985w, https://i2.wp.com/www.altnews.in/wp-content/uploads/2017/09/Selection_06_09_2017_005.jpg?resize=289%2C300&amp;ssl=1 289w, https://i2.wp.com/www.altnews.in/wp-content/uploads/2017/09/Selection_06_09_2017_005.jpg?resize=768%2C798&amp;ssl=1 768w, https://i2.wp.com/www.altnews.in/wp-content/uploads/2017/09/Selection_06_09_2017_005.jpg?resize=696%2C723&amp;ssl=1 696w, https://i2.wp.com/www.altnews.in/wp-content/uploads/2017/09/Selection_06_09_2017_005.jpg?resize=1068%2C1110&amp;ssl=1 1068w, https://i2.wp.com/www.altnews.in/wp-content/uploads/2017/09/Selection_06_09_2017_005.jpg?resize=404%2C420&amp;ssl=1 404w, https://i2.wp.com/www.altnews.in/wp-content/uploads/2017/09/Selection_06_09_2017_005.jpg?resize=32%2C32&amp;ssl=1 32w, https://i2.wp.com/www.altnews.in/wp-content/uploads/2017/09/Selection_06_09_2017_005.jpg?w=1224&amp;ssl=1 1224w" alt="Nikhil Dadich" width="640" height="665" /></p> <p>A journalist Jagrati Shukla wrote, “<em>So, Commy Gauri Lankesh has been murdered mercilessly. Your deeds always come back to haunt you, they say. Amen.</em>” Jagrati Shukla has a history of <a href="http://naradanews.com/2016/11/change-org-petition-demands-removal-of-zee-news-jagrati-shukla-for-bigotry/" target="_blank" rel="noopener">hateful tweets</a> against Dalits, Sikhs and other sections of the society.</p> <p><img class="aligncenter size-large wp-image-5338" src="https://i0.wp.com/www.altnews.in/wp-content/uploads/2017/09/Selection_06_09_2017_004.jpg?resize=640%2C423" sizes="(max-width: 640px) 100vw, 640px" srcset="https://i0.wp.com/www.altnews.in/wp-content/uploads/2017/09/Selection_06_09_2017_004.jpg?resize=1024%2C676&amp;ssl=1 1024w, https://i0.wp.com/www.altnews.in/wp-content/uploads/2017/09/Selection_06_09_2017_004.jpg?resize=300%2C198&amp;ssl=1 300w, https://i0.wp.com/www.altnews.in/wp-content/uploads/2017/09/Selection_06_09_2017_004.jpg?resize=768%2C507&amp;ssl=1 768w, https://i0.wp.com/www.altnews.in/wp-content/uploads/2017/09/Selection_06_09_2017_004.jpg?resize=696%2C459&amp;ssl=1 696w, https://i0.wp.com/www.altnews.in/wp-content/uploads/2017/09/Selection_06_09_2017_004.jpg?resize=1068%2C705&amp;ssl=1 1068w, https://i0.wp.com/www.altnews.in/wp-content/uploads/2017/09/Selection_06_09_2017_004.jpg?resize=636%2C420&amp;ssl=1 636w, https://i0.wp.com/www.altnews.in/wp-content/uploads/2017/09/Selection_06_09_2017_004.jpg?w=1174&amp;ssl=1 1174w" alt="Jagrati Shukla: So, Commy Gauri Lankesh has been murdered mercilessly. Your deeds always come back to haunt you, they say. Amen." width="640" height="423" /></p> <p>Another senior journalist who is the editor-in-chief of Vishwavani Daily and former OSD to BJP MP Ananth Kumar couldn’t resist labelling her as a Naxal Sympathiser while <a href="https://twitter.com/VishweshwarBhat/status/905093901551157248" target="_blank" rel="noopener">announcing</a>her death.</p> <p><img class="aligncenter size-large wp-image-5339" src="https://i0.wp.com/www.altnews.in/wp-content/uploads/2017/09/Selection_06_09_2017_007.jpg?resize=640%2C303" sizes="(max-width: 640px) 100vw, 640px" srcset="https://i0.wp.com/www.altnews.in/wp-content/uploads/2017/09/Selection_06_09_2017_007.jpg?resize=1024%2C485&amp;ssl=1 1024w, https://i0.wp.com/www.altnews.in/wp-content/uploads/2017/09/Selection_06_09_2017_007.jpg?resize=300%2C142&amp;ssl=1 300w, https://i0.wp.com/www.altnews.in/wp-content/uploads/2017/09/Selection_06_09_2017_007.jpg?resize=768%2C364&amp;ssl=1 768w, https://i0.wp.com/www.altnews.in/wp-content/uploads/2017/09/Selection_06_09_2017_007.jpg?resize=696%2C330&amp;ssl=1 696w, https://i0.wp.com/www.altnews.in/wp-content/uploads/2017/09/Selection_06_09_2017_007.jpg?resize=1068%2C506&amp;ssl=1 1068w, https://i0.wp.com/www.altnews.in/wp-content/uploads/2017/09/Selection_06_09_2017_007.jpg?resize=886%2C420&amp;ssl=1 886w, https://i0.wp.com/www.altnews.in/wp-content/uploads/2017/09/Selection_06_09_2017_007.jpg?w=1196&amp;ssl=1 1196w" alt="Vishweshwar Bhat Naxal sympathiser and Senior journalist" width="640" height="303" /></p> <p>Rita who claims to be a former presenter on All India Radio and is followed by PM Modi on Twitter gave not one but multiple labels. Apparently, that was the most important thing to do in this hour of grief.</p> <p><img class="aligncenter size-large wp-image-5343" src="https://i0.wp.com/www.altnews.in/wp-content/uploads/2017/09/Selection_06_09_2017_008.jpg?resize=640%2C358" sizes="(max-width: 640px) 100vw, 640px" srcset="https://i0.wp.com/www.altnews.in/wp-content/uploads/2017/09/Selection_06_09_2017_008.jpg?resize=1024%2C572&amp;ssl=1 1024w, https://i0.wp.com/www.altnews.in/wp-content/uploads/2017/09/Selection_06_09_2017_008.jpg?resize=300%2C168&amp;ssl=1 300w, https://i0.wp.com/www.altnews.in/wp-content/uploads/2017/09/Selection_06_09_2017_008.jpg?resize=768%2C429&amp;ssl=1 768w, https://i0.wp.com/www.altnews.in/wp-content/uploads/2017/09/Selection_06_09_2017_008.jpg?resize=696%2C389&amp;ssl=1 696w, https://i0.wp.com/www.altnews.in/wp-content/uploads/2017/09/Selection_06_09_2017_008.jpg?resize=1068%2C597&amp;ssl=1 1068w, https://i0.wp.com/www.altnews.in/wp-content/uploads/2017/09/Selection_06_09_2017_008.jpg?resize=752%2C420&amp;ssl=1 752w, https://i0.wp.com/www.altnews.in/wp-content/uploads/2017/09/Selection_06_09_2017_008.jpg?w=1226&amp;ssl=1 1226w" alt="Rita For those who don't know gauri lankesh leftist nexal sympathizer anti establishment anti Hindu" width="640" height="357" /></p> <p>Oxomiya Jiyori which is yet another account followed by PM Modi on Twitter said “<em>God had different plan for group of Bharat tere tukde Honge</em>” while commenting on Gauri Lankesh’s pictures with JNU students Kanhaiya Kumar and Umar Khalid. Those masked men who actually raised those slogans in JNU have never been arrested but the propaganda is being used to malign and justify Gauri Lankesh’s murder.</p> <p><img class="aligncenter size-large wp-image-5345" src="https://i1.wp.com/www.altnews.in/wp-content/uploads/2017/09/Selection_06_09_2017_009.jpg?resize=640%2C676" sizes="(max-width: 640px) 100vw, 640px" srcset="https://i1.wp.com/www.altnews.in/wp-content/uploads/2017/09/Selection_06_09_2017_009.jpg?resize=970%2C1024&amp;ssl=1 970w, https://i1.wp.com/www.altnews.in/wp-content/uploads/2017/09/Selection_06_09_2017_009.jpg?resize=284%2C300&amp;ssl=1 284w, https://i1.wp.com/www.altnews.in/wp-content/uploads/2017/09/Selection_06_09_2017_009.jpg?resize=768%2C810&amp;ssl=1 768w, https://i1.wp.com/www.altnews.in/wp-content/uploads/2017/09/Selection_06_09_2017_009.jpg?resize=696%2C734&amp;ssl=1 696w, https://i1.wp.com/www.altnews.in/wp-content/uploads/2017/09/Selection_06_09_2017_009.jpg?resize=1068%2C1127&amp;ssl=1 1068w, https://i1.wp.com/www.altnews.in/wp-content/uploads/2017/09/Selection_06_09_2017_009.jpg?resize=398%2C420&amp;ssl=1 398w, https://i1.wp.com/www.altnews.in/wp-content/uploads/2017/09/Selection_06_09_2017_009.jpg?w=1196&amp;ssl=1 1196w" alt="Oxomiya Jiyori God had different plan for group of bharat tere tukde honge" width="640" height="675" /></p> <div class="lastpara"></div> <p>The list is endless as who all among the right-wing thought that the horrifying murder of a person is the right time to spew all the venom they had against them because of ideological differences. There are many more on Facebook and Twitter who said things that are so hateful that it cannot be reproduced here. Is this the same hateful ideology that consumed Gauri Lankesh?</p> <p>https://www.altnews.in/right-wing-spews-venom-social-media-senior-journalist-gauri-lankesh-shot-dead/</p> <p>&nbsp;</p> </div> </div> </div> </div> " data-medium-file="" data-large-file="" class="alignright size-medium wp-image-77891" src="https://prod01-cdn07.cdn.firstlook.org/wp-uploads/sites/1/2016/08/UniqueFaces-e1470257178350-3-300x195.jpg" alt="UniqueFaces-e1470257178350-3" />

Photo: Alex Emmons

Instead, “Realtime Crowd Insights” is an example of facial characterization technology — where computers analyze faces without necessarily identifying them. Facial characterization has many positive applications — it has been tested in the classroom, as a tool for spotting struggling students, and Microsoft has boasted that the tool will even help blind people read the faces around them.But facial characterization can also be used to assemble and store large profiles of information on individuals, even anonymously.

Microsoft has traditionally adopted an “opt in” policy with facial recognition, requiring users’ consent before Microsoft can store an image of their face. The Kinetic Sensor on an Xbox, for example, allows users to sign in through facial recognition technology — but requires users to first give consent, according to Microsoft’s privacy policy.

Microsoft has a similar code of conduct for APIs, which requires developers to “obtain the consent of the people whose data (such as images, voices, video or text) are being processed by your app.”

Alvaro Bedoya, a professor at Georgetown Law School and expert on privacy and facial recognition, has hailed that code of conduct as evidence that Microsoft is trying to do the right thing. But he pointed out that it leaves a number of questions unanswered — as illustrated in Cleveland and Philadelphia.

“It’s interesting that the app being shown at the convention ‘remembered’ the faces of the people who walked by. That would seem to suggest that their faces were being stored and processed without the consent that Microsoft’s policy requires,” Bedoya said. “You have to wonder: What happened to the face templates of the people who walked by that booth? Were they deleted? Or are they still in the system?”

Microsoft officials declined to comment on exactly what information is collected on each face and what data is retained or stored, instead referring me to their privacy policy, which does not address the question.

Bedoya also pointed out that Microsoft’s marketing did not seem to match the consent policy. “It’s difficult to envision how companies will obtain consent from people in large crowds or rallies.”https://theintercept.com/2016/08/04/microsoft-pitches-technology-that-can-read-facial-expressions-at-political-rallies/