“Why do I look like Justin Timberlake?”
Facebook CEO Mark Zuckerberg was on stage wearing a virtual reality headset, feigning surprise at an expressive cartoon simulacrum that seemed to perfectly follow his every gesture.
The audience laughed. Zuckerberg was in the middle of what he described as the first live demo inside VR, manipulating his digital avatar to show off the new social features of the Rift headset from Facebook subsidiary Oculus. The venue was an Oculus developer conference convened earlier this fall in San Jose. Moments later, Zuckerberg and two Oculus employees were transported to his glass-enclosed office at Facebook, and then to his infamously sequestered home in Palo Alto. Using the Rift and its newly revealed Touch hand controllers, their avatars gestured and emoted in real time, waving to Zuckerberg’s Puli sheepdog, dynamically changing facial expressions to match their owner’s voice, and taking photos with a virtual selfie stick — to post on Facebook, of course.
The demo encapsulated Facebook’s utopian vision for social VR, first hinted at two years ago when the company acquired Oculus and its crowd-funded Rift headset for $2 billion. And just as in 2014, Zuckerberg confidently declared that VR would be “the next major computing platform,” changing the way we connect, work, and socialize.
“Avatars are going to form the foundation of your identity in VR,” said Oculus platform product manager Lauren Vegter after the demo. “This is the very first time that technology has made this level of presence possible.”
But as the tech industry continues to build VR’s social future, the very systems that enable immersive experiences are already establishing new forms of shockingly intimate surveillance. Once they are in place, researchers warn, the psychological aspects of digital embodiment — combined with the troves of data that consumer VR products can freely mine from our bodies, like head movements and facial expressions — will give corporations and governments unprecedented insight and power over our emotions and physical behavior.
Virtual reality as a medium is still in its infancy, but the kinds of behaviors it captures have long been a holy grail for marketers and data-monetizing companies like Facebook. Using cookies, beacons, and other ubiquitous tracking code, online advertisers already record the habits of web surfers using a wide range of metrics, from what sites they visit to how long they spend scrolling, highlighting, or hovering over certain parts of a page. Data behemoths like Google also scan emails and private chats for any information that might help “personalize” a user’s web experience — most importantly, by targeting the user with ads.
But those metrics are primitive compared to the rich portraits of physical user behavior that can be constructed using data harvested from immersive environments, using surveillance sensors and techniques that have already been controversially deployed in the real world.
“The information that current marketers can use in order to generate targeted advertising is limited to the input devices that we use: keyboard, mouse, touch screen,” says Michael Madary, a researcher at Johannes Gutenberg University who co-authored the first VR code of ethics with Thomas Metzinger earlier this year. “VR analytics offers a way to capture much more information about the interests and habits of users, information that may reveal a great deal more about what is going on in [their] minds.”
The value of collecting physiological and behavioral data is all too obvious for Silicon Valley firms like Facebook, whose data scientists in 2012 conducted an infamous study titled “Experimental evidence of massive-scale emotional contagion through social networks,” in which they secretly modified users’ news feeds to include positive or negative content and thus affected the emotional state of their posts. As one chief data scientist at an unnamed Silicon Valley company told Harvard business professor Shoshanna Zuboff: “The goal of everything we do is to change people’s actual behavior at scale. … We can capture their behaviors, identify good and bad behaviors, and develop ways to reward the good and punish the bad.”
Head movements are the most common metric being utilized by the handful of companies that form the still-nascent VR analytics industry. By recording a head-mounted display’s gyroscopic sensor data, commercial analytics software like SceneExplorer, created by a Vancouver-based company called CognitiveVR, is used to reconstruct “heat maps” of everywhere a user looks while in VR.
VR products like Oculus Rift are also well-positioned to be used for a burgeoning field known as emotion detection, particularly when paired with sensors used to map real-life body movements to a virtual world. Yotta Technologies, a VR company based in Baton Rouge, Louisiana, claims its platform can detect a user’s emotional state using an array of sensors mounted to a VR headset, reading microexpressions by tracking eye and muscle movements in the face. A benefit to end users is that such information can be used to give their VR avatars facial expressions that mirror their own. Speaking with Fusion, the company’s founder said that the company’s primary goal is to “unlock human emotion.” That goal is shared by Affectiva, an MIT spin-off company that offers “emotion detection as a service,” allowing clients to mine images and video feeds from webcams for emotional data revealing how peoples’ faces subtly react to certain cues.
Eventually, consumer VR systems will be able to capture a full range of human body motion, forming what Madary and Metzinger call a “kinematic fingerprint.” Much like the experimental gait recognition capabilities of some video camera-based monitoring systems, a kinematic fingerprint could be used to uniquely identify and analyze a person based on their body movements and posture, both inside and outside VR. VR sensors like the Oculus Touch and LeapMotion also show that hand tracking and gesture control are already becoming standard features in VR, allowing users to manipulate virtual objects and gesticulate to emphasize speech.
Advertisers are particularly excited about measuring and analyzing these movements. In a September 2016 industry report by the Interactive Advertising Bureau, the director of VR at one mobile ad company claims that the collection of this data is already “shedding light on user behavior and brand engagement for our advertisers,” and predicts that consumer VR will “bring unmatched value and measuring capabilities.” While VR adoption has been relatively slow due to prohibitive hardware costs and other factors, investment in the medium dramatically increased this year, suggesting ad companies are expecting to take advantage of those physical tracking capabilities in the near-future. - The Intercept
Add new comment