The Empathy Engine: VR Documentary and Deep Connection

“It is a factory of mental illness,” says the voice of a young man weary beyond his years. We turn and scan the hard-white walls. We are in a cell at the Harmondsworth immigration detention centre in the United Kingdom. “They kill your hopes,” he says. We don’t doubt him. “They kill your dreams.” Then we’re underwater. We spin our heads, we can just make out the walls through the swirling murk and, above us, a ripple of light from a high window struggles to penetrate the now submerged room. Someone – possibly us – struggles for air. We feel the man’s despair, the injustice of arbitrary borders, the lottery of birth. Or do we?

We are ‘in’ the Virtual Reality (VR) doc Indefinite (2016) – an eight-minute exploration into indefinite detention within the UK immigration system – recently released by the New York Times Op-Docs. If the sequence shocks, it’s meant to.

“We’ve done everything we can with this rectangular box,”¹ says founder of the trailblazing virtual reality production company Within, and all-round darling of the VR documentary community, Chris Milk. The box he’s referring to isn’t a cell at Harmondsworth but the flat 16×9 frame which Milk describes as “over”. Milk – whose latest VR docs include Clouds Over Sidra (2015), a film that allows us to share the experience of a young Syrian refugee in a Jordanian camp and VR: Millions March (2014), shot among the protesters of a New York demonstration – believes VR “feels like truth – you feel present in the world you are inside and present with the people you are inside of it with”.² Milk believes the constant immersion in a virtual world will change our relationship with the characters in the stories. VR, he boldly claims, is “the ultimate empathy machine”. ³

Clouds Over Sidra (Chris Milk, 2015)

Generating empathy in their characters is surely every director’s goal, but can virtual reality generate a genuine, deeper connection between the subjects of films and an increasingly connected/disconnected audience, and is that a desirable goal anyway? Not everyone in the VR doc community eulogises like Milk. Many believe the current technology is in a transitional, Betamax, phase. “Most of what these adventurous folks (myself included) are producing is terrible, which is as it should be,”⁴ says Janet Murray, Professor of the Digital Media Graduate Program at Georgia Tech. “Expanding new formats is difficult work.”  Or as Werner Herzog puts it: “We have a technology but no clear idea how to fill it with content.”⁵

In May 2016, MIT’s Open Doc Lab conference Virtually There: Documentary Meets Virtual Reality brought together VRs brightest minds to tackle this very dilemma. “VR requires a stylistic grammar of its own, rather than simply repurposing storytelling techniques borrowed from older media… are we fated to pursue ever-more accurate illusions of the real, or can we use VR to see and understand the world in new and critical ways?” says MIT’s Principal Investigator William Uricchio. ⁶  Here lies the potential and ultimately the potential threat of VR. Yes it’s a geeky, slightly dysfunctional technology that doesn’t really resemble its advocates’ acclaim. But like the kids who played Atari Tennis and pictured FIFA17,⁷ Milk and others see where it could go.

Of course, creating increasingly convincing virtual worlds and blurring the line between experience and virtual experience raises extreme ethical questions. In a world of progressively alone together (as Sherry Turkle would describe us⁸), self-obsessing ‘individuals’, could Milk’s empathy machine become the device that makes us jettison empathy for good? Or worse, could it become the ultimate tool of manipulation? If you thought fake news was confusing, wait till you encounter fake experience.

So What Is It?

VR is an immersive medium entered by putting on a headset. Users have the opportunity to look in all directions and the latest technology enables them to move around in space – both virtually and literally – and trigger actions and interactions within the experience.

In 2016 venture capitalists ploughed $2 billion into VR. For 2020 the predicted investment figure is $30 billion.⁹ Old world capitalism senses profit in this new world machine, and to sell the hardware – for now that’s headsets – they need ‘content’. Of course, gamers have hit the ground running with VR but, in the storytelling realm, movie makers are struggling to work out how drama will work on the medium, so documentary is catching much of the money.¹⁰

The Electronic Visualization Lab at the University of Illinois and Monash University in Melbourne have both built room-sized cylinders of LCD panels¹¹ and experiments are being undertaken in air jet driven force feedback¹² creating the illusion of physicality, but for now, it’s all about the headsets. Of course, you need £300 to buy one, and a computer to plug it into… and the electricity supply to power that, so this technology is beyond most of the world’s population. Google have a £3 cardboard headset – but you still need a £300 smartphone to put in it.

360° Video

In its simplest, most prolific form VR is little more than 360° video. Shot on 360° cameras or arrays of GoPros, 360° docs place the viewer ‘in’ a filmed environment. Vice News creative director Spike Jonze and Chris Milk made VR: Millions March at a New York demonstration against police killings. The film places you in the demo. “You never get that close to someone yelling their heart out without consequences. You feel their humanity on a deeper emotional level. More than on television,” says Milk. “Here, you’re existing without ego, the close proximity with another person” making “you feel more empathy for them.”¹³ Maybe describing your own creations of transcending ego is a little, well, oxymoronic, but when it comes to VR Milk borders on the messianic. If you purchase Google’s cardboard headset and watch VR: Millions March you can decide for yourself. Personally, I was no more – or less – existing without ego in VR: Millions March than I would in the cinema. Just as in the darkened theatre each viewer brings his or her own experiences, ideas, emotions, current mood and mind set to every doc – virtual or otherwise. I am yet to meet a filmmaker who assumes they can control each viewers’ reaction to their story.

VR: Millions March (Spike Jonze and Chris Milk, 2015)

In truth VR: Millions March – like many current VR docs – is a fairly pedestrian 360° observational documentary, direct cinema in the round if you will. And, if you struggle with Vice’s presenters online, you’re unlikely to warm to their VR versions. The most resonant impression VR: Millions March left on me was the belief that American protestors could really use a couple of weeks street-tactic training from their Athenian counterparts. To be fair to Vice, their stated aim is to use the technology in real time to allow people to VR live stream a demonstration or other event. Less doc, more ‘being there’ for those who can’t experience it.

The New York Times obs-docs strand has produced a number of docs that bring us into the lives of artists and therapists¹⁴ and several films, including Milk’s Clouds Over Sidra, which could be perceived as being geared more to elicit pity than empathy. VR ethnographic documentary makers seem to struggle, just as much as their flat frame counterparts, to find open water between their own intentions and the intentions of the NGOs they are working with in the field.

360° video is strongest when it plays to its own simple strengths. The New York Times’s The Click Effect – following free divers as they swim surrounded by dolphins is a beautiful thing but becomes immediately less interesting as the documentary proper kicks in on the surface. The New York Times’s Indefinite enhances 360° video with CGI: characters materialise inside the jail, their eyes dehumanising bands of white noise. One inmate recounts another setting himself on fire as a man engulfed in flames appears on the landing. The lost hope of bureaucracy is well captured by mounds of paperwork growing to block a doorway. Do we empathise? Possibly. Surely? In the end, it is the testimony that gets you. “In prison you count your days down. In indefinite detention you count them up.” As in every medium the creators of the doc realise good storytelling is less about telling and more about stories.

3D Capture

The 3D capture crowd appear to be aiming higher – as far as exploiting VR’s potential goes. In effect, 3D environments are created as copies of actual environments using a variety of techniques including photogrammetry and videogrammetry – the recreation of space from pictures, CT scanning and LIDAR¹⁵ – the laser mapping of actual space. Ersin Han Ersin, Creative Director for the design firm Marshmallow Laser Feast (who mapped a British forest for 2015’s Into The Eye of An Animal) says he is investigating “beyond the limits of our senses,” to “make the invisible visible”.¹⁶

RecoVR Mosul has made the no-longer-visible visible by using crowd sourced photos by tourists to create a walk-through of the Mosul Museum, destroyed in recent years by ISIS. The result is a worthy and undoubtedly important, but essentially dull, experience – not unlike walking around an old museum of antiquities.

More intriguingly, war photojournalist turned MIT OpenDocLab fellow Karim Ben Khelifa has 3D mapped combatants from some of the planets most entrenched conflicts for his 2014 VR doc The Enemy. Ben Khelifa is trying to “make the audience think more deeply about war” and to “humanise the combatants” to us and to each other.¹⁷ The algorithms – “mimicking the rules of everyday physics”¹⁸ in the software enable headset wearers to walk between Palestinian and Israeli fighters who explain their motivations, histories and actions. At MIT, neuroscientists have been measuring the audiences’ physiological reaction to the experience and in time they hope to discover “what kind of empathy has been created”¹⁹ in a kind of reality-virtual-reality loop.

Sound (and CGI)

The most empathy inducing examples of VR documentary seem to be those that truly embrace the differences of experience the technology offers. Several of the better ones appear to be created through CGI, though this is not their defining feature. Their defining feature is their use of sound. Into every VR headset plugs a pair of headphones. And the best VR doc creators have realised that it is the sound space that both divorces their audience from their own world while immersing them in the world of the story.

Notes on Blindness (2016) – the VR spin off of the doc of the same name that charts writer John Hull’s experience of going blind in the 1980s through his cassette audio diaries – begins with a click of a cassette recorder and Hull’s announcing: “A note on the nature of acoustic space.” What follows is exactly that – a lesson in listening. Hull points out the ambient sounds – the breeze, the distant traffic. “I hear a voice: ‘Daddy,’ and my children are suddenly there.”

Notes On Blindness: Into Darkness (2016) image from http://www.notesonblindness.co.uk/vr/

The CGI is a murky black canvas from which ghosted figures loom as we hear their footsteps or voices. The sound design is brilliantly three-dimensional – the road is “off to the left”, the ducks “down by the pond”. I catch myself closing my eyes to hear better – probably not what the VR team were hoping. The VR experience is disorientating – like joining in a game you don’t know the rules of. “Where there is no activity, no sound – that part of the world dies,” says Hull. “It’s isolating, eerie.” The doc cuts to black. It is isolating and eerie…and, I believe, empathy inducing. But later when I show people the app I am still encouraging them to track down and watch the feature-doc – 90 minutes in a 16×9 frame.

“At least (with virtual reality) sound is being taken seriously again,” says Zillah Watson from BBC Research and Development.²⁰ Like Watson, many VR doc makers come from radio backgrounds. One former radio producer is The Guardian’s special projects coordinator Francesca Panetta who ‘constructed’ a 6x9ft composite of actual segregation cells for her 2016 doc 6×9 tackling the extremely isolating experience of solitary confinement.

“Welcome to your cell, you’re going to be here for 23 hours a day.” The concrete is convincingly hard. The voices – of former inmates and a psychologist – are layered. They talk to ‘you’ the viewer. Panetta says this is a trick she learnt making geolocated audio (where sound files would be triggered by location in a walkabout audio experience). “I asked everyone to talk to me in the second person. Imagine me in the cell. Give me advice. Tell me what’s going to happen to me. Should I feel scared now?”²¹ The effect is powerful – the testimony all the more intimate and convincing.

“Now that you’re here what are going to do?” a voice asks. Panetta says she recorded straight interviews but found the second person approach more compelling. “Actually you are much more interested in what is going to happen to you. You’re much less interested in Johnny’s story, Victor’s story.”

“Interactivity within VR is difficult,” says Panetta. “But the questions ‘give you ownership.’”²² And within this doc, you do have some agency. As your gaze falls on a different aspect of the cell it triggers a sound file. You look at the rudimentary sink, a female prisoner’s voice says: “Everyday take bath, they call it a bird bath.” On the bed is a letter. Look at it and a man begins reading a desperate missive to a lost lover.

As the experience develops “things start to slip”. Text appears on the walls – minor infractions that land a person in solitary such as “looking at an officer the wrong way”. The hallways are alive with banging and shouting – the sound design is taken from actual recordings Frontline News made at Maine State Penitentiary. Prisoners tell you about becoming disembodied and ‘we’ float to the ceiling. “The night is worse than the day,” says a voice. The lights go out. The experience is intense. Blurry memories skip ghostly across the wall, the lights flash. Huge hallucinated cracks appear in the walls. “You turn on yourself,” says a voice. Though I am aware I am sat in a room with a pair of cardboard goggles held to my face I still feel claustrophobic and disorientated. I know for sure I don’t ever want to be in solitary confinement. When the psychologist tells me how high the suicide rates in solitary are I already know that locking people in a box is a depraved practise. Is 6×9 an empathy machine? Maybe.

Not a Film, Not an Empathy Machine

Georgia Tech Associate Dean and Professor Janet Murray, another big hitter in the VR world, doesn’t buy into the empathy machine model. “I hate to say this to people with a background in film, but (VR) really is not a filmic medium,” she says. “It’s not a film and not an empathy machine.” She continues, “It is an interactive medium…we want it to respond to us.” ²³

MIT OpenDocLab Principal Investigator William Uricchio feels the same way.  “Just as film’s pioneers spent their first decade emulating theatre,” he says. “Many of today’s VR makers are doing their best to emulate the logic of film. And just as early filmmakers finally shattered the proscenium arch and evolved new vocabularies, we can expect VR’s makers to find robust, exploration-based alternatives to the still-dominant film paradigm.”²⁴

Hunger in LA (Nonny de la Peña, 2013)

Hunger in LA (2013) creator Nonny de la Peña’s first rule of VR design (the VR-tech crowd love a rule) states: “Begin by thinking of your body in the space. The focus of VR design is not the camera frame, but the embodied visitor.”²⁵ Oscar Raby, creator of VRTOV’s Assent – a photogrammetric VR doc about Pinochet’s butchery in Chile puts it this way: “The photorealistic approach to reality is the reality of the camera that captures light. We are capturing the reality of another machine.”²⁶ “Instead of overhyping the inherent empathy value²⁷ of VR documentary.,” says Murray. “We should look to the genuine promise of the medium in creating compassionate understanding, and build on those.”²⁸

“Interactive documentaries often depart from the conventionally linear storytelling structure that, historically, characterises documentary,” writes Kerric Harvey, Associate Professor of Media and Public Affairs at the George Washington University as far back as 2012. “Multiple storylines may be pursued simultaneously in ways which exceed the convention of the ‘sub-plot’, while the hyper-textuality of the Web can let interactive storytellers abandon the idea of a dramatic arc with a beginning, a middle, and an end,”²⁹ Long before the streets of Western cities became awash with zombie-like youths welded to smartphone screens, Harvey talks about “the walk in documentary” re-conceptualising Alternative Reality Games as a participatory narrative merging the doc on the screen with the audience’s real-life world.³⁰ What may seem like smashing up narrative structure for the sake of it, Harvey sees as a way to connect with “generations of global citizens…for whom their mobile phone is the digital equivalent of the Swiss Army knife.”

Uricchio – in what sounds like an invitation to the tech giants to embark on Orwellian mind control experiments – talks about dreams being the way forward with neuroscience providing the research “as media move from ‘in front of the eyes’ to ‘behind the eyes’”.³¹ “VR is not film,” he says. “But if it’s not film, what is it … besides a philosophical claim of some kind or an investment opportunity?”³²

Hunger in LA (Nonny de la Peña, 2013)

Alarm bells probably should be ringing regarding the billions being ploughed into VR. Chris Milk gushingly tells his TED audience how delegates at the World Economic Forum in Davos watched Clouds Over Sidra.³³. More worrying is the spontaneous applause this elicits. Milk takes his empathy machine to the power-crazed, empathy-free, sociopaths responsible for war, poverty and global kleptocracy and gets a cheer from a room full of affluent American tech heads. Meanwhile Syria burns. Nonny de la Peña (“the godmother of VR”³⁴) took her VR doc Hunger in LA – in which a man at a food bank falls into a coma – to Sundance 2012. “People said they felt so frustrated as they couldn’t help the guy,” she says. “They took that back into their lives.”³⁵ Exactly how the film execs in Utah took the experience back to their lives she doesn’t say. Maybe lots of them began campaigning for greater equity or volunteering in food banks.

Is an Empathy Machine a Desirable Thing Anyway?

Dan Archer, Empathetic Media founder and Columbia University Tow Centre of Digital Journalism Research Fellow believes it’s more effective to maintain emotional space between audience and subject. “Too much empathy can cause distress or even terror in users, leading them to distance themselves from both an experience and the people depicted in it.”³⁶

Sam Gregory, program director at WITNESS who train video activists to expose human rights abuse says VR has a great potential for activism but only if we shift our focus from empathy to solidarity and compassion. “By allowing users to interact in real time, co-presence could be an effective route to mobilisation.”³⁷

Like traditional documentaries VR docs dealing with marginalised communities risk presenting narratives muddied by voyeurism and the colonial gaze. The rhetoric around VR and ‘empathy’ has become a catch-all response from the VR community to criticisms of poverty/trauma tourism.³⁸

Cognitive scientist MIT Professor Fox Harrell says: “Critical thinking around these issues is what we should be going for, rather than the idea that we can actually walk in somebody’s shoes without the physical repercussions or violence of the real world.”³⁹ “What immerses people, so that they can get over, for instance, their PTSD is to make it not completely real,” reckons Janet Murray. “It has to be reassuringly unreal enough that they can surrender to it, and act within it.”⁴⁰

Is VR the ultimate Me-generation add-on – a tool or a toy that lets us wallow in others’ misfortune? The blurred distinctions allow us to become the victims – like billionaire pillars of the establishment become challengers of the establishment. The agency of the user trumps the agency of the documentary subjects. As Sam Gregory points out there is intrinsic danger when corporate interests like Disney sink $65 million into a technology and Google and YouTube line up for a piece of the pie.⁴¹

The Plastic Mind

Rus Gant director of Harvard University’s VR Lab talks about the need to develop “intuitions” about how we can be manipulated in VR—the same intuitions we have all developed for TV and film.

Keyser’s work on mirror neurons and “tactile empathy” should raise some alarms when viewed alongside VR technology.⁴² Keyser found that the same area of the somatosensory cortex was active when participants experienced touch or witnessed others being touched. As VR technology develops is there a danger we could confuse empathy with experience or mistake compassion for action or become unable to differentiate between someone else’s opinion and our own first-hand knowledge?

Assmann’s theory of cultural memory tells us films and documentaries hold a strong influence in terms of the creation of collective memory.⁴³ Michael Madary and Thomas Metzinger’s Real Virtuality: A Code of Ethical Conduct is a tentative effort to begin establishing ethical standards for the field. “VR technology,” they say, “will eventually change not only our general image of humanity but also our understanding of ‘conscious experience,’ ‘selfhood,’ ‘authenticity,’ or ‘realness.’” They warn that “the potential for the global control of experiential content introduces opportunities for new and especially powerful forms of both mental and behavioural manipulation, especially when commercial, political, religious, or governmental interests are behind the creation and maintenance of the virtual worlds.” ⁴⁴

Poster for Activision’s Call of Duty: Black Ops (2010)

Both jihadi groups and the US military have developed games and social network techniques to stimulate enlistment⁴⁵ (Call of Duty anyone?) and there is evidence that advertising tactics using VR can have a powerful unconscious influence on behaviour.⁴⁶ Madary and Metzinger point out we don’t yet know the long-term effects⁴⁷ citing O’Brolchain et al.’s 2016 work on The Effects of Long-Term Immersion to social media⁴⁸ – the “need” to interact becoming more ingrained when “embodied in virtual spaces”.⁴⁹ The Stanford Prison experiment,⁵⁰, Milgram’s work on obedience⁵¹ and Asch’s work on conformity⁵² – where people were shown to adopt opinions of the group even when they knew them to be wrong – show how susceptible to influence the human mind is.

In VR, Nick Yee and Jeremy Bailenson call this the Proteus Effect when subjects “conform to the behaviour that they believe others would expect them to have” based on the appearance of their avatar”.⁵³ And if we lost self-determination would we know? Daniel Wegner’s I Spy experiments found people thought themselves in control of a cursor on a computer screen when it was actually controlled by someone else.⁵⁴

We see the results of VR decision making in bombed Yemeni weddings and Afghan schools upon which ‘godlike’ drone operators have brought death from above. In the online virtual world of LambdaMOO, a character called “Mr.Bungle” used a “voodoo doll” program to force other characters to perform disturbing sexual acts.⁵⁵ At least one victim of the virtual rape reported suffering psychological trauma. Madry and Metzinger recommend non-maleficence in all VR research and experimentation, but fear the at-all-costs technophilia at the sharp edge of VR development is an obstacle to good practice. Cyberbalkanisation⁵⁶ – where social media users create echo chambers of ideas that only chime with their own – becomes an even scarier prospect in the virtual realm where misogyny, porn, hate, violence and racism thrive.

The VR documentarians like to call themselves ‘creators’ not ‘directors’⁵⁷ – as their interest is in building spaces not narratives. But is this an abdication of responsibility and of decision-making – like putting six versions of an edit in a film rather than selecting the one that tells the story best?

The most convincing VR documentary makers understand we all experience the world differently and don’t try and batter us with ‘being there’ or empathy machining. Like all great storytelling, the journey to realisation is a circuitous one and the best VR docs reflect that. “We provide no answers or explanations,” says Karim Ben Khelifa who made The Enemy. “We aim to provide an experience and stimulate discussion beyond easy rhetoric.”⁵⁸

Approaching the medium from a technological perspective – as opposed to an artistic or humanistic one – will not lead to good documentary or improved public discourse.

The first casualty of war may well be truth. But for those who benefit by perpetuating eternal war, truth becomes the ultimate target. The financial interests pushing VR development and corralling the world’s wealth, resources – and ultimately populations – are the same. If the creation of empathy, and by extension the formation of our own identities and ideologies becomes even more intrinsically linked with the media we consume, we are undoubtedly in trouble. Allowing financiers to drive the medium may well land us in a situation where doc makers bring to understanding what big pharma brings to healthcare. It is revealing that there is no mention of responsibility in Janet Murray’s rules for VR documentary⁵⁹ or Nonny de la Peña’s ideas or Chris Milk’s euphoric projections.

“Empathy,” wrote Moshin Hamid, “is about finding echoes of another person in yourself”.⁶⁰ If we want to genuinely understand the experiences of others we will need to work hard to break out of our echo chambers. Virtual Reality documentarians may be uniquely placed to help us do that.  Eye contact, body language, touch, smell, connection cannot be recreated to be experienced alone in a headset in a room. Or, more worryingly, perhaps likenesses of them can.

Endnotes