To create movie special effects in the past, editors would splice film frames together and run them at the same time. Today, virtual tools like VR and generative AI not only make illusions seem real — they put us in the middle of them too. On the Signal stage in 2019, Loren Hammonds, director of special programming for the Tribeca Film Festival, showed us how virtual tools, like virtual reality, were transforming narrative, giving viewers a chance to nurse a baby elephant to health or follow the pathway of Syrian migrants.

Today, AI is the darling of the media world, reshaping the entire creative experience from the way we tell stories to how we experience them. But even as storytelling tools change, “….all of this means nothing without compelling content,” said Hammonds.

As the curtain raises on the 21st Tribeca Festival, you can hear more from Hammonds or read our lightly-edited transcript below.

Transcript:

Loren Hammonds
My name is Loren Hammonds, and I am the Senior Programmer of Film and Immersive at Tribeca Film Festival in New York City. So we’re a festival that enjoys a wonderful partnership with P&G. And I’m so thrilled to be able to join you all today to talk a little bit about our festival, and also about the future of immersive storytelling. Before I really get into it too far, I just want to thank Mark Pritchard for the invitation to contribute to Signal this year. Also so many thanks to the amazing team here, including Stan Joosten, John Battelle, Stacy Foreman for all of the assistance on the road to today.

I think it’s often most effective to look back at our origins to explain how we’ve gotten to the current shape of our festival as it stands now. Tribeca Film Festival was founded in 2002, as a direct response to the events of 9/11. Our founders, Jane Rosenthal, and Robert De Niro had already been operating a successful arts organization and film center in Tribeca at the time. In the wake of that terrible day, they decided to organize an event that could bring some joy back to the city. They recognized the fact that art has a unique ability to heal the spirit. That’s how Tribeca Film Festival was born. In the years since we have evolved to become much more than just a film festival. We’ve let storytelling become the backbone of every decision that we make at Tribeca.

Great storytellers are often not bound by a particular medium. In recognizing that we’ve continually expanded to include new elements of the festival to reflect this. At this point, we’re currently celebrating not just film, but also television, new online work, musicians and multi-disciplinarians that use storytelling within their work, and the reason why I’m here today, immersive storytelling.

In 2013, we showcased our first section of interactive and immersive storytelling, which is known as Storyscapes. That year, we included our first official selection of VR, virtual reality that was made by the woman who you see here. Her name is Nonny de la Peńa, and Nonny is a pioneer in the field. She’s often called the godmother of VR. She’s an Annenberg Fellow and an immersive journalist. She created a brilliant piece called “Use of Force.” What she did in that piece was she used actual audio of a deadly confrontation at the US Mexico border, to reconstruct the events within a game engine, and then give you the opportunity to actually stand in the shoes of a bystander there. Now, mind you, this groundbreaking piece of work was made on a homemade headset, it was presented on a homemade headset.I think spit and bubblegum were involved somewhere there. There was no commercial hardware available at that time. This was the only VR project that we exhibited that year.

I’d like to take a second to contrast that with this year’s 2019 festival, in which we showcase 32 VR projects, some of which included live actors motion capture performers, volumetric filmmaking, and more. This doesn’t even mention the various mixed reality and augmented reality and AI-powered storytelling that we also included as part of the festival. Tribeca has evolved in a few short years to become one of the leading showcases of immersive work in the world. Following the opening of this year’s exhibition, Forbes even called us the world’s most important XR festival.

A lot of our success does have to do with consumer experience. VR is by nature, a fairly difficult medium to exhibit. So audiences have the opportunity to experience amazing, transformative work. But throughput is an issue in a medium where the majority of projects require a minimum of 10 square feet, per viewer. In New York City, that kind of square footage is not as easy to come by as we’d like. So at Tribeca what we do is we allow groups of about 100 audience members or consumers to come into our Virtual Arcade for each three-hour session. We encourage everyone to explore and experience as many of the various projects on display as they can during their given time slot. There are a lot of factors to take into consideration when programming and curating a festival like ours. We have to think about the number of projects that we can accept, the number of headsets that we can present each of the projects on, the length of the projects themselves, space requirements, and more.

With this inherent challenge, as we started showcasing VR all those years ago, we wondered very quickly, what can the rest of the audience do while they’re waiting to have these experiences? We didn’t want lines, because no one wants lines, right, because who wants to wait in a line. So we implemented a virtual queuing system in which consumers would simply sign up for an experience with their cell phone. So they sign up for each experience, and then they get a text back when it was their turn. This freed our audience to walk about, and really just explore some of the more visual treats that our exhibition could present. So the natural solution for us was to create an entirely immersive space for the exhibition, one in which audiences became totally immersed as soon as they enter the arcade. They have hints of storytelling before they even put on a headset. So you’re seeing some of these examples here on the slide behind me right now.

Once we hit upon this idea, creators more than rose to the occasion. We suddenly had theatrical set design, we had decompression rooms after experiences, and even more added to the space. It really took off for everyone involved, but especially it was beneficial for the audience. We take the responsibility of showcasing the medium of VR very seriously. Because if you deliver a bad or mediocre experience to someone trying VR, for the first time, it’s not just Tribeca that sunk, it’s the entire medium. Because it’s very hard to get someone to give VR a shot for a second time if the first time wasn’t up to par.

What we recognized also was that unfortunately, VR headsets haven’t been widely adopted for home usage. How many of you have a VR headset at home? This is pretty wide adoption. Yes, Signal. They haven’t been totally permeating the market. So it’s really important for festivals like ours, and other location-based experiences to play a crucial part in the ecosystem of VR as a whole. But of course, all of this means nothing without compelling content.

For me, it’s always inspirational to look back at a seminal year in VR and a seminal year for our festival. 2014, that was a year in which hardware manufacturers had started giving creators access to their development kits. It allowed for more experimentation and a real richness in the storytelling started to emerge. The implications of putting an audience within a story, as opposed to merely watching it, really hit home. That year, we showed a piece called “Clouds Over Sidra,” which these are some images from, and it’s made by digital pioneer Chris Milk, along with Gabo Arora, who was a documentarian, who was working at the UN at the time. The piece followed a young girl living in a refugee camp in Syria. It did what VR does best, it transported you to a place that most rarely get to visit, and put you face-to-face with someone that you rarely get to meet.

There’s a moment in that experience where a line of children walk past you, the viewer or the visitor, and they seemingly look you directly in the eye. In that moment, you can see their curiosity. You can also see their fear, and their hopes and their dreams, and most importantly, you feel their presence. That year at the festival, this experience was set up in our registration area. I stood by and watched as unsuspecting festival visitors, most of them having their first ever VR experience, you know, put the headset on with dry eyes, and almost without fail to get off with tears. The power of the medium was undeniable. That was the point where we really started focusing on expanding the festival further to really focus on virtual reality. That’s when we introduced the Virtual Arcade. To truly highlight just how much that experience meant to people beyond even the Tribeca footprint, that piece was also showcased that year at Davos, and another fundraiser. After seeing the piece, some of the decision makers there ended up pledging $3.8 billion in relief funds for Syrian refugees.

The same year, we hosted the world premiere of “Invasion!,” and in “Invasion!” the fluffy little bunny on screen here becomes an unsuspecting hero, as he helps us save the world from an alien invasion. This piece was made from Baobab Studios. Eric Darnell, who’s creator of the “Madagascar” franchise, and his CEO, Maureen Fan saw VR as an opportunity to truly change the connection with audiences through narrative storytelling in the medium. This piece was wildly successful, and it became one of the most ubiquitous projects across the board. It was one that many people, including myself, often used to show first-timers as a way of explaining presence in narrative.

Interactivity in the space was also key. As technology continued to leap forward, further interactivity also became possible in the work itself. So I’m lucky enough to have Chris Campkin, Vision3, to share an example of what I mean.

This is related to something that we showed last year. So in 2018, we featured a project called, “My Africa,” which is a beautifully beautifully filmed 360-degree experience, narrated by Academy Award winner Lupita Nyong’o, was created by Chris and his team at Vision3in the UK, in conjunction with the amazing organization Conservation International. The experience follows a teenage girl in Kenya, as she is working and going through her routine of living on land that is shared with an elephant preserve. It’s really moving, and it’s gorgeous to watch, and it’s interactive in its own right. But what Chris and Vision3 were able to do at the festival was to take it even further and really enhance the experience of our audience. Because in the 360 piece, you do meet a baby elephant named Dudu, and after that 360 experience, they invited people to come and enjoy a room scale experience in which you actually got a chance to interact with a virtual version of the baby elephant. So Chris, can you tell us a little bit about that?

Chris Campkin
I’ll talk you through this, and I’ll give you a bit of a virtual live guide of what the experience actually is. The reason we made this experience was because talking with our partners, Conservation International, they had sort of found that through watching 360 films that people that watched them were getting an increased level of funding coming through from them. They’re becoming more immersed in it and feeling like they were there and then wanting to donate money through it. We thought, “Okay, well, we’re making a 360 film for them, we thought, how can we take that a step further, really increase that level of immersion.” If we increase that level of interactivity, we will boost that immersion level there. I’ll then put the headset on and show you guys, if we can get up on the screen. Loren you’re gonna have to guide me if it isn’t on the screen.

What we have here is an actual scene called Reteti Elephant Sanctuary. One of the big things that we’re trying to do here was make sure that it was as realistic as possible, as the technology would allow. So every post that you see, the beams in the ceiling, and even down to the sand on the floor was exactly as it was when it was there.

Hammonds
You had to take thousands of photographs.

Campkin
We took about 4,000 photos of this to recreate it and stitch it back. But what it did was open up that 360 world, it allowed us to now step from this fixed point to be able to move around and walk around a bit. We actually showed this to the carpenter and made this building and he’d put it on, and he burst into tears, because he said, It just took him straight back, which is a really, nice thing.

The story goes where we have to actually look after Dudu. She has been rescued, but now she needs nursing back to health. So we’re gonna have to do things like wet the area down. One of the things we’re doing is we’re using real-world tract objects has another level of bringing that immersion to people. People kind of freak out when they reach out and grab a sponge.

Hammonds
It’s actually such a weird feel to be in that world.

Campkin
You’ve got a stethoscope here, which you guys see as a weird plastic object, which is the tracker. But in my world, I’m feeling I’m hanging on to this stethoscope. So we can check for heart rates and see how she’s doing. If you go over the stomach you can hear her stomach gurgling.

Hammonds
I don’t think we’re gonna have time to truly revive Dudu.

Campkin
I think I think you’re right. I think you have to take some time, and if you do take your time and look after Dudu, then you’re able to bring it back to life. You can even feed her with this plastic bottle. But maybe you guys can give her a bit more attention than I have and bring her back to life.

Hammonds
Thank you so much, Chris. Before we before we wrap up, I just want to let you know that this experience along with “My Africa” is available upstairs at the Signal Immersive Experience on the second floor. And we’re also showing what’s see Yeah, we’re also showing “Bonfire,” which is by Eric Darnell, which is who I mentioned before, and I’ll just leave you with this. At Tribeca we continue to strive for new and innovative ways to showcase the groundbreaking work of XR creators across the globe. This medium continues to evolve and in response, so will we. Thanks so much for listening.