Tribeca Film Festival had an immersive theatre as part of the experience this year. It was heavily focused on VR and 360 degree video. The NYC Layerframe team went there to fill our buckets and see what’s going on with the future of tech, in order to bring some new ideas to our clients and creative processes. For this post I wanted to give some thoughts around the experiences and the festival itself, with an emphasis on what made each piece unique and interesting.
To start, the process of getting into the festival was both confusing and frustrating for a first timer, but I won’t spend a lot of time talking about it other than to say that there’s a lot of people, and not a lot of exhibits, and the process was not explained well. Some are single person experiences that last 7 minutes, some are 4 person experiences that last 5. You can do the math, but with only 3 hours to see as much as you can, you have to be ultra selective otherwise individual exhibit lists fill up and you won’t get to see them. Out of about 30 experiences, I only got to see 7, three of which you didn’t need to sign up for, which left me feeling like I missed out on some good ones.
Of the different pieces, there were a handful that told a story much like you’d tell a documentary, except you could look around. I didn’t find this use of VR impactful or innovative – people were using a new medium to tell stories in an old way. These included The Protectors, a short doc about people who hunt Elephant poachers. It wasn’t as immersive as I had hoped, but it did have a few cool moments where I was walking through tall grass and it felt like I was in the forests. Extravaganza is a short animation where you’re inside a music box as someone looks into it. It was supposed to be funny, but it fell flat all around. The Possible: Hoverboard had a ton of potential, but instead of using a 3D camera to let us ‘fly’ they used a 3D camera to watch someone else flying.
The other four were great for different reasons. The one that I purchased the festival tickets for to begin with was Hallelujah. There’s a ton of new technology behind it from Lytro and that links to a cool behind the scenes, and the video there will explain it much better than I can.
The gist is that they use an array of state of the art cameras to give you a more immersive experience. With traditional 360 cameras, you have a fixed viewing point. So wherever the original camera shot the video from will be where the viewer sees it from, even if the viewer moves closer or further away from what they’re looking at. With this camera, you can move (in a limited amount) closer or further from a subject, or left and right to see them from different angles. It’s a significant achievement.
As a tech demo and an experience, it was fantastic. They had one vocalist singing all the vocals, harmonies, sound effects and flourishes. They had him sing each part separately. They then took the 5 recordings and had him singing in harmony with himself, with the viewer at the center, so you can look around and see each ‘clone’ as it sings along. You can move around and it really feels like you’re inside that space with him. Eventually you end up in a church with a choir, which was more immersive but less interactive than the first section.
My gripe, which repeated throughout all my experiences, is that the festival itself wasn’t a good place to demo this tech, and the equipment used didn’t feel as state of the art as I was expecting for a premium venue. Many of the headsets were Samsung VR headsets and the resolution was low, the goggles were dirty (like looking through smudged glasses), and everything was hurried, so there wasn’t a lot of time to fine-tune each individual’s experience. Often the people working the booth were not people who created the experience, so their attention to detail and overall enthusiasm wasn’t there.
In the case of Hallelujah, we were using Lytro VR goggles and what I assume were good headphones. Unfortunately I was getting flares on the edges of the viewing area in the headset, which I couldn’t tell if it was because they were dirty or if it was part of the experience. For the first half of the experience the vocalist kept disappearing, and I thought it was part of the experience. That’s because I wasn’t informed that if I moved too far from the center of the viewing area, the screen would black out to let me know I needed to move back into the center. The audio was good, but my background isn’t in audio, so I’m not the best judge of quality there.
The next experience I did was Under a Cracked Sky. It’s basically a documentary about diving and exploring the sea underneath massive sheets of ice. While it suffered from a low res 360 degree camera and Samsung headset, I thought it was one of the best uses of VR in the whole festival.
We were taken under water, in an area that the majority of us will never visit, and got to dive and see a world that only a handful of people on Earth have seen. Not only was it a compelling story, it also let me experience a place I will never get any other change to experience. And while that may sound trivial, the experience of VR was so compelling in making it feel real.
At the end of this experience there was a short section where we’re looking out into the darkness of the ocean, and it felt so believable, and so vast, in a way that a regular TV just can’t do justice.
I compare it to the rim of the grand canyon. I always say that the canyon is so incredibly big that my brain just can’t grasp it from the rim. It’s just a big hole in the ground and the scale is lost on me. But once I hiked down in, and looked up, I realized just how absolutely massive the canyon is, and how infinitesimally small I am. That’s the experience you get with VR.
I hope they’re using this as a test piece to see if they can get more funding, and I hope they bring down a better quality camera and get some UHD footage, because I think an experience like that would be unlike any other people have had.
An experience that I didn’t expect to attend was To Be With Hamlet, which is a virtual world with a live performance inside. The actors were remote, performing from Brooklyn, but being motion captured and put inside the the VR experience. There were four of us experiencing the show simultaneously, and we were put inside a man made game world.
While the theatrical performance was a cool idea, what really hit home for me was exploring the world I was in. I could aim a joystick and click a button to teleport to that location, then physically move left and right and look over edges and around corners. I obviously tried to break the system, and had more fun than I should have trying to do it. It was, without a doubt, the most engaged and childish I felt all day, learning how the world worked and trying to find ways to get places I shouldn’t. The creators would do well to incorporate some Easter Eggs into the next version of that so people can really engage.
Overall, Hamlet had the coolest exploration possibilities, but the least engaging narrative. I can see how this could redefine MMOs and adventure games. While it was touted as a performance theatre piece, it felt more like a video game, and really blew my mind in that regard.
Treehugger: Wawona has been getting rave reviews around the internet, and for good reason. This was the experience that VR was created for; it was interactive, experiential, immersive, had a few glitches, but overall was the most unique and best use of VR at the show.
The experience and technology were top of the line. There was a large physical object in the center of the exhibit, shaped like a tree with 4 holes cut out, one for each participant. Each participant was wearing a full VR suit; haptic feedback gloves, haptic feedback backpack, VR goggles, UHQ headphones. The goggles even had the smell of sycamore tree (or jalapeno, I couldn’t decide) coming out of them, just in case the smell of NYC was too overwhelming to transport you away to the experience you were about to enjoy.
This experience was hard to explain (but this video is from the experience). It started really simply – you are standing in front of a great tree that has been built out of a point cloud. You can move around, within about a 5’ x 5’ square, and either interact with the tree, or the forest around you. As the experience progresses, you see nutrients, as particles, moving into the great tree and spreading out. Eventually it becomes less grounded, literally, as you start flying up and toward the top of the tree, which is endless. It turns quite whimsical and abstract, and really makes an effort to make it feel like you’ve become part of the nutrient stream.
But what makes it stand out from the rest of the shows? Well once again, it was exploring, interaction and immersion. You could wave your hand through the streams of particles and move them around, like a conductor in an orchestra. They would change positions, speed and colors to match your movement. You can walk around and decide if you want to interact with this giant tree, or maybe explore the branches around you. Or you can simple stand at the tree and dip your face inside the hole in the side, and just watch. See how the forest moves, and interacts with the tree.
It also didn’t suffer from any technical limitations of trying to use real life footage and stitching it for VR. It was completely built for VR – they used technology to collect point cloud data and rebuilt it for this exact purpose. They had full control. Marshmallow Laser Feast, besides being the best named company ever, was the team that built the experience, and also had previous VR work. Their original experience was having people move through the forest from the point of view of a series of animals. Their previous experience clearly bled through, making this top notch.
In the end, being in worlds I don’t normally live in is what made for amazing experiences. I think the act of exploring is what makes VR a worthwhile technology, and worthy of investing in. In some cases the creators were telling traditional stories, but it didn’t feel like a good use of new tech, and certainly doesn’t sell it. I look forward to seeing what’s created in the same vein as Treehugger and Hamlet next year.