Photo-real imagery cinema by Eymerce - one of the companies breaking new ground in movie-making. Photo / Supplied
Photo-real imagery cinema by Eymerce - one of the companies breaking new ground in movie-making. Photo / Supplied
Virtual reality and augmented reality, predicted to be part of a $200 billion industry by 2020, are set to transform our worlds. How? Jamie Morton explains.
The Rat Pack is headlining at the Sands Hotel.
Metres away from your stage-side seat at the Copa Room, Dean Martin, the king of cool himself, is bouncing around the house band, crooning a boozy, wisecrack-heavy take on When You're Smiling.
You can almost smell the smoke from the cigarette stuck between his ringed fingers.
The shine of his tuxedo sparkles in the spotlight and it feels every bit of Vegas in 1963.
A clown fish glides past the right side of your goggles and you reach out to touch it.
Or maybe you're behind the wheel of a Porsche 911 Carrera, trying to stay on the road at 200km/h while your competitors attempt to ram you into a fireball.
It's so close to the real thing that it's terrifying.
This isn't virtual reality, says Richard Taylor, but an entirely new entertainment format that doesn't require the viewer to wear a clunky head-set.
Instead, the viewer is surrounded by photo-real imagery wherever they look.
Each person wears active or polarised glasses and they are seeing imagery created by a high-resolution LED surface: There is no projection.
This is Eymerce, a Los Angeles-based company Taylor (not Weta Digital's Richard Taylor) says will be able to transport its audience to any environment imaginable.
It could be totally computer-generated, or a combination of location photography and any of the special effects studios use to make imagery for today's feature films.
Think photo-real human beings, at real human scale, being animated before you in real time.
How else does it differ from the 3D movies we can catch at IMAX today?
Well, Taylor says, the viewer doesn't just have to look straight ahead at the screen to experience 3D: they can look left, right, up, down.
Don't just think movie theatres.
The initial prototype Eymerce is developing, dubbed the Stadeon, is a four-metre high cylinder fitting an audience of between one and 20 that could be built into theme parks, cruise ships, museums and most anywhere else.
He's the co-founder of Fisherman Labs, another California-based tech company that's pushing virtual reality (VR) and augmented reality (AR) to new extremes.
Its innovations place over the viewer's eyes lenses that distort the imagery in a way that makes the user perceive visuals in way that almost envelops their peripheral vision.
"Combined with head tracking software, the user is able to look around in a fully simulated environment," Hu tells the Weekend Herald.
"Because our visual senses affect the body in so many ways, VR is able to induce physical reactions such as nausea, heart rate variation, even pain relief."
Hence, it could be as much a part of healthcare as entertainment.
Barry Sandrew, who founded Legend3D, a San Diego-headquartered company whose handywork can be seen in The Lego Movie, The Life of Pi and The Amazing Spider-Man, is a self-proclaimed "evangelist" for AR and another emerging format, mixed reality, or MR.
He sees a future where 3D sensors will allow anyone with a mobile phone to experience realistic AR and MR imagery overlaid onto their real world.
"One day when AR glasses become a reality, I believe people will be able to walk through the bio-florescent forest of Pandora as we remember it from Avatar together with their friends."
Only, their friends won't be CGI avatars, but themselves; something impossible with the isolation imposed by VR head-mounted displays.
"These immersive visuals, overlaid onto real world venues, could then potentially become a platform for cinema-like directed story-telling."
Predicted to be a $200 billion industry by 2020, the worlds of AR and VR are set to bring science-fiction-like situations into our everyday lives.
But, despite an explosion of superhero movies built to give audiences that rollercoaster-like sense of cinematic immersion, its reception in Hollywood has so far been mixed.
At a time where VR is still in its infancy, and there hasn't yet been a huge return on investment for VR content, some studios have embraced the technology while others remain extremely cautious of it.
Even in the longer term, Taylor, Hu and Sandrew (all in New Zealand next week for a two-day VR and AR expo at SkyCity, staged as part of Techweek 2017) doubt that true VR and AR will come to dominate the multiplexes we've long flocked to.
Rather, the next generation of viewers will be content with the small screen but high-tech offerings of ultra high definition and high dynamic range on TV, or streaming blockbusters via their smartphones.
While we'll see VR and AR in a new breed of fully-immersive, perhaps dome-shaped cinemas, the fare will be short films tailor-made for the industry, rather than event films.
After all, who would want to endure a 180-minute assault on the senses?
"VR can be disorientating and uncomfortable on the eyes but prolonged use can be downright nauseating," says Kiwi tech commentator Peter Griffin, who has already caught several VR movies and TV shows.
"It also has noticeably degraded image quality compared to the ultra high definition movie images we have become accustomed to and the headsets get uncomfortable before long."
Where VR and AR innovations eventually take us will depend on how on the development of what it will be used for, as well as the tech itself.
Associate Professor Leon Gurevitch, deputy head of Victoria University's School of Design, sees VR offering handy applications for platforms such as Skype; gaming or other immersive experiences usually reserved for individuals; or even an emerging VR film scene.
But, when it improves enough to hit the mainstream, Gurevitch says it's AR that will make the biggest waves.
Like traditional cinema itself, AR allowed for the consumption of virtual content at the same time as it allowed the user to be present and interactive in the real world.
"The importance of this can not be overstated," Gurevitch says.
Sharing the experience
More than a century ago, stereographic photographs were one of the most popular forms of photographic imaging, but the medium died because, unlike early cinema, it limited viewers looking at those images one at a time, and in isolation.
"Essentially we are social animals and wherever the content suits we like to appreciate media together, or at least have that option."
While we relish shutting ourselves off from the world to binge-watch an entire Netflix series in a weekend, it's those nights out with mates where we'll try out an AR experience.
Gurevitch doesn't just mean the pictures, but attractions like public artwork designed for AR lenses, or social games.
We've already seen the start of it with groups of Pokemon hunters following their smartphones around Auckland's Albert Park after dark.
Or it could just be used as a handy tool.
Imagine you've become separated from your friends at a music festival: Just hold up your phone to see a virtual pointer hovering over your friends with a numeral indicating their distance from you.
"Similarly if you are in a place where you know no one but you might want your AR system to identify friends of Facebook friends in a street: It is much easier to ask for help from a stranger who is a friend of only two degrees separation, as every Kiwi knows."
It's logical to assume it will rise from traditional technology like phones and computers, at the same time VR- and AR-specific devices begin to appear on the market.
But we shouldn't think we'll see an ultimate replacement of old media, says Dr Stefan Marks, a lecturer at AUT's Colab.
"We will merely get used to seeing more content in 360-degree panoramas instead of just photos, and content being 3D instead of 2D.
"And time will tell how many of today's products, technologies and ideas are over-hyped and how many of them are here to stay."
It's an all-too-familiar story with innovation: The tech we create can't immediately provide the giddy possibilities that we first predict for it.
VR itself was being talked up as the next big thing as far back as 30 years ago.
"The real question is when will it have its Apple iPhone moment that makes it an indispensable technology?
Schoolchildren watch a hologram that represents the earth during the opening session of the Climate Change Observatory called 'Mission earth' at the National History Museum in Mexico City. Photo / EPA
"Just now it still feels like a solution looking for a problem.
"But we had many years of supposedly 'internet connected' WAP phones looking for a market before Apple brought out its iPhone and finally capitalised."
Sandrew points out that it was Steve Jobs himself who remarked companies had to start with customer experience and then work backwards to the technology.
"There is a population of professionals in Hollywood who have been existing in a bubble when it comes to VR, feeding off each other's hype," he says.
"They've been attempting to fit a square VR technology peg into a round storytelling hole."
In the short-term, he says, look out for AR and MR in the home market for storytelling and gaming, or at theme parks and tourist attractions.
"Predicting that VR and AR have a bright future is a very different matter from predicting when and how that future will eventuate," Gurevitch adds.
"I would be surprised if we don't see some exciting breakthroughs in the next decade given the content and technology production curve we now appear to be on."
What is an immersive experience?
The term immersion gets chucked around a lot by tech geeks but what does it actually mean?
Dr Taehyun Rhee, founder and director of Victoria University's Victoria Computer Graphics Lab, says immersion can be defined as a "feeling of presence" in a virtual reality.
In a cinematic setup, if the viewer can fully engage themselves into the telling of a story, we may say that they're fully immersed.
Yet, interestingly, the real measure of success isn't just in the technology, but the story-telling itself.
"If we assume a movie has nice story, then recent immersive technologies may provide an ideal environment for maximising the immersion to heighten the engagement."
We see immersive displays in cinemas with wide fields of view, IMAX's Screen X technology being one example, or in those with stereoscopic or volumetric displays, such as 3D movie theatres, offering a better sense of depth perception.
More recently, the next generation of immersive cinema has arrived in the form of 4D theatres, which have now been installed in several theme parks.
But Rhee said all of these formats were still passive and didn't let us interact with the experience.
His research looks at whether 360-degree video and interaction can be combined to create an illusion for users to engage with 3D virtual objects seamlessly composited into a live background.
It would require lighting and composition steps typically found in the post-processing of visual effects films to instead be performed in real time.
This is called MR360: a mixed reality concept that's so far been incorporated into a commercial game engine and which was demonstrated in a live demonstration at Te Papa in March.
The latest version will be showcased at the Magnify Summit Expo, kicking off in Auckland tomorrow, and the Future Reality Conference in Wellington next week.
Rhee is also part of a collaboration between Kiwi and Korean researchers that would bring a combination of MR360 and 4D theatre to the living room.
