Is immersive experience becoming reality?

Mathys Beny11 mai 2020English

« I felt like I was really there… »

An immersive experience is often summed up with these seven little words since it can be defined as an illusory environment that completely surrounds you such that you feel that you are inside it and part of it. The objective is to put the subject in a state where he will forget his physical presence and be completely focused on what is happening around him.

Virtual reality, augmented reality… are often the first ideas that come to mind when we think about immersive experience. Indeed, it is true that these immersive technologies are more and more over-represented in our daily lives. An umbrella term has even been introduced to refer to them as a whole: extended reality (XR). By 2022, the XR market is expected to reach $209 billion, which is eight times what it is today, according to studies conducted by Raconteur a publisher providing content for business leaders via in-depth analysis. Recent research revealed that more than 60% of respondents believed XR will be mainstream in the next five years. [1]

First things first, what are the different types of immersive experiences related to new technologies?

Let’s try to deepen our understanding of this topic that is changing the way we behave as a society.

360-degree content. An interactive photo or video recording shot in all directions at the same time. It allows the user to see from all angles just by dragging or moving the phone or tablet without a headset. 360-degree videos are now increasingly popular especially on social media platforms, namely YouTube and Facebook. Brands are turning to 360° video as an essential part of their digital marketing strategy and it is not hard to see why. According to a report by Business Insider [2], the Hong Kong airline’s 360° advertising, launched in 2016, was 35 times more effective than the traditional 2d version of advertising.

Digital Twins. Digital Twins are virtual replicas of physical devices that scientists and computer professionals can use to perform simulations before real devices are built and deployed. The term was originally coined by Dr. Michael Grieves in 2002 [3]. NASA was one of the first to use this technology for space exploration missions. They connect the real and virtual worlds by collecting real-time data from installed sensors. The data collected is either decentralized locally or stored centrally in a cloud. The data is then evaluated and stimulated in a virtual copy of the assets. After receiving the information from the stimulation, the parameters are applied to the virtual assets. By integrating the data into real and virtual representations, the performance of the real assets can be optimized. Digital Twins can be used in a variety of sectors: manufacturing, automotive, construction, utilities, healthcare. They are the next big Thing in Fourth Industrial Revolution for the development of new products and processes.

Virtual Reality (VR). It is the term used to describe a three-dimensional, computer-generated environment that can be explored and with which a person can interact. That person is part of that virtual world or is immersed in that environment and, while there, is able to manipulate objects or perform a series of actions. VR tries to stimulate as many senses as possible to make the user feel as if they are really there.

Augmented Reality (AR). It improves the user’s perception of the real world. It does this by adding a layer of computer-simulated information. The most important advantage of AR is that the user perceives it as a natural extension. AR simply adds or hides data to the environment. In contrast, VR completely replaces the user’s perception of the real world. Augmented reality is used to enhance everyday experiences, manipulating digital content to transform physical space. Many industries are harnessing the power of AR to create better learning environments in educational settings or customer experiences for retail shopping and other business purposes.

Mixed reality (MR). It attempts to bring VR and AR together in an improved version of AR. Sometimes it is even called Augmented Reality 2.0 because MR aims to achieve better immersion than AR. It integrates entire virtual objects into the real world. They react to changes in the environment and user interaction in real time. Mixed reality is therefore the result of the fusion of the physical world with the digital world. MR is the next evolution in the interaction between humans, computers and the environment and unlocks possibilities that until now have been limited to our imagination. It is made possible by advances in computer vision, graphics processing power, display technology and input systems. The term mixed reality was originally introduced in a 1994 article by Paul Milgram and Fumio Kishino, « A Taxonomy of Mixed Reality Visual Displays » [4]. Their paper introduced the concept of a continuum of virtuality and focused on how the categorization of the taxonomy applied to displays. Since then, the application of mixed reality has gone beyond displays. It also includes environmental input, spatial sound and location.

Image: the interactions between computers, humans and environments. **[5]**

Image: the mixed reality spectrum. **[5]**

Concrete benefits for our society…

As you have understood, there are many possible extended reality applications for businesses, such as in the retail sector. It gives customers the opportunity to try before they buy. The watch manufacturer Rolex offers an AR application that allows you to try watches on your wrist. The furniture company IKEA offers its customers the ability to place furniture in their homes via their smartphone. In training, the XR can provide hyperrealistic training tools that will help soldiers, medical professionals, pilots/astronauts, chemists and others find solutions to problems or learn how to react to dangerous circumstances without putting their lives or anyone else’s life at risk. Finding buyers or tenants might also be easier if individuals could « walk around » the premises to decide whether they want to do so, even if they are in another location. This type of immersive experience is essential for research, as demonstrated by the experience on customer behavior lived by Daphné Doubleau, a student at Grenoble École de Management. She was able to contribute perfectly to the realization of this study whose goal was to improve marketing decision making thanks to an atmosphere ever closer to reality. « I knew that the situations were fake as well as the decors but it was well recreated and I really had the impression to be in the store ».

Extended reality lets us learn, feel, remember and process new ideas in a more experiential and deeper way than ever. It could even bring us from the Information Age to the Knowledge Age [6] thanks to its ability to simulate experiences. In the endless sea of digital information that defines the 21st century, the world’s next step will be driven by artificial intelligence which will process information for humans and turn it into knowledge, and in some cases, decisions. A white paper on Immersive Experiences in Education – New Places and Spaces for Learning [7] published by Alice Bonasio in collaboration with Microsoft Education at the 2019 International Society for Technology in Education (ISTE) demonstrates the possibilities that the mixed reality spectrum can bring to education and by extension to the democratization of knowledge. Humans are essentially programmed to learn best through experience, and that is precisely what immersive technologies enable. The main advantage of mixed reality is that it seems to significantly increase the transfer rate, i.e. our ability to apply and adapt learning to different real-world situations. MR facilitates transfer by simulating realistic scenarios in specific teaching competitions. It can literally bring a curriculum to life for students. This means that we can not only create more effective learning experiences, but also expand the range of experiences available, since we can simulate scenarios that would be too difficult, too costly or too dangerous for students to experience in real life.

MR can also help reduce cognitive load by allowing learners to visualize, manipulate and interact directly with complex structures autonomously, so that the brain is not so much busy trying to visualize data, but rather understanding, assimilating, recalling and applying it. Therefore, test scores for students using immersive technologies can improve by 22% especially for learners with dyslexia, autism and ADHD, as this technology is particularly useful for improving the learning conditions of students with disorders.

The extended reality is a new space that enjoys a strong reputation. Indeed, nearly one French person out of two has already experimented with virtual reality. It’s a new form of creation, one that particularly appeals to young people. According to Frédérique Bredin, ex-President of the CNC (Centre national du cinéma et de l’image animée), this is a real sector of the future [8]. Immersive experiences are a new way of approaching culture and leisure, as demonstrated by the Louvre in Paris or the MoMa in New York, which are among the museums that offer virtual tours of their collections, without moving from one’s couch, in the midst of the confinement caused by the pandemic. This notoriety even goes as far as the creation of the disputed myth of the « empathy machine ». This term coined by Roger Ebert to describe the medium of film, has emerged as a way to describe virtual reality after filmmaker Chris Milk’s presentation on TED 2015 [9]. This model of virtual reality as an empathy machine has often been criticized. For example, a 2014 study that asked participants to simulate blindness showed that participants tended to imagine what it would be like to suddenly become blind, rather than what it would be like to live as a blind person after a period of adjustment. But Milk justifies this myth with the screening of a VR documentary film about a Syrian refugee camp to participants at the World Economic Forum in Davos, Switzerland.

« These are people who might not otherwise be sitting in a tent in a refugee camp in Jordan, » says Milk, referring to the list of world and political elite at the summit. « But in January, one afternoon in Switzerland, they all suddenly found themselves there ». Thus, in the manner of Barry, this « virtual human » proposed by Talespin [10] to help teach people « soft » management skills, XR would be a way to understand the other more through immersion outside the comfort zone.

…but a use that remains risky in case of excess

However, a report published by Accenture notes that a growing number of companies are taking advantage of the « extended reality ». Entitled « Waking Up to a New Reality »: Building a Responsible Future for Immersive Technologies [11], the report stresses the need to be aware of the opportunities these technologies offer, but also the dangers they can pose. The authors argue that in addition to the dangers to privacy, extended reality also presents mental health risks that are linked to the damage caused by frequent and prolonged exposure to virtual environments. The merging of social networks and extended reality promises to provide us with more immersive experiences, with the ability to share our lives increasingly online. But it could also worsen the situation for people who use virtual space as a refuge from reality and increase cyberaddiction.

Then, let’s live immersive experiences in a different way

If it remains key in the development of immersive experiences, immersive technologies are finally not a prerequisite to immersion for Yann Garreau and Charlotte-Amélie Veaux who have achieved « a world tour of immersive experiences »[12]. So, in order to set out their research, they established a clear definition of the sector. To be successful, an immersive experience must: 1) bring the visitor into a creation, 2) blur the boundaries between the real and the imaginary, 3) influence his feelings, 4) motivate new behaviors. And the most surprising thing? There’s no need for tech in there. For Charlotte-Amélie, « Experiences that rely solely on technology are often anecdotal. Moreover, the experiments that work best are those where there is no revolutionary technology ». « Immersion is experienced in one way or another in our daily lives. When you read a book and at some point, you can’t stop reading, you’re immersed in the story and you identify with the characters, you’re in a form of immersion where there is no technology. What we call immersive experience is what will try to go further in the immersion » adds Yann Garreau. One example is the Jewish Museum in Berlin, opened in memory of the victims of the Shoah. Designed by the architect Daniel Libeskind, the institution is characterized by its lightning-flash architecture and its particularly poignant journey. « There isn’t the slightest trace of technology, but the emotion is very strong, » continues Charlotte-Amélie. « After seeing the personal belongings of deportees, you enter several spaces called voids. It’s dark and cold, and you can only see a bit of sky. There is total silence, but the walls are resonating. It’s overwhelming ». At a time when we no longer sell a product but an experience, we understand the interest of marketers for immersion, but the technology is sometimes poorly integrated into the experience. On the contrary, narrative frames and atypical scenographies are enough to create striking atmospheres. This is the example of immersive theatre, as demonstrated by the immersive play Sleep no more, based on Macbeth’s work and premiered in 2011 in New York, which is winning over more and more brands for its ability to physically and emotionally engage the public.

The journey remains at the center of the immersive experience… So, let’s keep combining the two and immersion will be all the more real.

Sources

[1] https://www.raconteur.net/infographics/what-is-xr

[2] https://www.businessinsider.fr/us/snap-debuts-spectacles-in-europe-2017-6

[3]https://www.researchgate.net/publication/307509727_Origins_of_the_Digital_Twin_Concept

[4] A Taxonomy of Mixed Reality Visual Displays. – Paul Milgram and Fumio Kishino

[5] https://docs.microsoft.com/en-us/windows/mixed-reality/mixed-reality

[6] Will virtual and augmented reality move us into the knowledge age? – Zenka – TEDxJacksonHole

[7] Immersive Experiences in Education – New Places and Spaces for Learning published by Alice Bonasio in collaboration with Microsoft Education

https://edudownloads.azureedge.net/msdownloads/MicrosoftEducation_Immersive_Experiences_Education_2019.pdf

[8] https://www.cnc.fr/professionnels/actualites/le-cnc-devoile-une-etude-inedite-sur-la-realite-virtuelle-et-les-experiences-immersives_978284

[9] How virtual reality can create the ultimate empathy machine – Chris Milk – TED 2015.