The Evolving Immersive Experience (VR/AR/XR)

---

Editor’s note: This feature ran first in our June newsletter Xpresso. To gain access to our top content faster and stay ahead of technology, please considering subscribing to Xpresso.

---

 

IN THIS FEATURE WE TAKE A BROAD LOOK AT IMMERSIVE technologies such as VR/AR—scanning across histories, present technologies, companies, and many of the key issues around their success.

Origins of VR

While it may seem that the VR-AR craze emerged only a few years ago, these technologies have had long invention cycles spanning multiple decades which sorted out the many difficulties that make immersive technology experiences so challenging. Even more, the very concepts behind “virtual reality” and the technologies that make it possible go back nearly two centuries.

In 1935 science fiction writer Stanley Weinbaum released Pygmalion’s Spectacles, a story about the main character that wears a pair of goggles that look like a gas mask which transports him to a fictional world where his human senses participate in holographic reality. Here’s a quote from the short story:

 

But listen—a movie that gives one sight and sound. Suppose now I add taste, smell, even touch, if your interest is taken by the story. Suppose I make it so that you are in the story, you speak to the shadows, and the shadows reply, and instead of being on a screen, the story is all about you, and you are in it. Would that make real a dream?

 

Weinbaum’s character is describing the concept of virtual reality at a level that we actually have not yet achieved in a mass-produced shipping consumer or industrial product. Yet, while this vision from 1935 is remarkable, key technical aspects of virtual reality head-mounted displays (HMDs)—like stereoscopy or what is also known as stereo imaging—first began with Sir Charles Wheatstone in 1838 when he was awarded the Royal Medal of the Royal Society in 1840 for his explanation of binocular vision, upon research that led him to construct the stereoscope.

His research proved that the human brain actually combines two images (one eye viewing each) of the same subject take from different points to make the image appear as one with a sense of depth and immersion or three dimensions. His Wheatstone mirror stereoscope is shown above.

However, David Brewster who rivaled Wheatstone developed a hand-held “Brewster-type stereoscope” that caught the attention and admiration of Britain’s Queen Victoria when it was demonstrated at the Great Exhibition of 1851. This device used lenses for uniting the dissimilar pictures in 1849, leading to lenticular stereoscopy, which allowed Brewster’s device to be much smaller than Sir Charles Wheatstone’s invention. Brewster’s device, made of wood, resembles the Google Cardboard in some ways.

These 19th-century stereoscopes would ultimately lead to one of America’s most well-known devices commonly purchased for children—the ViewMaster stereoscopes, which led to the craze for “virtual tourism” which is used in modern VR headsets today. End-of-life patients are sometimes allowed to travel to far-away places using VR as part of their care plans.

While stereoscopy was a critical element of the future of VR, another major piece of the modern-day puzzle was motion-tracking. If a user moved their head looking at the stereoscope-based image there was no sense of movement in the image, those devices only simulated depth and being in a far-away environment (“virtual tourism”), which was the big hit of ViewMasters that took millions of users to special places like the Grand Canyon in Arizona.

Morton Heilig, a cinematographer, created the Sensorama, a real VR machine that a user sat in and combined multiple technologies and stimulated the senses. Sensorama combined 3D video, audio, vibration in the seat the user sat on while using the device, plus smell and atmosphere effects like the wind. In many ways, the Sensorama was a forerunner of Walt Disney’s technologies behind their hit Disney World ride Avatar Flight of Passage. Combining so many sensation-based technologies increased the “immersion” into the virtual environment.

next page: Modern VR, Oakley Gate, and Apple and Immersive

Modern VR

Modern VR systems needed more than multimodal sensory systems, they needed motion-tracking. While Morton Heilig would patent the Telesphere Mask as the world’s first head-mounted display (HMD) in 1960, it would be one more year in 1961 when two Philco Corporation engineers (Comeau & Bryan) developed the first HMD precursor, known as the Headsight.

The Headsight HMD contained a video screen for each eye and a magnetic motion-tracking system which was linked to a closed-circuit camera. The device was created not for virtual reality (a term not yet coined) but for remote viewing of dangerous situations for military use. The Headsight controlled a camera not a computer-generated image of a 3D environment.

In 1965 famed computer scientist Ivan Sutherland, considered one of the fathers of CAD (computer-aided design) and computer graphics, presented his vision of the Ultimate Display. This was a vision about observing a virtual world through an HMD that could replicate reality including allowing the user to interact with objects. Sutherland and his student Bob Sproull created The Sword of Damocles, a head-mounted display connected to a computer rather than a camera and it displayed simple wire-frame primitives which would move in response to the HMD. The technology was innovative but it never left the lab.

MORE: The Wild is Changing the Realities of Work — The Pandemic Just Helped

Through much of the 70s and 80s technologies key to the development of modern immersive experiences would advance thanks to military and aerospace research and funding. Flight simulators were key developments that aided computerized immersive experiences. HMDs continued to advance as did the development of eye-tracking like in McDonnell-Douglas Corporation’s HMD technology in its VITAL helmet.

‘Virtual Reality’ — The Term

In 1985 Jaron Lanier and Thomas Zimmerman founded VPL Research, Inc. and Lanier coined the term ‘virtual reality’ during this time. The company developed multiple VR devices including a DataSuit with full-body sensors for measuring the movement of arms, legs, and the trunk. DataGlove, EyePhone, and AudioSphere were other devices VPL Research created.

In 1988 Autodesk with its Cyberspace Project was the first to implement VR on a low-cost personal computer. Project leader, Eric Gullischsen left Autodesk to form Sense8 Corporation which would develop the WorldToolKit virtual reality SDK. While popular the WorldToolKit and its VR SDK never took off.

Throughout the 90s and the first decade of the 21st century, the VR industry made steady gains, from Sega’s Sega VR headset to the development of virtual reality caves. Apple introduced QuickTime VR which actually produced linked 360-degree interactive panoramas, not true virtual reality. Nonetheless, the product was popular but expensive.

Oakley Gate

In the early 90’s serial entrepreneur Paul Travers, who came out of Eastman Kodak’s Labs was the founder and president of Forte Technologies, which developed the VFX-1, a helmet-sized VR headset that sold millions of units and was powered by a regular PC. In many ways, the VFX-1 was the closest thing to the eventual VR devices we have today and sold for similar price points suitable for gamers. In 2000, the device was superseded by Interactive Imaging Systems’ VFX3D.

In a recent ARK Invest podcast, Forte’s founder Paul Travers spoke about the history of his involvement in immersive technologies. Forte Technologies eventually ended and Travers launched Vuzix in 1997. He tells ARK Invest analysts and podcast host, Nicholas Grous, that prior to the late 90s and Vuzix’s switch to AR from VR, they took their VR technologies to Raytheon and the US Special Forces.

In the early 90’s US Special Forces were testing out Forte’s VR helmet-based HMDs but the soldiers were in need of something much lighter in weight and wanted something quite specific. He says in the ARK Invest podcast episode:

 

They told us they wanted to get rid of their Toughbooks. They came to [us] and said, ‘can you make a pair of glasses that look like Oakleys?’ — they call it Oakley Gate. ‘Oh man, if you can get through the Oakley Gate half the military is going to buy these glasses.’ 

 

Travers says that the AR market is where his company began to focus, not just to solve the military’s challenges but a wide range of industrial solutions where virtual reality serves little purpose. He also notes in the podcast that Covid accelerated his company’s products. Noting that in medical environments doctors were limited to possible exposer to Covid-19 patients so they would send one doctor to do the rounds wearing Vuzix immersive technology head-mounted glasses and the other doctors would be able to still participate.

AR and Vuzix

Vuzix today makes a range of products that Travers says are superior to Microsoft’s Hololens because that product weighs far too much for a person to wear it all day long without issue. Vuzix’s flagship product is the M4000, which Travers characterizes as similar to the failed Google Glasses but it is succeeding.

The M400 weighs under 4 ounces while the company’s competitors can weigh pounds. Called smart enterprise glasses, Vuzix products are targeted at manufacturing, field and remote assist, warehouse logistics, and telemedicine. We do a deeper dive on Vuzix’s AR smart glasses in the emTech section below. 

Apple and Immersive

Travers says in the ARK Invest podcast that his company may have the leading AR-based smart glasses technologies but they are likely not the company that will popularize AR smart glasses for the mass consumer market. That likely will be Apple.

He says while Vuzix makes AR smart glasses that are not that far from the Oakley’s US Special Forces like to wear, he says his wife won’t be caught walking down the street in them. “If they are big and bulky or they look odd looking it will fail,” he tells ARK Invest analyst Nicholas Grous. He says Vuzix is trying to solve the ‘Oakley Gate’ problem as it applies to industrial sectors that have much to gain. The consumer space is likely going to be solved by a company like Apple.

He says Apple’s AR technologies that are increasingly coming out more advanced are being “beta tested” on iPhones until the Cupertino company can completely solve the equivalent of the “Oakley Gate” problem for the mass consumer.

Former Apple exec Richard Kerris told Architosh back in 2016 that Apple’s failed Newton make valuable lessons for the company with regards to AR. Kerris gave the world a big clue as to why the company is taking so long to even introduce an Oculus Rift competitor in VR. Kerris—who has shared the stage with Steve Jobs more than just about any other Apple exec—says a good way imagine what Apple has planned for VR or AR is to imagine Steve Jobs making the keynote presentation on the new technology. Quoting from Architosh’s past article:

 

If you have paid a lot of attention to Jobs’ keynotes then you may have noticed a tried-and-true formula for how Apple does a ‘take down’ of competitors and their products in the market. It’s simple: go through all the weaknesses of a product or its technology and list their limitations. Then Jobs would introduce an Apple product that doesn’t have those limitations. And then wax on about how beautiful, thin, light, strong, and long-lasting the product is. And how it all fits into the Apple ecosystem.

 

One of the things Kerris says about VR which is troubling is that it’s not ideal for long-term use. On the AR side, Kerris says that the big challenge is the “faceware” that the user must wear. Most industry insiders believe that Apple will solve that problem and introduce AR smart glasses as stylish as any other pair of sun or prescription pair of glasses on the market. In fact, Apple will likely incorporate its advanced technology and increasing medical science expertise to deliver best-in-breed lens technology combined with frame styling that will upend the glass wearing industry much like it has upended the watch industry with Apple Watch.

ARK Invest is predicting the AR Smart Glasses market to grow from USD 1.2 billion globally by end of 2021 to USD 130 billion by 2030. Paul Travers agrees with Grous and says a tipping point is coming in the next few years. Apple is estimated to announce its Apple VR headset in 2023, while its AR smart glasses are estimated for 2025. By that time, Apple Silicon will be at a 3nm chip process enabling powerful computing within a tiny TDP (thermal design profile) suitable for glass frames.

Travers says to Grous in the podcast, “Nobody saw Uber when the iPhone was introduced. But here you go…without the iPhone revolution there would be no Uber.” The implication is clear: imagining the AR smart glasses market rising by 100x plus times in just nine years assumes Uber-like giants will emerge out of the AR smart glasses computing platforms once they begin to proliferate. Apple has the best track record making disruptive new technologies take off on a mass-consumption scale. They did it with the iPod, the iPhone, the iPad, and the AppleWatch. It doesn’t seem particularly smart to bet that Apple won’t do it again with AR smart glasses.

---

Editor’s Note

If you liked this article, sign-up for the INSIDER Xpresso newsletter (free) where our best feature stories like this are published first.

Additionally, Xpresso No. 28 has more hard-to-find information about immersive technologies, which you can find in our Curated EmTech section in the June Xpresso newsletter. See the newsletter here and subscribe!

Cover Image Credits:

Sir. Charles Wheatstone’s mirror stereoscope. His research and this instrument won him the Royal Medal from the Royal Society in 1840. In the article hero image, images ‘D’ and ‘C’ are brought together and combined by the human brain when viewed through lenses focused on two mirrors (‘A-B’) angled at 45 degrees to the images. (Image: Wiki Commons)