/ From the Archives

Will AR and VR converge?

Last Thursday I went to the April Virtual Reality Meetup here in NYC where Ken Perlin, professor of Computer Science and Director of the NYU Media Research Lab, defined the difference between Virtual Reality and Augmented Reality. Although talks at NYVR can typically be enjoyed by a general audience, most of them are geared towards a tech-savvy, geek-centric cohort of enthusiasts creating or vigorously consuming content with development kits of the Oculus Rift in tow.

Why did Professor Perlin spend so long talking about VR and AR with this audience? He offered the best definition of the difference between the two technologies that I have heard in a long time and hinted at the answer to a question often posed by my colleagues at IrisVR; Will virtual and augmented reality technologies be "the same thing" in a couple of years? Will we be able to seamlessly switch from real-world view, to augmented view, to fully immersive VR?

According to a Tech Times article that tops the Google search results list for "VR vs AR," augmented reality "is the blending of virtual reality and real life" and virtual reality is "all about the creation of a virtual world that users can interact with." Let me offer two different definitions, inspired by Perlin's talk:

Virtual Reality is when all of the light hitting your retinas is a pixel.

Everything you see in VR is fake, in a sense; computer-generated images cover your whole field of view. Literally all of the information that your visual system processes was first created deep within the GPU. Thus, it is possible to capture the real environment and then re-create it inside VR. It's also possible to convince your visual system that you've been transported to an entirely new world, completely divorced from the reality that existed before you put on the VR HMD, head-mounted-display (otherwise known by the technical term "goofy looking goggles"). Most of the experiences created for VR now fall into the second bucket - they re-create an entirely new visual universe. More and more, however, sensors can recreate parts of the "real world" to give you a mixed experience in VR where some of the visual elements actually exist and some are the product entirely of C++.

VR is hard because everything you see needs to be generated by a computer. All of the pixels need to be rendered really fast and because everything you see is computer-controlled, you feel sick if the computer lags. Thus, VR requires lots of computing power.

Augmented Reality is when some of the light absorbed by your retinas was reflected from a natural object and some other light hitting your retinas was born from pixels.

According to the above definition, any device that emits light as pixels in the real world is augmented reality. AR is additive, it embeds digital information into the world for human beings to consume and interpret alongside their visual interpretation of the natural environment. The billboards in Times Square are augmented reality in its simplest form; they convey more information when they are turned on than when they are turned off.

I have heard many people claim that Google Glass is not AR. On the contrary, I think it is AR. Glass (when it still existed), projected data that could be seen and interpreted by your eyes and brain, overlaid on top of the real world. A computer was augmenting your reality - that's AR.

Modern AR, and the kind of augmented reality that jumps to mind when people use the term in 2015, is being developed by companies like Magic Leap. The next step in the evolution of AR is 3D. Instead of projecting flat data in your field of view like a heads-up-display, new augmented reality devices will enable your visual system to perceive objects that don't actually exist. Think holodeck.

AR is hard because if you want to make it look like there is a miniature puppy sitting in the palm of your hand, the computer needs to know the size and location of your hand and it needs to accurately track its movement. This requires some pretty advanced 3D imaging, computer vision, and quick processing.

Will one win?

All over the internet, articles, posts, threads, and forums pit VR against AR in language reminiscent of the great Blu-Ray vs HD DVD debate. The technologies are different and VR provides for a set of experiences that actually supersedes AR. VR technology will be able to reproduce a view of the "real world," rendered as pixels. It's just a matter of processing power, and thus time, until this is technologically feasible, as Ken Perlin explained at NYVR. Three-dimensional AR is dependent on new 3D scanning and image processing technology before the first devices can ship. Thus, I think that VR will hit the market first, it will be adopted faster, and as sensing technology develops, it will be integrated into existing VR. If all of the light hitting your retinas is pre-processed, then there are more options when it comes to manipulating that data, or simply leaving scanned real-world data alone.