1. Introduction

Thanks to the wild success of Pokémon GO, augmented reality (AR) is getting a lot of attention in the mobile development world. The idea of augmented reality isn’t new, but only now is the technology finally getting close to maturity and commercial viability. But what is augmented reality exactly?

This a really broad concept that can take many different forms. The idea can be applied to entertainment, science, military, education, you name it. What these applications all have in common, though, is a real-time connection between the physical world and the digital.  

A live direct or indirect view of a physical, real-world environment whose elements are augmented (or supplemented) by computer-generated sensory input such as sound, video, graphics or GPS data. — Wikipedia

Concretely, AR is often realized in interactive experiences that overlay a camera feed with 2D or 3D objects, informed by some sensor data. 

However, creating an engaging augmented reality experience from scratch can be a real challenge. Fortunately, there are some solutions available that can make this job easier. One of the most solid augmented reality tools out there is Vuforia, an AR library that is compatible with Android, iOS, UWP, and some brands of smart glasses. 

In this tutorial we'll explore augmented reality using Vuforia. We'll explore Vuforia's features, possibilities, and challenges, and we'll also take a look at  how the SDK works and its main functions. We won't explore any code in this tutorial—future tutorials in this series will dig into coding in Vuforia, with step-by-step guides targeting all the library's main features.

2. Vuforia

Originally developed by Qualcomm and recently bought by PTC, Vuforia is on its sixth version. Its SDK is constantly evolving and is compatible with Android, iOS, UWP, and Unity. Unfortunately, Vuforia isn't open source, but its price range is reasonable, and there is no up-front cost for development or education. You can create almost any kind of AR experience on the most popular mobile platforms using Vuforia. 

2.1. How Does It Work?

Vuforia uses the device's camera feed combined with accelerometer and gyroscope data to examine the world. Vuforia uses computer vision to understand what it 'sees' on the camera to create a model of the environment. After processing the data, the system can roughly locate itself in the world, knowing its coordinates: where is up, down, left, right, and so on. What you do with this depends on your development goals.

There are multiple possibilities:

• Using World coordinates: Any kind of object can be exhibited using some arbitrary reference, ignoring real obstacles but considering the World localization obtained. The object can be out of view and continue to "exist" in the same position. This is a system similar to the one used by Pokémon GO.
• Recognizing targets: Vuforia can look for specific images as "targets". VuMarks, for example, a QR code-style image highly recognizable by the system, can be used as anchor points or references for any kind of object projected into the world. A magazine could be 'expanded' with this resource, with its pages opening rich interactions like videos, audio, and so on.
  
• Recognizing simple objects:  Simple objects like boxes and cylinders can be recognized and used as anchor points. This method is useful for recognizing packages and making them interactive.
• Recognizing complex objects: Vuforia can also look for complex objects, like toys, computer parts, gadgets, and so on. Those objects are previously scanned, conforming to specific requirements, and can later be recognized by the system. This functionality could be used, for example, to turn toys into live creatures, or to create interactive assistance for mechanics or service people.
• Looking for words: English words can also be understood by Vuforia. The system can look for specific words and target some kind of interaction on them. This could be useful for learning tools for children or for language translation.
• Recognizing the world's terrain: One of the most powerful features available on Vuforia is the ability to recognize the world as it is. Vuforia lets the user scan their surroundings and can do some hardcore processing to interpret the world by creating a 3D computer vision of the real world and its objects. Imagine a Pokémon GO-like game where the Pokémons can hide behind real objects. Wouldn't that be awesome?

2.2. Developing With Vuforia

Vuforia is compatible with multiple systems. Its software development kits (SDKs) are available for Android, iOS, UWP, and Unity. Once you download and install the SDK, you'll need to sign up for a developer account and create an app key before you can start to code.

Even though Vuforia is compatible with many systems, the easiest way to create engaging AR experiences using its SDK is definitely using Unity. Don't get me wrong, you can access almost everything that Vuforia offers when developing directly for Android or iOS. However, the process is much simpler when done using the prefabs offered on Vuforia's SDK for Unity.

3. Vuforia Targets

As mentioned before, Vuforia can seek 'targets' on the camera feed to create anchor points or references to be used by AR experiences. Those targets can assume many forms, and any file that has been previously processed by Vuforia target manager can be recognized. Targets can be created by the developer or created during execution by the user.

3.1. Image Targets

Any kind of image can be a Vuforia Image Target. However, the more detailed and intricate the image, the better it is to be recognized by the algorithm. A lot of factors will be part of the recognizing calculation, but basically the image must have a reasonable level of contrast, resolution, and distinguishing elements. A blue sky photograph wouldn't work very well, but a picture of some grass would work gracefully. Image Targets can be shipped with the application, created later and uploaded to the application through a cloud system, or directly created on the app by the user.

3.2. VuMarks

A VuMark works very similarly to a QR Code; however, it's much more versatile. It can assume many shapes and colors and adapt to a great number of environments. Its biggest advantage is that it's in total conformity with Vuforia's recognizing algorithm and will be easily 'found' by any Vuforia AR application. Vuforia also offers an Adobe Illustrator plugin that creates VuMarks.

3.3. Cylinder and Cube Targets

It's also possible to use cubes and cylinders as Image Targets. This is very appropriate to create engagement with product packages. Once recognized, the primitive object is used as an anchor point, and it seems to exist for the system's computer vision, allowing virtual objects to interact directly with the primitive. The virtual object could track the real object's position and orientation, for example.

3.4. Objects as Targets

One of the most interesting features of Vuforia is the possibility to recognize complex objects. A toy, a phone, a computer board, and other kinds of objects can be used as targets. 

To use this resource, you have to scan the object using the Vuforia Object Scanner tool, which runs only on Samsung Galaxy S6 and S7. The data scan is uploaded to Vuforia Target Manager, and the scanned object will be recognized by any compatible device using that data.

3.5. Managing Targets

There are multiple ways to create a target: 

• The application can ship with a series of targets embedded.
• The app can receive targets online through the cloud system offered by Vuforia.
• The user themselves can use the device's camera to create a target. 

As a developer, you don't have much control about how the user will create their target; all that you can do is provide the user with some tips about how to chose an appropriate image as a target. However, the situation is different for the targets processed using the Target Manager.

Vuforia's developer portal provides us with the Target Manager tool to help manage all our apps' targets. The manager is very simple to use: you create a database to hold the targets, upload a file that corresponds to a target, and wait until the file is processed. The target will receive a unique ID and a score related to how recognizable it is and will then become part of the database, available for download or for storage in the cloud. 

There are three different types of target databases:

• Device Databases are local databases of images or object targets that are stored on the user's device.
• VuMark Databases are local databases of VuMarks that are stored on the user's device.
• Cloud Databases are databases of Image Targets stored online and queried over the internet.

4. Smart Terrain

When developing a Vuforia app on Unity, there is a really exciting option available, the Smart Terrain. It allows an application to replicate 3D meshes of objects seen by the camera. 

The process works like some kind of scan, where the user uses the device's camera as a 3D scanner. As the camera scans through the ambient environment, the application creates 3D models of the recognized objects, giving the application the possibility to adjust the scenario to the real world, creating a deeply engaging experience.

5. Conclusion

There are hundreds of possibilities to explore with augmented reality, and we're just starting to scratch its surface. Many believe that AR will be a part of our future and that we'll use it on a day-to-day basis. This is a field that promises to grow a lot in the coming years, and Vuforia provides us with cool tools to create engaging experiences.

5.1 What's Next

In the following tutorials of this series, we'll work with some of Vuforia's most important tools. We'll develop a lot of small experiments to illustrate the SDK capabilities, and since Unity is the most AR-friendly environment available, everything will be developed on it. 

It won't be necessary to be a Unity expert to follow the tutorials—since our focus is the Vuforia SDK, I'll provide you with step-by-step guides that require only minimal previous experience with Unity.

If you want to learn a little more about Unity in the meantime, check out some of our other tutorials.

• Create a Basketball Free Throw Game with Unity

  In this tutorial, you'll create a basketball free throw game in Unity. You'll learn about 3D models, mesh colliders, physics materials, collision detection,...

• Creating a 3D City in Unity 5

  Welcome to our Unity tutorial where we will create a 3D city in the Unity 5 game engine. You can use this project for your first-person or third-person...

• Text Input for Games With Just an Intel RealSense Camera

  In this post, I'll share what I've learned about the best (and worst!) ways to let players enter text via gesture alone, and show you how to set up the Intel...

See you soon!