Getting Started with Augmented Reality/Virtual Reality Development

When I first began developing my Final Year Project (FYP), an Augmented Reality (AR) app in 2016, I started out with absolutely zero knowledge of Unity or Mixed Reality (XR) development. My knowledge base was limited to the curriculum. So, suddenly having to build a working a prototype as well as learn everything from scratch within a limited time frame was overwhelming.

I haphazardly went through tutorials I could find and hoped that they would be up-to-date, taking small baby steps from learning to finding out how to do things until I could develop my application. Since then, I began doing and learning more in the process, becoming the “go-to-person” for XR-development in my department, and advised Final Year students who are doing XR-related projects. I noticed a pattern in the problems and challenges they face, but the biggest problem of all is: “I don’t know how”.

As someone who had also undergone what they went through, I understand how it can be overwhelming. I would like to share the steps that I have taken and would recommend to anyone when they are looking into XR development for the very first time.

A brief rundown:

  1. Know the difference between Augmented Reality (AR) and Virtual Reality (VR)
  2. Know the different engines and SDKs required to develop AR/VR
  3. Start with the basics – learn how to an AR/VR project using the engine and Software Development Kit (SDK) you chose to use
  4. Alternately, start from the top – determine what it is you want to build and work your way to the finer details
  5. Forums and documentations are your best friends

Know the difference between AR vs VR

It always amazes me how AR and VR are often used interchangeably and misunderstood. Before we start, let’s define the two technologies.

Augmented Reality (AR) technologies allow you to augment virtual, interactive 3D objects upon the real-world in real-time. “Objects” include text, 3D models, buttons and so on. It supplements your reality. Whereas Virtual Reality (VR) completely immerses you in a new virtual environment.

It is important to know the difference because the display technologies and SDKs that you will use to render them are also different. With AR you can make do with a standard smartphone, tablet, or – if you go want to go the extra mile – high-end Head Mounted Displays (HMDs), such as Microsoft’s Hololens.

The device on which you deploy your app influences how your users interact with your application. A user interacts differently with an AR system than a VR system. For instance, touch interactions are more common for mobile AR apps. Apps deployed on Hololens often use hand gestures or voice commands. In VR, however, users interact with the virtual environment and objects via gestures or by clicking buttons on the controllers.

This is relevant because programming interactions differ in complexity depending on both the system you want to build on and the hardware you are using.

Know the different engines and SDKs required to develop AR/VR

There are many engines out there which you can use to develop AR/VR applications. Two of the most popular (that I know of) are the Unity Engine and Unreal Engine. I am personally more familiar with the Unity Engine because this is the engine that I started out with.

It is beneficial to find out more about the different engines available. There are many articles that highlights the differences. For instance, this article that compares the Unity Engine and the Unreal Engine. Next, you need to choose which SDK to use to help you develop your project. Simple put, “an SDK brings together a group of tools that enable the programming of applications”.

There are a lot of SDKs out there to build AR and VR apps. One of the most popular ones for AR are Vuforia, ARCore and ARKit. One of the most widely used SDK for developing VR applications is SteamVR. Each SDK has its own strengths and unique features.

Factors such as platform, hardware, the type of app you are building etc., all influence which type of SDK is suitable for you. Choose one that best suits your needs. For instance, EasyAR is one of the easiest SDKs (that I know of) to use to make marker-based AR apps; the SteamVR SDK is popular when building VR apps for the HTC Vive.

How to choose which AR SDK is suitable for you? This article might help. This article might help with choosing VR SDKs.

Start with the basics – learn how to create an AR/VR project using the engine and SDK you choose to use

AR tutorials for beginners typically
consist of augmenting squares.
This app
was made based on a Zenva tutorial
using the EasyAR SDK.

Although it is not wrong to just jump right into it and immediately start working on your brand new XR application, it really helps to build a “starter” or “dummy” app first. These would usually be the results of you following a tutorial or just trying stuff out. When you begin by building a dummy app, it helps you understand the process better because you are actually taking the time to understand it.

In this way, you might feel more free to make mistakes or try new things. After experimenting for a bit or after a while (depends on the individual), you will feel more confident and comfortable using the engine and the SDK.

Alternately, start from the top – determine what it is you want to build

If you have an end goal in mind, it really helps with planning out what you need to do. After you have detailed the steps you need to take to achieve your goal, you can then refine it even further or proceed straight to find out how to do it. Then you have a good game plan to help you go about developing your project.

For example, this is how I broke down my AR handwriting app step by step:

  • An animation to show how to write
    • how to render lines?
    • how to animate lines?
  • The user should be able to replay the animation
    • how is this done? 2D or 3D?
    • how to implement UI?
    • how to detect if user pushes a button?
  • Activity where they connect 3D dots
    • how to connect dots?
      • use another object to hit the dots
      • how to detect if objects are hit?
    • how to know if dots are connected properly?
    • does something happen when they do it correctly?
    • does something happen when they do it incorrectly?
  • The user should be able to reset the activity

I would find myself adding more and more items to the list as I go along and further develop, encountering new ideas, new questions, more knowledge gaps, and plenty of bugs and errors along the way. It happens organically, and before you know it you are building your app and enjoying the challenge.

Forums and documentations are your best friends

A lot of the times, you may not know how to achieve what you want to do – and that’s completely okay, especially when you are new to Unity or XR development and you are overwhelmed with the interface, SDKs, new jargons such as “Prefabs”, “Components, “GameObjects” and such.

From my experience, the key is knowing what you want to do. By clearly defining this, you have clear way of finding out how to do it. It is so easy to find out how to do something, especially now. You want to implement a scrollbar and you do not know how? Google it and you will get YouTube videos, blog posts, threads from official forums – all of which will more than likely help you or at least, point you to the right direction. Repeat for anything you do not know.

In some cases, you will not be able to find a direct answer, or the solution may be not be up to date – in that case try to improvise, or use a trial and error approach. Keep trying until you get it, and if you still cannot get it, then maybe it is time to post on the forums. Whatever the case, keep trying!

2 thoughts on “Getting Started with Augmented Reality/Virtual Reality Development

    1. Farzana Jeffri says:

      Hey Hakimin, I haven’t! Ever since I started developing with Unity I’ve just stick to it, and my experience with it thus far has been pretty good. You got recommendations for a different engine?


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s