OS X Development Environment for Vuforia Android Apps + Streaming Video Playback Sample

Qualcomm has instructions on setting up an environment to develop Vuforia based apps for Android (for those of you unaware, Qualcomm's mobile augmented reality SDK was renamed from QCAR to Vuforia). It focuses on Windows platforms with some tips/hints for Linux and OS X. This blog article will focus on setting up the same kind of development environment but on OS X. The plan is to document as I give it ago and give a bit more in-depth detail. At the end I'll cover running the Video Playback sample app and modifying it slightly to stream video off of the internet. I'll also cover debugging native code from within Eclipse.

Components to Install

• JDK (1.6.0_35)
• Eclipse IDE for Java (Juno)
• Android SDK (r20)
• Android ADT (Eclipse plugin for r20)
• Android NDK (r8b - native development kit )
• Android CDT (C/C++ development tools)
• Vuforia SDK (1.5.9 - Qualcomm's augmented reality SDK)
• Xcode command line/developer tools

Java, Eclipse and Android SDK

The Qualcomm setup docs suggest to use Java 1.7 but I'm going to stick with the stock standard JDK that Apple bundled with my Mountain Lion (Java 1.6.0_u35-b10 on OS X 10.8.2). They also suggest to use the the 32-bit version of Eclipse for Mac but I've already got the 64-bit of Juno so am going to stick with that too.

There's already a plethora of info on setting up a standard Android development environment with the Eclipse IDE and Android SDK so I suggest either searching Google or going straight to the official Android docs site for help with this first part. At least, you will require the Android SDK, Eclipse for Java and ADT (the Eclipse Android Development Tool) plugin installed.

Note: install the NDK Plugins at the same time as you install the ADT plugin in Eclipse.

Android NDK Installation

The following instructions assume you already have a basic Android development environment setup with Eclipse.

At the native level, Android runs its programs that were originally written in C/C++ (and no doubt some ASM). The majority of Android apps are written purely in Java, running on on the Android-optimized JVM called Davlik. Using the NDK (native development kit), developers can also themselves write part of their apps in C/C++. Such apps would be a blend of Java and C/C++ glued together with JNI - the Java Native Interface which is the bridge between code written in the two languages.

Vuforia itself is written in C++. The SDK library was no doubt written in native code because of the increased performance for its critical features like the computer vision algorithms and OpenGL code.

In order to compile NDK code on OS X you'll need Xcode and its Command Line Tools installed for programs like the GNU compiler gcc and the build tool make.

1. Download Xcode from here and once you've installed it goto Xcode -> Preferences -> Downloads and choose to install Command Line Tools (hopefully it's in your list). I am using Xcode 4.5.1. Alternatively, although I haven't tested it, you can try downloading and installing Xcode bundled with the Developer Tools package from here. In either case, you'll probably need to create a free developer account with Apple.
2. Download the Android NDK for OS X from here and unzip the archive in the same folder as you installed the Android SDK. I installed version r8b under /Users/wocky/Documents/android/android-ndk-macosx.
3. The installation path needs to be added to the environment variable called PATH. Do so by adding it to the end of the PATH configuration in ~/.bash_profile. For example: PATH=$PATH:/Users/wocky/Documents/android/android-sdk-macosx/platform-tools/:/Users/wocky/Documents/android/android-ndk-macosx; export PATH
4. Change to the samples/san-angeles directory (under the NDK installation directory) and run the command ndk-build from a terminal window to ensure the path setting in step 3 is correct. You should see some output saying the San Angeles project's C files have been compiled and the samples/san-angeles/libs/armeabi/libsanangeles.so library created.
5. Next we will install the CDT plugin for Eclipse. This is the C/C++ Development Tool plugin. Goto Help -> Install Software and choose the URL of the version that matches your Eclipse (http://download.eclipse.org/releases/juno for Juno). Under the Programming Languages category check C/C++ Development Tools and proceed to install it. Eclipse will need to be restarted after the installation (it should prompt you anyway).
6. Install the NDK Plugins for Eclipse that can be found on the same site as the ADT plugin. Following the same steps as above, the URL will be something like https://dl-ssl.google.com/android/eclipse.
7. Lastly, tell Eclipse where the NDK is installed by specifying the installation path in Eclipse -> Preferences -> Android -> NDK.

I won't go over the specifics of creating a new Android NDK project but here are some more articles to reference. They cover technical details of NDK/JNI and and setting up the environment and debugging. I haven't ran through these articles myself so can't attest for their validity.

• Official Android site on getting started with the NDK and projects
• Very nice and detailed step-by-step article for beginners to Android NDK
• Debugging native code with Eclipse (this is an area many people say is really difficult!)
• Tutorial on installing NDK and creating a basic project (a little old but perhaps still valid in most parts)

Vuforia Augmented Reality SDK Setup

1. Download the SDK for OS X from here. You will probably need to register a free developers account with Qualcomm. I am using SDK version 1.5.9.
2. Once the package was downloaded, I unarchived it and ran its GUI installation program. I chose the same folder as the Android SDK and NDK to house Vuforia as well. It's /Users/wocky/Documents/android/vuforia-sdk-android-1-5-9 in my case.
3. We need to make Eclipse aware of where the Vuforia SDK is. Open Eclipse ->Preferences -> Java -> Build Path -> Classpath Variables and create a new variable called QCAR_SDK_ROOT and point it to your installation folder.

Running Sample VideoPlayback App and Debugging with Eclipse

The Android NDK section above gives some links at the bottom to other blogs as an excellent source for learning more about the basics of Android NDK projects' structure and inner workings. I'm going to end this particular blog article documenting a couple of areas of interest; first the video playback feature that was made available in Vuforia 1.5 and debugging NDK applications from within Eclipse - something I had a lot of trouble with trying to setup on my Windows box a few months back.

Qualcomm announced a sample video playback app for Android and iOS earlier this year here. I downloaded the sample app for Android and placed the VideoPlayback directory in the zip file under the samples folder in the Vuforia SDK installation path.

Installing the App

1. Download the Eclipse project from the announcement link above (download).
2. In Eclipse, open File -> Import -> Android -> Existing Android Code Into Workspace and search for the folder where you copied the sample VideoPlayback app to.
3. Rename the Eclipse project to something simply like VideoPlayback if you prefer, instead of the default Java package name that Eclipse tends to use for imported projects.
4. Open a terminal and goto the VideoPlayback directory and run the ndk-build command to build the native library code. The sources should compile with no errors.
5. Right-click the VideoPlayback project in Eclipse and choose Refresh to make Eclipse aware of the new files created by step 4.
6. Right-click again and select Run As -> Android application to run it on your device.
7. You'll need either a printout (or display the image on your screen) of the stones and/or chips targets that you can find in samples/VideoPlayback/media under the Vuforia installation directory.
8. Once the VideoPlayback app is running on your Android and you see the video input from the camera, point the camera at the target and you should then see a play button to play a video like:
   
9. As an optional step you can modify the VideoPlayback sample to stream a video from the internet rather than play the mpeg4 videos packaged with the app. 

Streaming Video Media from the Internet

To stream a video from the internet instead of playing the default mpeg4 movies included in the sample app, open VideoPlaybackHelper.java and modify the following the load method to set the MediaPlayer object's data source to a file on the internet and comment out the lines that load the movie files from the asset resources.

try
{
    mMediaPlayer = new MediaPlayer();

    // This example shows how to load the movie from the assets folder of the app
    // However, if you would like to load the movie from the sdcard or from a network location
    // simply comment the three lines below
    //AssetFileDescriptor afd = mParentActivity.getAssets().openFd(filename);
    //mMediaPlayer.setDataSource(afd.getFileDescriptor(),afd.getStartOffset(),afd.getLength());
    //afd.close();

    // and uncomment this one
    mMediaPlayer.setDataSource("http://people.sc.fsu.edu/~jburkardt/data/mp4/cavity_flow_movie.mp4");
    ...
    }

I used this video as a sample. The file needs to be either over HTTP or RTSP and capable of progressive downloads (otherwise streaming wouldn't work). Here's a screenshot of it with the chips marker being display on my Mac air's screen.

I was even able to play a flash video so potentially YouTube videos could be used if you know the direct link to the FLV file (but apparently that breaches their TOCs). This option may only available on later versions of Android. I used ICS (4.0.4) on my Galaxy S3 but it didn't work on my 2.3.4 Nexus S.

Debugging Native Code in Eclipse

About a year ago when I first started playing with Vuforia and Android's NDK I remember it was a real nightmare to get debugging working. At that time I was using Eclipse on Windows 7. It seems the ADT plugin has come a wee way since then and the process has been simplified. I don't actually recall a direct "Run As -> Android native application" option. If memory serves me right, you needed to put a breakpoint in the Java code right before the C/C++ native code was called and then doing a normal Java debug session, it would jump into the native code space. Here's a summary of what I needed to do get ADT r20 running on my new setup:

1. Ensure you have set the path to the NDK installation in Eclipse's Android preferences and the CDT plugin is installed.
2. Right-click the VideoPlayback project and select Android Tools -> Add native support.
3. Eclipse will most likely automatically switch from the Java to C/C++ perspective (think of an Eclipse perspective is like an "editing mode" for that particular language/feature).
4. If you are using version r20 of the Android ADT Eclipse plugin there's a bug that will give you a bunch of warnings and errors with the cpp files in the jni folder. The actual building of these source files is handled externally by the ndk-build script but if you use Eclipse as an editor it's clutter on the screen you don't need. Following the advice here, I disabled all the items under project properties -> C/C++ General -> Code Analysis as a temporary work around.
5. Add a breakpoint somewhere in one of the cpp files. I chose the beginning of the 
   Java_com_qualcomm_QCARSamples_VideoPlayback_VideoPlayback_initApplicationNative function in VideoPlayback.cpp.
6. Lastly, select the project in Eclipse's package explorer, right-click and choose Debug As -> Android Native Application.

That's all folks.