Pages

Wednesday, 30 November 2011

Android e-mail intents with multiple attachments and conversion between path URIs

A recent Android app I was working on led me to spend sometime researching how to send multiple image attachments via e-mail. It turned out not to be quite as easy as I expected. For a simple single attachment e-mail you would have form an intent with code that looks like this:

String externalStoragePathStr = Environment.getExternalStorageDirectory().toString() + File.separatorChar;
String filenameStr = "MyPhoto.jpg";
String absPath = "file://" + externalStoragePathStr + filenameStr;
Intent emailIntent = new Intent(android.content.Intent.ACTION_SEND); 
emailIntent.setType("text/plain");
emailIntent.putExtra(android.content.Intent.EXTRA_EMAIL, new String[] {"wocks@gmail.com"}); 
emailIntent.putExtra(android.content.Intent.EXTRA_SUBJECT, "Test Subject"); 
emailIntent.putExtra(android.content.Intent.EXTRA_TEXT, "Here's your stuff"); 
Uri retUri = Uri.parse(absPath);
emailIntent.putExtra(Intent.EXTRA_STREAM, retUri);
startActivity(Intent.createChooser(emailIntent, "Send attachment with what app?"));

and to send multiple attachments you use the ACTION_SEND_MULTIPLE action intent with the multiple filenames passed in as an ArrayList of Parcelable Uris:

String externalStoragePathStr = Environment.getExternalStorageDirectory().toString() + File.separatorChar;
String filePaths[] = {
 externalStoragePathStr + "MyPhoto.jpg", //  /mnt/sdcard/MyPhoto.jpg
 externalStoragePathStr + "MyPhoto-2.jpg", //  /mnt/sdcard/MyPhoto-2.jpg
 externalStoragePathStr + "MyPhoto-3.jpg"}; //  /mnt/sdcard/MyPhoto-3.jpg
Intent emailIntent = new Intent(android.content.Intent.ACTION_SEND_MULTIPLE); 
emailIntent.setType("text/plain");
emailIntent.putExtra(android.content.Intent.EXTRA_EMAIL, new String[] {"wocks@gmail.com"}); 
emailIntent.putExtra(android.content.Intent.EXTRA_SUBJECT, "Test Subject"); 
emailIntent.putExtra(android.content.Intent.EXTRA_TEXT, "Here's your stuff"); 
ArrayList<Uri> uris = new ArrayList<Uri>();
for (String file : filePaths) {
 File fileIn = new File(file);
 Uri u = Uri.fromFile(fileIn);
 uris.add(u);
}
emailIntent.putParcelableArrayListExtra(Intent.EXTRA_STREAM, uris);
startActivity(Intent.createChooser(emailIntent, "Send attachment with what app?"));

But I ran into a bunch of various problems using different variations of the snippets of code out there on the web, namely many related posts you will find over at stackoverflow.com.

  • Attachment filenames would appear in the compose e-mail window but when they arrived at the destination they were corrupt (Thunderbird wasn't recognizing them as images)
  • Attachment filenames wouldn't even attach to the new e-mail (i.e. weren't displayed in the compose window)
Conclusion:

From what I can recall my major hurdle was the E-mail application was not attaching multiple filenames to the compose window but the G-mail application was. After many trials and errors, what I found was that the E-mail application didn't seem to like being passed an ArrayList of Uris that were made from absolute paths (e.g. /mnt/sdcard/DCIM/Camera/photo.jpg, file:///mnt/sdcard/MyPhoto.jpg etc.). What it did like was Uris in the form like content://media/external/images/media/1075.

This brought me onto my next big research and trial/error phase - converting the absolute path of my images which I know into content:// ones. Vice versa is an easy process but the way I needed was a bit more complicated.

I've put everything together into one activity with the code below. Check the source code comments for more detail. Basically, you define your filenames, run the activity and it will then prompt you to send a single or multiple attachments. It covers:

1) e-mail intents with single/multiple attachments that work with both E-mail and G-mail applications
2) converting absolute paths -> content URIs
3) converting content URIs to absolute paths

Click the Java filename below to expand the (very long) code. Alternatively, download the full Eclipse project here.

package ws.aroha.android.pilcrowpipe;

import java.io.File;
import java.io.FileNotFoundException;
import java.util.ArrayList;
import java.util.Stack;

import ws.aroha.android.pilcrowpipe.R;

import android.app.Activity;
import android.app.AlertDialog;
import android.app.ProgressDialog;
import android.content.ContentResolver;
import android.content.DialogInterface;
import android.content.Intent;
import android.database.Cursor;
import android.media.MediaScannerConnection;
import android.media.MediaScannerConnection.MediaScannerConnectionClient;
import android.net.Uri;
import android.os.Bundle;
import android.os.Environment;
import android.os.Handler;
import android.provider.MediaStore;
import android.widget.Toast;

/**
 * SendMultipleAttachments
 * 
 * @author Simon McCorkindale
 * 
 * Sample code to illustrate 2 concepts on Android:
 * 
 * 1. Opening a new e-mail intent with single or multiple image attachments using {@link Intent.ACTION_SEND} and {@link Intent.ACTION_SEND_MULTIPLE} 
 * 2. Converting between absolute paths and "content://..." type URIs
 * 
 * The reason for this sample code is for the situation where you only know the real (absolute) path of
 * the file(s) you want to attach. Unfortunately it seems E-mail/G-mail apps require the paths be a URI
 * in the format of "content://..." in order to attach them so we do the conversion here. Conversion
 * from "content://..." URI to an absolute path string is easy but not the other way around, which
 * requires a little a little multi-threading work with the Media Scanner API.
 * 
 * See more detail on my blog at:
 * {@link http://pilcrowpipe.blogspot.com}
 *
 * Disclaimer: Use this code however you see fit with no restrictions. Kudos to those
 * people who wrote code on the sites I've referenced in my comments.
 *
 */
public class SendMultipleAttachments extends Activity implements MediaScannerConnectionClient {
 /**
  * Path to external storage media (SD card in most cases).
  */
 private final static String EXTERNAL_MEDIA_PATH =
  Environment.getExternalStorageDirectory().toString() +
  File.separatorChar;

 /**
  * Alert dialog item IDs.
  */
 private final int SINGLE_ITEM = 0x10;    // Single file attachment
 private final int MULTIPLE_ITEMS = 0x1a;   // Multiple file attachments
 private int mSendTypeSelection = SINGLE_ITEM;  // Send single attachment by default

 /**
  * Define the absolute path to your multiple attachment files.
  * MODIFY TO MATCH FILES ON YOUR PHONE.
  */
 private final String mMultipleFilesPaths[] = {
   EXTERNAL_MEDIA_PATH + "MyPhoto.jpg",   // -> /mnt/sdcard/MyPhoto.jpg on my Android
   EXTERNAL_MEDIA_PATH + "MyPhoto-2.jpg", // -> /mnt/sdcard/MyPhoto-2.jpg on my Android
   EXTERNAL_MEDIA_PATH + "MyPhoto-3.jpg", // -> /mnt/sdcard/MyPhoto-3.jpg on my Android
   EXTERNAL_MEDIA_PATH + "MyPhoto-4.jpg" // -> /mnt/sdcard/MyPhoto-4.jpg on my Android
 };

 /**
  * Define the absolute path to your single attachment file.
  * MODIFY TO MATCH FILE ON YOUR PHONE.
  */
 private final String mSingleFilePath = 
  EXTERNAL_MEDIA_PATH + "MyPhoto.jpg";  // -> /mnt/sdcard/MyPhoto.jpg on my Android

 /**
  * The send e-mail {@link android.content.Intent}.
  */
 private Intent mIntent = null;

 /**
  * The number of URIs we expect to be converted to content:// type. This figure
  * is used so we can tell whether the conversion for all files has been complete or not.
  */
 private int mNumberExpectedConvertedUris = 0;

 /**
  * Handler to accept communiqué from the Media Scanner Client worker thread. "IPC" made easy!
  */
 private Handler mHandler = new Handler();
 
 /**
  * Progress bar dialog to display when converting URIs and waiting for the worker thread.
  */
 private ProgressDialog mProgressDialog = null;

 /** Called when the activity is first created. */
 @Override
 public void onCreate(Bundle savedInstanceState) {
  super.onCreate(savedInstanceState);
  setContentView(R.layout.main);
  
  // Check external media state
  String mediaState = Environment.getExternalStorageState();
  if (mediaState == null ||
    !(mediaState.equals(Environment.MEDIA_MOUNTED) ||
      mediaState.equals(Environment.MEDIA_MOUNTED_READ_ONLY))) {
   // Can't access external media so quit graciously notifying the user
   AlertDialog.Builder builder = new AlertDialog.Builder(this);
      builder.setMessage("External media is unavailable (not mounted?). Please check.");
      AlertDialog dialog = builder.create();
      dialog.setButton("OK",
                 new DialogInterface.OnClickListener() {
                     public void onClick(DialogInterface dialog, int which) {
                      dialog.dismiss();
                      finish();
                     }
                 }
             );
      dialog.setTitle("Tiene un problema"); // Houston, we have a problem
      dialog.show();
      return;
  }

  // Prompt the user to send either the one or multiple files registerd in
  // fields at the top of this class
  AlertDialog.Builder builder = new AlertDialog.Builder(this);
  builder.setTitle("Send one or multiple attachments?");
  CharSequence[] items = {"Single", "Multiple"};
  builder.setItems(items, new DialogInterface.OnClickListener() {

   @Override
   public void onClick(DialogInterface dialog, int which) {
    // Determine if user clicked to send single or multiple files
    ContentResolver contentResolver = getContentResolver();
    switch (which) {
    case 0: // Single selected
     // Check file exists
     try {
         contentResolver.openFileDescriptor(Uri.parse("file://" + mSingleFilePath), "r");
        } catch (FileNotFoundException e) {
         Toast.makeText(getApplicationContext(), "File " + mSingleFilePath +
        " doesn't exist",
        Toast.LENGTH_SHORT).show();
        }
        mSendTypeSelection = SINGLE_ITEM;
     break;

    case 1: // Multiple selected
     // Check files exist
     for (String filename : mMultipleFilesPaths) {
      try {
          contentResolver.openFileDescriptor(Uri.parse("file://" + filename), "r");
         } catch (FileNotFoundException e) {
          Toast.makeText(getApplicationContext(), "File " + filename +
         " doesn't exist",
         Toast.LENGTH_SHORT).show();
         }
     }
        mSendTypeSelection = MULTIPLE_ITEMS;
     break;
    }

    requestSendAttachments();
   }
  });
  builder.create().show();
 }

 /**
  * Request to send the attachments. This method invokes the thread that will
  * convert the attachments' URIs from absolute path strings to "content://..."
  * type URIs needed for attaching attachments. This Media Scanner client thread
  * will then initiate the actual sending process (i.e. opening of the application
  * to send the attachments by firing our intention).
  */
 private void requestSendAttachments() {
  mIntent = new Intent();

  // Set some basic e-mail stuff
  mIntent.setType("text/plain");
  mIntent.putExtra(Intent.EXTRA_TEXT, "Test e-mail with attachment(s)");
  mIntent.putExtra(Intent.EXTRA_SUBJECT, "Your file(s)");
  mIntent.addFlags(Intent.FLAG_GRANT_READ_URI_PERMISSION);
  
  // Because we wait on a thread show we're doing something meanwhile
  mProgressDialog = new ProgressDialog(this);
  mProgressDialog.setTitle("Converting URI(s)...");
  mProgressDialog.setMessage("Espere un minuto"); // Please wait a minute
  mProgressDialog.show();
  
  // Intent logic differs slightly depending on whether we are to
  // send single or multiple attachments.
  //
  // The stock standard E-mail / G-mail mail client applications
  // require the URIs to the attachments be in the form of
  // "content://...", not the absolute path type such as
  // /mnt/sdcard/MyPhoto.jpg. It requires some work to 
  // be done in a separate thread (see the Media Scanner code below).
  //
  // I've only tried this with images and it may not work with
  // other types of files for which the underlying Android system
  // doesn't generate content type URIs for.

  switch (mSendTypeSelection) {
  case SINGLE_ITEM: // Send single attachment file
   mIntent.setAction(Intent.ACTION_SEND);
   mNumberExpectedConvertedUris = 1;
   convertAbsPathToContentUri(mSingleFilePath);
   break;

  case MULTIPLE_ITEMS: // Send multiple attachment files
   mIntent.setAction(Intent.ACTION_SEND_MULTIPLE);
   mNumberExpectedConvertedUris = mMultipleFilesPaths.length;
   convertAbsPathToContentUri(mMultipleFilesPaths);
   break;
  }
 }

 /***************************** BEGIN URI CONVERSION CODE ***************************
  * Code snippet referenced from http://stackoverflow.com/questions/3004713/get-content-uri-from-file-path-in-android
  *
  * Logic is something like:
  * 1. UI thread invokes convertFileToContentUri() which causes a new Media Scanner Connection thread
  *    to be spawned.
  * 2. Once the thread has done it's stuff onScanCompleted() is called with the converted Uri
  * 3. My custom hack is to finally send a Runnable back to the UI thread via Handler to add
  *    the new URI to the list of attachments, and if all URIs have been processed fire the
  *    activity to open a new e-mail with attachments.
  */
 
 /** List of content URIs of files to attach */
 private ArrayList<Uri> mContentUris = new ArrayList<Uri>();
 
 /** Instance of our {@link android.media.MediaScannerConnection} client */
 private MediaScannerConnection mMsc = null;

 /** The path of the current URI being queried (scanned) */
 private String mAbsPath = null;

 /**
  * Convert absolute paths to content:// type URIs and store in mContentUris array list.
  * E.g. /mnt/sdcard/DCIM/Camera/photo.jpg -> content://media/external/images/media/1075
  * 
  * @param absPaths[] array of absolute path strings to convert
  */
 private Stack<String> mContentUrisStack = new Stack<String>();
 private void convertAbsPathToContentUri(final String absPaths[]) {
  // In order to achieve sequential processing of each path
  // we utilise a stack, and upon completion of converting
  // one URI it will be popped and the next processed. If we
  // don't process sequential then we get illegal state exceptions
  // from the Media Scanner client saying it isn't connected and
  // some of the paths don't get processed properly.
  for (String absPath : absPaths) {
   mContentUrisStack.push(absPath);
  }
  
  // Pop and convert
  if (!mContentUrisStack.empty()) {
   convertAbsPathToContentUri(mContentUrisStack.pop());
  }
 }

 /**
  * Convert absolute paths to content:// type URIs and store in mContentUris array list.
  * E.g. /mnt/sdcard/DCIM/Camera/photo.jpg -> content://media/external/images/media/1075
  *  
  * @param absPath absolute path string to convert
  */
 private void convertAbsPathToContentUri(String absPath) {
  mAbsPath = absPath;

  mMsc = new MediaScannerConnection(getApplicationContext(), this);
  mMsc.connect();
 }

 @Override
 public void onMediaScannerConnected() {
  mMsc.scanFile(mAbsPath, null);
 }

 @Override
 public void onScanCompleted(final String path, final Uri uri) {
  mHandler.post(new Runnable() {

   @Override
   public void run() {
    // Add converted content URI to list of content URIs to be attached to the e-mail
    synchronized (mContentUris) {
     mContentUris.add(uri);
    }

    // Just for sake of completeness here's demonstrating the reverse;
    // proving what the conversion is correct
    mHandler.post(new Runnable() {
     
     @Override
     public void run() {
      Toast.makeText(getApplicationContext(), "Original path:\n" +
       path + "\n\n" +
       "Converted content URI:\n" +
       uri.toString() + "\n\n" +
       "Reverse converted path:\n " +
       convertContentToAbsolutePath(uri),
       Toast.LENGTH_LONG).show();
      
      // Pop and convert
      if (mSendTypeSelection == MULTIPLE_ITEMS &&
       !mContentUrisStack.empty()) {
        convertAbsPathToContentUri(mContentUrisStack.pop());
      }
     }
    });
    
    // Conversion is complete, start our intent
    if (mContentUris.size() == mNumberExpectedConvertedUris) {
     mProgressDialog.dismiss();
     
     // Add the list of converted URIs to the e-mail intent so the
     // e-mail app can know their location
     switch (mSendTypeSelection) {
     case SINGLE_ITEM:
      // Single URI, add Parcelable Uri object only
      mIntent.putExtra(Intent.EXTRA_STREAM, mContentUris.get(0));
      break;
     case MULTIPLE_ITEMS:
      // Multiple URIs, add ArrayList
      mIntent.putExtra(Intent.EXTRA_STREAM, mContentUris);
      break;
     }
     
     // Open a new compose e-mail window
     startActivity(Intent.createChooser(mIntent, "Send with what application?"));
    }
   }
  });

  mMsc.disconnect();
 }

 /**
  * Convert "content://..." style URI to an absolute path string.
  * E.g. content://media/external/images/media/1075 -> /mnt/sdcard/DCIM/Camera/photo.jpg
  * 
  * Code snippet referenced from http://stackoverflow.com/questions/3401579/get-filename-and-path-from-uri-from-mediastore
  * 
  * @param contentUri {@link contentUri} object containing the "content://..." URI
  * @return string containing the absolute path (including filename)
  */
 public String convertContentToAbsolutePath(Uri contentUri) {
  String[] proj = { MediaStore.Images.Media.DATA };
  Cursor cursor = managedQuery(contentUri, proj, null, null, null);
  int columnIndex = cursor.getColumnIndexOrThrow(MediaStore.Images.Media.DATA);
  cursor.moveToFirst();
  return cursor.getString(columnIndex);
 }
 /*************************** END URI CONVERSION CODE ***************************/
}
// EOF

Tuesday, 29 November 2011

Installing Syntax Highlighter on Blogpost

public static void hola() {
    System.out.println("Hola, ¿cómo estás?");
}

Well the above Java code shows it - I got Syntax Highlighter working on my blog. For those of you who don't know, it's the JavaScript library that formats and highlights the Java syntax above making it all nice and easy for you to read. It supports most popular programming languages. There are already a tonne of articles already out there on how to install Syntax Highlighter so I'm kind of reinventing the wheel but below is an outline of the steps and ways I resolved some issues while installing it.

Getting Syntax Highlighter Running on Blogspot/Blogger.com

  1. Go and download Alex's Syntax Highlighter (SH) from here and unzip it once you've got it. Latest version was 3.0.83 at the time I wrote this article. You'll have a directory hierarchy something like:
  2. You need somewhere to save the SH's JS and CSS files on the web seeing as Blogspot doesn't allow you to upload files (or at least not as far as I could figure out). If you don't have a website or server to host them on then sign up for a free site at Google Sites.
    1. After signing up/logging on, go Create button -> enter details such as name and address of your site -> Create button
    2. New page button (top-right icon) -> enter name of folder (page) to store SH files, e.g. "sh" -> Create -> Save. You'll then have a location something like http://sites.google.com/sites/<your site name>/sh
    3. Manage site -> Attachments -> Upload button -> Choose file, select different location, choose page created in step 2 -> Upload.
      * Files to upload are talked about below so read the rest of the steps before uploading.
  3. Now login to your Blogspot dashboard/admin page (http://www.blogger.com/home) and assuming you're using the new interface (as I am as of 30th Nov 2011):
    1. Open top-right  options menu (beside goto post list icon ) -> Template -> Edit HTML button for the template you current use (under the old interface it's Design -> Edit HTML)
    2. Add the following code directly above the </head> tag. I am using the "Simple" template by the way. Read more below for what edits you will need to do before saving.
    3. <link href="http://sites.google.com/site/arohacorp/sh/shCore.css" rel="stylesheet" type="text/css"/>
      <link href="http://sites.google.com/site/arohacorp/sh/shThemeDefault.css" rel="stylesheet" type="text/css"/>
      <link href="http://sites.google.com/site/arohacorp/sh/shCoreEmacs.css" rel="stylesheet" type="text/css"/>
      <link href="http://sites.google.com/site/arohacorp/sh/shThemeEmacs.css" rel="stylesheet" type="text/css"/>
       
      <script src="http://sites.google.com/site/arohacorp/sh/shCore.js" type="text/javascript">
      </script>
      <script src="http://sites.google.com/site/arohacorp/sh/shAutoloader.js" type="text/javascript">
      </script>
      <script src="http://sites.google.com/site/arohacorp/sh/shBrushJava.js" type="text/javascript">
      </script>
      <script src="http://sites.google.com/site/arohacorp/sh/shBrushPhp.js" type="text/javascript">
      </script>
      <script src="http://sites.google.com/site/arohacorp/sh/shBrushPerl.js" type="text/javascript">
      </script>
      <script src="http://sites.google.com/site/arohacorp/sh/shBrushCpp.js" type="text/javascript">
      </script>
      <script src="http://sites.google.com/site/arohacorp/sh/shBrushBash.js" type="text/javascript">
      </script>
      <script src="http://sites.google.com/site/arohacorp/sh/shBrushJScript.js" type="text/javascript">
      </script>
      <script src="http://sites.google.com/site/arohacorp/sh/shBrushCss.js" type="text/javascript">
      </script>
      <script src="http://sites.google.com/site/arohacorp/sh/shBrushXml.js" type="text/javascript">
      </script>
      <script src="http://sites.google.com/site/arohacorp/sh/shBrushSql.js" type="text/javascript">
      </script>
       
      <script type="text/javascript">
      SyntaxHighlighter.config.bloggerMode = true;
      SyntaxHighlighter.all();
      </script>
      
      1. I have included a few extra files (more than the minimum required) to support an Emacs theme and syntax highlighting for a variety of programming languages). Below is a detailed description of the files and their purpose. Read over that.
      2. Once you've decided what files you will require, upload them to the sh folder you created on Google Sites and then modify the appropriate URLs inside the code you will add to your blog's template.
    4. Now SH should be all setup nicely on your blog, here's what you need to do to actually utilise it.
      1. Create a new post
      2. Open the "Edit HTML" box, and to enter your code copy/paste it between the <pre> and </pre> tags. For example the following code:

        <pre class="brush: java">
        class Foo {
            Foo() {
                System.out.println("Hello, world");
            }
        }
        </pre>

        results in formatted code like:
        class Foo {
            Foo() {
                System.out.println("Hello, world");
            }
        }
        
        A few things to note:
        • You replace the "java" with whatever language you're copy/pasting; js, html etc. Make sure you have the corresponding brush JS file installed (see below).
        • The HTML editor may give you errors saying your tags are not closed if you enter something like <link .../>. It will accept <link ...></link> and may even convert it to the former after you save it.
        • Using <pre>...</pre> tags means that right angle brackets (< and >) in your code will need to be HTML escaped, i.e. be entered > as &gt; and < as &lt;. The solution to that is <script>...</script> tags but these break RSS feeds so pre is considered better for blogs. The trick for using pre is not to directly copy/paste any code that will have right angle brackets in it, but run it through this Postable utility which will automatically convert them for you (Note; as of 2011/12/01 I noticed that this script was deleting + characters in my Java code. I've contacted the creator with the bug and have since resorted to manual search-n-replace in Notepad++ for < and >). More info here.
        • The old flash plugin of yonder days of SH used for copying and printing the contents has been done away with. Now to select all code inside a SH formatted box, just double-click a blank area inside it, all text will be selected then Ctrl+C / Edit->Copy.

    Syntax Highlighter files

    Warning: Do not use the files inside the src folder. You only need worry about the scripts and styles folders in the SH zip file. I was using shCore.js from here and getting lots of XRegExp not found errors when loading my blog.

    At a minimum you will need to install (i.e. upload to Google Sites and link from your template's HTML code) shCore.css and shThemeDefault.css from the styles folder. From the scripts folder you will need shCore.js and at least one brush file for highlighting the syntax of a language (for e.g. JavaScript). The code would look like:
    <link href="http://sites.google.com/site/arohacorp/sh/shCore.css" rel="stylesheet" type="text/css"/>
    <link href="http://sites.google.com/site/arohacorp/sh/shThemeDefault.css" rel="stylesheet" type="text/css"/>
     
    <script src="http://sites.google.com/site/arohacorp/sh/shCore.js" type="text/javascript">
    </script>
    <script src="http://sites.google.com/site/arohacorp/sh/shBrushJScript.js" type="text/javascript">
    </script>
    <script type='text/javascript'>
    SyntaxHighlighter.config.bloggerMode = true;
    SyntaxHighlighter.all();
    </script>
    
    If you want to highlight pure HTML then you can use the Xml brush script and enter "html" as the brush class in the pre tag. Add other languages as you require them.

    That's all. I pieced together this tutorial from memory and half re-doing things as I wrote after I first spent a few hours trying to get it all running. If you find some mistake in this article please let me know.

    Tuesday, 8 November 2011

    Lesson 01: Examples of AR apps and which AR library/SDK to use

    Examples of Augmented Reality Apps on Android


    I started off a month back by firstly looking to see what Android apps were already in the market claiming to be AR apps.

    A rather basic one that shows the augmenting of 3D primitives (basic polygonal shapes) along with basic interaction by manipulating the objects onto the surrounding environment is Virtual Graffiti. This is an app that uses the black and white pattern markers to track the camera's movement. From this it calculates the camera's relative position in order to determine the correct location on the screen where to render the 3D objects.


    Googling for Android AR apps will yield a tonne of sites all with their own ranking and list of the top XX apps. One comprehensive list I found with videos was a site that lists the 33 coolest apps.

    One of the coolest, and from what I can gather, most popular AR apps is Wikitude. This app uses the GPS/location info services and other sensors available on your Android phone to overlay POI (points of interest) on your screen as you view the area where you are. Imagine simply holding the camera up in some random direction and nearby restaurants and shops etc popup on your screen letting you know where they are.


    Another one that I particularly like is Google Sky Map which shows the real relative position of stars, planets and constellations by just pointing your camera in that direction.


    In order to come up with your own idea for your first project, I suggest first watching lots of videos and trying out the ones that catch your eye. Then research into the technical aspects of what would be required to get an idea for the level of complexity. Don't go overkill with your first idea if your main aim is to learn the ropes - you can always create cooler apps down the line once you get more confidence. God knows I've already changed my mind about 3 times over the past few weeks!

    Choosing an AR Library

    Unless you are an overly eager beaver, don't believe in the concept of reusing code, do believe in reinventing the wheel (in which case you probably wouldn't bother reading my blog in the first place) then your next item of business is to choose an AR library to build your app upon. As somebody who has forget most of his high school linear and matrix algebra then using an AR library is an almost must. It will provide all the functions you require for tracking and funky dunky matrix calculations which are needed to place your objects in the correct position in 3D space. The inner workings of the computer vision algorithms is a topic for another day (ha! I recall my computer vision lecture from my compsci uni days and am glad that's over!).

    A fellow blogging developer over at www.evilcodingmonkey.com has provided some similar articles about developing AR on Android which I referenced during my research phase. He has an excellent writeup including reviews and summaries of the various libraries available for AR on Android. I won't bother re-writing my own list here so please do check out his site.

    Qualcomm's QCAR Platform

    Suffice to say, I chose to go with Qualcomm's QCAR SDK. Some of the main reasons being:

    Pros:
    • Seems to be an active project with a big corporation behind it and lots of documents, references and lively forum available
    • Available for both iPhone and iOS
    • Videos show it to have pretty robust tracking (as compared to some AR apps I've seen with poor AR tracking where the objects are always disappearing off the screen as it fails to register the tracker in the screen)
    • Has Virtual Buttons which allow for some cool looking interaction with your app
    • Doesn't require the traditional black & white marker patterns for tracking; you can use photos


    Cons:
    • Registering photos as trackers must be done through Qualcomm's website's online tool (you upload the photo and then download some files to bundle with your application). The UI itself for the tool is pretty lame. Being able to do this programmatically would allow developers to add some cool functionality to their apps.
    • The resource files containing your tracker data must be bundled with your app when you release it, i.e they cannot be dynamically loaded at run-time (although this feature is apparently going to be added in an upcoming version according to a post I read on Qualcomm's forums)
    • The SDK bundled in your app will automatically sends anonymous data (usage statistics I suspect) to Qualcomm's servers. While this isn't a problem for me, I can imagine some situations (especially corporate ones) where the network access of an app must be strictly controlled to adhere to compliance regulations etc.
    • The library is not open-source and written using the Android NDK (read more in the next post) which means moving from Java to C/C++ coding using JNI (bridge interface between Java and native level C/C++ code). This increases complexity of your app, the learning curve and also drastically decreases efficiency for debugging (which I have slightly improved on with my Wogles utility which I will write about in a future post).


    Watch the intro video below that demonstrates the sample apps using QCAR that Qualcomm has released.




    Or, alternatively, view the video on the Qualcomm website. When Qualcomm initially released QCAR they held a competition to encourage development with their platform (first price was something like over USD$100,000!). You can view a video from the winner and also a list of all participating apps under the "Featured Commercial AR Apps" section of the QCAR top page.


    In conclusion


    So now we have an idea of what can be achieved with AR apps on Android and furthermore what kind of functionality QCAR can offer as a versatile AR library. Sometime down the line I'm hoping to play with QCAR on iPhone and maybe one day experiment with other libraries out there, but for the meantime I've already got a lot on my plate.

    Monday, 7 November 2011

    Android AR 101

    We need a kind of roadmap to know where we're going with our "studies". I'll be up front right now and say I will not be talking about nor releasing the source code to my little pet project but I will be releasing the source code to projects of demo apps I write to learn how to implement various AR technologies.


    Think of this post more like a table of contents for up and coming articles.


    First up, lets itemize what I intend to write about.
    1. Examples of AR apps and which AR library/SDK to use
    2. The Android NDK and setting up an Eclipse development environment
    3. OpenGL ES and debugging with Visual Studio (meet Wogles)
    And advanced topics I hope to write about as I figure them out for myself:
    1. 3D modeling and importing models into OpenGL ES
    2. Rendering 3D animations made in 3rd party applications like 3DS Max or Blender into OpenGL ES for the AR app
    3. Rendering video onto a small surface in the AR app
    4. Interacting with the AR app - implementing Virtual Buttons
    I'll add hyperlinks to the above index as I publish the articles.


    Developing an AR application, where to begin?

    In deciding to develop my very first AR application, I decided to go mobile - on the Android platform (iOS is on the drawing board but not for some time yet...). After a month of research and writing demo programs to learn the basics of various things I came to realise I had a bunch of information in my head gathered from many sources in a rather haphazard manner. I decided to take a little break from coding and reading, and instead spend some time gathering my thoughts and experiences into a somewhat organized structure by writing a blog.


    I intend to go forward writing articles in the order of the different problems and challenges I have faced so far and will no doubt face in the future. I'm hoping the end result of this order will serve as a kind of learning guide to other people who are in the same boat as me - wanting to program augmented reality applications but  not sure how or where to begin. For others who are already experienced in various areas, perhaps there will be at least one article, or maybe even a small section of info which one of your Google searches will stumble upon and help you solve whatever problem it is that is plaguing you at that point in time.


    Programming AR on Android - A Beginner's Guide


    Why Android?


    If you're like me, you know Java, have a rusty knowledge of C (which I haven't really touched in about 5 years) and a desire to learn about C++. I do know others like Perl, PHP etc but they aren't going to be much help here are they ;-)


    Considering the fact I haven't got past writing a simple GUI'ed Hello World equivalent in Objective C for iPhones, I decided to spend my limited time on hardcore AR stuff rather than learning a new language with weird syntax at the same time. So for me, the choice of wanting to develop a smartphone application that was AR based was simple - do it on Android.


    How to write Android apps?


    Re-inventing the wheel by covering the fundamentals of Android programming is outside the scope of this blog. I haven't been programming for Android that long myself, having only written a few incomplete prototype apps that focused on bluetooth, GPS and Google Map functionality. 


    Trust me when I say it doesn't take long to learn the basics to start writing Android apps and then finding out the rest Googling as ya go. If you already know what Activities and Intentions are (in the context of the Android SDK framework, not the evil politically motivated kind!) then I say go for it; starting Android AR hacking right away!


    For those of you who really are new to writing Android apps (but at least know rudimentary Java) then here's a good starting point: Android tutorials with sample code

    OK, enough chitter chatter, I wanna start already!


    From now on I'll try and keep my posts concise and focus on only a specific topic or related topics per post. Lessons will start in the next blog posting.

    First blog entry - what is AR (augmented reality?)

    So this is my first post. I started this blog with the intention of sharing my knowledge and experiences in the fields of smartphone programming (focusing on Android and iPhone). I live in Asia so every now and then I might throw in the odd post about life here too.


    What is Augmented Reality (AR)?


    During the past month I've been researching the various technologies available in the area of mobile augmented reality. Some people also called it mixed reality. For those of you unaware as to what augmented reality is, a simple video can illustrate the concept.




    Unlike virtual reality where the entire environment is usually computer generated, imagine the real world around which you see normally everyday but overlaid (augmented) with digital information such as videos or computer graphics. The classic example is watching the view from your webcam on your computer screen and seeing little animated 3D objects pop out, seemingly as if they were there in that physical environment with you.


    There are a number of various displays upon which an augmented reality system can be displayed; the most common is a normal computer screen, the next would be handheld devices like smartphones and tablets. Then you get onto less popular things like HUDs (heads up displays), wall projectors and goggles etc.


    Vuzix Wrap 920AR goggles with USB cameras
    and little displays on each eyepiece capable of
    producing a stereoscopic (think, depth perception)
    affect for a 3D feel.


    The classic AR system uses "trackers" (also called markers or patterns) like the following printed on a piece of paper to track the camera's relative position (distance, height, angle) to the printed marker.




    Most AR-powered applications will usually utilise some library which provides various functionality. Think of this library as the engine which does the nitty gritty work such as using some pretty clever math and computer vision algorithms to scan what's coming through the camera and see if it can recognize any markers that it's been told to look for. Once it's found a marker it's aware of it, then it does the fancy math of calculating where the camera is relatively to the marker. Armed with that information we can then go about drawing our pretty 3D animations (or whatever) over top of the environment in the correct perspective and scale.


    The two AR libraries that I have played with a little is ARToolKit and QCAR. ARToolKit originally came out of the HIT Lab at the University of Washington. I first encountered ARToolKit during my university years at the University of Canterbury in New Zealand which has a sister HIT Lab NZ. QCAR was recently released by Qualcomm and is what I've been focusing on recently.


    Your AR application will usually sit on top of the library, using what functionality you need and then integrating whatever content you want to overlay.


    Besides trackers, location based services such as GPS offer another way to position the digital content we want to overlay onto the world around us. Think of pointing your camera in some random direction at some random objects (buildings etc.) and then information instantly popping up on your screen about what's there. Checkout Wikitude as a cool example of this type of AR application.


    I had an interest in AR for years but never really took the opportunity to get involved with it until recently. Although the hardware used in some AR systems today is bulky and too cumbersome for many mainstream products aimed at the average end-user consumer, it's a field I strongly believe holds a very interesting and bright future. It has applications in all kinds of fields from entertainment, medicine, building/architecture/landscaping, engineering, education, collaboration for multiple geo-located meetings to even military. Don't get me wrong when I say this - as there are many AR products already out there on the market - but it seems this field is still much in its R&D phase at institutions, corporations and universities around the world. My dream is one day that it becomes as ubiquitous with the consumer markets like smartphones themselves have achieved over the past few recent years.


    In this post I haven't even really done justice indescribing what AR really is and what can be done with it. My aim today was pretty simple - learn the basics of writing posts on blogger.com and wet your appetite for things to come with a simple introduction. For more info just Google "augmented reality" and read, read, read! In the coming days, weeks, months, (years?) ahead I'll be writing some more useful posts about my actual adventures trying to develop AR applications.


    Stay tuned!