Chroma Keying (transparent background) with Vuforia AR Video Playback

It is possible to make augmented reality video playback with the Vuforia platform just that little bit more realistic by chroma keying the video. Chroma keying, aka the green screen effect, is an old trick used in the television industry of putting a solid colour background behind the person being filmed so they can replace that one colour in post-production with a completely different scene. Obviously this requires the person in the foreground to avoid wearing anything close to that colour otherwise they'll appear patchy/transparent in places.

I managed to come up with a basic way to implement this with the video playback functionality offered in the Vuforia SDK. Here's a snippet of the OpenGL ES shader code that I used:

static const char videoPlaybackFragmentShader[] =
  "#extension GL_OES_EGL_image_external : require \n"
  "precision mediump float; \n"
  "uniform samplerExternalOES texSamplerOES; \n"
  "   varying vec2 texCoord;\n"
  "   varying vec2 texdim0;\n"
  "   void main()\n\n"
  "   {\n"
  "       vec3 keying_color = vec3(%f, %f, %f);\n"
  "       float thresh = 0.8; // [0, 1.732]\n"
  "       float slope = 0.2; // [0, 1]\n"
  "       vec3 input_color = texture2D(texSamplerOES, texCoord).rgb;\n"
  "       float d = abs(length(abs(keying_color.rgb - input_color.rgb)));\n"
  "       float edge0 = thresh * (1.0 - slope);\n"
  "       float alpha = smoothstep(edge0, thresh, d);\n"
  "       gl_FragColor = vec4(input_color, alpha);\n"
  "   }";

UPDATE: I forgot to add I also needed to run these commands in order for the fragment shader above to work:

glDepthFunc(GL_LEQUAL);
glEnable(GL_BLEND);
glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);

UPDATE #2: 2013/04/04:

Some people were reporting they had problems with their video content turning black. Re-reading this article I realised I'd missed out the vertex shader code as well so here it is. If I recall rightly, the variable "texCoord" that is used in the fragment shader must have the same name as in the vertex shader. Using the sample Video Playback app with the Vuforia SDK, if there are any problems with compiling your shader source code then there'll be some errors printed in LogCat that you can use to help debug.

static const char* videoPlaybackVertexShader =
    "attribute vec4 vertexPosition; \n"
    "attribute vec4 vertexNormal; \n"
    "attribute vec2 vertexTexCoord; \n"
    "varying vec2 texCoord; \n"
    "varying vec4 normal; \n"
    "uniform mat4 modelViewProjectionMatrix; \n"
    "void main() \n"
    "{ \n"
    "   gl_Position = modelViewProjectionMatrix * vertexPosition; \n"
    "   normal = vertexNormal; \n"
    "   texCoord = vertexTexCoord; \n"
    "} \n";

In the VideoPlayback sample code, then in VideoPlayback.cpp, inside the renderFrame() function I inserted these 3 lines above all the calls to SampleUtils::scalePoseMatrix. For those of you who have studied the source code of the Video Playback project that can be downloaded with the Vuforia SDK you can see this is a modified version the source code for the fragment shader for the video frames. What it does is, for any colour that is close to the specified colour that is processed by the fragment shader (any pixel to render on the screen from rasterization), that colour will be made transparent by modifying its alpha channel. The same code can be used for the keyframe's fragment shader too. Notice I have 3 %fs. These, in order, are the RGB values of the colour to chroma key - i.e. to make transparent. I used sprintf to replace the string with actual float values. Note the range of these numbers is 0.0-1.0. They're float numbers, not the integer range you often see of 0-255.
Unfortunately I don't have a project I can share this time as the source code I originally implemented this in is mixed in with other code I'm not able to release publicly.

To be a little more adventurous and dynamic you could consider implementing a function to dynamically change the value of the colour that is made transparent on-the-fly using a colour picker dialog or some other input method. Other input methods could be a text input field for the 0x RGB value or a different kind of colour picker like the ones often found in paint programs where the user touches a part of the screen and the colour value of that pixel is read in as the input. This would be handy so users could just touch the area of the background they want to remove and it would automatically detect the colour.