Kevin Hoyt

MJPEG Component for Flex 4

Generally, when we think about Phidgets, we think of plug-and-play hardware via USB. Recently however, a new product was released called the Phidget SBC (Single Board Computer). The Phidget SBC offers direct network access via ethernet (cable included) and USB wireless card. It also supports streaming video from a USB web camera in MJPEG format. Although Flash Player doesn’t natively support MJPEG, that doesn’t mean it can’t support it; and here’s how.

Phidget SBC

First, a little more about the Phidget SBC. The Phidget SBC is designed for standalone deployment of your Phidgets projects. The requirement to be tethered to a main computer via USB is removed. The Phidget SBC itself is an 8/8/8 Interface Kit and a USB hub, which means you can plug your existing Phidget sensors and controls directly into the board.

It runs a Linux operating system and even has support for C and Java. It is administered via a web-based interface, and runs the Phidgets web service driver so you can connect to it just like any other Phidget project. The USB hub is nice for sensors like the Phidget RFID which are USB already, but it also gives extensibility to other devices. As mentioned two of these devices are a USB wireless card, and a USB web camera.

While it is clearly compelling to be able to deploy your Phidgets project in the wild, and communicate to it via wireless, it is the web camera that fascinated me the most. Getting remote sensor data is one thing, but actually seeing what is going on adds a whole new range of possibilities. You can get standalone still images on demand, but I was most interested in streaming video from the web camera. The Phidgets SBC supports this, but through MJPEG, which isn’t natively support by Flash Player.

MJPEG Overview

MJPEG is an interesting format. It is essentially a never-ending stream of JPEG images. One could certainly debate the efficiency of sending full JPEG image across the network at twenty (20) frames per second, but it does make it fairly open to other technologies. In this case, Flash Player does support JPEG images, so all we have to do is figure out where the JPEG images in the stream start and stop, strip them out, and then load them.

I know, it sounds hard, but JPEG has nice markers for the start and end of the file, which makes them easy to locate. If you Google on Flex and MJPEG you will find earlier works on this front by Doug Hughes from Alagad. Doug has a Flex component for this, and I’ve seen the source extracted for non-Flex usage as well. I’d encourage you to check out that content on the subject.

One of the problems of MJPEG in a Flash world is that you have a high potential for flickering as you swap the images coming across the stream. Doug’s original Flex component solved this by having two image placeholders available and then swapping them with each incoming frame. Doug’s work was for Flex 3, and no longer works in Flex 4, so I spent a few hours updating it for the changes in the framework.

MJPEG in Flex 4

The first change I made was in moving from a Canvas subclass to a Group. You can think of a Group as a Canvas in Flex 3. It is a low-level container for components. The only components the Group holds are two SWFLoader instances. The SWFLoader is important here because BitmapImage is designed to load images from a URL, not from local bytes. You also cannot use Bitmap because it is not a UIComponent. By comparison, SWFLoader is a UIComponent subclass, and it supports showing an image from a ByteArray instace.

An MJPEG viewer will need to know where the stream is coming from in order to display it. Doug took the time to expose those parts as component properties. In this fashion, the properties were exposed as attributes on the component. My problem with this was that I needed to read the stream location from a file on disk, and couldn’t rely on starting the stream straight away. To solve this, I still expose the stream location properties, but I don’t start trying to play the stream until you tell me to via a Webcam.stream() call.

More work could be done to make the component support both use-cases, but that was beyond the scope of what I wanted to accomplish. I did however add a “last” property. This property holds the last image parsed from the stream. The reason for this is that on occasion, I wanted to save the pictures to disk. Rather than incur the overhead of the old BitmapData-to-JPEG trick, I figured I’d just shave off the last incoming image, already encoded, and let you grab it at your leisure.

When I had all of this in place, I was still suffering the problem of a flickering as the images were swapped. After a little digging, it occurred to me that the flicker was happening because I swapped the images immediately after telling the SWLoaders to load the bytes – when they hadn’t yet finished loading or painting the image. The solution to this was to listen for the COMPLETE event on the SWFLoader and then swap the images out.

Conclusion

The result is a pretty tight little component that you can use to display, and capture, MJPEG video streams from web cameras – specifically the Phidget SBC in my case.

I want to be clear here that I am not the main innovator – this work is an addition to the existing work to figure out how to access and parse MJPEG streams. I simply molded the source to the Flex 4 framework. What follows is a small example of how to use the component. You can also download the source. If you do leverage this code in a project, please let me know! Perhaps one day we can get it added to the Flex framework itself.

<?xml version="1.0" encoding="utf-8"?>
<s:WindowedApplication 
  title="MJPEG"
  showStatusBar="false"
  width="800"
  height="600"
  creationComplete="doCreationComplete( event )"
  xmlns:fx="http://ns.adobe.com/mxml/2009" 
  xmlns:s="library://ns.adobe.com/flex/spark" 
  xmlns:mx="library://ns.adobe.com/flex/mx" 
  xmlns:local="*">
 
  <fx:Script>
    <![CDATA[
      public static const PORT:Number = 81;
      public static const FILE:String = "/?action=stream";			
      public static const HOST:String = "192.168.15.108";
 
      protected function doCreationComplete( event:Event ):void
      {
        webcam.stream( HOST, PORT, FILE );
      }
    ]]>
  </fx:Script>
 
  <local:Webcam 
    id="webcam" 
    horizontalCenter="0" 
    verticalCenter="0"/>
 
</s:WindowedApplication>