Kevin Hoyt

Ryan Stewart and I are crazy about GPS and geospatial information. We both actually carry a Trackstick device wherever we travel which records our adventures. The data file can be accessed via USB as any number of formats, but it is usually XML-oriented. Of course we’re both building applications that use this data, which is when I ran into the problem of waiting for a server to process several megabytes of XML data. Using Flex I was able to monitor the long-running transaction without unnecessarily delaying the responsiveness of the user interface.

The first step I took was simply to upload the data. Depending on how full I let the Trackstick get, this is generally in the range of one to two megabytes. Of course I wanted to let the user know how far along that upload was, so I added a Flex ProgressBar to my applications, registered for the appropriate events, and started displaying the bytes as they were sent.

public function doComplete( event:Event ):void
{
    file = new FileReference();
    file.addEventListener( Event.SELECT, doSelect );
    file.addEventListener( ProgressEvent.PROGRESS, doProgress );
    file.addEventListener( DataEvent.UPLOAD_COMPLETE_DATA, doData );
}

public function doSelect( event:Event ):void
{
    btnUpload.enabled = false;

    barProgress.setProgress( 0, file.size );
    barProgress.label = "Uploading file ... %3%%";
    barProgress.visible = true;

    file.upload( new URLRequest( "/messaging/gpx/upload.cfm" ), "gpxdata", false );
}

public function doProgress( event:ProgressEvent ):void
{
    barProgress.setProgress( event.bytesLoaded, file.size );
}

public function doData( event:DataEvent ):void
{
    var result:XML = new XML( StringUtil.trim( event.data ) );

    records = new Number( result );

    barProgress.source = null;
    barProgress.setProgress( 0, records );
    barProgress.label = "Processing data ... %3%%";
    barProgress.visible = true;
}

Great, so the file is on the server, right? The next problem I encountered was that I originally had a single thread process the upload, and then the processing of the XML file into the database. Given that a one megabyte GPX file contains about 2,300 records, this obviously took some time. The experience on the client was that the application had hung once the file had been upload. This wasn’t true of course; the server was simply trying to cram all that information into the database.

It was pretty clear to me that I needed to spin off a thread to handle this process so the user could get a response from the upload and know that the application wasn’t hung or broken. Normally threading is a pretty complex subject, but thanks to the CFTHREAD feature of ColdFusion 8, I simply added a couple tags, and a name for the process. When the file arrives, ColdFusion quickly inspects the document, and determines how many records are in the file, starts the thread to insert the data into the database, and then returns a result containing what will eventually be the total record count.

<cfset source = GetDirectoryFromPath( GetTemplatePath() ) & "data/" />

<cffile action="upload" destination="#source#" filefield="gpxdata" nameconflict="makeunique" />

<cfinclude template="udf/DateConvertISO8601.cfm" />

<cfset source = CFFILE.ServerDirectory & "/" & CFFILE.ServerFileName & "." & CFFILE.ServerFileExt />
<cffile action="read" file="#source#" variable="gpx" />
<cfset doc = XMLParse( gpx ) />

<records><cfoutput>#ArrayLen( doc.gpx.wpt )#</cfoutput></records>

<cfthread name="insert">

    <cfset dao = CreateObject( "component", "kevin.gpx.GPXDAO" ) />

    <cfloop index="sub" from="1" to="#ArrayLen( doc.gpx.wpt )#" step="1">

        <cfset gpx = CreateObject( "component", "kevin.gpx.GPX" ) />

        <cfset gpx.latitude = doc.gpx.wpt[sub].XmlAttributes.lat />
        <cfset gpx.longitude = doc.gpx.wpt[sub].XmlAttributes.lon />
        <cfset gpx.elevation = doc.gpx.wpt[sub].ele.XmlText />
        <cfset gpx.stamp = DateConvertISO8601( doc.gpx.wpt[sub].time.XmlText, 0 ) />
        <cfset gpx.variation = doc.gpx.wpt[sub].magvar.XmlText />
        <cfset gpx.name = doc.gpx.wpt[sub].name.XmlText />
        <cfset gpx.description = doc.gpx.wpt[sub].desc.XmlText />
        <cfset gpx.source = doc.gpx.wpt[sub].src.XmlText />
        <cfset gpx.satellites = doc.gpx.wpt[sub].sat.XmlText />

        <cfset id = dao.create( gpx ) />

    </cfloop>

</cfthread>

Now the user uploaded the file, saw the progress of that upload, and quickly got a response telling them that the server was processing “n” number of records. This was better, but still incomplete. When should the user expect their sent data to be returned to them for visualization? What I really wanted to do was show the user just how many records the database had actually processed. The two options here are to incrementally poll the server to see where the thread is at, or to push the processing information to the client. Since data push is easy with Flex and LiveCycle DS, I chose the data push route.

I want to be perfectly clear here that this isn’t a ColdFusion specific article. The concept of having a separate thread process the data can be accomplished using any number of languages. LiveCycle DS is easily exposed to ColdFusion and Java, so those are obvious choices. I could also use BlazeDS for the message push, but as BlazeDS doesn’t include an RTMP channel, there would be some lag in the notification process, and I wanted the interaction to be as smooth as possible.

On the server I added a new message destination, and then some code in the thread to put a message on the bus. A note here is that if you’re using ColdFusion, you’ll want to use a CF Gateway. If you’re using Java, this is a great place for JMS. While the XML data is being processed, the thread puts the current row index on the message. That in turn is returned to the client. Since we already know how many records there are in total from the file upload response, we can calculate total completion.

...

<cfthread name="insert">

    <cfset dao = CreateObject( "component", "kevin.gpx.GPXDAO" ) />

    <cfloop index="sub" from="1" to="#ArrayLen( doc.gpx.wpt )#" step="1">

        <cfset gpx = CreateObject( "component", "kevin.gpx.GPX" ) />

        ...    

        <cfset id = dao.create( gpx ) />

        <cfset cfevent = StructNew() />
        <cfset success = StructInsert( cfevent, "body", sub ) />
        <cfset response = SendGatewayMessage( "GPX Feed", cfevent ) />        

    </cfloop>

</cfthread>

...

Normally, if you’re using messaging for collaboration, such as chat, you’ll want to use a Producer instance, to send messages, and a Consumer instance to receive messages. Since we’re only really listening here, we can get away with only using a Consumer instance. When a message arrives, we can catch that event, and pull the index value from the message body. A little math against the total number of records we know to be in the file, and we can then tell the user just how far along the database is in processing their data.

<mx:Consumer
    id="consumer"
    destination="gpx"
    message="doMessage( event )" />

...

public function doMessage( event:MessageEvent ):void
{
    var count:Number = event.message.body as Number;

    if( records == count )
    {
        btnUpload.enabled = true;
        barProgress.visible = false;

        svc.getList();
    } else {
        barProgress.setProgress( count, records );
    }
}

When the records have been completely processed, I then use remoting to get the data itself and display it in a DataGrid component. I use remoting here for AMF and the speed of processing the 2,300+ records. I’ve even gone as far as having a client data type that parities the server data type so that this could be easily tied into LiveCycle DS’s data management feature. With that additional step, I could keep the data in sync and even work with it offline, which is a great place to use Adobe AIR.

There’s a variety of different approaches that one could take here, but what I think is most valuable is the pattern. I like the message approach because it doesn’t try to over-extend HTTP by keeping that connection open any longer than it really needs to be. It also gives me the ability to easily differentiate between the upload of the file and the processing of the data. An alternative approach I’d like to explore in the future would be to keep that connection open, incrementally respond with the index being processed, and then compare performance and ease of implementation. In the meantime, feel free to download the source code for this version.

This entry was posted on Tuesday, June 24th, 2008 at 4:22 pm and is filed under ColdFusion, Flex, LiveCycle. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.