Adobe Flash provides a powerful suite of multimedia tools for rapid user interface prototyping. Its flexible timeline and movie clip concepts allow designers to quickly capture and test complex interactions. Its ECMAScript-based ActionScript language extends the graphical workflow, allowing for procedural and event-driven designs that would be difficult to execute with Flash’s timeline tools alone. ActionScript is a mature, Object-Oriented language that is even capable of tasks such as basic video processing. However, Flash is far from a complete application development platform. By design, a Flash movie is trapped inside the Flash player—there aren’t many ways in which it can interact with your hardware. Flash is intended for the web and, as such, it has all of the security-related limitations one would expect of an environment that hosts potentially malicious software. Because Flash is otherwise flexible and ubiquitous, one might wish to circumvent these limitations so that Flash may act as a rich UI layer for hardware demonstrations. Luckily, there are several ways of doing so.

Let’s briefly look at what Flash movies are and how they run. A Flash movie is a collection of multimedia resources (shapes, photos, movies, sounds) and scripting instructions that are packed into a binary file. Flash movies generally have a file extension of “.SWF”. A Flash movie is created in an authoring environment—usually Adobe Flash; however, because Adobe has made the Flash movie format specification publicly available, third-party programs are also capable of generating Flash movies.

Flash Movies are loaded into the Flash player, which is provided by Adobe. The Adobe player is closed-source; therefore, while some 3rd party Flash players exist for embedded computing platforms, the Adobe player is the “standard” player and the most widely used. The standard Flash player is distributed as a platform-specific plugin, such as an ActiveX control, so that it can be embedded in documents such as web pages. Because it is not a complete application, the Flash player plugin runs in a host application such as Internet Explorer.

Adobe also distributes a standalone Flash player, which is a complete application that provides movie debugging functionality as well as standalone operation.

Now, imagine you are developing a novel input device and would like to quickly demonstrate its functionality. Let’s say the device is a USB knob. It functions as a USB Human Input Device (HID), but it does not appear to Windows as either a mouse or a keyboard. As you may know, packets from a USB HID that is not a mouse or a keyboard (and a few other specific types of devices) don’t go anywhere by default in Windows or Mac OS X. These HID packets are just dropped unless an application has connected to the HID and registered to receive its packets. As of version 9, Adobe Flash offers no mechanism for enumerating and connecting to HIDs. Adobe Flash does offer a mechanism for capturing mouse and keyboard events. Thus this problem is commonly solved by writing a small application (shown as an olive box in the illustration below) that connects to the HID and translates the device’s data into emulated keyboard or mouse movement. In Windows, this application could sit in the Taskbar Notification Area and perform the emulation via SendInput() or SendMessage(). USB knob rotations could, in this way, be translated to mouse movement or keyboard button presses for the Flash demonstration.

This perfectly acceptable solution works well for HID-type applications. Because working with mouse and keyboard input is a common task in Flash, it is well documented and generally well understood by Flash developers. However, there are a few drawbacks worth mentioning:

• On Windows, If SendInput() is used to perform the emulation, only the foreground application receives the emulated keyboard/mouse input. This may be undesirable as the Flash player must then be in the foreground to receive the input. It is possible to send messages directly to the Flash host window with SendMessage() instead, but this requires more complex programming.
• The keyboard/mouse inputs that you reserve for your device can no longer be used for standard mouse/keyboard entry. For example, if we chose to send the keyboard key “R” for a right rotation of our USB knob, and “L” for a left rotation, a text field would be flooded with these letters if the user rotated the knob while the text field had focus. In this case, the demonstration application probably can’t require keyboard entry, or the user must be warned not to rotate the knob while he/she is entering text.
• This technique only allows us to send data into a Flash application. If our USB knob had an output such as an LED or force feedback, we would still have no way to control it from Flash.

Since version 5, Adobe Flash has had the XMLSocket class, which has traditionally served as the one flexible mechanism for getting data both into and out of a Flash movie over TCP/IP. This class has the one minor drawback of requiring that all data be converted to a null-terminated string for transport. In Flash 9, Adobe introduced the Socket class, which allows for binary or text data of arbitrary formatting. Both of these classes are quite suitable for our bidirectional HID demonstration of a USB knob with an LED.

To use the XMLSocket or Socket, a developer must write a small server application that connects to the device and opens a listening socket on a port greater than or equal to 1024, but otherwise of the developer’s choosing. The flash movie connects to this socket. Once connected, the movie may send data to the application and the application may send data to the movie. An XMLSocket is best suited for sending data as XML, a flexible, but relatively verbose markup language. To reduce overhead, a Socket can be used to send binary data. Other than requiring that it be well-formed and null-terminated, the XMLSocket does not make assumptions about the actual XML string transferred, so both the XMLSocket and the Socket are rather flexible regarding data semantics. The designer is free to choose a protocol that best suits the demonstration’s needs.

A programming language with good socket and XML classes, such as Microsoft’s C#, facilitates development of the server application. Both the server application and the Flash movie may reside on the same computer. Flash does impose some security restrictions on how a Flash movie connects to a server; however, these are generally easy to work with in a demonstration setting. Compared to the emulated mouse/keyboard input technique, the most significant disadvantage to this technique is that it requires advanced programming on both the application and Flash movie side. Using the XMLSocket or Socket requires a Flash developer who is comfortable with network programming concepts and the ActionScript language. It will also require closer communication between the application developer and the Flash developer during protocol specification and testing phases.

The emulated input and XMLSocket/Socket techniques both assume that the Flash movie runs in a standard Flash player host such as a web browser or Adobe’s standalone Flash player. A third, more powerful, option is to embed the Flash player into a custom host application. This option allows for very flexible, bidirectional communication between a Flash movie and the application while eliminating the complexities associated with network programming. Additionally, it allows the application to control aspects of the Flash movie’s presentation that are not otherwise easily accessible. For example, the custom host application could maximize, minimize, show, or hide the Flash movie as desired.

Creating this custom host application in Windows with Microsoft C# is as simple as dropping the “ShockwaveFlash” ActiveX control onto a C# form. Doing this will create a reference to “AxShockwaveFlashObjects” in your project. This reference contains the “AxShockwaveFlash” interface. Taking a peek at this interface in the Visual Studio Object Browser, we can see that it provides many interesting methods, members, and events.

Now, Adobe does not support embedding the Flash player ActiveX control in this manner, so they do not document this interface. However, much of it is self-explanatory, and the rest is documented in smatterings across the web. For example, the Object Browser tells us the following about the “Movie” member: “public virtual string Movie { set; get; }”. From this, we might surmise that setting this member to a string representing a path to a .SWF file might cause the player to load this file. This is, in fact, the case. So, by looking at the methods and members, one should be able to figure out how to perform simple tasks such as loading, starting, and stopping a movie.

Getting data into and out of a Flash movie inside an embedded Flash player is accomplished via the ExternalInterface class. This class is supported by Adobe and is well documented. When a Flash player is embedded inside a web browser, this class can be used to communicate with JavaScript running inside the browser. When we embed the Flash player in our own application, we use the ExternalInterface class to communicate with our C# application code.

To receive data from Flash, our application must handle the FlashCall event. We receive notifications of this event whenever a flash movie invokes the ExternalInterface.call() method. The passed “functionName” and any arguments are handed to our FlashCall handler as very simple XML. To send data to flash, we invoke the AxShockwaveFlash method CallFunction(). This takes a string, which is the same simple XML format that Flash hands to us in the FlashCall handler. It looks like this:

Error: Could not open UnboxingFlash/FlashCallFunctionXML.xml

In this example, the Flash function “MyFunction()” would be invoked with a single argument, “MyString”.

A Flash movie may send data to our application by invoking the ExternalInterface.call() method with any “functionName” it desires. There is no built-in mechanism to translate the data that arrives in the FlashCall event into a C# function call. Therefore, our C# application must parse the resulting XML, identify functions it cares about, and execute them. However, when our application invokes CallFunction(), Flash does translate this directly into a Flash ActionScript function call. To do this, Flash requires that we register any ActionScript functions we wish to expose to our C# application via the ExternalInterface.addCallback() method. If we attempt to invoke a function that has not been registered, an exception will be thrown.

The C# application is free to use system resources, so it can connect to devices, access the network, write to files, etc. Thus, it’s possible to expose an API to the Flash player that’s not normally available in a standard host such as a web browser. Flash movies that are written for this API could invoke its functions via the ExternalInterface to communicate with devices directly.

Though it may sound complex due to the XML shuffling, this technique can be simple and powerful once C# code is written to parse and generate the required XML. This work is facilitated by C#’s included XML-handling classes. From a Flash developer’s perspective, it is quite easy to add the required functionality to a Flash movie. The Flash player keeps the developer from having to deal with XML by converting it directly to simple ActionScript calls.

One drawback to this technique is the inability to redistribute Flash player with your application due to Adobe’s licensing restrictions. If you run your application on a computer that does not have the proper version of Flash installed, or does not have Flash installed at all, then your application will not work properly. In a demonstration environment, it’s usually easy enough to ensure that all computers are properly configured. However, if you’ll be distributing the application to a wider audience, you’ll have to write extra code that checks the Flash version and possibly prompts the user to upgrade Flash. Alternatively, commercial wrappers such as Multidmedia’s Zinc™ come with a Flash Run Time Distribution License that allows for the redistribution of the Flash player. Some of these wrappers also provide file system and basic device connectivity, but they can be more limiting or have steeper learning curves than a simple C# application.

With the input emulation, XMLSocket/Socket, and External API techniques for getting data into & out of Flash, developers can quickly produce compelling UI demonstrations for hardware devices. The technique selection is driven by the demonstration’s data requirements and the skill set of the development team.

Further Reading:

• Swocket – an XML Socket server framework implemented in Python
• Fun with C# and the Flash Player 8 External API

Tags: Adobe, C#, Flash, HID, software, USB

This entry was posted on Friday, June 8th, 2007 at 1:53 pm and is filed under MindTribe Tech. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.