kReese.net

The software being developed, referred to as “Mocoa”, is being developed primarily as an accessibility tool for computing. Mocoa will work in a very similar manner as a mouse by allowing control of software through use of hand motions. The original design will be implemented to work within select games, such as: “Pong” to show Y-coordinate control, “Archanoid” to show X-coordinate Control, and “Snake” to show X- and Y- coordinate hot-spot controls.

Section two will discuss the Overview of the System being developed. It will provide a general description of the system including its functionality and other matters related to the overall system. Section three will discuss the Design Considerations for the system. It will discuss general assumptions, design constraints, goals, and development methodology. Section four will discuss architectural strategies. It will describe all design decisions and strategies that affect the overall organization or the system and its higher level structures. Section five will provide a high-level overview of the functionality and responsibilities of the system. Section six will discuss policies and tactics which will not have a significant impact on the overall organization of the system and its higher-level structures. The document will end with section seven, which will discuss the components mentioned in Section Five with a much more detailed discussion.

For more information on specific requirements for this software, which are not taken into account in the “Design Considerations” section, please see the included “Requirements Document”. For more information on the time-line and how each milestone will be met, please see the included “Planning Document”.

When a hand is detected on an image, a red box will be drawn around (or rather near) the hand. This rectangle will contain x- and y- coordinates of each corner, as well as the height and width of the rectangle. The center of this rectangle will be used as the (x, y) coordinates for controlling the application. Finding the X- or Y- coordinate will use one of these simple equations:

x = (rect.x – rect.width())/2

y = (rect.y – rect.height())/2

In the development of this application, a calibration tool will need to be devised to help in control of each individual game. X- and Y- coordinates will be mapped based on the received image. The maximum and minimum X- coordinate and Y- coordinate that a user can achieve will need to be calibrated to allow for best control of the secondary application. By using calibration we can also achieve a 3rd dimensional coordinate (z- ) by comparing the ratio of the original rectangle to the new rectangle. Though finding the Z- coordinate can be achieved, the math is a bit more complex:

Z = rect_n.height() / rect₀.height()

Z > 1 –> Positive Z coordinate

Z = 1 –> No determinable Z change

Z < 1 –> Negative Z coordinate

The Z coordinate will be implemented though initially no application will be developed to show the Z concept (future work).

Using the coordinates recieved from the application, we should be able to implement these coordinates as a sort of mouse in each of the three games described in this section.

Though Cross-Platform compatibility would be a plus, for the initial project, we will stick to a single platform before working on Cross-Platform compatibility. Though this has been decided prior to the development of the application, choosing a framework to develop GUI application for multiple platforms (Windows, Mac OS X, and Linux) would be a consideration worth looking into prior to beginning development (though not essential to the success).

Determining our platform environment must also be considered. We must take into consideration the language we are going to be using, as well as a compiler for that language. If an Intergrated Development Environment is desired, we must also take into consideration an IDE that integrates with the desired compiler and works in the desired platform.

After determining the framework or API to be used for development of GUI applications inside of our development environment, we must then turn our attention to necessary hardware for grabbing an image. This of course will be taken by a webcam, but we must consider which webcams will be compatible with the platform in which we will be working. Other issues arise in the programming of how we will get an image from the camera programatically.

Our final decision for determining required hardware and/or programming language libraries is determining a way to programatically detect an object. This can be acheived in multiple ways whether by programming one of the many object detection algorithm or by using a framework or library which already has the object detection completed.

Looking into the software design considerations there is potential for very heavy usage of the processor. Refreshing a video’s frame from the webcam will require heavy usage of the processor. Many of the detection algorithms are also very processor intensive. Knowing this we should also consider for the possibility of this becoming a multithreaded application to take advantage of the most recent Dual and Quad Core processors from both Intel and AMD. However, more information on this will be required as the development progresses.

Since there is an issue of Mocoa being very processor intensive, performance will be a major issue in the development process as well. Though many applications can tolerate a degree of inefficiency, Mocoa will not be able to tolerate much inefficiency. C++ is designed to be both usable and developed for efficiency.

For the development Environment it has been decided that Linux (more specifically Ubuntu) will offer the best development Environment. Linux offers free tools for the development of applications. With Linux, we will be using the GNU GCC compiler for development of Mocoa. This will be used inside of the Eclipse IDE with the CDT Development toolkit. This Toolkit offers both an integrated debugger and compiler, and any C/C++ library can easily be integrated inside of the IDE.

There are many C++ GUI frameworks and libraries that can be chosen from. In Linux, the primary frameworks are GTK+ and QT. GTK+ is primarily a C library but is not cross-platform compatible. QT 4.3 is a framework developed for C++ and with cross-platform compatibility in mind. Because QT 4.3 works in most Operating Systems, Mocoa will be developed with the QT API for future development for all operating systems.

We then turn our attention to finding both a library and a webcam that will work in Linux for development. For this portion Mocoa will initially be developed with a Creative Live! Notebook Pro webcam which works in Linux using the ov51x-jpeg drivers in Linux. Though this will be the most tested webcam, other webcams using the V4L and V4L2 drivers will also be tested. Accessing these drivers and getting the image from the webcams is another issue. However OpenCV, a library originally developed by Intel in 1999 and now actively developed by Willow Garage, offers the tools for grabbing an image from a webcam through use of the Operating System drivers. OpenCV offers the tools for developing object recognition in software by using the Viola-Jones algorithm for Object Detection.

If Multithreading becomes necessary, a C++ library developed for the purpose of multithreading applications called OpenMP offers the tools necessary for forking threads. If it becomes necessary that Mocoa requires multiple cores to operate smoothly, OpenMP will be used for this purpose.

MainWindow – This is where the GUI for Mocoa is set up. This GUI contains “slots” to receive signals for individual QWidgets for each Game. It also contains “slots” for receiving (x, y) coordinates from the Camera Widget Object. It will emit a signal to the game.

camWidget – This sets up the widget for capturing the image from the webcam. The widget will capture a frame every 0.1 seconds. This will receive coordinates from ObjectDetector and will transmit these coordinates to MainWindow where they will be sent to secondary “applications”

ObjectDetector – This class will look for a specified object (through Haar-like classifiers). It will then draw a red box around the object and calculate the center coordinates of the box. This will be accessed through accessors in the code by the Camera Widget.

Pong/Archanoid/Snake – These classes are the receivers of the (x, y) coordinates. These applications will ultimately be controlled by the change in the (x, y) coordinates.

• File
  • Pong
  • Archanoid
  • Snake
  • Quit
• Edit
  • Calibrate
• Help
  • Content
  • About

Screenshot showing Mocoa's image capture

The Main Window will also continuously update the Status Bar on the location of the hand.

The Central widget is derived from the CamWidget class which captures a frame using the CvCaptureFrame function in OpenCV. This is then refreshed every 0.1 seconds. This image is passed through Object Detector where it will use a Haar Classifier XML file to detect the specified object. OpenCV uses the “cvHaarDetectObjects” function to detect the object.

The coordinates received from the ObjectDetector class are filtered back up to the MainWindow where they are continuously updated. This is the information which is used for motion controls. When one of the three games are started the Main Window will begin to transmit signals to the QWidget where the (x, y) coordinates sent will be used to control the game.

Each game will pop up in their own window and will contain only the game (see a sample below).

Screenshot showing a simple game of Pong in Mocoa

This Motion Control of each application will have a calibration tool for determining the maximum, minimum, and potentially relative 0 coordinate for each game. This will allow for a much more responsive game. The Content and About Page will contain information on how to use and how to develop new games for the application.