CS 475: Senior Project

Requirements Document

Part I – Application Overview

The software being developed, referred to as “Mocoa”, is being developed primarily as an accessibility tool for computing. Mocoa will work in a very similar manner as a mouse by allowing control of software through use of hand motions. The original design will be implemented to work within select games, such as: “Pong” to show Y-coordinate control, “Archanoid” to show X- coordinate Control, and “Snake” to show X- and Y- coordinate hot-spot controls.

Objectives

Mocoa is a Proof-of-Concept Accessibility tool designed with the purpose of showing the potential usages of Hand/Motion tracking as a controller for computing applications. As mentioned, three separate game applications are being developed to show this concept in action: “Pong” to show the usage of Y- coordinate controls, “Archanoid” to show the usage of X- coordinate controls, and “Snake” to show Hot Spot Controls based on X- and Y- coordinates.

The Overall Goals of this project are to:

  1. Build a tool that can detect and track an object based on an image from a webcam.
  2. Build smaller applications (games) which show a potential use of the Motion Controls.
  3. Describe in a document how this type of tool can be used in HCI for Accessibility purposes.
Prior Examples of Motion Controls

Using a sort of webcam has been done in gaming prior to the development of Mocoa. More specifically, Sony’s Playstations 2 and 3 offer a device known as the EyeToy which does what is proposed in the development of Mocoa. Though the EyeToy is an example of this form of HCI in action for a specific purpose (gaming), but little has been done to show or explain how this type of technology could be use in all forms of computing.

Integration with Other Systems

This Proof-of-Concept application will initially developed to work in applications specifically integrated into Mocoa. This will include “Pong”, to show Y-coordinate control; “Archanoid”, to show X-coordinate Control; and “Snake” to show X- and Y- coordinate hot-spot controls. After the Proof-of-Concept application is completed (and beyond the scope of this initial project) it would ideally be well integrated into the operating system to act as a mouse.

Replacement of “Legacy” Systems

The mouse just celebrated it’s 60th anniversary in 2008. For a technology that has become such an essential tool to computing, the mouse has had very minor changes over the past 61 years of it’s life. A few extra buttons have been added, and Apple’s touchpad mouse is by far one of the most innovative takes on the mouse/touchpad. Since many laptops on the market today come with webcams, Mocoa could easily be implemented on these computers and controlled without the aid of a mouse. And the technology that Mocoa will get better as the technology in home computers continues to get better.

Part II – Functional Requirements

This section of the paper describes the Functionality that will be required through this project. As well as a description of what functionality will be delivered during each phase of the project. It describes in more detail the Functionality of the entire project in the “Statement of Functionality” section of this document.

Statement of Functionality

Mocoa is a Proof-of-Concept Accessibility tool designed with the purpose of showing the potential usages of Hand/Motion tracking as a controller for computing applications. As mentioned, three separate game applications are being developed to show this concept in action: “Pong” to show the usage of Y- coordinate controls, “Archanoid” to show the usage of X- coordinate controls, and “Snake” to show Hot Spot Controls based on X- and Y- coordinates.

When a hand is detected on an image, a red box will be drawn around (or rather near) the hand. This rectangle will contain x- and y- coordinates of each corner, as well as the height and width of the rectangle. The center of this rectangle will be used as the (x, y) coordinates for controlling the application. Finding the X- or Y- coordinate will use one of these simple equations:

x = (rect.x – rect.width())/2
y = (rect.y – rect.height())/2

In the development of this application, a calibration tool will need to be devised to help in control of each individual game. X- and Y- coordinates will be mapped based on the received image. The maximum and minimum X- coordinate and Y- coordinate that a user can achieve will need to be calibrated to allow for best control of the secondary application. By using calibration we can also achieve a 3rd dimensional coordinate (z- ) by comparing the ratio of the original rectangle to the new rectangle. Though finding the Z- coordinate can be achieved, the math is a bit more complex:

Z = rectn.height() / rect0.height()
Z > 1 –> Positive Z coordinate
Z = 1 –> No determinable Z change
Z < 1 –> Negative Z coordinate

The Z coordinate will be implemented though initially no application will be developed to show the Z concept (future work).

Using the coordinates recieved from the application, we should be able to implement these coordinates as a sort of mouse in each of the three games described in this section. The flow of the Program will be as follows:

Scope

There are 5 individual phases of this project:

  1. Phase I: Proposal. Propose one or more topics to the Senior Project Coordinator.
    Deliverable(s): Topic Proposal
  2. Phase II: Planning, Requirements, & Specification Phase. Upon acceptance of the topic proposal, begin to work on required project artifacts.
    Deliverable(s): Project Planning Document, Requirements & Specification Document, Architecture & Design Document
  3. Phase III: Implementation & Testing. Upon acceptance of the previous phase documents, begin work on implementation and testing.
    Deliverable(s): Code base, Testing Documentation (project dependent)
  4. Phase IV: Project Paper.
    Deliverable(s): Project Paper, Project Poster, Project Documentation (TBD)
  5. Phase V: Project Defense & Presentation.
    Deliverable(s): Oral Presentation & Defense

The first phase of the project was completed on January 27. For this section of the project a proposal was delivered. This document, the Design Document, and the planning document are all deliverables of the second phase of the project. The Fourth and Fifth Phases of this project require the writing and presentation of a paper discussing the potential usages of Mocoa, especially for accessibility purposes. During the third phase, the entire application will be delivered by the end of this phase of the project.

Performance

Performance will be an essential issue for the development of Mocoa. Each image will be refreshed every 0.1 seconds. This aspect of Mocoa will require heavy usage of the processing power of the computer. Alongside of this usage of the Processor, we will need to use an Object Detection algorithm for finding the user’s hand. This will require heavy processing power as well. In order to suit the performance needs of this program, it will likely require the use of a library for multithreading these processes. This will allow for the application to make use of multi-core processors.

Usability

As this is a Proof-of-Concept Application, there are no specific requirements for Usability. The Menu system on the application is small enough that the Menu should not become overly complex. It would be ideal that the application’s performance not hinder the User’s experience with the application, but other than User Experience, there is no usability requirement.

Concurrency
System Architecture of Mocoa

System Architecture of Mocoa

Part III – Behaviour Model and Validation Criteria

Proper Behaviour and Layout

Hot keys will be used for accessing specific portions of Mocoa; however, every Hot Key will contain a Menu Item as well. When a game is opened, a new window will appear containing the game. This game will be controlled solely by the location of a user’s hand. The hand will have to be within the range of the camera’s lens. Initially, the application will not attempt to avoid false positives. However, if time permits, research on how to better the Haar classifiers in the XML file will be conducted. The status bar should be continuously updated with coordinates as it detects the objects.

Overall the behaviour and responsiveness of the program will be fairly simple. The GUI layout will also be simple. The Main Window will look similar to:

Screenshot of Mocoa

Screenshot showing Mocoa's image capture

While the Game’s window is even simpler:

Screenshot of Mocoa Pong

Screenshot showing a simple game of Pong in Mocoa

Validation Criteria (Testing)

Multiple tests will be conducted during the development of the application. Upon completion of an Beta-like version, testing will move from use by myself to use by other people to test the object detection works properly in multiple situations. This will help in determining if false positives will be a major issue in Mocoa.

Various performance tests will also be conducted on Mocoa. As this system may require heavy CPU performance, it will be essential to keep track of the CPU usage of the system through various aspect of Mocoa’s development. These Performance test phases will be done after:

  • Web Camera image capture
  • Addition of Object Detector
  • Running various games (no motion controls)
  • Running various games (w/ motion controls)
  • Upon completion of the entire project

By conducting these performance tests at these locations, we will be able to better understand which portion of the application will need to be better developed for performance reasons.