CS 475: Senior Project
Testing Documentation
>> Introduction
Because this software has very little input, testing the software should be less difficult. The primary goal of this testing will be to determine whether Segmentation Faults or other runtime errors may occur that should not be accepted. From the outcomes of this testing, we can begin to prioritize issues in the software that will need to be fixed for the final product.
We will begin testing the behavior of various components of the software. The primary focus of the testing will be on the camera display widget, which is the most demanding portion of the application. Secondary focuses will be on the games, which display the capabilities of the application. We will also do some minor performance testing on Single-Core and Multi-Core processors. This will compare the processor loads on each processor core rather than focusing on the Multi-core processor as a whole.
Section Two of this document will discuss the Testing Plan for the software. This will include a discussion of various components and how they should react in certain situations. This will then lead into an examination of methods for logging and recording tests, tools and environments which will be used, and a schedule in which testing will be completed. Section Three will discuss more in depth the details of the Unit Test(s), System Test(s), and provide information on what the final products will be.
>> Testing Plan
Mocoa (Motion Control Application) will be the software that will be tested during this phase of development.
The primary focus of these tests will be on the core system (MainWindow.h/cpp, CamWidget.h/cpp, ObjectDetector.h/cpp). The secondary focus of the testing will be on the various games which have been implemented. These games will be tested at a much higher level.
System Architecture of Mocoa
Unit Test(s)
The Unit Test will be conducted solely on the primary system. Mocoa has been built to be fairly modular so that games could easily be implemented into the system without having to affect the main system. Because of this, the Primary function of the system should be unit tested.
The primary system includes the MainWindow, CamWidget, and ObjectDetector classes. However, due to the image processing in the ObjectDetector class, Unit Testing will be skipped in this class. Therefore Unit Testing will only be completed on the CamWidget and MainWindows. We will use the QT Test Framework to test various aspects of the Application from the GUI to the overall system capabilities. This will include test for receiving and transmitting SIGNALS from other aspects of the application, and simulate all events used in the application for automated testing. We will also ensure that all of the mathematics used in the program provides numbers that make sense.
Integration Testing
The Integration Testing will use a Bottom Up approach for each of the modules in Mocoa. We will begin by creating various test cases for the User Interface of the Primary Module which has successfully completed Unit Testing. Upon completion of this module’s check, we will begin integration testing for all other modules to ensure that the components do not conflict with previous components. The Integration Testing will be completed by hand with specific test cases, discussed in detail in section three.
Validation Testing
In this phase of testing, we will validate that all portions of the code as a whole work properly together. We will validate that the QT signals are transmitting the proper data and that all algorithms provide the proper data from specified output. This test phase will be very closely tied into the Unit and Integration Testing. This testing will also provide insight as to the CPU usages in both single-core and multi-core processors.
User Acceptance Testing
In this phase of the test, we will get the thoughts of everyday users and take into consideration their suggestions without sacrificing accessibility for an interesting feature. We will also view how the user uses the program and make assumptions based on this information. Mocoa has been built to show how this can be used to control multiple aspects of a computer in an accessible manner for those who cannot use a mouse. The primary goal of this project is to show potential usages in accessibility.
Various Tools will be utilized for testing. Of these we will use the GNU Debugger that is packaged with G++ (the primary compiler for this project). We will also use the QT Framework for simulated testing and to log information to files and to test various aspects of the program during Unit Testing. The log files from these tests will be the primary work product. These log files will be simple text files that will store logged information in a chronological order.
At the outcome of all testing we hope to have information that will give us the Quantity of Tests Prepared vs. the number of defects found (as well as the type of defects found) and the processor usage of the application. With this information we will begin to prioritize bugs that need to be fixed, determine if more testing will be needed (if we have a high Tests prepared to defects found ratio, we will prepare more tests), and begin to optimize code if necessary.
Testing Schedule
Gantt Chart of the Testing Schedule for Mocoa
>> Testing procedure
During this test, Mocoa (Motion Control Application) will be the overall system being tested however, for a majority of the tests,
we will focus on the core system of the application (MainWindow.h/cpp, CamWidget.h/cpp, ObjectDetector.h/cpp). All other modules will be tested, but not to the extent that the core is being tested. All of the modules will go through a High Level test.
The following sections will discuss the test cases for High Level Cases and Unit Testing Cases. This information will be used to develop the Unit tests and provide a standard testing case for High Level testing. Below is the UML diagram of Mocoa:
UML Diagram of Mocoa Classes
Unit Test Cases
The Unit test will be conducted on the core classes of Mocoa. This Core module of Mocoa is made up of the the CamWidget class, the QOpenCVWidget class, and the ObjectDetector Class.
Libraries – Mocoa requires 3 different libraries to function properly. Two of these libraries are necessary while the third library being a performance enhancement. If the third library is missing, the application will still function properly, but may function somewhat slower. Below is a table of required libraries for each of the 3 classes being tested:
| Class | Library | Library Classes |
|---|---|---|
| CamWidget | QT 4.3 | QWidget, QVBoxLayout |
| OpenCV | cv.h, highgui.h | |
| QOpenCVWidget | OpenCV | cv.h, highgui.h |
| ObjectDetector | OpenCV | cv.h, highgui.h |
Since the camWidget just handles the timer event for refreshing QOpenCVWidget every x seconds, The Unit Testing will primarily focus on testing the ObjectDetector and QOpenCVWidget classes. The Unit Tests will determine if the Object Detector class was able to successfully detect an Object based on an image which the object has been successfully detected. The Unit Test will also determine the accuracy of the image based on the (x, y) coordinates returned from the QOpenCVWidget class.
These two items are the most important portions of the application. Without being able to determine the location of an object, we cannot use the application as it was built. And the accuracy of the detection algorithm is necessary or we could get errors in location and ultimately control over the application, even if the application is calibrated. These tests will be very simple unit tests that will not capture excessive amounts of data, but will be essential for determining that future code does not adversely affect the object detection algorithm(s).
Integration Testing
The integration testing will be done manually using a specified test matrix to assure that the information being recieved is fairly consistent and works the way it is supposed to be working. Each component will be tested by running the application through GDB (GNU Debugger). Any runtime errors should be recorded in the test matrix.
Because QT is so reliant on Events, very few stubs and drivers need to be written. There are two cases that require stubs. Since the game “Snake” is not completed a stub has been put in place to catch the Snake launch event. We have the same issue with the help menu since it has not been built yet.
Test Cases & Expected Results:
See Test Matrix template in Appendix A
- Application Launch – The launch of the application should be tested first without the camera, then with the camera. No Segmentation Faults should occur. Without the camera, the image should appear as static. With the camera, you should see an image. Window should be full Size on load of the application
- Test Object Detection – Tester will hold their hand (in the sign letter “A”) and should see a box appear when the object is detected.
- Test Coordinates – When object is detected, Tester will move up and should see the y-coordinate decrease, then move down to see the y coordinate increase. Then move to the right to see the x-coordinate decrease, then move left to see the x-coordinate increase. NOTE: NO CALIBRATION SHOULD BE DONE! The second number (in the parenthesis in the status bar) should be the x coordinate divided by 100 [e.g. 55 should appear as (0.55)]
- Test Calibration – Tester will calibrate to the highest point that they can reach (typically 70) and the lowest poing that they can reach (typically 170), as well as the rightmost and leftmost points (typically 70 and 210 respectivly). Hit the calibrate button then reload the calibrate to check that the calibration remains. Hit Cancel.
- Launch Pong – Tester will Launch Pong. Press Spacebar to begin the game. Play the game for a bit using the motion controls, count the number of times the ball hits the left wall (non paddle side). Then let the ball hit the right wall (paddle side) check the score. Repeat and check the score. Click the “X” to close the window.
- Launch Archanoid – Tester will Launch Archanoid. Press Space to start the game. Play the game a bit then let the ball hit the bottom. You should see “Game Over”. Then replay until all block are gone. You should see “Victory”. Close Game.
- Launch Snake – Tester will Launch Snake. You should see a message pop up. This is a stub. Upon completion of the game: Press Space to start the game. Move hand to the top, your Snake should move up. Move hand to right, your snake should move right. Move hand to bottom, you snake should move downward. Move hand to left, your snake should move left. Repeat but during each other the repeats test that moving hand in the opposite direction [e.g. Snake move left you try to turn it right]. You should see no change in the direction. Close Game
- Launch About – You should see an About Box pop up.
- Launch Help – You should see a stub pop up. Upon completion: click around to get to various aspects of the help menu.
- Exit Application – Using the menu close the entire application. Check that process properly closes all processes and any zombies properly.
- Relaunch Application – Test that all Hotkeys work properly:
- Ctrl+1 – Launch Pong
- Ctrl+2 – Launch Archanoid
- Ctrl+3 – Launch Snake
- Ctrl+4 – Launch FiggityWidget
- Ctrl+C – Launch CalibrationWidget
- Ctrl+H – Launch Help Content
- Ctrl+A – Launch About Window
- Ctrl+Q – Quit Application
Validation Testing
For this portion of the test, we will test the application based on the requirements document. We will test again use the same test matrix used in the Test cases but rather than capturing all of the runtime occurances, the Tester will use a pass/fail to determine if that portion of the application meets the requirements for the project. We are not capturing Bugs during this Testing, we are simply doing a pass/fail test based on the requirements stated in the Requirements Document.
High-Order Testing
Recovery and Stress Tests: This application uses very little resources outside of the processor. Because of this, using the application in a system which is already consuming a large portion of the memory or Hard Drive will cause no issues (it has been tested). However, because this application is very processor intensive, the application should be tested on a system with very little CPU. This should be a Single Core system preferably with a clock cycle of less than (or equal to) 2.00GHz. The application should run very slow, but we should test that the application will not cause the operating system to crash. If the system crashes due to Mocoa, we need to fail this test and return to find a way to make the application prevent the crash of the system.
Performance Tests: Because this application is very processor intensive, we want to capture the processor usage on Single-Core, Dual-Core and Quad-Core processors. The application should see a significant increase in performance on Multi-core systems. OpenMP was used to multithread the application to take advantage of the multiple-cores in todays computers. If we do not see performance gains through processor loads due to the multiple threads, we should fail the performance and work to find solutions to boost the performance of the application.
Upon completion of all tests, we should have various work products, including logs and a filled Test Matrix. These test logs can be found in Appendix B.
>> Appendix A
| Integration Tests | Validation Tests | ||||||
|---|---|---|---|---|---|---|---|
| Test | Expected Result | ||||||
| TEST 1 | TEST 2 | TEST 3 | TEST 1 | TEST 2 | TEST 3 | ||
| Application Launch w/o Camera | Static Image | ||||||
| Application Launch w/ Camera | Image from Camera | ||||||
| Object Detection | Red Box near/around hand | ||||||
| Move Hand Up | Y-coordinate should decrement (Check Status Bar) Y-percentage should be # / 100 [e.g. 55 (0.55)] |
||||||
| Move Hand Down | Y-coordinate should increment (Check Status Bar) Y-percentage should be # / 100 [e.g. 55(0.55)] |
||||||
| Move Hand Right | X-coordinate should decrement (Check Status Bar) X-percentage should be # / 100 [e.g. 55 (0.55)] |
||||||
| Move Hand Left | X-coordinate should increment (Check Status Bar) X-coordinate should be # / 100 [e.g. 55 (0.55)] |
||||||
| Change Calibration. | Maximum X- & Y- Number 1000 Minimum X- & Y- Number 5 |
||||||
| Hit “Calibrate” | X- and Y- percentages should now reflect calibration | ||||||
| Relaunch Calibration. | Previous Calibration should remain | ||||||
| Change a Number | — | ||||||
| Hit “Cancel” | Nothing should change!!! | ||||||
| Launch Pong! | Should launch the Pong Game. | ||||||
| Press Space | Ball should begin moving. | ||||||
| Play Game. Count # balls hitting left side. | — | ||||||
| Let ball pass paddle (hit right side) | Score should be the same as your count! | ||||||
| Repeat without closing | Score should be the same as your count! | ||||||
| Close Pong | Should close Pong properly | ||||||
| Relaunch Pong! | Should launch the Pong Game. | ||||||
| Play Game. Count # balls hitting left side. | — | ||||||
| Let ball pass paddle (hit right side) | Score should be the same as your count! | ||||||
| Launch Archanoid | Should launch the Archanoid Game | ||||||
| Press Space | Ball should begin moving. | ||||||
| Play Game. | — | ||||||
| Let ball pass paddle (hit the bottom) | “Game Over” should appear | ||||||
| Repeat without closing | — | ||||||
| Close Archanoid | Should close Archanoid properly | ||||||
| Relaunch Archanoid | Should launch the Archanoid Game | ||||||
| Play Game until all bricks gone. | “Victory” should appear | ||||||
| Launch Snake | STUB | ||||||
| Launch Help Content | STUB | ||||||
| Launch About | About Window should pop up. | ||||||
| Quit Application | Should close ALL windows. All processes killed, all Zombies killed |
||||||
| Relaunch Application w/ camera | Application should start up | ||||||
| Check Hotkeys | — | ||||||
| CTRL+1 | Launch Pong | ||||||
| CTRL+2 | Launch Archanoid | ||||||
| CTRL+3 | Launch Snake | ||||||
| CTRL+4 | Launch figgity widget | ||||||
| CTRL+H | Launch Help Content | ||||||
| CTRL+C | Launch Calibration | ||||||
| CTRL+A | Launch About | ||||||
| CTRL+Q | Quit Application | ||||||
>> Appendix B – Test Results
Unit Test (QOpenCVWidget) Log
Object Detected in image Rectangle Validation: Passed Expected Output x: 81 | y: 75 Received Output x: 81 | y: 75 X-coordinate: Passed Y-coordinate: Passed Unit Test Successful
Single-Core Processor Log
Processor Log of a Single Core running Mocoa
After running Mocoa on a Single Core 2.00GHz clock speed, we see severe application speed degradation but this does not appear to slow down the operating system as much as had been hypothesized. Because there is no descernable need to optimize this code to stop if issues arise with regards to processor speed, this will not be done. It will be recommended that this code run on a Multi-core processor.
Dual-Core Processor Log
Processor Log of Dual Cores running Mocoa
Quad-Core Processor Log
Processor Log of Quad Cores running Mocoa
Integration & Validation Test Matrix
- Title: Test Matrix
Caption: Integration Test Matrix
Description: A Test Matrix for the overall testing of Mocoa.
File: TestMatrixNew.pdf
Hood College
Univ. of Louisville