Overview:
List available test cases and select for execution
Typically you will use TestExecWindow to get an overview of available test cases contained within a test application. By selecting a single test suite or a single test case and then pushing "Run selected" TestExecWindow will automatically set appropriate command line parameters and start the corresponding test case.
It depends on the structure of your specific test application if test suites or test cases can be executed independently from each other. If this is not the case you should think about improving your test application. A good advise could be to ensure that at least each test suite could be executed alone.
TestExecWindow supports you to check which test suites or test cases can already run independently. By pushing "Run each" TestExecWindow will start your test executable separately for each item in the list of test suites or test cases. The result of each call to your test application is displayed within the log pane together with the used command line params.
Remark:
If you simply want to execute all test cases as quick as possible it is recommended to use button "Run" at the top of the window. In this case your test application will be started only once to execute all tests.
Searching for errors, extending test cases
If you want to extend your test cases or you want to search the reason for some test failure the following features may be useful:
- Protocol file
Pushing "Out" opens the protocol file of your test application within Visual Studio. Usually you could find more detailed information in this file which may allow a better understanding of the error situation. It depends on your specific test application whether there exists a protocol file. TestExecWindow assumes a protocol file with the name "YourTestApp.out" besides the test executable "YourTestApp.exe".
- Debugging
When searching for an error it may be useful to start test execution within debugger. You can simply switch debugging on and off by selecting / deselecting checkbox "Debug".
Remark: The command line params used to execute your test within debugger are stored within the project settings and will not be restored to the original values. You may have to reset them manually when running the debugger outside of TestExecWindow.
- Src - jump to source code
Pushing "Src" immediately opens the selected test function or test group within your source code. This may be useful for inspecting the test instructions when looking for a test error or for adding a new test case before or after the selected one.
- Refresh
If you have just added a new test case press "Refresh" to trigger a refresh of the list of available test cases. This may also be necessary when you have opened TestExecWindow after you have already loaded your test project. TestExecWindow currentlly listens only to notifications about a changed startup project or a change of Debug/Release. If TestExecWindow is not opened it cannot listen to such events.
Batch testing and automatic shutdown
After some time or when you are working within a team there may be a greater number of test applications and test cases. It is a good idea to execute all unit tests within the nightly or weekly build you may have introduced. You should care for a sufficient level of test output or a detailed test protocol to have enough information for later analysis of possible errors.
TestExecWin also offers the possibility to execute an arbitrary set of test applications in batch mode (for details see Batch file support). This feature may be useful in the following situations:
- execute a specific subset of tests with one button click
As a starting point for your batch you may use "Import from solution" to get a complete list of test executables from your test solution.
- verify that all your test suites can be executed alone
Use "Import test suites from startup project" to get a list of separate calls to your startup project with appropriate start args to select each test suite.
verify that all your test applications are free of memory leaks
Ensure that within detailed settings "leak check" is activated. The status of each test execution is written to log pane and has one of the following values:
- OK:
correct test result, no memory leak detected
Warning: if your test application is not configured to write memory leaks to stdout or if you have activated "no leak check" within detailed settings then you may get this status despite of having memory leaks!
- FAILED:
there are functional errors (e.g. a BOOST-Check has failed), in this case the check for memory leaks was skipped
- FAILED (MEM_LEAKS):
correct test result but memory leaks were detected
Precondition:
Your test application is configured to write memory leak messages to stdout when program terminates (C runtime supports leak check, see for keywords "_CrtSetDbgFlag", "_CRTDBG_LEAK_CHECK_DF").
- automatic shutdown and log file
Before leaving your office you can start a test sequence. If you activate checkbox "Shutdown" your computer will be switched off when all tests have terminated. The next day you can find the test results from log pane within a protocol file with the name "TestExecWin.[InfoAboutExecutedTests].[TimeStamp].Log.txt". The protocol file is located within the target directory of your startup project. If there was no startup project selected at time of execution (via batch feature) you should check your temporary directory as defined via environment variables TMP, TEMP or USERPROFILE.
- Read next chapter: Parsing of source files - current limitations
- Read previous chapter: Extended settings - defaults args and configuration of layout