The role of the in-circuit test harness as the final link in the software QA testing chain.

The need to improve embedded software quality has prompted the development of some extremely powerful code analysis tools. Foremost amongst these are the static analysers that take in C, C++ or sometimes assembler source code and perform extensive syntax and grammar checks, data flow analysis, complexity metrics and other detailed examinations of programs and their structure. The simplest analysis tools like PC-Lint can spot language misuse whilst other entry-level tools like the Ristanovic DAC can produce flow-charts and complexity metrics that help assess the testability of code. Heavyweight packages like LDRA, ATTOL, CANTATA and others can go beyond this and actually run the source code under simulation on the host PC to perform statement and decision coverage measurements, plus execution time analysis. They automatically generate test harnesses that can be used to exercise functions, modules or complete programs so that repeatable testing can be undertaken. Thus some measure of dynamic testing, albeit in an artificial environment, is possible before the final target hardware has even been designed.

Once the final hardware and CPU are available, LDRA for example, allows measurements to be repeated using an instrumented version of the code. Instrumentation is a technique whereby small code sections or "probes" are added to the user's program that are able to report back to the test tool. The means by which this is achieved varies but in systems that use an operating system, a new IO stream may be opened that is monitored by the analyser's agent running in the real time operating system (RTOS). Where no RTOS is present, the probes can communicate with fixed address ranges that are monitored on the CPU bus via a simple hardware address monitor add-on.

All these tools are able to provide a very high degree of confidence in the correctness of program coding. The basic ones are, as a minimum, essential prerequisites for a successful embedded software project.

However the nature of embedded systems is that the program will be cross-compiled and then run on a microcontroller, often with very limited memory and system resources. A fundamental problem then arises which can seriously compromise the effectiveness and validity of these analysis tools: Heisenberg’s uncertainty principal identified a basic tenet of physics that "by measuring something, you inevitably change it". Likewise, any software measurement method that relies on instrumenting code with software probes will change the very thing that is being tested, whether it be memory usage or execution time. The impact of these code additions is largely dependent on the power of the CPU involved. On large devices like the PowerPC, Tricore, Pentium etc. that use operating systems, the run time and memory resources required are insignificant. In some instances the probes are left in place in the final released software so that at least the code tested is identical to that which is to be shipped. However on 8 and 16-bit microcontrollers like the C167, 68HCxx or 8051 that form the mainstream of today's embedded system, the overhead of the probe code can be unacceptable.

The measurement of dynamic code coverage is perhaps the most compromised by pure software methods. As an example, some Pentium testing tools can perform a coverage analysis of code in the final target. This is done by constantly setting breakpoints using the Pentium's breakpoint unit after each branch or jump instruction. Thus the program proceeds from one jump to the next with the tool constantly monitoring which branch was taken. Thus over a period of time the program flow can be reconstructed by filling in the gaps between branches so that those instructions that were not executed can be identified. This sort of technique can only be applied on CPUs that have some sort of JTAG-driven on-chip debugging systems. An important drawback to this approach is that the code no longer executes in real time. In many systems that are driven by hardware-generated events, this loss of real time operation renders any measurements so made highly questionable.

With the increasing use of microcontrollers with on-chip ROM, there is no bus for an external hardware address monitor to use, giving no alternative to software probe methods. Thus for 8 and 16 bit CPUs software tools on their own are not sufficient to ensure the highest levels of software integrity. Some form of hardware-assistance is needed to allow code to be in its final operating environment and its final memory configuration when tested.

The shortcomings of measurements derived from analysis tools that employ probe methods are known about but are generally ignored or glossed over. This can be attributed to many factors but the predominant ones are:

Most software engineers are from a PC-based background where true real time operation is largely irrelevant
Analysis tool manufacturers generally only possess software expertise and cannot implement the complex hardware that is needed to replace software probes for truly non-intrusive measurements.
To allow the production of very complex software, engineers are forced to assume that software runs in isolation from the real world and that cross-compilers and CPUs are perfect. The embedded software engineer knows that this is not a valid assumption.

Hardware-Assisted Test Harness Techniques

A popular approach is to use an in-circuit emulator-based test script technique such as the HiSCRIPT embedded test language. HiSCRIPT is C-like and is executed by a command interpreter running on the host PC. Test harnesses produced are effectively written in an independent language that is not influenced by the cross-compiler or target CPU in any way. Thus errors in these items do not influence the integrity of the test. Uniquely for a test environment, HiSCRIPT has simultaneous access to:

(i) Embedded system hardware

Embedded program’s symbols
CPU internal busses in single-chip microcontrollers
Emulator trace, trigger and coverage RAM
Mass storage on the host PC hard disk

The inclusion of the emulator means that no changes are required to the code to test it, i.e. there is no software instrumentation and because the final target hardware is used, the memory map and stack usage remain the same. These last two points are absolutely critical as the majority of residual microcontroller software errors are caused by misuse of the cross-compiler, linker and the CPU architecture and are exactly the ones that a pure software test tool will not reveal. Finally, the use of the real CPU even if it has no external bus, means that tests including coverage, can be performed in true real time. Thus the code tested is exactly the same code that will be shipped to the customer.

The creation of test harnesses is a manual process (but can be made semi- automatic) and so represents some additional software design effort which needs to be budgeted for. However, this is often more than compensated by the time it saves in regression testing and the increased efficiency in the test phase.

Points at which test data is to be inserted into, or retrieved from the program are usually chosen to be near function entry and exit. The program’s parameters and variables are accessible from the script as the full symbol table is available. Run/stop control of the program is under harness control as are hardware-based execution timers and coverage RAM. Thus it is possible to conduct tests that exercises code whilst measuring true run times, code and data coverage in an automatic and repeatable way on the final CPU.

Of course these "in-circuit" test harnesses rely on there being a proper hardware emulator available for the CPU in question. Fortunately the smaller 8 and 16-bit microcontrollers where the method is most relevant, are well provided for in this respect. Indeed several of the UK’s aerospace, medical and automotive companies have used the technique to good effect on devices as small as the 2k EPROM 87C751.

Hitex, based on the University Of Warwick Science Park, is happy to advise on the application of the HiSCRIPT language and can assist in the production of test harnesses if required.

Application Note 1

The following is an extract from a real HiSCRIPT test harness produced by an automotive company in the UK for the 68HC12D60 microcontroller. It exercises a one-dimensional linear interpolator function in a vehicle sub-system. Data is taken from stimulus files on the PC hard disk and is placed into the received parameters in the interpolator function under test. The code is run and the result is compared against an expected result, also read from a disk file. A PASS/FAIL status is generated. The entire interpolator test is automated and repeatable. The overall program and the test function is not altered in any way and runs in real-time. As a refinement, the emulator’s coverage analyser simultaneously records the execution of individual instructions and dumps the results to another disk file. Thus if the test run gave the correct interpolated result AND 100% coverage was achieved, the function is deemed to be correct - any unexecuted instructions could contain residual bugs so proving complete execution is vitally important.

1. Open stimulus and results data files in host PC...

// Open the input data file
%input_data = 0
%input_data = %file_open("C:\uk-nsi\interp_data.dat","r")
// Open the output data file
%output_data = 0
%output_data = %file_open("C:\uk-nsi\s00248s.dat","w")

2. Get the number of test cases from disk file and read first test data into HiSCRIPT local variables...

// Read number of test cases from the input data file
%file_read(%input_data,"%u\n",%number_of_tests)
// Main test loop
while(%test_count < %number_of_tests) // Main test loop
{
%test_count++ // Read test data from file
%file_read(%input_data,"%u,",%i_val)
%file_read(%input_data,"%u,",%i_lim1)
%file_read(%input_data,"%u,",%i_lim2)
%file_read(%input_data,"%u,",%o_lim_1)
%file_read(%input_data,"%u,",%o_lim_2)
%file_read(%input_data,"%u\n",%result)

3. Transfer data read from file into interpolator function and call it. Start a hardware timer running to measure total test run time.

i_val = %i_val // Load test data into application variables
i_lim_1 = %i_lim1
i_lim_2 = %i_lim2
o_lim_1 = %o_lim_1
o_lim_2 = %o_lim_2
TIMER ENABLE // Enable timer on go
TIMER RESTART
GO // Start the ICE

while(%ice_running) // Wait until ICE halts
{
wait
}
%interp = interpolated_value // Read result into hiscript

4. Check interpolator result against correct value previously read from disk file. Write result and PASS/FAIL status to PC screen and disk file - if not the final test case, repeat process from 3...

%write(" %u ",%test_count) // write test result to message window

if(%interp == %result) // test for Pass or fail

{

%write (" Pass ")

}

else

{

%write (" Fail ")

}

%time_stamp = (float)%ice_timer_100ns/10 // convert the time stamp

// to microsecs

%write (" %.1f microSeconds \n",%time_stamp) // write time to

// message window on PC

// write results to disk

%file_write(%output_data,"%s %u","\n\n\n Test Number",%test_count)

if(%interp == %result) // test for Pass or fail

{

%file_write(%output_data,"%s"," PASS")

}

else

{

%file_write(%output_data,"%s"," FAIL")

} // Write test data set to disk

5. Write a summary of test rests, including test runtime to a disk file. Store coverage information to a second file.

%file_write(%output_data,"%s"," \n\n Data Set\n\n")
%file_write(%output_data,"%s = %u %s = %u %s = %u %s = %u %s = %u"," i_val",%i_val,"i_lim1",%i_lim1,
// "i_lim2",%i_lim2,"o_lim_1",%o_lim_1,"o_lim_2",%o_lim_2)
%file_write(%output_data,"%s"," \n\n Results")
%file_write(%output_data,"%s %u %s %u"," \n\n Expected Result",%result,"Computed Result",%interp)
%file_write(%output_data,"\n\nRun Time %.1f Microseconds ",%time_stamp)
%printwrite_file("CC","c:\cover.txt",1) // Write coverage
// data to a file

The complete original test harness can be found here.

Application Note 2

A typical software testing process might be:
Static Phase Produce source code Static analysis (syntax and semantics) Real code, no hardware: NOT Real time Dynamic Phase 1 Simulation Statement coverage Decision coverage Real code No hardware: Not Real time	Dynamic Phase 2 Target CPU with probes Statement coverage Decision coverage Real Hardware, Instrumented code: Not Real time Dynamic Phase 3 In-circuit testing with scripts Real code, Real hardware: Real time
Phase 1 should find 80% of the software bugs.	Phase 3 will find the remainder of software and hardware bugs.