Snapp Action

Friday, February 16, 2007

SCons Unit Testing (with CxxTest)

In working with SCons I found unit testing critical to verify modules of C++ code. What I want to do here is let others know the method I discovered that fit my criteria:

  • Lightweight. It needed to be lightweight so we could add tests easily. Nobody wants to add tests if it they are difficult to create.

  • Executed automatically with each build. They are executed with each build so bugs can be caught early and before checking in code.

  • Extensible. Some pieces of our code modify or depend on aspects of the system (registry, smbios, network). While unit testing the modules of code could test the "recipe", so to speak, I also wanted a way to "bake the cake" to make sure it actually ran in certain environments (in other words, system tests). I also had other ideas in mind. Basically, by being extensible I wanted to be able to extend unit tests into other areas later on.

I researched several different implementations (see http://www.scons.org/wiki/UnitTests for starters). I think what I ended up with was a union of some the best suggestions given there.

CxxTest

CxxTest is a slick project for doing unit tests that does not require a lot of framework code just to run simple tests. See http://cxxtest.sourceforge.net to download it, and http://cxxtest.sourceforge.net/guide.html for instructions on how to use it. Yes, I did look into CppUnit, and in fact it is used by other members of my team, but I found CxxTest to be superior in terms of simplicity. You can read the links for more information, but the short of it is that by using Perl/Python, it can generate framework code based on a simple unit test class in a header file. I have used JUnit and felt right at home with coding CxxTest unit tests.

One customization I did was to convert the CxxTest python script cxxtestgen into an executable. That way I could call it like any other SCons tool. I don't know that that was necessary, but it was simple enough using py2exe.

Source tree structure

My tree is structured in the following way:

trunk/
build/ <-- this contains all build tools, and they are checked into our source control
scons/
cxxtest/
apps/ <-- this has the different applications we build
lib/ <-- the library of code that we share is in here
scons/ <-- a place for all my scons tools
UnitTest.py
util/
String.h
String.cpp
... (more source files)
test/
_MainPlaceHolder.h <-- I will describe this later
TestString.h
... (more unit tests)
out/ <-- This is the directory we build to

Adding the UnitTest tool to SCons

I took advantage of the fact that CxxTests are header files, and added a custom Builder for the ".h" suffix. This way, I could add the ".h" files themselves to the Program() call and it would know how to generate the .cpp files and build them.

If you are wondering about the _MainPlaceHolder.h, CxxTest will generate the test framework around as many test classes as you want, but it needs to also generate the main() entry point into the unit test. Since I don't want to create individual unit test binaries for each unit test header, _MainPlaceHolder.h is an empty file simply used as a file to key off of and generate the main() method for all tests in the "test" directory. There are probably a number of ways to work around this (i.e. pick the first test in the group to be the one with "main()", or generate/delete a place holder file automatcially when generating the unit test cpp files), I simply took the simplest choice at the time.

Here is the code in UnitTest.py:

import os
import string
import sys
import re

import SCons.MyHelper
import SCons.Util
import SCons

from SCons.Script import *

## in order to add unit test function
from SCons.Script.SConscript import SConsEnvironment

# A Custom Glob method that operates on the source directories rather than the BuildDir.
# This is actually factored into a different file (SCons.MyHelper) in my own build environment, but I
# have included it here for reference.
def Glob(env, pattern):
path = string.replace(env.GetBuildPath('SConscript'), 'SConscript', '')

result = []
for i in glob.glob(path + pattern):
result.append(string.replace(i, path, ''))

return result

def UnitTest(env, testlib, **kw):
testdir = 'test' # default test dir
runtests = 1 # automatically run tests when built

if (kw.has_key('CPPPATH')):
kw['CPPPATH'] += ['#/build/cxxunit', '.', '#']
if (kw.has_key('TESTDIR')):
testdir = kw['TESTDIR']
if (kw.has_key('RUNTESTS')):
runtests = kw['RUNTESTS']

if (kw.has_key('LIBS')):
kw['LIBS'] += testlib
else:
kw['LIBS'] = testlib

# automatically get all the .h files from the 'test' directory
source = SCons.MyHelper.Glob(env, testdir+'/*.h')

foundMain = 0
for s in source:
if '_MainPlaceHolder.h' in s:
foundMain = 1
break
if foundMain == 0:
source += [testdir + '/_MainPlaceHolder.h']

# create the name of the test executable (i.e. "test_mystuff.exe" or "systest_mystuff.exe")
name, ext = os.path.splitext(testlib[0].name)
name = testdir + '_' + name
test = env.Program(name, source, **kw)

# run tests if user has not overridden their use from the command line with 'runtests=0'
if ( env['RUNTESTS'] == 1 ):
# run tests at the end, if runtests is true (this may be a system test, which is not meant to be run immediately)
if runtests:
env.AddPostAction(test, test[0].abspath);
env.Alias(testdir, test) # Alias to testdir which should be unique
env.AlwaysBuild(test)

return test

def build_unit_test(target, source, env):
for t, s in zip(target, source):
if (str(s).endswith('_MainPlaceHolder.h')):
env.Execute('cxxtestgen --error-printer --root -o %s %s' % (t, s), strfunction='')
else:
env.Execute('cxxtestgen --part -o %s %s' % (t, s), strfunction='')

return 0

def unit_test_string(target, source, env):
h, t = os.path.split(str(source[0]))
return "unit testing '%s'" % (h)

## regex to find unit test headers
header_re = re.compile(r"([^.]+)\.h$")

def unit_test_emitter(target, source, env):
target = [] # for some reason this is coming in with a prefilled value

for item in source:
dir, file = os.path.split(str(item))
match = header_re.search(file)
trunkdir, testdir = os.path.split(dir)
target += [testdir + '/' + match.group(1) + "_gen.cpp"]

return (target, source)

def generate(env):
"""
Add builder and UnitTest wrapper for unit testing
"""

unit_test_action = Action(build_unit_test, strfunction=unit_test_string)
bld = Builder(action = unit_test_action, emitter = unit_test_emitter, src_suffix = '.h') #, suffix = '.cpp')
env.Append(BUILDERS = {'Test' : bld})

# adding test builder to src builders
static_obj, shared_obj = SCons.Tool.createObjBuilders(env)
static_obj.src_builder.append('Test')
shared_obj.src_builder.append('Test')

SConsEnvironment.UnitTest = UnitTest

def exists(env):
"""
Make sure this tool exists.
"""
return env.Detect('cxxtestgen') # this is why I converted cxxtestgen.py into an executable

In my SConstruct, I simply added UnitTest to my Environment tools:
env = Environment(platform=target_info['build_os'],
tools=target_info['tools'] + ['UnitTest'] + ['msvs'] + ['VSProj'],
ENV=target_info['env'],
toolpath = ['.', 'lib/scons'])

Now, in order to perform unit tests, all we need is to pass the result of the build into UnitTest (in this case "lib"). Note that we don't need to specify the header files that contain unit tests since the builder will automatically gather them from the TESTDIR. Here is an example:
# define sources, includes, defines, dependant libraries and lib paths, etc.
...

# create the library or executable binary as usual
lib = env.StaticLibrary('util', sources, CPPPATH=includes, CPPDEFINES=defs)

# Build unit/system tests, unit tests run immediately
test = env.UnitTest(lib, CPPPATH=includes, CPPDEFINES=defs, LIBS=libs, LIBPATH=libpath) # TESTDIR is 'test' and RUNTESTS is 1 by default
systest = env.UnitTest(lib, CPPPATH=includes, CPPDEFINES=defs, LIBS=libs, LIBPATH=libpath, RUNTESTS=0, TESTDIR='systest')
Here is an example test case for lib/util/String that shows how simple the test cases can be:
#include <cxxtest/TestSuite.h>
#include "String.h"
using namespace std;

class TestString : public CxxTest::TestSuite
{
public:
void testString_1( void )
{
String str1 = "This IS a Test STRing";
String str2 = "THIS is a TEST strinG";

// test case insensitive comparisons
TS_ASSERT_EQUALS( str1, str2 );
}
}

That's all it takes to create a unit test. Pretty cool, huh?

Conclusion

By using a simple tool for building unit tests (CxxTest) and the SCons build tool, we can create a simple, but powerful way to test modules of code that requires minimal effort on the developer's part.

Feel free to respond with suggestions for improvement.

1 Comments:

  • It will be great if you put full source code of this tutorial to github or bitbucket.

    By Blogger Unknown, at 11:05 PM  

Post a Comment

<< Home