Functional Coverage and Coverpoints

I recently posted an article called Setting up and using Code Coverage.   As I mentioned there, code coverage is useful for seeing what portions of your code have been exercised.    What code coverage can’t do though is to ensure functional correctness – this is where functional coverage comes into play. Functional coverage allows you to determine when you have exposed your code to a sufficient enough set of stimulus, including hitting corner cases,  that you have a high confidence in its functionality.

My background includes experience as a hardware verification engineer, where the teams I was part of used both code coverage and functional coverage as essential tools in test development.     In the Wikipedia section on SystemVerilog (a hardware description and verification language) there is a nice write-up on Coverage where the difference between code coverage and functional coverage is discussed.

What is a Coverpoint?

I wanted to employ some functional coverage as a guide to my unit test development.   In the SystemVerilog language mentioned above, they have the concept of coverpoints and covergroups (group of coverpoints).   I found a nice document called SystemVerilog Testbench Automation Tutorial where there is some good information about these concepts starting in the section titled “Functional Coverage”.

A coverpoint is essentially a measurement point where an event (such as values taken on by a variable) can be recorded, along with bins for how many times specific values (or range of values) occurred for that event.

What does this have to do with Xcode Unit Testing?

I’m going to borrow the term coverpoint and its basic concept to allow me to implement some functional coverage in my Xcode unit testing.

In this context, I’ll define a coverpoint as having the following characteristics:

  • It has a name.
  • It has a condition that enables it.
  • It has an associated counter (a simple bin).
  • It can placed anywhere in my code.
  • Thread safe.
  • Comes with a set of utility methods.
  • Only active during development.

An instance of a coverpoint is a macro that is called such as:

COVERPOINT(CoverPointStep4ReconcileCat2CaseC, parseSecondaryRecordIdUniqueNum && ![parseSecondaryRecordIdUniqueNum boolValue])

The macro is defined as follows.   Note how I use the DEBUG define to ensure that this macro only expands to the desired code during development.

#ifdef DEBUG
#define COVERPOINT(Name,Condition) if(Condition) { [[CommonStateSingleton sharedCommonStateSingleton] incrementCoverpoint:Name]; }
#else
#define COVERPOINT(...)
#endif

Let’s look at some of the characteristics above more closely and then show an example of how it can be used.

Coverpoint name

I chose to implement coverpoints using a singleton.    For the coverpoint names, I decided to use a  NSString constant (rather than an enum), to make it easier to use coverpoints descriptively in NSLog calls, etc.

In CommonStateSingleton.m (my singleton class) I define coverpoints such as:

NSString *const CoverPointStep4ReconcileCat2CaseC = @"CoverPointStep4ReconcileCat2CaseC";

In CommonStateSingleton.h I export these names making them available to all classes that include this header:

FOUNDATION_EXPORT NSString *const CoverPointStep4ReconcileCat2CaseC;

Coverpoint condition

The condition is what triggers the coverpoint to record the event.  This condition can be any expression such as the example given above:

parseSecondaryRecordIdUniqueNum && ![parseSecondaryRecordIdUniqueNum boolValue]

Coverpoint counter (bin) & Thread Safety

Every time the condition for a coverpoint is met, I record the event by incrementing a corresponding counter.

The counter is just a dictionary value that I increment using the method shown in the macro above –  incrementCoverpoint:  The dictionary is a property in CommonStateSingleton.m.

In order to do this in a thread safe fashion I use an NSConditionLock, which is something I discussed in my post Unit Testing View Controllers and Asynchronous APIs.

@interface CommonStateSingleton ()

// Functional Coverage

// We will define a coverpoint as having a name, condition, and one associated bin that keeps track of how
// many times the condition has been met.   This will be implemented using a dictionary.
@property (strong, nonatomic) NSMutableDictionary *unitTestCoverpointMutableDict;

// Control access to unitTestCoverpointMutableDict with locking
@property (strong, nonatomic) id unitTestCoverpointConditionLockObj;

// An array of all known coverpoint names.
@property (strong, nonatomic) NSArray *coverpointNameArr;

@end

Here is what my incrementCoverpoint: method looks like in my singleton class CommonStateSingleton.m:

// Increment the bin value for the named coverpoint
- (void)incrementCoverpoint:(NSString *)name
{
    [self.unitTestCoverpointConditionLockObj lockWhenCondition:0];  // Locked

    NSNumber *valueNum = [self.unitTestCoverpointMutableDict objectForKey:name];

    NSInteger value = [valueNum integerValue];
    value++;

    valueNum = [NSNumber numberWithInteger:value];
    [self.unitTestCoverpointMutableDict setObject:valueNum forKey:name];

    [self.unitTestCoverpointConditionLockObj unlockWithCondition:0];  // Unlocked
}

Initialization

The following is some of the initialization that I perform in CommonStateSingleton.m:

- (id)init
{
    self = [super init]; // call our super’s designated initializer
    if (self) {

        _unitTestCoverpointMutableDict = [[NSMutableDictionary alloc] init];
        _unitTestCoverpointConditionLockObj = [[NSConditionLock alloc] initWithCondition:0];

        _coverpointsToCoverArr = @[];

        // Keep an array of all coverpoints in self.coverpointNameArr
        self.coverpointNameArr = [NSArray arrayWithObjects:CoverPointAddAllToParse,
                                                    CoverPointAddAllToParseZeroRecords, CoverPointNoParseUpdatesNeeded, ..., nil];

        [self resetCoverpoints];
    }
}

Other utility methods

Besides incrementCoverpoint:  I also created some other utility methods for working with coverpoints.

Set all coverpoint bin values to 0:

- (void)resetCoverpoints
{
    [self.unitTestCoverpointConditionLockObj lockWhenCondition:0];   // Lock

    for (NSString *name in self.coverpointNameArr) {
        [self.unitTestCoverpointMutableDict setObject:[NSNumber numberWithInteger:0] forKey:name];
    }

    [self.unitTestCoverpointConditionLockObj unlockWithCondition:0];  // Unlock
}

Return the coverpoint value for the specified name:

- (NSInteger)coverpoint:(NSString *)name
{
    NSInteger value;

    [self.unitTestCoverpointConditionLockObj lockWhenCondition:0];  // Locked

    value = [[self.unitTestCoverpointMutableDict objectForKey:name] integerValue];

    [self.unitTestCoverpointConditionLockObj unlockWithCondition:0];  // Unlocked

    return value;
}

Return a copy of self.unitTestCoverpointMutableDict:


// Return a copy of the current state of self.unitTestCoverpointMutableDict for a snapshot
- (NSDictionary *)coverpointDict
{
    NSDictionary *unitTestCoverpointDict;

    [self.unitTestCoverpointConditionLockObj lockWhenCondition:0];  // Locked

    unitTestCoverpointDict = [self.unitTestCoverpointMutableDict copy];

    [self.unitTestCoverpointConditionLockObj unlockWithCondition:0];  // Unlocked

    return unitTestCoverpointDict;
}

Get a dictionary of all coverpoints that were hit with their counts:


// Call provides a coverpoint dictionary (previous
// copy of self.unitTestCoverpointMutableDict obtained with coverpointDict)
//
// Method returns a Dictionary of all coverpoints that were hit along with their counts.
// A hit is counted as a count in self.unitTestCoverpointMutableDict that is greater than
// in unitTestCoverpointDict (the baseline) for a given name.
//
- (NSDictionary *)coverpointsHitReferencedToCountsFromDict:(NSDictionary *)unitTestCoverpointDict
{
    NSMutableDictionary *hitMutableDict = [[NSMutableDictionary alloc] init];

    NSInteger baseline;

    for (NSString *name in self.coverpointNameArr) {
        if(!unitTestCoverpointDict){
            baseline = 0;  // No dictionary provided for baseline
        }
        else {
            NSNumber *baselineNum = [unitTestCoverpointDict objectForKey:name];
            baseline = [baselineNum integerValue];
        }

        NSInteger currentCoverpointCount = [self coverpoint:name];

        // Our current count should never be less than baseline.
        NSAssert(currentCoverpointCount >= baseline, @"Current coverpoint count for %@ is less than baseline.  currentCoverpointCount=%d  baseline=%d",name,currentCoverpointCount,baseline);

        // If current count exceeds baseline, than we did hit current coverpoint.
        NSInteger numHits = currentCoverpointCount - baseline;
        if(numHits > 0){
            [hitMutableDict setObject:[NSNumber numberWithInteger:numHits] forKey:name];
        }

    }

    return [NSDictionary dictionaryWithDictionary:hitMutableDict];
}

Check that a set of coverpoints have been hit, referenced to a baseline snapshot:


// Caller provides an array of coverpoint names and a coverpoint dictionary (previous
// copy of self.unitTestCoverpointMutableDict obtained with coverpointDict).
//
// Method returns an array of those coverpoints that were not hit during testing.
// A hit is counted as a count in self.unitTestCoverpointMutableDict that is greater than
// in unitTestCoverpointDict (the baseline) for a given name.
//
- (NSArray *)coverpointsNotHitFromArray:(NSArray *)namesArr
             referencedToCountsFromDict:(NSDictionary *)unitTestCoverpointDict
{
    NSMutableArray *notHitMutableArr = [[NSMutableArray alloc] init];

    NSInteger baseline;

    NSUInteger namesCount = [namesArr count];

    for(NSUInteger i=0; i<namesCount; i++){                  NSString *name = [namesArr objectAtIndex:i];                  if(!unitTestCoverpointDict){                          baseline = 0;  // No dictionary provided for baseline                  }                  else {                          NSNumber *baselineNum = [unitTestCoverpointDict objectForKey:name];                          baseline = [baselineNum integerValue];                  }                  NSInteger currentCoverpointCount = [self coverpoint:name];         // Our current count should never be less than baseline.                  NSAssert(currentCoverpointCount >= baseline, @"Current coverpoint count for %@ is less than baseline.  currentCoverpointCount=%d  baseline=%d",name,currentCoverpointCount,baseline);

        // If current count does not exceed baseline, than we did not hit current coverpoint.
        if(currentCoverpointCount == baseline){
            [notHitMutableArr addObject:name];
        }
    }

    return [NSArray arrayWithArray:notHitMutableArr];
}

Using Coverpoints in Unit Testing

I’ve shown how to create a macro for a coverpoint and instance it in your code, and provided some methods for working with them.

The last thing I wanted to show was how to use coverpoints in your unit testing.   For a given unit test case, you can define the coverpoints that you expect to see and fail the test if they aren’t covered.   If you expect a certain coverpoint to be executed a specific number of times in a test case, you could use the methods I provided to check for this as well.

ContactsOnlineBasicUnitTests.m:


// Make sure that we covered the coverpoints from coverpointsToCoverArr using unitTestCoverpointDict as a baseline
- (BOOL)checkCoverpointsNotHitFromArray:(NSArray *)coverpointsToCoverArr
             referencedToCountsFromDict:(NSDictionary *)unitTestCoverpointDict
                                 errStr:(NSString **)errStr
{
    // We use the counts in unitTestCoverpointDict as our baseline.
    NSArray *coverPointsNotHitArr = [[CommonStateSingleton sharedCommonStateSingleton] coverpointsNotHitFromArray:coverpointsToCoverArr referencedToCountsFromDict:unitTestCoverpointDict];

    if([coverPointsNotHitArr count] != 0){
        *errStr = [NSString stringWithFormat:@"Did not cover the coverpoints: %@",coverPointsNotHitArr];
        return NO;
    }
    return YES;
}

- (void)dumpCoverpointsHitReferencedToDict:(NSDictionary *)unitTestCoverpointDict
                                fromMethod:(const char*)methodCStr
                               fromSection:(NSString *)section
{
    NSString *methodName = [NSString stringWithCString:methodCStr encoding:NSASCIIStringEncoding];

    NSDictionary *coverPointsHitDict = [[CommonStateSingleton sharedCommonStateSingleton] coverpointsHitReferencedToCountsFromDict:unitTestCoverpointDict];
    NSLog(@"The coverpoints hit in %@, section: %@  %@", methodName, section, coverPointsHitDict);
}

- (void)testParseAddDelete_1
{
    NSString *errStr = nil;

    // Test case specific coverpoints to cover.
    NSArray *coverpointsToCoverArr = [NSArray arrayWithObjects:CoverPointStep4ReconcileCat2CaseA,
                                      CoverPointStep4ReconcileCat2CaseB,
                                      CoverPointStep4ReconcileCat2CaseC,
                                      CoverPointStep4ReconcileCat3CaseA,
                                      CoverPointStep4ReconcileCat3CaseB,
                                      nil];

    // Get a snapshot of the current coverpoint dictionary (counts for coverpoint names).
    NSDictionary *unitTestCoverpointDict = [[CommonStateSingleton sharedCommonStateSingleton] coverpointDict];

    // ----- Body of the test case code -----

    // Check that expected coverpoints hit for testcase.
    if(![self checkCoverpointsNotHitFromArray:coverpointsToCoverArr referencedToCountsFromDict:unitTestCoverpointDict errStr:&errStr]){
        STFail(errStr);
    }

    // Dump all coverpoints that were hit in testcase.
    [self dumpCoverpointsHitReferencedToDict:unitTestCoverpointDict fromMethod:__FUNCTION__ fromSection:@"End of testcase"];

}

Why do I dump all the coverpoints that were hit at the end? I want to make sure that I completely understand which coverpoints are being hit in a particular testcase. There should not generally be any coverpoints dumped at the end that aren’t also included and checked in coverpointsToCoverArr.

Advertisements

One response to “Functional Coverage and Coverpoints

  1. Pingback: The development of Contacts2Web | Finalize.com: My journey with iOS and other code adventures

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s