Error reporting for complex unit tests

System level unit testing

I was recently putting together the structure for some system level unit tests of a Swift project.   The structure extended beyond a single file for these tests.   I currently have:

  • SystemTests – Base class for a suite of system level unit tests.
  • BasicSystemTests – Class for my first group of unit tests.  Derives from SystemTests.
  • Various utility classes, each specific to a major section of the project.

Here is an example chain of calls coming from a test case:

BasicSystemTests

   func testModified () {
        
        // Test modification to Project
        //
        createProject(name: "Test Project", bundleId: "TestBundleId", appDelegate: "TestAppDelegate", fsConfig: "TestFsConfig", description: "Test Project Description")
        

SystemTests


   func createProject(name: String, bundleId: String, appDelegate: String?, fsConfig: String?, description: String?) -> String? {
        
        // Is Add New button enabled?
        project.verifyAddNewButton(enabled: false)

ProjectTestUtils

    func verifyAddNewButton(enabled: Bool) {
        XCTAssertEqual(addNewButton.isEnabled, enabled, "Add New button enabled = \(enabled)")
    }
  

The problem – what is the path?

My test case has a bug.  When I run the test case, I see the following in Issue Navigator and the source code editor:

The error is being reported from the verifyAddNewButton() method in ProjectTestUtils, but where is the path from the test case code?

The Test Navigator doesn’t provide any more information:

A report also isn’t helpful:

A better way to report the error

Instead of adding variations of XCTAssert throughout all the various classes, I reserve their usage to the file where the test case itself is defined which in this case is BasicSystemTests.

For cases where the test case calls out to base class or utility methods from other files, I use an entirely different approach.

Strategy

Instead of asserting where the problem is first detected, I will report any issues back to the caller.   Each method in the call chain from the test case will return an optional String.  If there is no failure nil is returned, otherwise I return an error message.  The returned message will include the file, function and line number of the reporting code.

TestUtils

Let me introduce a utility method I keep in a class called TestUtils:


    class func errorMsg (_ message: String, filePath: String = #file, function: String = #function,  line: Int32 = #line) -> String {
        
        let ns = filePath as NSString;
        let file: String = ns.lastPathComponent as String
        
        let prefix = "\(file) -\(function)(\(line)):"
        
        let msg = "\(prefix) \(message)\n"
        
        return msg
    }

This method is called with a message string and returns a new string that is formatted with the file, function and line number of the caller.

Rewrite the call chain with our new reporting methodology

Let’s now rewrite the code in my chain of calls.   Note that I have renamed verifyAddNewButton() to checkAddNewButton() which is just for my convenience in preparing this article.

BasicSystemTests

   func testModified () {
        
        // Test modification to Project
        //
        if let result = createProject(name: "Test Project", bundleId: "TestBundleId", appDelegate: "TestAppDelegate", fsConfig: "TestFsConfig", description: "Test Project Description") {
            XCTFail(result)
        }
        

SystemTests


   func createProject(name: String, bundleId: String, appDelegate: String?, fsConfig: String?, description: String?) -> String? {
        
        // Is Add New button enabled?
        if let result = project.checkAddNewButton(enabled: false) {
            return TestUtils.errorMsg("") + result
        }

ProjectTestUtils


    func checkAddNewButton(enabled: Bool) -> String? {
        if addNewButton.isEnabled != enabled {
            return TestUtils.errorMsg("Failed checkAddNewButton")
        }
        return nil
    }
  

New report – including the path

When we look at the failure now, we can see a couple differences:

  • The failure is attributed to the originating call in the test case.  In this case it is the call to createProject()
  • The error message has a complete path to the problem.   File, function and line number are all included.

By examining each part of the call chain for the error, I can now easily figure out the problem.  In this particular case the call to checkAddNewButton() should have had a value of true for the enabled argument.

 

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s