Automatic Unit Test Strategy and Process

1         Introduction


The purpose of this document is to provide principles and road map for software development practices in integrating automatic unit test development activities into software development process. It does not cover any testing activities (manual or automatic) conducted by QA team.  Software development engineers are typical target audiences. However, SDM, Archest, and DBA could find it has some reference value.
The unit test or unit test case or test case in the scope of this document is defined as following:
·         It is the test case can be automatically executed, as comparing the test case that requires manual execution.
·         It is test case developed by the developer using the same language used to develop our production code as comparing to the test case developed by QA using tools like QTP. In our case most likely our test case will be developed using VB.Net or C#
For abbreviation, the rest of document refer the unit test case defined hereby simply as “test case” .

2         Basic principles on test case development


From time to time, software development projects shy away from engaging in test case development process. The following is a short list of some common reasons (excuses):
  • It imposes too much initial investment before we can realize its pay back.
  • We are in the middle of the development project; there are 100s of projects, 100s of classes in each project and 100s members (methods and properties) in each class. To write one test case for all members of all classes of all projects is a huge undertaking, and we have no time (no budget) for it.
  • I need to write 100 lines of code to test a method of 20 lines
  • It adds development time to develop the test cases and to maintain them.
  • Unit test should only test the “unit”, it is too time consuming to write the stub and mock-up beforeI can test the method. We do not have the framework to de-couple various layers to test specific layer.
Well, all these considerations are valid if we set our goals to high percentage code coverage, and most precise for all test cases whilst we do not have any test case developed, and more than often our code and our design is not unit test friendly.  However, it is possible to be benefited from engaging in test case development process if we start small.

Let’s not to be frightened by high percentage of code coverage goal.  Let’s not to be restricted by the definition of unit testing. Let’s not hope to have some kind of break-through in code quality or in software development productivity within a short period of time. Finally, let’s not hope it does not take extra time.

Let’s lay down some basic principles on test case development:

2.1       It never be too late to start writing test cases


Any project in any stage can be benefited from engaging in test case development if we adjust our expectation to project reality. Those are some don’ts:
·         Do not set high percentage of code coverage as our goal.
·         Do not hope to have some kind of break-through in code quality or in software development productivity within a short period of time.
·         Do not hope it does not take extra time.
·         Do not be restricted by some academy definition of unit testing.

 Face the reality and start small are the key success factories to the success test case development. Another key success factor to the success of test case development is the commitment to the maintenance of developed test cases from all levels from project management to each developer in the team. 

2.2       Writing test case is not a task by itself, rather is part of every development tasks


If I do not have a development task in hands, I will not write new test cases.  Developers like this because, as a developer, I do not need to go back to write test cases for the millions line of code developed in years by others. Managers like this, because as a PM, I would like to see the project making progress in feature development in production code base (of course, with fewer bugs)     

2.3       For developers, writing test case is one of the tools at your disposal rather than your obligations


Testing is the process of establishing confidence that a program does what it is supposed to do.
Glenford J. Myers in his book “The Art of Software Testing” [1]
As a developer, your confidence vary from time to time when you write code, when your feel your confidence is low, you write some test case and execute them to improve confidence. When your confidence is high, you continue write production code.
When the code you wrote is very simple, like constructors setting some initial values, like some passing through methods. You have confidence there is only remote chance you could put bugs in it. You will not write test cases.  On the other end, when you wrote a code block of hundreds LOCs, you start to have the feeling that you are not sure it works as it should, you will stop and write some test cases and make sure it does what it should before moving on.

2.4       Regression Testing and Continuous Integration testing


A Set of test cases developed for production code base are quality guardians of your production code base, it is to project’s  interest to keep on maintaining them ( make sure they all pass) before every deployment. All parties need to understand that it costs to maintain them.
It is highly recommended to include test case execution as part of continuous build process. It should be given highest priority to resolve it when the build computer reports the build is broken or some test case is failed.

2.5       Balance between quality and quantity


In the traditional software development process, we write production code, and we run some manual testing then we deploy the system to test region for QA testing. In TDD (Test-Driven Development) process, we write test cases first and then we write production code.
According to TDD process, creator of Extreme Programming process, Ken Beck described the Test Driven Development process as “Development is driven by tests. You test first, then code. Until all the tests run (pass), you are not done. When all the tests run, and you can’t think of any more theses that would break, you are adding functionality”. (Page 9 of [3])
Bill Hetzel said in his book; The Complete Guide to Software Testing [2] “Complete Testing Is Not Possible”.  For a simple feature, “you can’t think of any more theses that would break” could take very long time to achieve.
When you strictly follow TDD process, you would write lots of test cases. (High quality, low quantity)  When you follow traditional software development process, you would not write test case at all (low quality, high quantity).
There many practical guidelines between them. From time to time, the project situations call for adjustment of the balance point. In chapter 3, I will provide some general guidelines as a baseline.

3         Development process with integrated test case development activities

3.1       New feature development

 

When you are going to write a new unit (new function, new sub, or new property), you write the production code as you did in the past. When you feel your confident level on the correctness of the code goes a bit lower, you write a test case (and run it). When it passes, your confident level gets improved, you continue. When you are done with the unit, you write a test (driver test) to execute the unit for sizable methods and complex properties.

3.2       Defect Fixing


When you are given a defect to fix, You want to write a test (bug detector test) to prove the existence of the bug before you touch the production code. Before you fix the bug, your bug detector test should fail. Then you change your production code try to fix the bug. When you bug detector test pass, your bug is fixed.

3.3       When you decide to refactor a code unit


 Assertion test, when you decided to refactor a code unit, you write some test cases to execute the code unit. As the code block function as you expected, the test cases should pass.
Then you kick off your refactoring effort, during the refactoring process, your assertion test should still pass.  When the assertion tests fail, it indicates somewhere in your refactoring process you alter the business logic of the code unit.

3.4       Referencing third party component


Assertion test, when you are going to use third party (non-trust worth) component, you write some test (assertion test) to prove the third party component is functioning as you expected. You will only touch your production code after the assertion test all past

4         Process for making your codebase “testable” and how to test different layers


In today’s .net world, most of the systems are we developing are architecturally a simplified version of the following the architecture.

Figure 1: Common application architecture ( page 10)
For client side, Model-View-Controller pattern are widely used to address a host of development challenges, one of them is testability. Specifically, Model-View-ViewModel pattern is specifically designed to work well with WPF for Window application. Microsoft implementation of MVC pattern is specifically designed for web application. Both of them addressed testability very well. Please refer to respective documentation for detail.

The following describes the detail steps on how to make server components unit test friendly or testable.

4.1       Test Database

Establish a database instance specifically for Unit Test purpose. The database schema should be the same as that of the development database instance.  Database team provides a set of insert scripts to populate data in this database as starting point. The development team takes over from that. This statement should be true “by running a set of data base scripts, the development team should be able to refresh the data in the database as they wish.  The initial “set of insert scripts” provided by the database team could be as small as none, or as comprehensive as a complete workable database with all reference data and some transactional data. It is depends on the resource availability of various team.

Microsoft offered a project template for both Visual Studio 2008 and Visual Studio 2010, call SQL Server Project.   With SQL Server Project, the development team could manage database schema objects, pro-deployment, post deployment script from within Visual Studio. It also provides delta based deployment facility. With one click, one could refresh the database instance in a specific region. It is recommended to use SQL Server Project to manage database in general, specifically, It is highly recommended to use SQL Server Project to manage our test database instance.
The key point for this test database instance is that, development team should have the full control of what data should be in the database, and it should be able to refreshed or reestablished easily.  (please refer to my article on SQL/Server Project in my blog)

4.2       Data Access Component

For Data Access Component, all DAC classes should have a default constructor which takes no parameter. This contractor will use the default connection string defined in the configuration file to establish database connection.  They all should have an overloaded contractor which take a string as connection string as parameter. The database connection should be instantiated using the connection string provided. In either case, all their methods should use the database connection established in either constructor to conduct their operations.
Once these are in place, we will be able to write test case against DAC methods. When we write test case for DAC methods, we use the over loaded constructors to instantiate DAC class, (while the Business component class use the default constructor to instantiate DAC class) the connection string we use will be for the test database described above. All test cases should not alter the data in the test database. If it does, we should write some more methods to un-do what the test case did. This is so that we can re-run the unit test case as many times as we wish.  These are the detail steps you want to follow to write test case for DAO methods:
·    Instantiate the DAO classes by using the overloaded constructors and provide the connection string to point to Unit test database
·         Exercise  the method
·         verify the result using Assert
·         ( optional) undo what the test case did when needed

Finally, to prepare DAC for Business Component testing, we need to break the dependency between Business Component and Data Access. This is achieved via abstracting DAC classes into Interfaces.  If you are using C#, IDE provided a refactoring feature.  You can simply right click on the class, then <Refactor><Extract Interface> then follow the <extract Interface> wizard. If you are using VB.Net, you might need to use some third party refactoring tool.
There are two most popular refactoring tools in the market (free and commercially available). Resharper and Refactor! for Visual Basic (aka  CodeRushXpress).  Code RushXpress is free (well, MS paid for the license) and Resharper is more sophisticated but costs money). While Code RushXpress is free, the software maker also offers Refactor!Pro  as its commercially edition. At the time of writing, Resharper is more expensive than Refactor!Pro.
You can visit their web sites for product details:
http://www.devexpress.com/Products/Visual_Studio_Add-in/CodeRushX/
http://www.devexpress.com/Products/Visual_Studio_Add-in/Refactoring/
http://www.jetbrains.com/resharper/
The following code snippet demonstrates how DAC class , its respective interface  and a sample of test case look like:
interface ICustomerDAC
    {
       IEnumerable<Customer> Load(CustomerType customerType);
    }

public class CustomerDAC :ICustomerDAC
    {
        private string _connectionString  ;

        public CustomerDAC()
        {
            _connectionString = "default Conenction String"
        }

         public CustomerDAC(string connectionString)
        {
            _connectionString = connectionString;
        }

         public IEnumerable<Customer> Load(CustomerType customerType)
         {
             throw new  NotImplementedException("under construction");
         }
    }

  [TestClass()]
    public class CustomerDACTest
    {
       …

        [TestMethod()]
        public void LoadTest()
        {
            CustomerDAC target = new CustomerDAC("Test Database Connection String");      
            IList<Customer> actual;
            actual = target.Load(CustomerType.individual);
            Assert.IsTrue(actual.Count > 0);          
        }
    }

4.3       Business Component

For BC Component, all BC class should have DAC interface defined as a private variable.  The interface will be only been assigned with solid class through BC constructors.
Each BC class should at least have 2 constructors. The default constructor takes no parameter. This contractor will use the default constructor of DAC to instantiate DAC class. The overloaded constructor takes an interface of DAC.  All their methods should use the private DAC interface to conduct their operations.
Once those are in place, we can write unit test case against BC  methods. In detail, this is how we do in the unit test case:
·         Instantiate all necessary DAO classes by using the overloaded constructors and provide the connection string to point to Unit test database
·         Instantiate BO class by using the overloaded constructor and provide the DAO instances created in previous step.
·         Exercise  the method and verify the result using Assert
·         ( optional) undo what the test case did when needed
The following code snippet demonstrates how BC class , its respective interface  and a sample of test case look like:

public interface ICustomerBC
{
    IList<Customer> Load(CustomerType customerType);
}

[TestClass()]
    public class CustomerBCTest
    {
        [TestMethod()]
        public void LoadTest()
            ICustomerDAC TestDAC = new CustomerDAC("Test Database Connection String");
            CustomerBC target = new CustomerBC(TestDAC);
            IList<Customer> actual = target.Load(CustomerType.Business);          
            Assert.IsTrue( actual.Count > 0, "there should be at least one");        
        }
    }

4.4       Service Interface 


In almost identical process described for Business Component, one could make Service Interface layer testable and write test case against it. For briefness, I would not go into lengthy detail. The following code snippet demonstrates how Service Interface class and a sample of test case look like:
  public class CustomerService
    {
        private ICustomerBC _customerBC;

        public CustomerService()
        {
            _customerBC = new CustomerBC();
        }

        public CustomerService(ICustomerBC customerBC)
        {
            _customerBC = customerBC;
        }

        public IList<Customer> Load(CustomerType customerType)
        {
            return _customerBC.Load(customerType);
        }
    }

[TestMethod()]
        public void LoadTest()
        {
            ICustomerDAC TestDAC = new CustomerDAC("Test Database Connection String");
            ICustomerBC TestBC = new CustomerBC(TestDAC);
            CustomerService target = new CustomerService(TestBC);
            IList<Customer> actual = target.Load(CustomerType.Business);
            Assert.IsTrue(actual.Count > 0, "there should be at least one");        
        }

4.5       General guidelines on developing test cases


·          Name your test project after its respective production code project, name the test class after the class you want to test, and name the test method after the method you want to test.
·          You need to ensure your test case execution is repeatable
·          The test case should leave no trace in the system. If you test case committed a Transaction, you need to un-do the Transaction as part of test case execution.
·          To test the up layer, you could either create an instance of lower layer component pointing to the test database, or develop a stub implementation of the lower layer component confirming to the interface. At the same time, we could choose to write a stub method of the lower layer component when the supporting method is not available in production code or it cost more time to setup the test case than to write a stub implementation.
·          To provide test data, you could either add data to the test database, or establish its own data and then remove them after testing.  You will choose whichever ways requires less effort.
·            Use <ClassInitialize()> ,  <TestInitialize()> , <ClassCleanup()>  <TestCleanup()>  attributes to setup and teardown test cases.

5         Further development


A very promising further development from here would be employing Depend Injection (DI). By introducing depend injection, the hardcoded dependency among layer are totally removed.  You would not see any “New” key word in our production code nor in test case code… they all controlled by the configuration file. Unity Application Block is a member of Microsoft Enterprise Library, which is a light weight depend injection solution. Click this link for detail information on Unit Application Block. http://unity.codeplex.com/

6         Reference

1.                   The Art of Software Testing (Glenford J. Myers  published by John Wiley & Sons   1979)( Chapter 2: The Psychology and Economics of Program Testing)
2.                   The Complete Guide to Software Testing, Second Edition  ( Bill Hetzel  published  by John Wiley & Sons © 1998 (Chapter 2 - Principles of Testing)
3.                   Test-Driven Development: By Example (Kent Beck,  published by Addison-Wesley Professional, 2003)
4.                   Microsoft Application Architecture Guide 2nd Edition (patterns and Practices)

No comments:

Post a Comment