Test Automation

Mar 3, 2009

SimpleTest has brought a technology to Drupal, but also a methodology, both of which have greatly enhanced the Drupal development process. The test automation methodology is common to all testing frameworks. It is the topic of this article and will also serve as an introduction to the Intro to SimpleTest session this Friday at DrupalCon DC.

Before we start automating anything, let's define the word testing, which has many similar but distinct meanings. In our case, when we say that a piece of code has been tested, we mean that executing it gives the desired outcome. Of course, this is only relevant if we have previously defined the desired outcome. Defining what your code needs to do before writing it is like making a shopping list before going to the grocery store: it makes it easier not to forget something, and keeps you from getting things you didn't need. Having this certitude about the completeness of your code makes it possible to deal confidently with complex requirements, whether they come from a client or from yourself.

Of course, testing code can be done manually by following specific steps from the user interface and looking at what happens, or executing code with a debugger to follow the state of the system internally. In any case, manual code testing is a tedious and time-consuming process. Automating this process takes some extra time to define the desired behavior formally, but it gives us the benefit of being able to “run” those tests as often as we want with almost no time investment.

To automate testing, we write tests that define the desired behavior of our code, and these tests are parsed by our testing framework (SimpleTest in the case of Drupal) which runs them and give us some feedback. Before we take a look at the syntax for writing such tests, let's take a step back and look at what a test is really made of. In its essential form, a test consists of three definitions:

  1. A context
  2. A specific action
  3. A desired outcome

As the test is run, SimpleTest prepares the context, executes the action (such as calling a function or simulating a user clicking a link) in that context, and compares the actual outcome to the desired outcome. This comparison is called an assertion: if there is a match we get a pass, otherwise we get a fail. A fail indicates a mismatch between what the code does and what the test expects. The goal is to resolve these mismatches and to get only passes, improving our code in the process. What happens while setting up the context is not actively tested, we just assume that it works.

Let's take a look at an example: a flashlight. A test for the switch on the flashlight would look something like this:

  1. context: there are batteries in the flashlight (again, this is not tested, we assume this is true)
  2. action: press the switch
  3. desired outcome: there is light

Note that even though we are testing the switch, nobody would stare at the switch looking for a reaction; the outcome is not part of the action. This distinction might not always be this clear, but tests always involve a correspondence between actions and assertions.

Here are some things to keep in mind when writing tests:

  • A test can only results in a pass or a fail.
  • The action should be specific. “Press the switch, eat a sandwich and do a little dance” is not a good action and should be broken down into multiple tests.
  • The outcome can be decomposed into multiple assertion, in which case the test passes when all assertions pass.
  • When your code has multiple layers, start by testing the more primitive layers first. Example: core functions, then public API, then user interface.

With this knowledge in hand, writing tests using any testing framework should simply be a matter of learning the proper syntax. For SimpleTest, we will cover more details on Friday.