Here is a commandment for you: seek happiness in sorrow. Work, work tirelessly. 

—Fyodor Dostoyevsky

I have been in software testing business since 2002 and have never heard a satisfactory explanation of test execution techniques. I accept that test design doubtlessly requires more intelligence, analytical thinking, and creativity than test execution; but still, you can experience numerous benefits from good test execution.

I will divide the execution process into three main parts: test execution states, test execution priorities, and test execution phases. When you combine all of them, you may reach a point where you have a good test execution framework and you are ready for the battle.

Test Execution States

We are all executing test cases; however, we are grading them improperly many times. Following are some of the most important execution states (you may call them test results as well) and their definitions. If you are using them correctly, keep on with what you are doing. If not, you might consider changing the way you mark test cases. Sure, you can even add to this list, but with the ones here, you are more or less addressing all the possible situations/outcomes.

  • Pass: test procedure is executed and the expected result is satisfied.
  • Fail: test procedure is executed and the expected result is not satisfied.
  • Inconclusive: test procedure is executed and requires further analysis to have a clear result.
  • Block: test procedure cannot be executed due to the fact that at least one of the test case preconditions are not met.
  • Deferred: test procedure is not executed yet and deferred for a future test cycle/release for execution.
  • In progress: test procedure is currently being executed.
  • Not run: test procedure is not executed yet.

Test Execution Priorities

In almost every mobile software project, test budgets are plundered and testers are given less than enough time for testing. A tester is always a person who needs to do more with less. Because mobile software projects are full of baseless commitments and time-to-market madness, an additional test hour is as precious as an oasis in a desert.

Under these circumstances, a tester should be very structured and wise while she or he is executing test cases. At any time of testing, your budget can be cut without any logical explanation, and you may be given less than enough time for completing your “not run” test cases. In such a case, you need to know that you are executing them by correct priorities.

Test execution priority can be calculated by the following variables. If you assign weights for each item and count them all, you may conclude on a final test case priority. I will not suggest a formal numbering scheme here, because I believe that the weight of the following items may vary for every test organization, so a generic scheme cannot be formulated. So please take a look and create your own.

  • Risks covered by the test case can be explained by which risk items are being exercised by the test case and how important/critical they are.
  • Platforms covered by the test case can be explained by which mobile OS is being exercised by the test case.
  • Complexity of the test case can be explained by how complex the test case is in terms of interfaces, business rules, boundary values, data entry points, and software components/features. You can also count the total number of test steps (procedures) of the test case, or you may define its dependencies.
  • Depth of the test case can be explained by how deep the test case exercises in-depth/full functionality of a single feature.
  • Breadth of the test case can be explained by how broad the test case exercises the variations of different functionalities/features in a high-level perspective. In other saying, how many features does the test case involve?

Test Execution Phases

I am a huge soccer fan, and while I was recently watching a Trabzonspor (the Turkish soccer team that I support) game, I realized that in-game phases of soccer are quite parallel to the phases of test execution. To explain it more clearly, when a soccer team is in possession, you can say that the team is in attacking phase. If the team is not controlling the ball, it is in defending phase. And if the team loses or wins the ball, it goes through a transition phase from attacking to defending or from defending to attacking.

When we apply these definitions to mobile testing, they can be defined as follows:

  • Attacking phase While a tester is executing a test case, she or he is running the test procedures one by one. And while the tester is in action—for instance, she or he is clicking on a button, entering data, or selecting an option—she or he is trying to exploit open areas and catch defects. The main focus in this phase is to attack the software. Finding a defect means the tester is scoring a goal.
  • Defending phase While a tester is comparing the actual outputs with the expected results, he or she is in defending phase. The main issue here is to deny the opposition (software). If expected results are fulfilled by the software, this means that the tester didn’t defend well. A passed test procedure means that the software is scoring a goal.
  • Transition phase This phase refers to the period of adaptation just after the possession changes. It can be regarded as the shortest phase. Once a tester completes running the test procedure (ending the attacking phase), he or she should quickly switch to analyzing the expected results (starting the defending phase). In order to adapt to the new phase, you need to change the mind-set.

While you are attacking, you need to be more suspicious, hypercritical, relentless, creative, and crafty. While you are defending, you need to be more patient, accurate, attentive, and objective. Oftentimes it is quite difficult to switch between these two testing modes, and this can be referred to as the inherent limitations of a tester’s mentality of eagerness to detect defects.

After reading this section, I realize again that test execution is not a piece of cake. So please quit saying:

  • Test execution can be done by anyone.
  • Test execution requires no special thinking, skills, and mind-set. It is an autonomous action.
  • You just execute a test procedure (click on a button and wait for the result) and compare the output with the expected result. There’s no special logic behind it.
  • Test execution is kid stuff.