Achieving flexibility in test automation

You will probably want a flexible way of creating test suites, so you don’t have to run the same test cases grouped in the same way at every execution. You may want to run a single test case with specific input data, every test case you have in random order, a few test cases in a specified order, or every test case marked as regression test, depending on the situation at hand. Some test cases may be too time-consuming to execute often, or require a very special configuration. Some test automation frameworks lets you create a test suite using drag and drop, some use XML files, others create suites during compile time. Some frameworks let you tag test cases with different labels that can be used when creating suites. What are your requirements?

How much in your test automation framework needs to be configurable? Do you want change the servers you talk to, the way the simulators are set up, what hardware you run on? How do you provide that input to the test cases? What if not all test cases can run on your selected configuration: should they just be skipped or do you want the whole suite to fail? Do you want to parse the configuration once and for all, or do you want all test cases to parse it themselves? Or do you even give each test case a selection of options to choose between and let the test case decide which options it prefers? Or perhaps you give all test cases in the suite a set of options to run and make them decide on a single option that works for all of them in order to avoid potentially time-consuming reconfiguration? These are some questions you should ask yourself in order to get the flexibility you need.

In our next blog contribution about test automation we will bring up some lessons learned about facilitating trouble shooting.