Most of you probably know or have at least heard that Model Based Testing (MBT) is a new innovative technology where tools help you create better tests with automated generation, and with Conformiq the generation is automatic and provides a direct path to automated test execution, but that’s not the topic of this blog. What I want to talk about is that in many client engagements I get asked the question: “How can model-based testing generate tests that I [the tester] can’t think of?”
Typically, when testers start modeling, they first sketch test cases. In a second step, they refine these cases by adding data to their model and perform tests. I find this very understandable. As testers, we have already specified so many tests during our career that it seems natural for us, when we model, to model the tests.
Generally speaking, I have no problem with this approach. There is no single right way to create a test model – there are many. But in this case, it is also obvious that this approach will only generate test cases that you already know – because that is exactly what we modeled in the first place. In the end, you get a number of similar test cases but with different data. If this result is acceptable to you, then it is acceptable to me – but then you have to accept that model-based testing in this case does not generate tests that you did not think of.
By following this approach, you can also get requirements traceability, a direct path to test automation and many more test cases. You get 100% coverage of your « test case model ». But what does 100% coverage mean when it comes to the actual functionality to be tested in the first place? 100% of our test model represents exactly what percentage of your functionality? Maybe by spending enough time thinking about, analyzing, and modeling more and more tests, you could reach that 100% coverage goal – but how long would it take, and what would such a model look like?
In my experience, they have been large, messy, and difficult for others to understand and reuse. Also, how can you be sure that you have covered the entire functionality to be tested? Are you sure that even on your bad days you will think of all the cases? Do you know which cases you missed ? Probably not, because the tests you missed are tests you didn’t think of.
Next, let’s talk instead about modeling the actual functionality described by the application requirements. Let’s talk about automated test design. At first glance, this approach seems more complicated than modeling tests, because it requires us to change our way of thinking. After all, we are testers and we have been designing and writing tests for years. Thanks to this experience, we know better than anyone the critical and difficult aspects of how an application works.
This changes in automated test design, because the experience is not as critical because testers simply model the operation to be tested, for example, as specified in the requirements specifications, from the perspective of « I am the application under test and this is how I interact with my environment » instead of « I am the tester and this is how I interact with the application under test ». In this approach, we don’t think in terms of tests but in terms of expected interactions and decision-making within the application under test itself: “How should I [the application] react when I get this kind of data from this interface? ” This type of question often reveals insufficient or imprecise documentation of requirements before we have a model or run a test.
Once all our requirements are reflected in the modeled functionality, the automated test design tool analyzes the modeled functionality based on the modeled data to cover all aspects of the system’s operating model [the tester] specified. Then, for the first time, you may find parts of the functionality or data that were not covered by test cases – which are exactly those “tests we couldn’t think of before”, that are now covered and documented. A computer will not have a bad day.
A computer will always “remember” to try all possible cases on a given part of the functionality. A computer will cover the really complicated operations, especially negative paths, which may be too difficult to take the time to manually design tests with scenarios. Ultimately, a computer will be able to trace in which test, at which step, a requirement or case was covered. Traceability is generated automatically – one mouse click – and no longer requires additional analysis of each test in a test set. In the same way, always up-to-date documentation is automatically generated.
To conclude, not all model-based testing approaches will give you test cases you wouldn’t have thought of, because many of these tools are based on automating tests from user-developed scenarios. Only automated test design based on system models can achieve this goal. At the same time, it is clear that automated test design requires testers to model the functionality to be tested.
Testers must identify and direct the computer to the critical parts of that functionality. Modeling the system is where the tester comes in, because a computer can’t think or read your mind (at least today). This approach doesn’t eliminate testers; it just makes them better and faster. The more data and features you add to your model, the more you will reap the benefits of automated test design.
Read more at: www.conformiq.com/blog/
We thank you.
