A few weeks ago, I wrote a LinkedIn post mentioning how people are spending most of their time fixing (or discussing) the automation library/IDE issues instead of testing the actual system-under-test. Given that many testers don’t understand testing beyond point-and-click, this should come as a no-brainer. But how to overcome this problem and as a community, what can we do to make people start understanding the importance of generating value out of their automation work that is both measurable and observable?
To pull up this issue from the roots, we need to step back and understand a few core things which are driving businesses and software engineering for decades.
Composite Entities and Systems
As humans, every problem we come across involves composite entities that come in two forms – “Sets” and “Systems“. “Sets” are entities with which we usually don’t deal most of the time in businesses and engineering. These are the entities that are made of elements that are unorganised, not interrelated and hence do not perform reasonable functions together that bring value to society. Most of our problem-solving focus, in businesses and engineering, revolves around the other type – “Systems“. These are the entities that have interdependent elements that work together to affect some joint outcomes. In an ideal context, these outcomes are the ones that “should” drive businesses forward and add value to society.
Systems Understanding
Before we start testing, to derive a good understanding of the system-under-test, the learning focus needs to be on the integration of the different parts that form the system and also on the individual parts themselves. This will help us to derive relations between the different elements that form the complete system and how these relations drive the system’s functions and behaviour. Simultaneously we need to keep in mind that system behaviour will not only depend on the individual parts and their relations but also may depend largely on the environment in which it will operate, the data being used and the entities who will interact with them.
Systems Modelling
To understand the system-under-test in-depth, the system needs to be “modelled” (represented/abstracted at a high level for better understanding). It is essential to model the system to systematically define the problems intended to be solved by the system. It may be in a very basic form to start at the beginning and slowly modified as more system and problem understanding emerges. This will create a productivity increase for the test engineers who will be exploring it. Once the engineers start connecting the things, ideas and intuitions from these system models, they will start deriving valuable insights about the system and that’s what will set them apart. From that perspective, it does not matter whether the engineers are using tools to derive an understanding of the system (Test Automation) or not.
From a tester’s perspective, the understanding of the problem definition that the system is intended to solve is paramount. A difficult but important task for him will be to do problem formulation, problem conceptualisation and problem specification of the system as well as how the computer will help to solve the problem. Why?
The synergy between the different elements of the system can be constructive and destructive in nature and these synergies combined define the overall functionality of the system. In a functioning system, that provides value, constructive relations between the elements are greater than destructive relations. The tester’s effort will be to help maximise this ratio as much as possible. Too much isolated analysis of the system’s internal program structure, which we usually try to do by using traditional Test Automation ways, removes/shifts away a significant amount of focus from how the system should function as a whole. Systems Thinking and Systems Modelling can help us here to keep our focus balanced.
A complex system exists in many levels – Macro, Mesh, and Micro. We need to zoom in and zoom out of all these levels frequently to understand the overall system dynamics. System modelling can help us with this by guiding us with the understanding of the structure, function, data, platform and operations of the system.
Systems modelling techniques can be of various types and each type has a certain context where they are preferred.
In this article, I will aim to show you how using the narrative text + symbols-based “Use Case” technique of systems modelling, you can prepare to design your test automation work. The target will be the “uses” of the system.
Consider the below diagram:
Here, we have a system that we have to “test”. This, like most systems, can be represented using 3 different aspects:
1) Process
2) Software
3) Component
At a high level, the “Process” will involve the activities performed on the system. The “Component” will represent the independent, re-usable blocks of the system. The “Software” will represent the program structure/coding instructions (with data) that enable the system to act. The system is also defined by the system boundaries (which can be sometimes vague to completely apprehend) but important to be defined.
The system can be made up of multiple components which should work together based on some defined relationship between them.
[Note: I am not going into the details of how a monolith architecture or a microservices architecture (with its different service components) will look here as the principal message of this article may get lost in that discussion]
There are “Actors” (sometimes referred to as “Roles” or simply “Users”) who interact with the system and perform activities (processes) on the system. They may be anything outside the system that exchanges information with it. Also, they may not necessarily be humans but can be something different too – a software agent, an installer, an external system, a hardware, an OS and everything you can think of. They have roles, responsibilities and personalities that define how they interact with the system.
The system-under-test may interact with an external system too. This can be for anything e.g. sending or retrieving data or extracting some information on validations done by that system from its end. There can also be actors involved here who initiate the processes (acting as initiators) or facilitate processes on someone else’s behalf (acting as facilitators) in this external system.
All these can make up the whole system and several use cases can be derived from these elements and their relationships.
How can your Automation Design be mapped to this modelling?
To utilise the maximum benefit that Test Automation can offer and also to ensure that all the efforts that you will put into automation development have definite reasons to do so, I propose to map your System Models to your Automation Design. How can you do that?
First, create interfaces for each of the components. These interfaces will define the problem specifications that the components will help in resolving. It’s important to formulate, conceptualise, specify and define the problems first before putting them into the automation design. During the automation run, each of these problems should be acted upon appropriately in different circumstances and when acted upon by different actors (roles/users/systems) and data. Next, you should start writing implementations of these problem specifications using your preferred programming paradigms like Object-oriented programming or Functional programming. Using polymorphism and other OOPs/Functional programming concepts, make the automation program work appropriately for different circumstances (actors, environment, data). All these problem specification implementations should help you gather valuable insights into the state of the system-under-test. This is certainly not limited to the confirmatory checks, with assertions, that people usually write. You should use these implementations to their full potential:
– Detect anomalies in the system – detect the outliers (the data points and behaviour) that do not conform to how the system usually works
– Understand the performance behaviour of the system
– Gather reports on different aspects (process, component, software) of the systems for a given period or a given role
– Generate and share information logs – These are the logs that will only gather data at specific points of the system exploration. e.g. How much time it took to get a response from an external API/DB? What was the datetime when the network was slow? What happened to a particular element of concern in the webpage when you did a particular screen flow (e.g. imitating as an installer agent)? Later, you can feed these data to some system analyser tool that will help you to understand the system more.
– Generate Monitoring logs – These are the logs that can be specifically set up to check on how a particular part of the system will behave based on different events, at different times or when interacted with by different users.
– Recommendations. With the gathered data and other insights/patterns from the logs, you can write recommendation engines to provide recommendations based on what areas of the system seem to be more problematic and which engineers need to focus more deeply on.
– Visualization. Using visualisation tools, you can also visualise the behaviour of the system in its different states and then share those visualization outputs with the stakeholders. Everybody loves visual representations.
If external systems are involved, you don’t need to always interact with the external system if its testing is out of your scope. You can write mocks, and stubs or virtualise those systems’ services and use the responses in your automation.
All these implementations and automation work should be driven by a core automation framework. This reusable framework will utilise different libraries under the hood and hide/abstract out all the automation-related complexities. It will operate on the test infrastructure that you need to set up for integrating with CI pipelines or to perform automation execution on an ad-hoc/scheduled basis.
All the feedback gathered from these automation efforts should be shared with the Direct Shareholders (e.g. users, operations people) or Indirect Shareholders (e.g. customers, management, developers, testers) of the product for helping them know the present state of the system and take informed decisions based on those supplied information. We have to keep in mind that each of these stakeholders may have a very different idea/expectation of what they want from the system.
At a very high level, the whole structure will look like this:
