Successful testing needs clear context and appropriate documentation

They caution that just starting testing at random is a mortal sin. Broes explains: "First and foremost, the focus should be on the context: what are the expectations? Do not start from assumptions. You need to be familiar with the context and the customer requirements. The change program needs to be specific and crystal clear. And there must be an answer to the question: do cost and time determine which risks are acceptable or is it the other way around and do the acceptable risks determine how much cost and time should be spent?"

Eric hooks up and explains: "A tester tests. Testing is about verifying what works and what doesn't. Those who test are not responsible for solving problems. That has to be clear. Just as it should be clear if the customer wants it otherwise. It is important that you don't just test what has been requested. Test more broadly: positive, negative testing and exception flows, functional and non-functional testing strategies. Use your business expertise to go outside the scope. Test a lot, but be aware that testing everything is not possible. Time and money limit testing, but there are also the things you haven't thought of. A tester should ideally keep the mitigating actions in mind and think in terms of risk response."

Broes continues that change projects take place while the business keeps going. "There is a team that runs the bank and for which it is business as usual. Another team develops the changes that will change the status quo. A project like this doesn't come without risks. Project risks are for example not completing on time or going over budget."

"Change risk management, on the other hand, focuses on changes that do not have the desired effect in the aforementioned business as usual environment. What are the operational risks to the existing environment? Are there implications for operational performance and for the control framework? Testing is an excellent tool for minimizing risk. Once risks are identified, it is up to the project team to formulate a risk response."

"It is important to determine what is considered an acceptable level of risk. Context and constraints must be explicitly defined and documented. Documentation is the second cornerstone of change risk management. What you know today and tomorrow, you'll have forgotten six months from now. That's also why you need to have decisions validated by senior management."

There are different types of testing and Eric lists the six most common ones, which apply in an IT change process, but he believes are also applicable in other change processes. "You have the 'non-functional testing,' which has to do with security and performance, for example. Too often it happens that those are not tested or only at the end of the process when there is no time left to solve problems."

"The 'unit testing' deals with individual components or modules, unlike 'integration testing,' which deals with complex issues, built around multiple ‘units’. 'Regression testing' happens after every change because it can always have an impact. Something that was working may no longer do so. 'User acceptance testing' checks how things are going end-to-end when the customer uses the new environment. Finally, you have 'clueless testing,' where you assume that a user will do something at some point - press the same key six times, for example - that you didn't expect and that hasn't been tested. Five to 10 percent of the testing phase should be used for the latter."

Furthermore, Eric advises "Document what you test outside the scope. Define the stakeholders. Also, the management team and the end-user, who is often forgotten. Record your way of working. For shorter projects, use an existing test method, even if it is imperfect. Creating your own test method or dealing creatively with an existing one takes time which you cannot use for actual testing. For the same reason, too many meetings are not a good idea. Finally, establish whether the reporting is intended to be purely informative or supportive."

For Broes, it is appropriate, especially on larger projects, to include in the documentation the regulatory requirements involved and what you have done to meet them. "That helps to avoid difficulties when the regulator would scrutinize the results. Don't hesitate to involve the legal, risk, or compliance department in a timely/early manner. It will bring a strong added value. At the end of the ride, you don't wish to experience any nasty surprises."