CONTINUOUS VALIDATION - HOW IS IT DONE? - PART 2: VALIDATION
This blog post is Part 2 in this series. What are the key steps involved in continuously validating a Cloud App? (click here for Part 1)
The above diagram depicts the key elements of a Continuous Validation Program for a Cloud App. One has to bear in mind that the underlying IaaS and PaaS infrastructure is constantly changing. In fact, the Cloud App itself is continuously changing. In the new cloud world very rarely you are given the option of pinning to an "ancient" version. Thus this Continuous Validation Framework is designed to mitigate these risks and ensure that your cloud app is maintained in a validated state.
Requirements Definition: This step provides the foundation for the continuous validation framework. You need to clearly specify your Functional, Non-functional, Regulatory, Performance, Security, Logging, Disaster Recovery, Interface requirements.
Risk Assessment: Perform a thorough risk assessment to ensure business continuity, regulatory compliance, etc. are considered. In my experience very rarely a logical, useful risk assessment is performed, let alone applying it to testing strategies. The output of the risk assessment must be applied to your testing strategies to determine: What features to test? What Platform combinations to test on? What should be the extent of negative testing? What type of testing strategies to utilize (for eg: datasets to use, N Pair Testing)?
Specification Definition: Specify IaaS/PaaS, Configuration, Workflow, Interface, Security, Log Management, etc.. specifications to meet your requirements.
Test Automation Scripts: I recommend using a Model Based Test Automation framework for developing various models to validate your cloud app. Always separate your model from data which will be utilized during runtime. You can use a data designer to generate test data including randomized data. You can also use combinatorial testing strategies to reduce the number of iterations while increasing your test effectiveness.
Test Model Validation: The test automation model must be validated to ensure that it is meets the specified objectives. This validation effort can be minimized by designed your model to generate good execution reports. A well designed model execution report provides enough evidence so that it can to the extent possible becomes self-validating.
Test Execution: The model based test automation approach provides you with the flexibility to repurpose the same model to conduct various types of tests (smoke, regression, greedy path, optimal path, load, performance, etc..). Also, such a framework lends itself more conducive to updates (remember your cloud app is constantly changing...so will your test automation framework!). Such framework is well suited for continuously running validation test scripts (say on a daily basis) - you can even randomize your smoke and regression tests continuously!
Validation Reporting: A robust ALM (Application Lifecycle Management) tool is a must and forms the heart of your Continuous Validation Platform. Your goals must be to remove paper and manual generation of reports (in short: Microsoft Word will not play any role here!). Your ALM tool will provide real-time dashboards, KPIs, summary reports, test deviation reports and more. Your real-time Validation Health Dashboard becomes a reality.