A Step-by-Step Guide to "continuously" Qualify Microsoft Azure PaaS and Serverless Architectures

I very frequently get asked whether Azure Platform-as-a Service (PaaS which includes Serverless Architectures) can be qualified as well?  Surprisingly, this question is asked even by Azure users who agree that Azure IaaS can be qualified.  In this blog post I will provide a prescriptive guide to intelligently qualify Azure PaaS and maintain it in a qualified state (QS). 

Before reading ahead, one must understand that a Software App that resides on IaaS/PaaS services is “validated” while Iaas/PaaS services are “qualified”.   A SaaS application can be truly “validated” only if the underlying IaaS/PaaS services are fully qualified.

“Continuous qualification (CQ) is providing documented evidence to certify that a PaaS service not only met the pre-established acceptance criteria, but “continuous” to meet thus mitigating the risk of unknown changes.”

Step-by-Step Guide

Step 1:  Establish the “formal” user requirements for your Azure PaaS service.  The user requirements are the “intended use” of the PaaS service across your organization.  Once this Azure PaaS service is qualified for its intended use, any development team within your company can use the “qualified” PaaS service in their projects.  

When developing the intended use, I recommend that you categorize your requirements into:

  1. Functional

  2. Security & Compliance

  3. Availability

  4. Performance

Step 2:  Establish a risk-based approach to qualifying each requirement.   Your risk-based approach can be achieved by assigning a Risk Priority to each requirement as follows:

High – A risk priority of High shall be assigned to a critical requirement which meets the following criteria:

- is not “out of the box” (OOTB) functionality AND
- is a legal/regulatory requirement.

All High priority requirements will be tested (both positive and negative testing).

Note:  High risk priority may not at all be applicable in this context.

Moderate – A risk priority of Medium shall be assigned to an important requirement which meets the following criteria:

- is achieved with “out of the box” PaaS features; AND
- is a legal/regulatory requirement;

All Moderate Priority requirements will be tested (positive testing) or verified (configuration verification).

Minimum – A risk priority of Low shall be assigned to a “nice to have” requirement which meets the following criteria:

- is achieved with “out of the box” software features.

Minimum Priority requirements will not be tested.

Step 3: Build a "continuous" qualification framework to qualify your Azure PaaS service. This framework can automatically perform various tests to ensure all the applicable High/Medium Priority requirements are met. The test execution reports with all evidence are automatically generated.   IT Quality reviews the results and then certifies the Azure PaaS service for its intended use.

Step 4: Make your qualified Azure PaaS Service available in your global catalog.  Your global teams can deploy this PaaS service any number of times without worrying about qualification (Qualify it once and use it many times!).

Note:  When such a qualified service is provisioned, the end-user needs to review your SLAs (in this case the intended use requirements that the PaaS service is continuously qualified for).  If they have additional requirements, then such requirements need to be managed under change control and the continuous qualification framework updated accordingly.

Step 5: Establish a qualification schedule wherein you constantly (for example: daily) run your qualification tests and automatically provide evidence that the “qualified state (QS)” of the Azure PaaS service has not drifted (See FAQs below for more details).
 

Conclusion

By following the above five (5) steps, you can build and maintain a qualified PaaS service that is GxP compliant and always "audit ready".   The above framework uses the mantra "qualify it once and use it many times.

Frequently Asked Questions (FAQs)

Microsoft Azure releases changes constantly. How will I maintain change control?

A true "Cloud" is designed to constantly release changes so that the end customer can leverage these innovations and thus increase productivity. Azure PaaS is no exception here. However, this presents a dilemma for the traditional validation folks who are used to reviewing each change and then addressing it one way or the other.  

If you want to embrace the Cloud, the compliance perspective must change from examining every change by the Cloud Provider (which is practically impossible considering the velocity of changes) to ensuring your requirements are met constantly. This can be achieved with the "continuous" validation framework whereby you are constantly (for example: daily) testing to ensure your requirements are met in spite of the changes. 
 

How does "continuous" qualification really work?

Continuous Qualification is GxP compliant and based on a sophisticated Model Based Testing Framework.  Once the Continuous Qualification model is built, it can be used to perform initial qualification of a PaaS service. It can be run at regular intervals to "continuously qualify" with no human intervention ("lights out mode").  This approach enables cost effective testing on a continuous basis thus enabling deployment of GxP workloads in the public cloud.
 

Why should monitoring requirements be included in Step 1 above?  Isn’t addressing only Functional & Regulatory Compliance requirements sufficient?

Once the PaaS service is built and deployed, you need to monitor its health and also ensure that the Qualified State (QS) drift has not occurred.  QS drift can occur when changes to the deployed service are made (either intentional or unintentional) after it is qualified by bypassing the change control process.  Also, Microsoft Azure provides various services that will help you provide further assurance that the “qualified” service is functioning as expected.

  1. Automation & Control enables continuous services and compliance with automation and configuration management. You can apply and monitor configurations using a highly available pull service, and fix configuration drift without manual intervention. You can combine change tracking with configuration management to identify and apply configurations and enable compliance. This service will enable compliance with the Qualified State (QS). It will bring to your notice if a QS Drift has occurred.

  2. Log Analytics enables you to quickly connect and collect log data. You can correlate and analyze using powerful machine learning constructs. You can transform your Azure activity data into actionable insights. You can automate and trigger remediation with Azure Automation, Logic Apps and Functions.

  3. Azure Monitor will help you get detailed, up-to-date performance and utilization data, access to the activity log that tracks every API call, and diagnostic logs that help you debug issues. Azure Monitor gives you the basic tools you need to analyze and diagnose any operational issue, so you can resolve it efficiently.

  4. Azure Advisor helps you optimize across four different areas – high availability, performance, security, and cost – with all recommendations accessible in one place on the Azure portal. You can follow recommendations based on category and business impact.

  5. Security & Compliance helps you analyze events across multiple data sources and identify security risks. You can understand the scope and impact of threats and attacks to mitigate the damage of a security breach. You can understand the security posture of your entire environment regardless of the platform. You can capture all the log and event data required for security or compliance audits.

  6. Azure Policy helps you turn on built-in policies or build your own custom policies to enable security and management for Azure PaaS resources. You can choose to either enforce policies, or audit policy compliance against best practices.

Why are performance requirements included in Step1 above?  Isn’t addressing only Functional & Regulatory Compliance requirements sufficient?

Azure provides various tools to monitor the performance of your PaaS services.  By continuously monitoring the performance, you are also ensuring compliance.  For example, if a particular performance parameter falls outside the pre-set acceptable range limit, as compared to its historical data, an investigation may be warranted.   This may be a symptom resulting from a change or a patch that has been applied. 

  1. Azure SQL Database Intelligent Insights lets you know what is happening with your database performance. Intelligent Insights uses built-in intelligence to continuously monitor database usage through artificial intelligence and detect disruptive events that cause poor performance. Once detected, a detailed analysis is performed that generates a diagnostics log with an intelligent assessment of the issue. This assessment consists of a root cause analysis of the database performance issue and, where possible, recommendations for performance improvements.

Post, PublicationNagesh Namacontinuous validation, azure qualification, xLM, managed service, IT Infrastructure QualificationComment