Landscape data observability for Azure

Introducing Glass...

Rapidly raise and diagnose any issues with your data and pipelines, enabling fixes before your business users even notice any issues.

Intelligent [cloud] Data Platforms (IDPs) such as Azure have become critical components of enterprises, supporting not only analytics but integrations, business apps, AI automated decision making/bots, and more. IDPs provide this service backbone, facilitating the flow of data in and out, across the enterprise ERP landscape including 3rd party sources.

Mature Enterprise IDPs will have hundreds if not thousands of data pipelines moving and transforming the latest datasets into Common Data Models (CDMs). Data from these CDMs is subsequently transferred into Data Marts, Datawarehouses and Semantic layers to be consumed by Intelligent processes. The question is – how do you know if the data in your CDMs is fresh and of good (gold) quality? There are lots of possible failure modes across the enterprise landscape components that can result in data pipelines not firing as expected, leaving CDMs out of date, losing integrity - from credential expiration (source ERP/Azure Service Component), network issues, delta volume timeouts and more.

Normally, data integrity issues caused by mis-firing data pipelines are spotted by end-business users, days or weeks after the failure event(s) and take a considerable amount of time for admins to trace through audit logs across the landscape to root cause and resolve the issue(s)…it can be weeks before issues are actually fixed….all reactively.

In partnership with Microsoft, we've developed a new Platform as a Service (PaaS) product named Glass. It provides a centralised control panel for monitoring all data flows, from their entry into and exit from Common Data Models (CDMs), proactively ensuring data integrity across all consumer processes. Beyond mere pipeline monitoring, Glass actively tracks data as it traverses the data estate, identifying anomalies, detecting metadata changes, monitoring volume drifts, and evaluating data quality. Glass stands as a unified data observability solution for your data estate, offering immediate issue alerting capabilities and facilitating a rapid, centralised review of landscape audit logs for swift root cause analysis.