Here’s everything you should keep in mind when building or buying a data observability platform.
To address this need, teams are becoming increasingly bullish on data observability, a new layer of the modern data stack that allows teams to keep tabs on the health of their most critical analytics and build organizational trust in data at scale. Just as robust observability and monitoring helps engineering teams prevent application downtime, data observability tools provide visibility, automation, and alerting that help data teams minimize and prevent data downtime.
Investing in data observability is becoming increasingly important as companies collect more and more (often third-party) data. Introducing new data sources and expanding access to new data consumers also naturally leads to more complex pipelines—which increases the opportunities for missing, stale, or otherwise inaccurate data to negatively affect your business.
With data observability, teams can answer common questions like:
- Is my data reliable?
- Where does my data live, and when will it be updated?
- How do I know when changes have been made to this data?
- Which data assets are most important to my business? How are they being used? Are they fresh and accurate?
As the need for data observability increases, some teams are building their own tools, while others look to existing solutions. While we couldn’t be more excited about this growing demand, we also understand that it can be challenging to cut through the noise and understand exactly what your data observability platform needs to deliver.
The following checklist will cover specific outcomes that a robust data observability platform should deliver, organized into four key categories:
- Knowing about data downtime
- Troubleshooting data issues
- Preventing data downtime in the first place
- Protecting your data with a security-first approach
Let’s dive in.
Knowing about data downtime
At a fundamental level, a data observability platform should monitor your data’s health and alert your team when pipelines break or jobs don’t run well before your downstream data consumers notice the issue in their dashboards or queries. Knowing what issues to monitor for can be overwhelming which is where a ML driven approach is critical. Instead of relying on manual tests to check for data quality issues, your data observability platform should intelligently surface when freshness, volume and schema changes are noticed.
The following are two examples of common data issues your data observability platform should automatically detect.
In the first example, we see a significant unexpected drop in data volume within a table, indicating a deletion of records which may or may not be planned.
In the second example, we see a schema change to JSON data stored within a table field, which similarly may or may not be planned, but will absolutely require downstream changes when parsing the JSON values.
Proactively monitor and alert for data quality issues
This is table stakes for any data observability platform. Just as software engineers rely on monitoring to alert them to any outages or downtime, data teams should be able to rely on monitoring to automatically notify them of any anomalies in data health rather than waiting for an internal consumer or customer to notice something’s gone wrong (which happens to a staggering 57% of businesses.)
Alerts routed intelligently based on data set assignments and incident type
Alerts are only helpful if the proper team (i.e., the data owner) receives them. Your data observability platform should enable routing based on assignments and incident types.
Centralize, standardize, and automate data quality monitoring processes
Data observability should apply across your entire organization, standardizing and centralizing data quality monitoring to ensure every team is speaking the same language and operating from the same playbook.
Automatically generate rules based on historical data
With machine learning, your data observability platform should be able to analyze trends in your data to automatically develop hundreds of rules without anyone in your organization needing to write a single line of code—identifying anomalies, null values, or missing data, and alerting your team accordingly for further investigation.
Provide the ability for data engineers to set their own rules and thresholds based on business-specific requirements
While automation saves manual labor, your data organization knows your data assets best. As a result, your data observability platform should include the flexibility for data engineers, data analysts, and data scientists to set their own thresholds to monitor data according to business needs or as new pipelines and projects are introduced.
Automated anomaly detection based on historical trends at the field level
The aforementioned ML-based anomaly detection should function at the field level—not just the table level–based on historical metrics and statistics about your data.
Data observability doesn’t just cover anomaly detection. A strong data observability platform will enable users to the data they need by indexing all assets (including BI reports and dashboards) for search and discovery. Data health metrics, augmented with business context, give your data consumers super powers when it comes to finding the right data for any given job.
Continuous monitoring and alerting against standard data quality dimensions as well as lineage
Alerts against quality dimensions are great: they notify you when data is missing, incomplete, or otherwise inaccurate. But without lineage—tracking all data integration operations on a record-by-record basis, across all pipelines and environments—you’ll have a much harder time figuring out what went wrong. Both should be provided by your data observability platform.
Troubleshooting data issues
The second category of outcomes — troubleshooting data issues so you can fix them, fast — should focus on making it easy to understand what data broke and why, while also giving your team the tools to track downstream and upstream dependencies before they impact the business.
Understand the business impact of issues and prioritize issue resolution
An intelligent platform will be able to assess the likely business impact of data downtime based on downstream data consumption—and prioritize issue resolution accordingly.
Intelligently route alerts to the appropriate team(s) via their preferred channel(s)
Alerts should be meaningful communications, not white noise. Not only should your data observability platform support intelligent routing of alerts to the right team members—ideally, those alerts should be delivered via their preferred channels (think Slack, email, or SMS text message).
Search for and explore data assets with a simple UI
When it comes to addressing broken data, speed and accessibility matters. A clean, simple interface helps your team search for and review data assets swiftly, using Catalog and Lineage functionalities to identify the root cause of incidents.
Collect and display information required for investigating and resolving issues
Your platform should deliver all the relevant information required to resolve issues at the alert or case level, reducing the need for manual investigation and speeding up the time to resolution.
Root cause analysis via end-to-end lineage
When your platform provides full visibility into your end-to-end lineage, you can take the guesswork out of root cause analysis by immediately accessing the full lifecycle of your data.
Collaborate to resolve incidents and mark them as notified, resolved, no action required or false positive
Your platform should make life easier by keeping everyone affected by a given incident on the same page with visible status reports on every issue.
Access incident alert logs with complete incident information for analysis and reporting
Comprehensive logs are a must: they help every team member stay in the loop, reduce time-to-resolution for common issue types, and enable data leaders to analyze and identify recurring issues.
API integrations to ensure that monitoring and alerting about key elements of your pipeline can be routed to the appropriate party
A modern data observability platform should offer thorough API access to all attributes so you can customize workflows that fit into your existing tech stack and empower your team to save time, avoid duplicative efforts, and solve problems swiftly.
Preventing data downtime
No matter how great your data observability platform is at identifying and alerting you to data downtime, your best efforts at maintaining reliable data pipelines will be all for naught if you can’t be proactive about incident prevention. A true end-to-end data observability solution will take data quality monitoring a step further by leveraging machine learning to understand historical patterns in your data through intelligent catalog and lineage capabilities.
Data quality and health information for all assets
Apart from alerting when something goes wrong, data observability tooling should provide a holistic view of data quality and health for all assets at all times based on both historical patterns and the current state of data affairs.
Understand fields and field types at a glance through lineage
When a data observability platform includes lineage, you should gain a “catalog”-level view into data fields and field types that provides an at-a-glance overview of the relationships between data assets and overall data health.
Understand critical reports and dashboards tied to table and field lineage
Data observability should enable data teams to understand which BI reports use which fields—helping ensure schema changes don’t unintentionally break BI reports or frustrate downstream data consumers.
Analyze field-level metrics/ statistics and how they’ve changed over time
Your data observability platform should be able to understand how the data is changing and evolving over time, so you can track important metrics at a granular level.
Dynamically update schema metadata and information
With a data observability platform, your metadata should update dynamically, without requiring manual changes.
Identify what your key assets are and how they are being used
Your data observability platform should answer pertinent questions, including: who is using this data, what are they using it for, when are they accessing this data, and how often does this data go “down” (i.e., stale, missing, etc.)? Data observability should include insights that help you understand how key data assets are being used, helping enable version control and allowing you to identify and deprecate outdated assets—ensuring you’re not using the wrong data for important business decisions or to power digital products.
Assess timeliness and completeness by analyzing table load pattern metrics
Your data platform should help you verify assumptions about your data’s freshness and volume, ensuring that your expectations meet reality.
Protecting your data
Data observability platforms should never require data to leave your environment — full stop. The best approaches will not require having to store or access individual records, PII, or any other sensitive information. Instead, data observability should only extract query logs, metadata, and aggregated statistics about data usage to ensure that your most critical data assets are as trustworthy and reliable as possible.
Monitoring data at rest
Your data platform should be designed with a security-first approach that monitors data at rest and does not require access to your data warehouse.
Your data observability platform needs to be SOC2-certified, which measures controls including security, availability, processing integrity, confidentiality, and privacy.
Identify breaking changes quickly from query logs and schema change logs
Your platform should use metadata such as query logs and schema change logs to automatically alert when changes break your data pipelines. By using this metadata instead of the data itself, your platform provides an additional layer of security.
Getting started with data observability
Ultimately, your data observability should enable everyone who interacts with your data, from an engineer to an end consumer, to answer a complex question with confidence: “Is this data trustworthy?”
Whether you build your own solution or buy one from a vendor, true data observability will bolster trust throughout your organization and open up new opportunities for teams to leverage data. At the same time, your data engineering team should get valuable time back from constant fire drills and manual investigations, allowing them to focus on innovating, solving problems, and helping your business grow.