What is Data Observability? 5 Key Pillars To Know

Editor’s Note: So much has happened since we first published this post and (created the data observability category) in 2019. We have updated this post to reflect this rapidly maturing space. You can read the original article linked at the bottom of this page.

---

What is Data Observability?

I coined the term data observability in 2019. Here is how I define it today:

Data observability provides full visibility into the health of your data and systems so you are the first to know when the data is wrong, what broke, and how to fix it.

The solutions in this category feature AI-powered anomaly detection, accelerated root cause analysis features, end-to-end data lineage, and observability agents that help teams create monitors and resolve issues. This approach results in healthier data pipelines, increased team productivity, enhanced data management practices, and ultimately, higher customer satisfaction.

In this post, I’ll dive into these concepts in more detail, including what data observability is, what data observability isn’t, how it’s evolving for AI, and how to choose the right data observability solution for your platform.

---

The origins of data observability

I started thinking about the concept that I would later label “data observability” when I was serving as the former VP of Customer Success Operations at Gainsight.

Time and again, we’d deliver a report, only to be notified minutes later about issues with our data. It didn’t matter how strong our ETL pipelines were or how many times we reviewed our SQL: our data just wasn’t reliable.

Unfortunately, this problem wasn’t unique. After speaking with hundreds of data leaders about their biggest pain points, I learned that data downtime tops the list. 

Data downtime — periods of time when data is partial, erroneous, missing, or otherwise inaccurate — only multiplies as data systems become increasingly complex, supporting an endless ecosystem of sources and consumers.

Much in the same way DevOps applies observability to software, I thought it was time data teams leveraged this same blanket of diligence and began creating the category of data observability as a more holistic way to approach data quality.

When I originally created this category, it was defined by five pillars of data observability: 

1. Freshness
2. Quality
3. Volume
4. Schema
5. Lineage

Together, these components provided valuable insight into the quality and reliability of enterprise data pipelines. But with the maturity of the category, the continued significance of data for the enterprise, and the advent of accessible production AI, both the problem—and the data observability category—have expanded dramatically.

These 5 categories are still critically important, and form the foundations of reliable data products today. But as you’ll soon see, to be effective in the modern data + AI landscape, data + AI teams need to go even further. 

Let’s jump in.

---

What are the five pillars of data observability?

“Observability” is a concept many data leaders are probably familiar with from DevOps, but when it comes to data, what is it that we’re actually observing?

As I said previously, I initially created the data observability category to observe what Monte Carlo defined as the 5 pillars of data observability. It was on the basis of these five initial pillars that the entire category—and what would become data + AI observability—was developed.

Freshness

Freshness seeks to understand how up-to-date your data tables are, as well as the cadence at which your tables are updated. Freshness is particularly important when it comes to decision making; after all, stale data is basically synonymous with wasted time and money.

Quality

Your data pipelines might be in working order but the data flowing through them could be garbage. The quality pillar looks at the data itself and aspects such as percent NULLS, percent uniques and if your data is within an accepted range. Quality gives you insight into whether or not your tables can be trusted based on what can be expected from your data.

Volume

Volume refers to the completeness of your data tables and offers insights on the health of your data sources. If 200 million rows suddenly turns into 5 million, you should know.

Schema

Changes in the organization of your data, in other words, schema, often indicates broken data. Monitoring who makes changes to these tables and when is foundational to understanding the health of your data ecosystem.

Lineage

When data breaks, the first question is always “where?” Data lineage provides the answer by telling you which upstream sources and downstream ingestors were impacted, as well as which teams are generating the data and who is accessing it. Good lineage also collects information about the data (also referred to as metadata) that speaks to governance, business, and technical guidelines associated with specific data tables, serving as a single source of truth for all consumers.

---

Why is data observability important?

Data observability is critical to the success of any data team. Bad data doesn’t simply mean bad insights (though it certainly means that)—it also means wasted resources, damaged trust, poorly adopted products, and untold damage to your organization’s reputation and its bottom line.

Gartner recently stated that they believe 50% of enterprise companies implementing distributed data architectures will have adopted data observability tools by 2026 – up from just ~20% in 2024.

If any organization wants their data or AI products to be valuable, the first thing they need to be is reliable.

Let’s look at some of the specific benefits of data observability in detail. 

Benefits of data observability

The benefits of data observability include improved trust, scalable monitor creation and incident management workflows, improved efficiency, reduced risk, and immediate time-to-value over manual data quality practices. The benefits of data observability extend far beyond the data team and data practitioners to the stakeholders, business users, and external consumers who leverage their data’s insights.

Enable data trust at scale

Scalability is one of the greatest deterrents to data trust. While traditional data quality approaches like data testing can address some of the most basic problems that appear within data pipelines (more on that below), they lack the scalability to be effective for all but the smallest teams—and they lack the kind of comprehensive approach that’s required to actually impact health or deliver greater trust over time.

Unlike traditional quality approaches that are predominantly manual, data observability tools leverage automation and AI to make creating monitors, resolving issues, and enabling teams fast and simple at any scale. This allows data teams to quickly improve the health of their critical products, empowering business users to make decisions and build data and AI products with confidence. 

Accelerate operational efficiency

Efficiency often follows scalability, and the same is true for data observability. Data observability doesn’t just accelerate the creation and maintenance of monitors, but it also provides automated lineage and resources like troubleshooting agents and root cause insights to make managing and resolving those incidents fast as well.

By making it easier for anyone to build monitors and manage incidents—and delivering a single pane of glass to see and measure the health of critical products across owners and stakeholders—data observability reduces the time spent on data quality issues and the time it takes to maintain a quality strategy in one fell swoop.

Avoid reputational risk

One of the primary ways that data observability delivers demonstrable value to the business at large—particularly executive leadership—is through its ability to reduce the frequency and the impact of financial and reputational risks.

From maintaining regulatory compliance through automated checks to AI profiling that programmatically identifies coverage gaps, data observability is a non-negotiable for regulated industries and beloved brands alike.  

Deliver immediate time-to-value

The best tool is the one you can use. One of the greatest differentiators between traditional data quality practices and a comprehensive data observability solution is its ability to deliver immediate value.

Through deep integrations, user-first design, and features like automating routing and ownership controls, data observability tools like Monte Carlo empower data leaders, data governance teams, data engineers, and practitioners alike to get up and running fast, and start receiving value on day one.

Quick set-up and thoughtful AI (like Monte Carlo’s monitoring recommendation agent, AI profiles, and automated lineage) is a stark contrast to the manual and unsustainable practices of manual testing and even traditional ML monitor creation.

---

How to calculate the impact of data observability

How you measure success for your data quality practice is just as important as the tool you use. From improving efficiency and increasing adoption to reducing the impact of data incidents in production, data observability delivers not just impact but real ROI for data leaders. That being said, it’s unlikely the chief financial officer is going to take your word for it.

So let’s take a look at how data teams measure the impact of data observability for their organizations and their bottom line.

A simple calculation for the estimated number of incidents you have each year (whether you are currently catching them or not) can be done by dividing the number of tables you have in your environment by 15.

You can then multiply this number by your average time-to-detection and average time-to-resolution. If you aren’t currently capturing these metrics, don’t worry, you are not alone. 

Our industry research revealed the industry average is about 4 hours and 9 hours respectively–feel free to use or adjust those estimates based on your organization’s data quality maturity.

Congratulations, you have just calculated your data downtime! Now let’s calculate its cost, and thus the value of a data observability solution.

The first part of the calculation, labor cost, is relatively straightforward. Since we know data quality professionals spend around 40% of their time on inefficient data quality practices, we can use this formula:

Total data engineers x 1804 (average working hours in a year) x $62 (average salary) x .4

The operational cost of poor data quality is a bit harder to quantify. A data incident can be as harmless as a broken dashboard no one uses or as painful as reporting incorrect numbers to Wall Street.

One way to estimate this is to measure the overall risk. If an organization invests in its data team to increase overall efficiency 10% (or insert your own value here) then for each hour of data downtime we can assume the organization’s productivity has been reduced 10%. The formula is then:

Overall data downtime x % reduction in hourly revenue generated.

We put together a data observability value calculator to help.

---

How Gartner defines data observability

With the current level of enterprise demand—particualry in response to the rise of AI—it’s not surprising Gartner has published its first Market Guide for Data Observability Tools. And this guide provides some fantastic benchmarks for enterprise organizations considering their first data observability solution. 

Here’s how Gartner officially defines the category of data observability tools:

“Data observability tools are software applications that enable organizations to understand the state and health of their data, data pipelines, data landscapes, data infrastructures, and the financial operational cost of the data across distributed environments. This is accomplished by continuously monitoring, tracking, alerting, analyzing and troubleshooting data workflows to reduce problems and prevent data errors or system downtime.”

Why Gartner’s definition matters

While Monte Carlo is unquestionably the data observability category’s creator, we’re certainly not the only game in town – and that’s a good thing. Competition not only validates the value of a category, but it drives innovation as well.

The greater the competition, the more important the problem—and the better the solution.

But with more vendors also comes more confusion.

According to Gartner:

“Vendors are offering a range of different capabilities branded as data observability, causing confusion in the market and tool adoption issues.” 

The truth is, it’s often easier to tweak a definition than it is to develop more comprehensive features. That’s where tools like the Garnet Market Guide become important. Analyst reviews give leaders an objective third-party’s opinion on common features within a category—and forward to that, what’s really a differentiator.

Key features of data observability

As Gartner sees it, data observability has several critical features, each with the express goal of helping data teams “resolve and prevent” data quality issues:

1. Monitor and detect: (What went wrong?)
2. Alert and triage: (Who should be notified and when?)
3. Investigate: (Why did it happen and what’s the impact?)

Gartner goes on to explain that some data observability solutions will also go one step further by even recommending solutions to answer the ultimate question: “How can this be fixed?”

As they describe it, “[some] issues are critical and require immediate solutions” and specific vendors will extend their value by leveraging root cause analysis tools to provide tangible recommendations to fix those problems, although “Only vendors with advanced technologies offer this feature in their data observability tools… This is a differentiating factor among vendors.”

What data observability should monitor

According to Gartner, data observability solutions should monitor five key areas: 

1. Data content: ensuring the quality and validity of the data itself
2. Data flow and pipeline: preventing interruptions within the pipelines
3. Infrastructure and compute: monitoring the evolution of code and compute consumption over time
4. User, usage, and utilization: helping teams understand how data is being used and by whom
5. Financial allocations: helping teams optimize the resource cost of their data and pipelines over time

For example, by extending automated data quality monitoring, data quality testing, and lineage features to the data, system, and code levels of the data stack, Monte Carlo empowers data teams to observe, manage, and measure the quality and performance of each of these areas at scale over time.

---

How to choose the right data observability tool

The Gartner description of data observability—and its capabilities—directly aligns with Monte Carlo’s own. When I created the category in 2019, I did so under the belief that the ability to detect and resolve data quality issues faster—and improve data health over time—was the most important value we could provide to enterprise data teams. 

Six years and one Market Guide later, I’m more convinced than ever. Evaluation criteria can be tricky when you may not even have a strong answer to the basic question, “what are data observability tools?” You can see what analysts have used as data observability criteria as well a a sample RFP, but in short a great data observability platform has the following features:

• It connects to your existing stack quickly and seamlessly and does not require modifying your data pipelines, writing new code, or using a particular programming language. This allows quick time to value and maximum testing coverage without having to make substantial investments.
• It monitors your data at-rest and does not require extracting the data from where it is currently stored. This allows the data observability solution to be performant, scalable and cost-efficient. It also ensures that you meet the highest levels of security and compliance requirements.
• It requires minimal configuration and practically no threshold-setting. Data observability tools should use machine learning models to automatically learn your environment and your data. It uses anomaly detection techniques to let you know when things break. It minimizes false positives by taking into account not just individual metrics, but a holistic view of your data and the potential impact from any particular issue. You do not need to spend resources configuring and maintaining noisy rules within your data observability platform.
• It requires no prior mapping of what needs to be monitored and in what way. It helps you identify key resources, key dependencies and key invariants so that you get broad data observability with little effort.
• It provides rich context that enables rapid triage and troubleshooting, and effective communication with stakeholders impacted by data reliability issues. Data observability tools shouldn’t stop at “field X in table Y has values lower than Z today.”
• It prevents issues from happening in the first place by exposing rich information about data assets so that changes and modifications can be made responsibly and proactively.

---

Data observability vs. data testing

Similar to how software engineers use unit tests to identify buggy code before it’s pushed to production, data engineers often leverage tests to detect and prevent potential data quality issues from moving further downstream. 

This approach was (mostly) fine until companies began ingesting so much data that a single point of failure just wasn’t feasible.

I’ve encountered countless data teams that suffer consistent data quality issues despite a rigorous testing regime. It’s deflating and a bad use of your engineers’ time.

The reason even the best testing processes are insufficient is because there are two types of data quality issues: those you can predict (known unknowns) and those you can’t (unknown unknowns).

Some teams will have hundreds(!) of tests in place to cover most known unknowns but they don’t have an effective way to cover unknown unknowns.

Some examples of unknown unknowns covered by data observability include: 

• A Looker dashboard or report that is not updating, and the stale data goes unnoticed for several months—until a business executive goes to access it for the end of the quarter and notices the data is wrong.
• A small change to your organization’s codebase that causes an API to stop collecting data that powers a critical field in your Tableau dashboard.
• An accidental change to your JSON schema that turns 50,000 rows into 500,000 overnight.
• An unintended change happens to your ETL, ELT or reverse ETL that causes some tests not to run, leading to data quality issues that go unnoticed for a few days.
• A test that has been a part of your pipelines for years but has not been updated recently to reflect the current business logic.

In a Medium article, Vimeo Senior Data Engineer Gilboa Reif describes how using data observability and dimension monitors at scale help address the unknown unknowns gap that open source and transformation tools leave open.

For example, if the null percentage on a certain column is anomalous, this might be a proxy of a deeper issue that is more difficult to anticipate and test.

Choozle CTO Adam Woods says data observability gives his team a deeper insight than manual testing or monitoring could provide. 

Without a [data observability tool], we might have monitoring coverage on final resulting tables, but that can hide a lot of issues. You might not see something pertaining to a small fraction of the tens of thousands campaigns in that table, but the [customer] running that campaign is going to see it. With [data observability] we are at a level where we don’t have to compromise. We can have alerting on all of our 3,500 tables.

To summarize, data observability is different and more effective than testing because it provides end-to-end coverage, is scalable, and has lineage that helps with impact analysis. 

---

Data observability vs. data monitoring

Data pipeline monitoring involves using machine learning to understand the way your data pipelines typically behave, and then send alerts when anomalies occur in that behavior (see the 5 pillars).

Some tools within the modern data stack, like Airflow for instance, will have the ability to monitor their portion of the ETL pipeline. While helpful, data teams need to monitor their entire pipeline end-to-end from ingestion to landing and through transformation all the way to consumption in the BI layer.

It is also important that data pipeline monitoring is supplemented with a process for monitoring the data quality itself. This is because while the pipeline may be operating fine, the data flowing through it may be garbage.

For example, the data values may be outside the normal historical range or there could be anomalies present in the NULL rates or percent uniques. Monitoring the data itself can be done automatically with machine learning as well as by setting custom rules, for example if you know a monetary conversion rate can never be negative.

When automated data monitoring is combined with features to accelerate incident resolution, understand the impact of those incidents, and illustrate data health over time, it then becomes data observability.

---

Data observability vs data quality

Data observability enables and improves data quality. Data quality is often expressed in the six dimensions of accuracy, completeness, consistency, timeliness, validity, and uniqueness.

I’ve found that among business stakeholders, the reality is data quality is considered a binary metric. Your CFO doesn’t come up to you and say, “the data was accurate but out of date so I’m considering it to be of average quality.”

Those other data quality metrics are helpful for data professionals to understand what’s not working and where to focus their resources, but for your data consumers either the data quality is good or it’s bad. Just like a SaaS solution, either it’s working or it’s not. That’s why we created the metric of data downtime.

With a data observability solution in place, data teams can ensure they have high data quality.

---

Data quality vs data reliability

These terms are often used interchangeably, which is fine. However, another way to think about it is that solving for data quality requires you to think beyond a point in time, and consider how the quality changes over time in a variety of real-world conditions. 

That is true data reliability.

For example, the quality of an airline might be measured based on its timeliness (percent of flights on-time), safety (major incidents) and food service approximating that of a diner. 

But in order for the airline to be reliable, it’s expected to maintain those levels of quality consistently over time, across various routes, weather conditions and holiday weekends. 

Similarly, a data product’s quality might be assessed by its availability at 9am, the completeness of records, and its consistency versus a source-of-record. For it to be reliable, you must assess whether it maintains these service levels over time, across holiday traffic spikes and product launches.  

In solving for reliability you must not simply measure data quality (at a point in time and space), but also establish expected levels of quality and service (i.e. how quickly you’ll communicate and respond to incidents), and have the toolkit to rapidly diagnose and resolve data incidents. 

---

Data observability vs data reliability engineering

Some data observability companies have started to describe themselves or their tools in the framework of data reliability engineering.

This makes sense as data observability borrows heavily from observability and other concepts of site reliability engineering (SRE). While different solutions or tools may have significant differences in features offered, there is no real difference between data observability and data reliability engineering. 

Both terms are focused on the practice of ensuring healthy, high quality data across an organization.

---

Signs you need a data observability platform

This 2019 talk helped introduced data downtime and data observability. It is still relevant today.

From speaking with hundreds of customers over the years, I have identified seven telltale signs that suggest your data team should prioritize data quality. 

They are:

• You’re data platform has recently migrated to cloud
• Your data stack is scaling with more data sources, more tables, and more complexity
• Your data team is growing
• Your team is spending at least 30% of their time firefighting data quality issues 
• Your team has more data consumers than you did 1 year ago
• Your company is moving to a self-service analytics model
• Data is a key part of the customer value proposition

---

How data observability is changing

Data observability is at the heart of the modern data stack, whether it’s enabling more self-service analytics and data team collaboration, adoption, or working alongside dbt unit tests and Airflow circuit breakers to prevent bad data from entering the data lakehouse in the first place. It is also a core component of important and emerging data quality best practices such as data mesh, data SLAs, and data contracts.

Google Trend data on data observability from the start of 2021 through the start of 2024.

But the modern data stack isn’t just for data anymore. The burgeoning AI industry is upending the data platform, and forcing pipeline architecture to evolve at the fastest pace we’ve seen since the cloud revolution—accelerating the scale and complexity of quality issues with it.

Here’s how the data observability is evolving to meet the challenge. 

Data observability for unstructured data

Observing the data itself has always been critical to ensuring the reliability of data pipelines. However, due to the limitations of the technology available, that data was predominantly structured.

However, in the age of AI, teams are leveraging the power of LLMs to activate unstructured data as well in order to provide critical context for their applications, improve access to new insights, and derive more value from their data + AI products. If a platform is going to provide true data observability, it needs to increasingly include unstructured data as well.

Solutions like Monte Carlo’s data + AI observability empowers data teams to identify, triage, and resolve issues in their unstructured data in Snowflake, Databricks, and BigQuery.

Check out this blog to learn more.

Observability for AI applications

Unlike traditional data pipelines which have been informed by observability approaches for years, AI is still largely a black box. We can sometimes see what went wrong, but it’s not always clear why or how. Natural language inputs, dynamically generated outputs, and foundational models trained on largely unknown datasets are all muddying the waters for agentic reliability—and traditional approaches to data quality aren’t enough to mitigate it. 

At its most basic, AI is a data product—and the success of any data product is wholly dependent on the reliability of the first-party data that powers it. It’s no longer data and AI indepedently—it’s data plus AI in a single end-to-end system.

And that means we need to manage it that way. So, let’s take a look at some of the challenges inherent to the unification of data + AI—and how the future of data + AI observability aims to solve. It.

---

What is data + AI observability?

Data + AI observability is an end-to-end reliability platform that extends observability into all four components of AI (data, system, code, model) with AI agents and automated incident management to improve the reliability of any data + AI application. 

As I said before, data and AI are no longer two separate technologies—they’re one and the same. AI failures often begin with data issues, but that failure can cascade into silent model drift, buckling systems, unforeseen code changes, and more—and that risk is compounded in agentic applications where multiple AI workflows can be strung together to deliver a single output.

Ultimately, these failures can be boiled down into one of four root causes:

• Data—for example, bad data from a problematic data source (sent a bunch of NULLs)
• System—you can have a system failure (like an Airflow job failed), can be dbt or Informatica, or just transformations you are doing in Snowflake (and let’s be clear, all systems will fail at some point)
• Code—breaking transformation code, a breaking schema or a bad join, a change in the transformation pipeline, OR breaking code that goes in the AI application—or prompting engine—that can influence reliability of outputs
• Model—even with the perfect prompt and the most perfectly curated context data, the model itself can generate an output that’s unfit for purpose, e.g. hallucinations

Like traditional data pipelines, it would be impossible to write a rule or a data test to catch every possible anomaly within an agentic workflow. (And even if you could, you wouldn’t be able to maintain an ever-growing catalog of rules and checks over time.)

By providing automated observability into each of the core components where these breaks happen, data teams can effectively scale the reliability of their agentic pipelines over time in the same way they would traditional data products.

Now, let’s take a look at each of those core components in a little more detail. 

The 4 Foundations of Data + AI Observability: Data, System, Code, & Models

I just discussed the four ways a data + AI system can break, and each of those breaks aligns to a core component of an AI agent—data, system, code, and model.

These four core components form the foundation of data + AI observability. 

Observing data

First, you have the data feeding your modern data and AI platform. At its most basic, AI is a data product. Both the foundational models and the enterprise applications they support rely on a vast collection of structured and unstructured data to create outputs and deliver stakeholder value. From model training to the RAG pipelines, data is the heart of the AI—and any data+AI quality strategy needs to start here first. 

Data observability was already the preeminent solution to build and scale data health for the enterprise—and just like Gartner identified in their own AI-readiness report, this modern data quality solution has become the foundation for production-ready AI as well. 

Observing the system

Much like its origins, data + AI applications—and agents more specficically—rely on a complex and interconnected web of tools and systems to manage and facilitate model outputs. Among its most basic system components, you might include your warehouse or lakehouse, ETL tools like dbt and Fivetran, orchestration tools like dagster and Airflow, or Langchain to lash it all together with your respective LLM. 

Each of these systems has the potential to cause catastrophic breaks in an AI agent, which means that each of these systems needs to be observed carefully. If it matters for your data pipelines, it matters even more for your agentic pipelines. 

Any solution that’s going to protect the reliability of your AI system needs to provide comprehensive integrations into the infrastructure that powers it.

Observing code

Just because you have a data problem doesn’t mean you have a problem with your data. Bad code has always been a challenge for software and data products. Bad pull requests, breaking schema changes, and simple human error have been wreaking havoc on data pipelines for years. 

But code takes on new weight in the data+AI system. Observing your code in the context of an AI agent could include traditional codebases like SQL to move and transform your data, the miscellaneous code used to build and control your agents, and even the natural language prompts that trigger a model. 

Observing the model

Finally, you have the model responses themselves. Much like the dashboards and ML models of old (but not that old), the response is your customer-facing “product.” But as we discussed earlier, the AI responses of today aren’t nearly as simple as the data products of yesteryear (were they simple?). These model responses rely on an everything bagel of data and black box generative algorithms to deliver its final outputs. Coverage for metrics like response relevance and precision will be considerable focus areas here.

---

The future of data observability is data + AI.

Data observability is the foundation of reliable AI. However, to deliver a truly reliable AI pipeline we need to extend that coverage into the systems and responses that bring these two powerful technologies of data + AI together. 

Without scalable end–to-end visibility into each of the four core systems of data + AI simultaneously—and the resources to immediately act on an issue—you allow technical blindspots that can easily cascade into multi-point failures within an agentic workflow. 

An effective approach to AI reliability should provide coverage for each of the four systems (data, system, code, and model response) within a single, continuous, and programmatic solution that goes beyond detection to make alerts actionable.

What’s more, the future of data + AI observability won’t just support AI; it will be powered by it. A tailored AI-first approach like Mont Carlo is the best way to scale coverage for AI—including thoughtful features like SQL authoring with AI, Monitoring Agents, and soon, Troubleshooting Agents.  

This is the future of data observability. Are you ready for it?

[The original article that launched a category: What is data observability? Hint: it’s not just data for DevOps ]

---

If you want to learn more, reach out to Barr Moses and book a time to speak with us via the form below.

Our promise: we will show you the product.

---

Frequently Asked Questions

---