Application Observability vs Data Observability

In my former career as a software engineer ,I spent years transforming manual processes, typically done on paper, to digital workflows as a software engineer. 

Applications were not as complex then as they are now with modern, cloud-native microservice applications.

Older monoliths typically had a web front-end, a back-end written in Java or .NET, and a SQL database to “store all the things.” Monolith applications were all hosted on-prem because, at that time, “the cloud” literally meant the puffy white things in the sky above. 

So, when something broke in the application, there were relatively few places to look, or in my case, people to blame. My favorite was to point fingers at the network team–I’ll never forget the one time they unplugged my production SQL server because “they thought no one was using it.”

But, if it wasn’t the network or physical server hosting the application, I would look into the application logs or SQL. Pretty easy.

Data has much of the same story. There were no data warehouses or data lakes, nor were there data pipelines. Most of the time, data transformation was done by the code or in the database itself. So, again, not a lot of places to look when things went south with your data.

Modern architectures make simple monitoring obsolete

You might be thinking, “thanks for the history lesson, but what’s your point?” Glad you asked! 

When things were more simple, you could use traditional monitoring to be alerted when things broke, because there weren’t a lot of places where things could break. 

However, monitoring is a reactive stance — you have to know what you’re looking for to catch issues. Now, fast forward to cloud-native architectures and modern data platforms and pipelines — monitoring simply doesn’t cut it. 

There are too many places where things can go wrong and the volume of data is more than any human can sift through to find where something broke. 

In modern architectures, you need tools that are proactive. You need observability, which is measuring how well internal states of a system can be inferred from knowledge of its external outputs.

While the term observability is relatively new, the origins are not — they were a product of control theory which stemmed from research by James Clerk Maxwell in the late 1800s then furthered by Rudolf Emil Kalman at MIT in the 1950s-60s. The entire crux of observability is to dig through the vast noise of outputs to draw correlations and get to the root cause quickly.

Where app observability falls short

While working as a solutions engineer at a well known platform observability company, I helped customers with full-stack platform observability with tools like infrastructure monitoring, application performance monitoring, synthetics tests and real user monitoring. 

Many times, customers would also ask about the ability to monitor data for things like freshness, quality, volume, schema and lineage. But, application observability tools were not built for data observability.

Application or infrastructure observability will help you track when your application is throwing errors or is latent, and will usually tell you if it’s the code or the infrastructure or network that was the cause of the problem. 

It won’t tell you if your ELT jobs have failed, why they failed or what downstream reports are affected by that failed job. So to use a practical example, your application observability tool will tell you if Airflow goes down, but it won’t tell you if the DAG running your transformations for your sales attainment dashboard goes down.

Other common scenarios that will be missed include situations where the system is running fine but:

• The data running through the pipeline is incomplete or dupes
• The data running through the pipeline is hot garbage
• The data is correct, but someone changed the logic in the code so now the metric is incorrect
• The query failed
• The query ran successfully but no data was changed (empty or futile query)

Catching all of this requires a data observability tool, like Monte Carlo. 

In fact, Monte Carlo invented the term data observability and the Monte Carlo product was specifically built to track freshness, quality, volume, schema and lineage. You can read a much more thorough description of data observability in this blog, What is Data Observability? 5 Key Pillars To Know.

You can’t draw a line with a single point

The way data + AI observability tools like Monte Carlo work is they automatically correlate between the data quality issue and the root cause.

First, they leverage a combination of data tests and anomaly detection to detect the issue and then they correlate that issue with a change in the behavior from either the data source, ETL systems, or query code.

Application observability tools do not monitor at the data level, so they cannot draw the correlation between bad data and the root cause. And again, they also are only monitoring one small category of root cause (overall system performance) rather than the three main ways data can break (unreliable data source, failed job, query code change).

Apples and oranges

So, application observability tools will help solve why did my application break, while data observability tools will tell you why and where your data went south, so that the people who consume your data can trust your data. 

With many corporations today, data is one of their most valuable assets. Data trust is a huge concern, as it should be. The cost of data downtime can not only immediately and directly impact revenue but can also indirectly and have a long-lasting impact on revenue through loss of reputation.

And, as well all know, data is the number one ingredient we feed AI. It is therefore imperative that your data is AI ready. 

Are you confident you have quality source data? Even the smallest issues in data, embeddings, prompts, or models can lead to dramatic shifts in a system’s behavior. Do you have a system in place that can detect, triage, resolve and measure your data + AI? 

Each of these principles (detect, triage, resolve, measure) require the right tool that can empower your AI team to scale your reliability loop effectively across your data + AI estate and without this continuous baseline loop supporting your data in production, data + AI systems cannot be operated reliably — no matter how good the underlying model might be.

So, be sure your teams are using the right tool for the right task. Need help? Download the Data Observability Evaluation Guide.

Our promise: we will show you the product.