If you’ve worked in data for any amount of time, you’ve felt this tension:

Data is at the center of everything… and it’s also one of the easiest places for trust to quietly break.

For the last few years, I was leading data engineering, data warehouse, and BI teams at a high-velocity company. Our data org supported nearly every corner of the business: product, engineering, finance, GTM, customer support, HR – you name it. Each function had different questions, different definitions, different time horizons. And somehow, data always found its way into the heart of decisions that mattered.

On a good day, data told the story of the business – health metrics and KPI trends for leaders, board updates, forecasting, risk exposure, marketing performance, support staffing models, recruiting funnels. It wasn’t just “reporting.” It was an operating system.

And with that kind of reach, the margin for error shrinks fast.

When “good enough” becomes brittle

Like most growing teams, we built what made sense at the time, with the resources and urgency we had. But as we scaled, we started to feel the pain:

• brittle pipelines
• unanticipated edge cases
• downstream breakage that reached stakeholders before we did
• debugging sessions that felt like archaeology

We invested in the modern data stack to mature the platform—ELT automation, orchestration, transformation frameworks, and testing. 

These tools gave us strong unit test coverage across pipelines and robust dbt tests across our models. But even with automation and thoughtful manual testing, you can’t catch everything. There are simply too many edge cases, too many permutations, too many ways for data to drift in subtle but meaningful ways.

And that’s the uncomfortable truth: 

You can’t test what you can’t see.

We still had incidents that slipped through and hit dashboards, reports, and all sorts of integrations. That’s the moment data quality stops being a “data team problem” and becomes a company trust problem.

Why we explored observability—and why we chose Monte Carlo

We started evaluating observability tools because we needed a way to:

• detect issues earlier than our stakeholders
• reduce time-to-triage and time-to-resolution
• prioritize what mattered (we had thousands of assets)
• build confidence that our data products were reliable at speed

This is where Monte Carlo clicked for us. We weren’t trying to boil the ocean with bespoke testing frameworks or homegrown monitoring that would never keep up with the surface area. We needed broad coverage fast, with smarter signals than “did the job run?” 

Onboarding in the real world: fast, imperfect, and still worth it

Our implementation moved quickly—because it had to. The business wasn’t going to slow down so the data team could build a perfect reliability program from scratch.

A few things helped us ramp:

• We organized around data domains (ownership matters).
• We used Monte Carlo’s signals around table importance to focus on what the business actually depended on first.
• We uncovered assets safe for deprecation, which helped us cut tech debt instead of adding more surface area to maintain.

The other reality: onboarding observability is inherently cross-functional. Upstream sources are often owned by teams outside of data. When upstream contracts change, downstream pipelines break—sometimes silently. Having observability watching hundreds (or thousands) of tables wasn’t just “nice”; it became the mechanism that made cross-team accountability possible without finger-pointing.

What I would do differently (and what I’d recommend now)

Looking back, I’d still make the investment—but I’d be more intentional in a few areas:

1) Don’t try to monitor everything on day one—start small, tune fast, then expand

If I could do our rollout again, I’d be much more intentional about pacing.

Our initial instinct was to maximize coverage quickly. We turned on monitoring broadly across a large set of assets, thinking more monitoring would immediately equal more reliability.

In reality, broad coverage without tuning can create a new kind of problem: alert fatigue.

Because most data teams already operate with an “always-on” mindset—there’s an on-call rotation, there are pipeline failures, there are stakeholder pings. If observability alerts become one more noisy stream, even a great platform can get labeled as “just more alerts.”

The better approach is a phased rollout:

Start with the assets that truly matter.

Pick the critical tables/models and dashboards tied to:

• executive reporting and core KPIs
• revenue, risk, or customer-facing analytics
• high-visibility stakeholder workflows

Tune early and continuously:

• aligning freshness expectations to real business rhythms
• adjusting thresholds to reduce false positives
• routing alerts to the right owners so issues get handled quickly
• reviewing alert outcomes and iterating

Expand coverage once signal quality is high
Once alerts are consistently actionable in one domain, expand to the next. That’s how you build trust in observability—both for your data team and for the stakeholders who rely on the outputs.

The lesson I learned the hard way: if you try to blanket-cover everything before you tune, you risk creating noise before you create confidence.

2) Make reliability visible to stakeholders (because “trust” needs a scoreboard)

One thing I wish we had done earlier is treat data reliability like a first-class business metric, not just an internal engineering concern.

In the data team seat, it’s easy to assume stakeholders will “feel” the improvement once incidents go down. But stakeholders don’t experience your best weeks — they remember the day an exec dashboard was wrong, or a customer-facing report broke, or finance numbers didn’t tie out. Trust is emotional, and it doesn’t rebuild quietly.

Most Data Quality Initiatives Fail Before They Start. Here’s Why.

What I’d do differently next time:

• Build a reliability scorecard early (even if it starts simple).
• Share it on a predictable cadence (monthly is great).
• Slice it by data domain so accountability is clear and teams can see their progress.
• Tie it back to business outcomes: fewer escalations to leadership, fewer “pause the meeting” moments, faster decision-making.

Because once reliability becomes visible, you unlock two things:

1. you get buy-in for the work, and
2. you turn reliability into a culture — not just a tooling decision

3) Contain the blast radius when issues happen (detection is step one; containment is the multiplier)

The other big thing I’d do differently is move beyond “detect and alert” toward detect and prevent downstream impact.

Because here’s what every data leader learns eventually:

Most data incidents don’t hurt you at the point of failure — they hurt you at the point of consumption.

If a freshness issue happens upstream but still allows downstream transforms to run, your BI layer and downstream data products can happily refresh with incomplete or stale inputs. That’s how bad data ends up in the dashboards your C-suite sees on Monday morning. That’s also how you create a trust setback that takes months to undo.

This is exactly why I’m a huge believer in adding circuit breakers to orchestration workflows like Airflow.

Monte Carlo’s Circuit Breakers are designed to “stop pipelines when data does not meet a set of quality or integrity thresholds,” including checks between transformation steps or after ETL/ELT jobs execute but before BI dashboards are updated.

What this could look like (practical pattern):

• Your DAG runs ingestion and staging tasks.
• Before expensive downstream transformations (or before publishing/report-refresh steps), you add a gate task.
• That gate task checks a Monte Carlo condition (freshness, volume anomaly, schema drift, custom SQL validation, etc.).
• If the check fails, the DAG short-circuits / stops downstream tasks.

Why this matters in the real world:

• Protects executive-facing (or really any) dashboards: you’d rather show yesterday’s trusted data than today’s broken refresh.
• Prevents cascading failures: one upstream issue doesn’t trigger ten downstream incidents.
• Saves compute and time: expensive models don’t run on garbage inputs (and you avoid multi-hour reruns).
• Improves incident response quality: your team triages one contained issue instead of chasing symptoms across the stack.

If I’m honest, this is one of those “I can’t believe we didn’t do it sooner” things. We spent real engineering hours fixing downstream breakage that could have been prevented with a simple gating step tied to observability signals.

In hindsight, circuit breakers are less about being fancy and more about being disciplined:
when data quality is uncertain, don’t publish.

Why I’m excited to be at Monte Carlo now

It’s easy to romanticize perfect maturity when you’re looking backwards. But in the moment, the business is moving fast, priorities are competing, and data teams are expected to deliver high quality at high velocity.

After living that, I don’t see data observability as a “nice-to-have category.” I see it as the missing piece of the modern data stack—the connective tissue between pipelines, warehouses, BI, and the people who depend on the outputs.

Now I’m on the other side of the table as a Sales Engineer at Monte Carlo—helping teams build trust in data systems that are only getting more critical (and more complex). If you’re leading data initiatives or just want to talk about all things data, I’m always happy to connect and talk through what you’re seeing and where you’re headed.

Our promise: we will show you the product.