Circuit Breakers: A New Way to Automatically Stop Broken Data Pipelines and Avoid Backfilling Costs

Did you ever wish you had a pause button for broken data pipelines?

Well, today is your lucky day.

Monte Carlo is excited to announce the release of a new suite of data observability capabilities to help data teams automatically stop broken data pipelines at the orchestration layer — before they impact the business.  

Data engineers spend upwards of 30 percent of their time tackling data downtime, meaning periods of time when data is missing, erroneous, or otherwise inaccurate. These issues cost companies millions of dollars per year, eroding trust in the data that informs decision-making and powers digital services. 

Monte Carlo’s Circuit Breakers automates testing and manages all quality checks across the entire data pipeline within a single interface. Circuit Breakers limit the downstream impact of bad data by stopping data quality issues closer to the source. With this release, Monte Carlo becomes the first data observability provider to deliver both automatic and custom orchestration-based tests to monitor for data quality issues. 

A better way forward for managing data reliability at the orchestration level

Traditionally, data teams leverage manual tests to ensure that poor quality data doesn’t get passed downstream to stakeholders before going to production systems. To troubleshoot, teams must toggle between dozens of tools and platforms, slowing down the root cause analysis (RCA) process and making it difficult to develop a central source of truth about their data. 

By stopping data processing jobs when data quality rules fail, Monte Carlo’s Circuit Breakers reduce time to detection and resolution of data issues, ensuring that teams avoid backfilling costs associated with cascading data failures. Simultaneously, Apache Airflow logs, dbt models, and other metadata related to an incident are made available in Monte Carlo, alongside additional root cause analysis capabilities. 

In addition to helping prevent backfilling costs and improve engineering efficiency, Circuit Breakers allow data teams to: 

• Deploy data reliability checks directly into your data pipeline. For the first time, data teams can integrate Monte Carlo’s data quality tests directly into the orchestration layer, helping identify and stop bad data at its source. 
• Manage rules and tests within a single platform: Data engineering teams can create and manage unscheduled data quality checks with Monte Carlo. Updates to the rules are automatically reflected in Airflow DAGs and other data processing jobs.
• More easily triage data quality issues: Once a job is stopped, Monte Carlo alerts data engineering teams via Slack, Microsoft Teams, email, webhooks, and other communication channels so they can prioritize associated issues and reduce time-to-resolution.
• Prevent broken dashboards and reports downstream: Circuit Breakers that fail rules automatically “circuit break” or stop the data processing job, preventing tables, downstream dashboards and reports from being populated with bad data. As a result, negative business impact from wrong decisions made with inaccurate, stale, or duplicate data are eliminated.
• Set both automatic and custom rules for Circuit Breakers: Monte Carlo automatically generates Circuit Breakers based on common data quality checks, including null values, freshness, and distribution, with the option to set custom rules and alerts based on the needs of your business. 

With Circuit Breakers, data teams can more easily achieve data quality KPIs and service-level agreements (SLAs), reducing time-to-detection (TTD) and time-to-resolution (TTR) for data issues from hours or days to minutes. By automating Circuit Breakers, data engineering teams can better scale data reliability across their entire data stack, without needing to update code each time a change is required. 

Circuit Breakers reduces data downtime by 88 percent

Circuit Breakers takes the promise of reliable data a step further by making it easier for Monte Carlo customers like Optoro and Red Ventures to identify, solve, prevent, and communicate data quality problems before they affect the business, reducing data downtime by 88 percent.

“Optoro is committed to making the returns process more sustainable and seamless for retailers worldwide, from customer initiation to warehouse processing for restock and resale. Reliable data and analytics are critical to ensuring that we can deliver on this vision,” said Patrick Campbell, Lead Data Engineer, Optoro. “With Monte Carlo’s Circuit Breakers, we can catch data downtime with Airflow at the orchestration layer, avoiding backfilling costs and preventing cascading data quality issues from affecting downstream dashboards or data science models. With data observability, my team saves 44 hours each week that would otherwise be spent tackling broken data pipelines and responding to support tickets.”

“Red Ventures is committed to delivering reliable and accurate data products for teams across our portfolio of media companies. With Monte Carlo’s data observability, our data engineers and analysts can automatically and collaboratively detect, alert on, and resolve data downtime before it becomes a problem for the business,” said Brandon Beidel, Director of Product, Data Platforms, Red Ventures. “Features like Circuit Breakers will make it faster and easier to detect and resolve data quality issues at the orchestration layer, allowing us to gain greater visibility into our dbt and Airflow assets.”

Here’s wishing you no data downtime – broken pipelines!

To learn more, visit our docs or book a time to speak with us in the form below.

Our promise: we will show you the product.