Monte Carlo Announces dbt Core Integration to Help Companies Ship Reliable Data Faster

When it comes to trusting your data, Monte Carlo, the leading data observability platform and dbt Core are better together.

“Why didn’t my job run?” 

“What happened to this dashboard?” 

“Why is this column missing?”

“What went wrong with my data?!”

If you’ve been on the receiving end of a broken data pipeline, these questions probably look familiar to you. And as data ecosystems become increasingly distributed and complex, the likelihood of this “data downtime,” or in other words, periods of time where data is missing, inaccurate, or otherwise erroneous, only grows.

Fortunately, there’s hope.

Drawing corollaries to application downtime and the world of software engineering, data downtime can be addressed through a robust approach to developing, testing and observability. Just as an engineer would never deploy code to production without testing it first, or run production software without application performance monitoring and observability, data teams can apply a similar “testing + observability” framework to their data pipelines.

To this end, I’m excited to announce Monte Carlo’s integration with dbt Core, helping teams ship more reliable data, faster, through robust testing and monitoring. This new integration will allow data teams and business stakeholders to focus on the work that adds the most value, with confidence in the integrity of their underlying assumptions.  

Monte Carlo and dbt: better together

Our latest dbt integration helps data teams ensure data is reliable and trustworthy at each stage of their data pipelines, during transformation with dbt tests all the way across production environments with Monte Carlo’s end-to-end, fully automated Data Observability Platform. Additionally, data teams relying on dbt Core to transform and test their data before it goes off to the end-user will be able to leverage Monte Carlo’s automated anomaly detection, monitoring, and end-to-end lineage to prevent bad data from eroding trust and slowing down teams.

Previously, Monte Carlo customers could import dbt tags and descriptions into our data observability platform to centrally manage all of their metadata in one place. With the latest release of our dbt Core integration, Monte Carlo and dbt Labs customers can:

• Troubleshoot data incidents by checking associated dbt models and tests run results in Monte Carlo.
• Map out dbt models to tables in their database, gaining insight into each table’s dbt model name, location, model code, and last run time.
• Import dbt tags and descriptions on table and field levels, allowing developers to manage all metadata in a single place.

 The integration will also enable data teams to automatically monitor for failed dbt model runs and conduct root cause analysis through our Incident IQ dashboard to determine if any dbt run or test failures contributed to the incident. 

Check out the latest features from our dbt integration below:

Here’s what some of our mutual customers have to say about the Monte Carlo – dbt Core integration:

Optoro

Washington, DC based Optoro is on a mission to make retail more sustainable by eliminating waste from returns for industry leaders in the retail space like Ikea, Target and Best Buy. Their organization uses data to re-route inventory to the best locations. To achieve data trust and tackle data quality at scale, they turned to dbt and Monte Carlo to solve their data quality issues. 

“We use dbt to test and transform data after it enters our warehouse and Monte Carlo to monitor for data quality issues at every stage of the data pipeline,” said Patrick Cambell, Lead Data Engineer at Optoro. “Now, we are the first to know when data quality issues arise, rather than stakeholders downstream.”

Auto Trader UK

Manchester-based Auto Trader is the largest digital automotive marketplace in the United Kingdom and Ireland. Their platform connects millions of buyers to sellers, and handles thousands of customer interactions a minute. As the data team migrated to trusted, on-premises systems to cloud they needed a way to ensure data is trustworthy to more than 50% of all Auto Trader employees. 

“With dbt and Monte Carlo, we know our data is reliable and trustworthy,” said Edward Kent, Principal Developer at Auto Trader. “This integration signifies a commitment to adopting software engineering best practices for DataOps, particularly as it relates to validating and monitoring data as it evolves across its life cycle. We look forward to the continued innovation and collaboration!”

Kolibri Games

Berlin-based Kolibri Games has had a wild ride, rocketing from a student housing-based startup in 2016 to a headline-making acquisition by Ubisoft in 2020. While a lot has changed in five years, one thing has always remained the same: the company’s commitment to building an insights-driven culture based on accurate and reliable data.

“With over 100 million unique events produced per day across 40 different event types, our games generate an unprecedented amount of data, and in order to trust it, we need to prevent bad data from entering our pipelines and know when incidents arise downstream,” said António Fitas, Head of Data Engineering at Kolibri Games. “Monte Carlo and dbt are the perfect tools to help us achieve the level of trust and reliability as we scale our data platform in 2021.”

If you are a current Monte Carlo customer and want early access to the integration, follow our documentation to get up and running. 

Want to learn more about our latest integration with dbt core? Check out our documents or book a time to speak with us using the form below.

Our promise: we will show you the product.