Announcing Monte Carlo’s End-to-End Field-Level Lineage to Help Teams Achieve Data Reliability
Monte Carlo’s field-level lineage helps data teams track column-level dependencies from ingestion in the data warehouse or lake to dashboards and reports in the BI layer.
It’s Friday evening, and you are wrapping up after a long week of work. Before logging off, you make a schema change to a table in your warehouse – and don’t think twice as you close your laptop and get started on your weekend.
Monday morning, chaos strikes. You’re woken from your relaxing weekend to messages flooding your inbox and Slack channel:
“The data doesn’t seem right.”
“What happened to this table?”
“Why is my data in this report missing?!”
Remember that one small schema change? Well, apparently it was the deployment heard around the world, now causing confusion, frustration, and lack of trust among your company’s Business Intelligence team (let’s not get started on your COO…).
Making matters worse, the majority of your time on Monday is spent trying to figure out what other tables were affected by this change downstream, alerting the appropriate people that rely on those tables for analysis, and working backwards to fix the issue. All of this could have been avoided if you could understood the relationship between table columns and fields in downstream reports and dashboards.
Here is why we are excited about field-level lineage and you should be too:
- Field-level lineage makes it faster and easier for conducting root cause and impact analysis for critical data issues. Users are now able to look below the basic lineage relationships to understand the intricate and complex relationships between fields in your tables. Users can gather a better understanding of how fields are used, i.e. directly SELECT’ed across tables, or indirectly filtering results of a downstream table using a WHERE clause.
- Access to field-level lineage allows our customers to reduce the Time to Resolution when data incidents occur using Incident IQ.
- Field-level lineage allows teams to understand the effects of changes as they relate to columns that may be shared across tables or parsed for use in specific reports, narrowing down the scope of investigation when it comes to understanding the root cause and impact of data issues. Instead of needing to figure out which of the hundreds of fields from table-level lineage might be the source of the problem, data teams now need only to investigate a shortlist of fields upstream, saving valuable time and resources.
How it works
Figuring out field-level lineage across tables is not easy without a fully automated data observability solution. In most cases, data teams need to manually inspect complex SQL code to understand how data flows from one table to another. Due to this complexity, data teams were often flying blind, avoiding changes or requiring tedious efforts to approve and test them.
Now, with field-level lineage fully automated, data engineers and analysts can confidently make changes to tables on Friday evening and log back on Monday morning without having to worry about losing trust and visibility in their data, at each stage of its life cycle.
Monte Carlo customers like Vimeo, ShopRunner, and Software.com are already using field-level lineage to help them understand the root cause and impact of downtime across their data ecosystems.
“With Monte Carlo’s field-level lineage, our team can automatically and seamlessly connect the dots between upstream field changes in our Snowflake data warehouse to downstream reports and dashboards in Looker. It’s been a game changer for our team, helping us reduce time to detection and resolution for critical data incidents that would otherwise go undetected,” said Pamela Dalal, Head of Data, Software.com.
Interested in learning more about field-level lineage with Monte Carlo?
Check out our documentation and book a time to speak with us using the form below!