Data Observability, Announcements

Monte Carlo Launches Insights to Help Data Teams Understand What Data Matters Most to Your Business

monte-carlo-insights

Lior Gavish

CTO and Co-founder, Monte Carlo. Programming wizard and lover of cats.

Monte Carlo Insights is the first solution on the market to offer customers operational analytics about their data platform.

“Is the marketing team using this dashboard?”

“What data can we deprecate?”

“Which tables should we test?”

“Why was my storage bill so high last month?”

“What data matters most to our business?”

If you’ve asked these questions of your data systems, you’re not alone. While great solutions already exist to help teams understand operational analytics for their products, there doesn’t exist a holistic, end-to-end way to understand the operational analytics and performance of your data platform. Until now.

Today, I’m excited to announce Insights, a new Monte Carlo capability that helps companies understand which data is most important for the business, and in turn increase data trust.

Built on top of the Monte Carlo Data Observability Platform, Insights leverages machine learning for monitoring and ranks events and assets based on their usage, relevance, and relationship to other tables and assets. Insights is the first solution to offer customers operational analytics about their data platform. With Insights, customers can measure and optimize the reliability, performance, cost, and effectiveness of their data initiatives.

Monte Carlo Insights gives data teams unprecedented visibility into the operational analytics of key data initiatives, including data governance, data quality, performance, and cost. Image courtesy of Monte Carlo.

On average, companies lose over $15 million per year on bad data, with data engineers spending upwards of 40 percent – or 120 hours per week – of their time tackling broken data pipelines. And all too frequently, data teams have trouble understanding what their most critical data is, preventing them from focusing on data that actually matters when it comes to ensuring quality and reliability. As a result, teams are either wasting cycles trying to figure out what datasets they should be prioritizing and end up missing tables when setting up coverage.

Monte Carlo’s mission is to accelerate the adoption of data by eliminating data downtime – in other words, giving data teams the tools necessary to trust their data. Insights puts the metadata Monte Carlo generates in the hands of data engineers to help them answer the most important questions around how their efforts ultimately lead to higher quality data. In partnership with rich, field-level lineage, businesses can get a holistic, end-to-end view of data health and utilization across the business.

With Insights, data teams can access the synthesized metadata Monte Carlo generates to build dashboards, analyze data platform team performance, and even commit to and track SLAs. The data itself can be downloaded as CSVs via the Monte Carlo CLI or in the app, and for Snowflake customers, can be accessed directly in their Snowflake environment via secure data sharing. This level of detail, common in software engineering and DevOps tooling, makes it possible for data teams to understand what data matters most to the business based on usage, access, data quality checks, and automatic lineage. Additionally, Insights makes it easy to create and share high-level reporting with CTOs and CDOs, fostering greater data trust and ownership across the company.

With the release of Insights, Monte Carlo is the first solution that can help companies understand:

  • Which tables are most widely used
  • Which tables are most frequently used
  • Who uses which tables
  • Which tables can be deprecated to reduce storage and processing costs
  • Data incident trend analysis
  • Data service-level agreement (SLA) dashboards
  • Which tables drive key assets for the business, such as financial reports or executive dashboards

After teams have developed processes to address data quality incidents with Monte Carlo, they often ask for help quantifying the impact of those efforts on the organization. Requests range from wanting to define SLAs and track performance, to tracking which teams are following data quality best practices, to which data assets merit investment. Insights addresses this need by providing end-to-end visibility into operational analytics across the data stack. 

Here’s what our customers have to say

ShopRunner to be Acquired By FedEx – Multichannel Merchant

“ShopRunner leverages data-driven insights to power business decisions and improve the user experience on our platform. To set our stakeholders up for success, we need to ensure that our data pipelines are performant and our data is reliable. As part of this vision for data trust, we’ve integrated our Snowflake and Looker data stack with Monte Carlo’s Data Observability Platform to gain unprecedented visibility into our data ecosystem through end-to-end lineage and ensure that data is accurate and up-to-date at each stage of its life cycle,” said Valerie Rogoff, Director of Data Analytics Architecture at ShopRunner. We’re excited to partner with Monte Carlo on their vision for Insights, a new and powerful way to understand what data assets matter most to our business and how we can better drive an impact with our data across the organization.”

PagerDuty Drives Auto Trader UK's Incident Response | PagerDuty

“Auto Trader leverages data-driven insights to power business decisions and improve the user experience on our platform. To ensure that data stakeholders are set up for success, we need to keep tabs on the reliability and health of our data systems, including usage, access, quality checks, and SLAs,” said Edward Kent, Principal Developer at Auto Trader UK. “Monte Carlo’s Insights Reports provides a novel and powerful way to understand what assets matter most to our business and how we can better drive an impact with our data across the organization.”

“As a data-driven organization, Kolibri Games produces over 100 million unique events per day across 40 different event types. To ensure that this data is trustworthy, we leverage Monte Carlo’s end-to-end Data Observability Platform,” said António Fitas, Head of Data Engineering at Kolibri Games. “With Monte Carlo’s latest product, Insights, we can better understand the operational analytics of our data platform, allowing us not just to trust our data but also optimize the storage, reliability, usage, and cost of our data products.”

How does Monte Carlo Insights work?

Monte Carlo’s field-level lineage traces data issues from the data warehouse or lake all the way to the field or column level in reports or dashboards. Image courtesy of Monte Carlo.

Companies have tried to build this kind of solution in-house, but the efforts are often ad hoc, manual, resource-intensive, and unable to evolve with business needs. Insights is fully automated so it can cover a company’s entire data stack. Our machine learning scores data assets based on active users, average daily reads, their relationship to upstream and downstream assets, and their connection to important service-level indicators and data quality checks. Such insights make it easy for teams to set data strategy and prioritize accordingly based on what data matters most to the business.

To learn more, check out our Insights docs or book a time to speak with us using the form below.

Our promise: we will show you the product.