Data Observability, ML Model, and Agent Observability In A Single Pane Of Glass

Monte Carlo just launched machine learning metric monitors allowing users to monitor and validate ML model performance— no SQL, no formulas, no data science overhead required.

Monitoring ML inputs and outputs

Data + AI teams need to monitor the features (essentially metrics) that are input to ML models to generate the prediction, or inference. When the data quality of the inputs are poor, so are the predictions.

Not only can poor performing models be expensive to run, but operation inefficiencies and cost quickly add up when bad model predictions automate operations.

For example, the digital advertising platform Kargo leverages ML models to acquire digital advertising inventory. They avoided a five digit data quality incident thanks to a Monte Carlo alert.

But it is also important to monitor the quality of the ML model outputs as well. The data is usually there–most ML teams log predictions and actuals to a table somewhere. The problem is what happens next — or rather, what doesn’t.

Monitoring model performance has always required someone to write the math: the RMSE formula, the classification accuracy query, the logic to compare predicted versus actual at scale. That means needing a data scientist just to set up a monitor. For teams without that bandwidth, it simply doesn’t get done. Models degrade quietly.

Teams told us the same thing, over and over:

None of our production machine learning models — which support 50 to 60 percent of revenue — have any kind of modern tooling built in for observability.

Our ML team wants to use Monte Carlo to alert on data drift…Have you looked at the ML observability side of things — drift detection, statistical test calculations? At the end of the day, there’s math being applied on a bunch of columns.

What you can do now

ML Metrics are available today inside Metric Monitors — the same workflow teams already use to monitor key business metrics and ML model features No new product to learn, no new setup overhead. Monitors include:

Regression performance — measures how far off a model’s predictions are from actual values:

• RMSE (Root Mean Squared Error) — penalizes large errors disproportionately; ideal for catching serious model degradation early
• MAE (Mean Absolute Error) — the average size of prediction errors in the same units as your data, easy to interpret and act on
• MAPE (Mean Absolute Percentage Error) — percentage-based error that travels well across teams and stakeholders
• R-Squared — how well the model explains variance in the data; a meaningful drop signals the model has stopped fitting
• Mean Error — catches systematic bias, where predictions consistently skew high or low in one direction

Classification performance:

• Accuracy — tracks how often the model predicts the correct label over time, with automated alerting when it degrades

Just select the prediction column and the actual column. Monte Carlo handles the computation, builds a baseline, and alerts when performance degrades.

Up next: Out-of-the-box drift detection metrics — PSI, KS Test, and JS Divergence — to automatically flag when a column’s data distribution shifts over time, before it affects model outputs downstream.

Check out the documentation to configure ML Metrics on your model tables today, or schedule a demo to see it in action.