The Battle of Models: Sensible Machine Learning Improves FP&A Accuracy

In the dynamic machine learning (ML) landscape, precision is paramount. However, selecting the right forecasting model can be difficult and time-consuming, often resembling a maze of complex algorithms and intricate performance metrics. As FP&A teams strive for more accurate data models to enable informed decision-making and optimize strategies, Sensible ML introduces a groundbreaking concept – the Model Arena. How? Much like in a sports bracket, models compete head to head, vying to be the most accurate predictor for specific product-location combinations. In this third post of our AI for FP&A series, let's delve into this analogy to uncover the inner workings of Sensible ML's Model Arena.

Miss the first two posts in the series? Catch up now with post #1 (AI for FP&A Starts with Data Quality) and post #2 (The Secret Ingredients of Machine Learning for FP&A: Features).

The Analogy: Forecasting Models as Sports Teams

Sensible ML orchestrates model competitions that mirror the intensity of a sports bracket. The objective? To pinpoint the most accurate models tailored to each individual product-location combination for which Sensible ML is creating a forecast. This tailoring sets the stage for a groundbreaking approach to ML optimization.

Much like a sports bracket, Sensible ML divides models into a structured competition format. Each model is a contender, entering the arena with the aim to outperform other models. The criteria for success? Producing the most accurate prediction for each target or product-location combination line-item.

Why Is the Model Arena Unique?

While some Corporate Performance Management (CPM) competitors opt for the simplicity of applying a single model across all targets, Sensible ML handpicks individual models for each target, ensuring tailored precision that goes beyond a one-size-fits-all approach.

This is the real power of AI For FP&A. To actually do all the hard work so FP&A teams don't have to.

Polaris, a global leader in powersports whose products have vastly different characteristics, uses Sensible ML to forecasts for specific products and locations with distinct models across the business.

Contrast this with a one-size-fits-all approach, where reliance on a single forecasting model would fail to account for significant differences between their boating and snowmobile product categories for example. Below are some of the differences that various models consider for Polaris:

• Seasonality: Boat sales are highly seasonal, peaking during the summer months and declining in the winter. Snowmobile sales, on the other hand, have different seasonality trends than boats.
• Demand factors: Factors influencing boat sales – such as consumer spending, fuel prices, and weather conditions – differ significantly from those affecting snowmobile sales.
• Product lifecycles: Boat and snowmobile lifecycle stages are different. Boats typically have a longer lifecycle and are less susceptible to rapid technological advancements. Meanwhile, snowmobiles undergo more frequent design changes and are influenced by trends in the winter sports industry.

A single ML model would simply fail to capture the above nuanced factors in the Polaris product categories, leading to inaccurate forecasts, missed opportunities and poor decision-making. And if that's the case, that's a missed opportunity for AI and ML.

With OneStream's Sensible ML, the Model Arena is built right into the solution. That means, with a single click of a button, business planners can automatically get the most accurate model selected for each forecasted line-item. AI for FP&A at scale!

Accelerate Model Train Times and Experimentation in Sensible ML's Model Arena

Model monitoring is also a challenge for traditional ML tools, as model performance begins to degrade once placed into production. Manual efforts also only further hamper model performance. For these reasons alone, effective and scalable ML solutions must automatically compare models and contain the following capabilities to create speed to value for Finance and Business analysts:

• Accelerated model training time and experimentation to efficiently produce forecasts and reforecasts
• Agility to utilize Sensible ML to address different business use cases
• Continuous monitoring of model health score and performance over time and auto-retraining

In other words, users can continuously monitor model health scores and performance over time and auto-retrain in Sensible ML's Model Arena. All the model variations in the arena compete against each other in a sports-bracket structure to identify the best-performing models and determine which ones to deploy. Below, Figure 1 depicts Sensible ML's Model Arena selecting the most accurate model to predict the sales of menu items for a fictitious organization.

Figure 1: Sensible ML Model Arena Call Target

In Figure 1, model A (CatBoost) won the sports bracket. Why? The model received the lowest error score compared to the rest of the models. Therefore, model A was used to create the forecast number for the call product in the Rochester location. When a different target or product location combination is selected, however, a different ML model wins (see Figure 2).

Figure 2: Sensible ML Model Arena Pints Target

As shown in Figure 2, when the Pints product for the Rochester location is selected, model B (XGBoost) beat the others to win the sports bracket. Model B was the most accurate model and was therefore used to create the forecast number for Pints in the Rochester location.

Conclusion

In sum, OneStream's Sensible ML's Model Arena automates a targeted, personalized approach for FP&A teams that surpasses the limitations of generalized, one-size-fits-all approach. Bottom line – this helps FP&A create value from AI, unlock the true potential of their data and create competitive edge through unparalleled forecasting accuracy and tailored insights.

The future of accurate, personalized AI for FP&A is here – be a part of it with Sensible ML.

Learn More

To learn more about Sensible ML, stay tuned for additional posts from our Sensible ML blog series. You can also download our white paper here.