In the Sellforte's Model Validation view, you can validate the model performance by using the model validation metrics.
At the top of the view, you can select a date range and group the view by day, by week, or by month. You can also set the graph to show absolute validation or validation difference.
In the line chart you can compare in-sample and real sales.
In-sample is a model estimation of total sales based on investment data and historical performance of media. When the model is trained, for each media it learns some parameters that describe how effective this media is. Then those parameters can be plugged back into the model equation that we used and we see how much are estimated sales - this is then in-sample. So it tells us what are 'the predicted sales' based on trained parameters in the model.
The output of the in-sample and actual sales comparison is the validation metrics, R2 and MAPE.
Validation metrics
R2
R2 = precision quality (higher is better). It stands for the coefficient of determination. It measures how well the model's predicted values match the actual values. R2 ranges from 0 to 1, with 1 meaning a perfect fit and 0 meaning no fit at all. R2 takes into account the variation in the actual values and how much of that variation is explained by the model's predicted values. A higher R2 value means the model is a better fit for the data.
MAPE
MAPE = Mean absolute percentage error (smaller is better). It measures how far off the model's predicted values are from the actual values, as a percentage of the actual value. For example, if the model predicted 100 sales and the actual number of sales was 110, the error is 10%, because the model was off by 10 sales. MAPE calculates the average error as a percentage of the actual value, so it tells us how accurate the model is, on average.
Validation metrics limitations
MAPE and R2 metrics are important measures of the accuracy of Marketing Mix Models, but they have their limitations. These metrics can tell us how well the model is able to predict sales, but they don't necessarily tell us how well the model is able to capture the impact of specific marketing activities, such as advertising or promotions.
Therefore, the impact of marketing activities can be limited compared to other factors that affect sales, such as seasonality or pricing. In addition, there are often many different media channels and campaigns to consider, which can be difficult to accurately represent as independent variables in the model.