In our blog post on how to conduct a cohort analysis to increase your game's retention rate, we talked about the importance of cohort analysis to make informed decisions, the types of cohort analysis and how you can utilize the UAhero platform for your analyses to squeeze the most out of your data. In this post, we’ll dive deeper into the intricacies of cohort metric predictions and how they can become more reliable with in-depth examples.
Assessing cohorts' performance beforehand is crucial for running a successful user acquisition campaign. It can be used for reasons such as finding the target CPI, target CPA, target ROAS, or to allocate budgets. High prediction accuracy is essential for understanding and redirecting the future to align with your needs.
Let’s recall; in the previous blog post, we divided cohort analysis types into two: time-based and segment-based analysis. In time-based cohort analysis, you analyze the behavior of your user base in a selected time range.
When assessing performance on cohort metrics based on time, you should be calculating the metric at hand (e.g., Revenue) by addressing a date limitation on cohorts (i.e., starting from day A to day B) and a horizon of calculation (i.e., daily period). To elaborate with an example, say you want to calculate ROAS D30 for cohorts from 2022-02-01 to 2022-02-28. To be able to calculate it (assuming today is later than 2022-03-30), you need to sum the revenue generated by your cohorts at the mentioned period interval and calculate the total spend for the days 2022-02-01 to 2022-02-28. Don’t forget that you must wait until at least 30 days after the last cohort starts using the app before starting the analysis.
How can you increase the accuracy of cohort metric predictions?
The signal-to-noise ratio is the ratio between the desired information and the undesired background noise.
For the AI and ML domain, predictability is generally defined as a by-product of the target variable's signal-to-noise (SNR) ratio; therefore, it defines the lower/upper bounds for the accuracy of the problem. Noise generally incurs a less meaningful variance on what we want to predict. So to be able to analyze meaningful patterns rather than noise, you need to have a high signal-to-noise ratio and to achieve your targets, you need to put great effort into noise reduction.
Remember that reducing noise and accurately predicting ROAS D30 for a cohort group is equivalent to accurately predicting Revenue D30. The same argument applies to ARPU D30 without the loss of generality and for ARPPU D30, given the Revenue is IAP’s and the number of unique purchasers is predicted.
Let’s look into a case to understand the methods used for generating more accurate predictions.
Given a cohort group from day A to day B, let’s estimate ROAS D30 on an arbitrary day of choice (you can do the calculation at day B+2 days, or B+7 days and so on).
There are various approaches to estimating the revenue (or ARPU, LTV… etc.), but there is one thing in common, the D0 of each cohort is the period with the highest variance and a low SNR ratio. Therefore where X is greater than 0, if DX is observed, predicting D30 from D0 is naturally more prone to errors than predicting D30 of the same cohort from DX.
However, where X and Y present the number of days, and if we think that X is a smaller number of days than Y and (DX -> predicted ROAS D30) stands for predicting ROAS D30 from data available on period DX, with high probability; it can be conjectured that var(DX -> predicted ROAS D30) is greater than or equals to var(DY -> predicted ROAS D30).
The reason for this inference is simple: the later periods include more data, and therefore, their SNR is higher. So simply put, more data equals more accurate predictions.
Assume you have a method M that calculates D30 revenue from a given DX, and you would like to estimate the ROAS D30 of cohorts that arrived between (today-9 and today-2). The today-9 cohort has info on D7; the today-8 cohort has info on D6 …and so on.
Because of the conjecture above, if the method predicts the revenue from D0 for all the cohorts, it predicts using the highest variance period and ends up in an unstable and low accuracy prediction. And given that we have information for later periods on all cohorts except today-2, we can utilize what is available and make a higher quality prediction for aggregate level ROAS.
Suppose you want to predict revenue for D30 for a specific cohort, for instance, today-5. This cohort has D0, D1, and D2 data available. How would you predict D30 revenue for this cohort? Would you use D0 because that is what you can do, or would you shoot for the moon and try to utilize all the data at hand (like using all the available periods)?
It takes no effort to see that for UA experts to predict the revenue correctly; they should use maximum information. Otherwise, money and performance loss is inevitable with the less accurate predictions you get due to a lack of data.
Comparing methods for prediction accuracy
What if you want to compare two methods for prediction accuracy? Say you’ll make the comparison for revenue D30.
One method forces UA experts to use D0 for cohorts where D7 is unavailable; the other one utilizes maximal information to calculate each unobserved period for individual cohorts. How do you compare the two assuming you would like to predict your ROAS D30 with the highest accuracy?
Of course, you compare what you value most, which here are predictions on individual cohorts for D30 without caring about aggregation. Then it takes no effort to see that the method that uses maximal information makes wiser choices just because they abide by the simple laws of ML. Would you want the second method to drop down all guns and use a knife in a gunfight, or like the first one to improve itself, gear up and then return to the battle? Now that you’re comparing predictions and not cohorts, you should choose a method that’ll bring the most accurate predictions to ensure you won't waste your budget.
In real life, we observe partially complete cohorts and their various periods. A method that uses all the information available makes better choices than a method that uses static periods, given that later periods are observed in the picture. Suppose that a cohort started with high revenue on D0 but lost most of the users and revenue on D3. The method which uses maximal information can see this drop and refine the predictions; however, the method that predicts from D0 still predicts high revenues and makes you believe the illusion that you are doing OK when in fact, you are not. The same scenario may also happen after one week. Still, the methods that predict by using static periods cannot identify this.
This is where UAhero’s superpowers step in. In the UAhero platform, our revenue/ROAS/ARPU/ARPPU prediction algorithms use all the available information, use the most recent to identify potential increases/decreases and help you decide with the most accurate predictions.
To increase predicted ROAS accuracy, and boost the performance of your campaigns and your profits, drop us a line at hello@uahero.ai. And don’t forget to subscribe to our newsletter to learn more about AI-based predictions and user acquisition strategies.
%20copy.png)
