We’ve improved our A/B testing feature on GameAnalytics so you can optimize further your games and try out new great ideas and concepts. You can now stop user acquisition and exclude offline users from your tests.
A/B Testing: Stop acquiring users
We notice many game developers want better control over the allocation of users in an experiment. There are situations where developers would prefer a cohort to remain active; however, no more users join this experiment.
For this reason, we introduced a functionality where you can choose if and when you’d like your experiment to stop allocating more users.
How does it work?
While an experiment is running (active), the number of users allocated to the test is displayed on the page. When you want your test to stop assigning more users, click the Stop Acquiring Users option from the experiment menu. Confirm your action, and that’s all.
GameAnalytics take care of the rest:
- The status on the experiment page will update to confirm that the test is still active but not acquiring any more users
- Users assigned in the experiment so far will continue to be in the test cohort as long as it is active
- Users continue to receive variant config values in their game
- The experiment will remain active until you choose to Stop from the menu or until it reaches its end of life after two months
We recommend letting your experiment run for at least a few days before hitting the ‘stop acquiring users’ button. This way, sufficient statistical certainty is achieved for the best analysis of your experiment.
Use cases and benefits
- An active experiment is populated with enough users, and the model has calculated results. You are satisfied with the experiment results but want to keep the experiment active so users continue to receive variant config values.
This means that the events data from such users in the game will continue to include A/B Test information. So you can continue to query metrics data for your experiment users in GameAnalytics Dashboards using these Filters.
- You can start another (concurrent) A/B experiment with new user traffic directed towards the new experiment or once you are satisfied with the current experiment userbase.
Complete documentation is available here, and as usual, feel free to get in touch so we can help with your specific case.
A/B Testing: Exclude offline users
If a user is offline during their first session, we can’t assign that player to an experiment until they’re back online (that’s when our backend servers place them into an experiment and send the values to their device).
Based on your feedback, some game developers prefer that the experiment ensures such users are never allocated to a test.
So we added a checkbox in the experiment set-up flow called ‘exclude offline users’ to provide this control.
Read more about how to set up an experiment here.
You can tick the checkbox ‘exclude offline users when setting up a new experiment. This will ensure our backend servers will not consider new users who were offline during their first session. Once they are online, an attempt to assign them a variant will not be made. These players will experience the default game settings.
Use case and benefits
Enabling this option for your experiment will ensure users’ experience in the game remains unaffected. New users who first opened your game when they were offline will continue to see the default game behavior as our backend servers will never attempt to assign them to a variant once they are online.
Technically this means that such users will not influence the experiment by causing a bias in the statistical results.
However, please note that this will mean the actual population of your new users assigned to an experiment will be lower than what is defined in the experiment setup.
Practical example:
Let’s assume you start one experiment set up to target 50% of your new users. A new player that qualifies all criteria will have a 50% chance of getting assigned to this test. Assuming no other experiment is active, approximately 50% of your new installs will join the test.
However, if “exclude offline users” is checked, some offline users could be in their first session, despite qualifying the test criteria. Since they will be excluded, less than 50% of your new users are likely to join the test.
