Skip to content

Commit 962b054

Browse files
update Open Source Docs from Roblox internal teams
1 parent 6d1337c commit 962b054

File tree

3 files changed

+6
-1
lines changed

3 files changed

+6
-1
lines changed
Lines changed: 3 additions & 0 deletions
Loading

content/en-us/discovery.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ Engagement signals from sponsored ads, search, charts, friends, teleport, notifi
2020

2121
**Ranking (stage 2)**: The ranking stage takes the input from the retrieval stage and selects the most relevant experiences to be ranked in a personalized way and shown to each user. There is also some exploration besides the relevancy ranking to users. How far your game goes and how much organic distribution it gets depends entirely on the engagement of users who come through Recommended for You.
2222

23-
We do not count the engagement or monetization of users first acquired from ads, friends, or social media in the ranking stage of Recommended for You.
23+
We do not count the engagement or monetization of users first acquired from ads, friends, Today's Picks, search, social media, or any other source in the ranking stage of Recommended for You.
2424

2525
### Key signals
2626

content/en-us/production/experiments.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -203,6 +203,8 @@ For convenience, the results page lets you replace the default config value with
203203

204204
Roblox calculates MDE using your goal metric and number of players per variant, which is based on daily active users, rollout percentage, experiment duration, and variant splits. If the MDE is too high for your goal metric (for example, more than 100%), it's unlikely you can reach statistical significance. Experiences with fewer than 1,000 daily active users might struggle to get useful data from experiments.
205205

206+
<img src="../assets/analytics/configs/experiment-mde.png" alt="Insufficient MDE screen during creation." width="300" />
207+
206208
- **Let experiments run for their full durations.** The novelty effect (temporary interest in a change not because it's better, but because it's new) can heavily skew early results, sometimes causing them to swing in and out of statistical significance. Ending experiments early increases the odds of you taking premature action based on anomalous spikes that more data would have smoothed out or even contradicted.
207209

208210
- **Don't act without statistical significance.** Even seemingly large changes in player behavior might not be statistically significant, generally due to small sample size. If a change isn't statistically significant, ignore it.

0 commit comments

Comments
 (0)