diff --git a/content/collections/web_experiment/en/actions.md b/content/collections/web_experiment/en/actions.md index fa16c6a73..74992ee16 100644 --- a/content/collections/web_experiment/en/actions.md +++ b/content/collections/web_experiment/en/actions.md @@ -31,6 +31,38 @@ URL redirects load a new URL when a targeted user lands on a targeted page in yo URL redirects retain any query parameters on the original page URL. For example, you create a variant to redirect users from `https://example.com` to `https://example.com/get-started`. If a user clicks a link `https://example.com?utm_source=facebook`, Web Experiment redirects that user to `https://example.com/get-started?utm_source=facebook`. +{{partial:admonition type="note"}} +It's possible for the URL redirect test to have a [Sample Ratio Mismatch (SRM)](/docs/feature-experiment/troubleshooting/sample-ratio-mismatch). Redirect tests work by loading the redirected page, ideally as fast as possible. The sequence for this is: + +**Control flow** +- Load control page HTML. +- Browser parses and loads dependencies (including the experiment script). +- Experiment script initializes, evaluates the user, and logs an impression if they're in control. + +**Treatment flow** +- Load control page HTML. +- Browser parses and loads dependencies (including the experiment script). +- Experiment script evaluates the user → assigns them to treatment → triggers a redirect. +- Load treatment page HTML. +- Browser parses and loads dependencies (including the experiment script again). +- Experiment script initializes and logs the impression for treatment. + +Because the treatment flow involves more steps before the impression is logged, users who bounce quickly (for example, after clicking an ad by mistake) are more likely to be counted in control than in treatment. This imbalance can show up as Sample Ratio Mismatch (SRM). + +Researchers have observed similar effects: if treatment slows performance, more users may leave before logs are generated, leading to fewer recorded impressions. Conversely, faster performance can lead to more recorded users in treatment than in control. + +**Further reading:** +* [Causes of SRM](https://www.lukasvermeer.nl/srm/docs/causes/) +* [Pitfalls in Metric Interpretation (KDD 2017)](https://exp-platform.com/Documents/2017-08%20KDDMetricInterpretationPitfalls.pdf) +* [Diagnosing SRM in Online Experiments (KDD 2019)](https://exp-platform.com/Documents/2019_KDDFabijanGupchupFuptaOmhoverVermeerDmitriev.pdf) + +**What to do** + +* **Reduce latency in evaluation**. The longer the delay before impressions are logged, the more pronounced the SRM effect. +* **Use local evaluation where possible**. For example, target only on browser properties instead of slower remote attributes like Country. This reduces the chance that users drop off before logging. Here’s a configuration example in Amplitude Experiment. + +{{/partial:admonition}} + ## Custom code {{partial:admonition type="note"}}