Vanguard A/B Test

An end-to-end analysis of Vanguard's March–June 2017 digital-interface experiment. The project asks whether a redesigned flow increased completion, improved speed, and met a 5-percentage-point business threshold.

Executive summary

• Completion: Control 65.58% → Test 69.29%, a +3.71 percentage-point lift.
• Statistical evidence: A one-sided two-proportion z-test finds the Test completion rate significantly higher (p < 0.001).
• Business threshold: The observed lift does not reach Vanguard's required 5-percentage-point threshold.
• Navigation friction: Backward transitions account for 6.82% of Control web events and 9.19% of Test web events. This is an event-level navigation signal—not a percentage of clients.

The redesign improved completion, but the higher backward-transition rate and missed business threshold argue against an unconditional rollout.

Methodology

1. Join the experiment roster, client demographics, and web-event logs.
2. Validate client counts and experiment-group coverage.
3. Calculate completion at the unique-client level.
4. Measure time on each process step.
5. Define a backward transition as an event whose step order is lower than the previous event for the same client and visit.
6. Test the completion-rate lift and the 5-percentage-point business threshold.

Reproduce the analysis

The notebook reads the four original Ironhack datasets directly from their version-pinned GitHub URLs, so no manual download is required.

git clone https://github.com/vildanpir/vanguard-ab-test.git
cd vanguard-ab-test
python -m pip install -r requirements.txt
jupyter notebook notebooks/vanguard_analysis.ipynb

Run the notebook from top to bottom. The source datasets remain at the version-pinned Ironhack location.

Running the notebook also creates notebooks/df_final.csv, which the Tableau workbook uses through a repository-relative connection. The generated 40 MB file is intentionally excluded from Git.

Repository structure

notebooks/vanguard_analysis.ipynb   Full cleaning, EDA, KPI and test workflow
images/                             Exported KPI charts
tableau/project_charts.twb          Tableau workbook
requirements.txt                    Reproducible Python dependencies

Presentation: Canva deck

Limitations

• Completion means that a client reached the confirm step at least once.
• The backward-transition rate uses events as its denominator and should not be interpreted as the share of clients who made an error.
• The dataset does not identify concurrent product, campaign, or seasonal changes during the experiment.

Team

Vildan Pirpiroglu · Sandra Fernández Pascual · Elena Garrigues