Vanguard, a US-based investment management company, ran an A/B test between 3/15/2017 and 6/20/2017 to evaluate a redesigned online client process. The goal was to determine whether a more intuitive, modern User Interface (UI) with in-context prompts (cues, hints, and instructions shown directly within the client's task flow) would lead to a smoother experience and a higher process completion rate compared to the traditional interface.
- Control Group: experienced Vanguard's traditional online process.
- Test Group: experienced the new, redesigned digital interface.
Both groups navigated the same process sequence: an initial page (start), three intermediate steps (step_1, step_2, step_3), and a final confirmation page (confirm).
As data analysts on the Customer Experience (CX) team, our task was to analyze the experiment's results and provide Vanguard with a clear, data-driven recommendation on whether to roll out the new design company-wide.
Would the redesigned UI encourage more clients to complete the online process, and is that improvement large enough — and reliable enough — to justify launching it?
Three datasets were provided and combined for this analysis:
| Dataset | Description |
|---|---|
Client Profiles (df_final_demo) |
Client demographics: age, gender, tenure, number of accounts, account balance, and platform usage. |
Digital Footprints (df_final_web_data_pt_1 / pt_2) |
Event-level log of client interactions on the site (visit, step, and timestamp), split into two files and merged prior to analysis. |
Experiment Roster (df_final_experiment_clients) |
Identifies which clients belonged to the Test or Control group. |
Key fields include client_id, visitor_id, visit_id, process_step, date_time, variation, clnt_tenure_yr, clnt_age, gendr, num_accts, bal, calls_6_mnth, and logons_6_mnth.
- EDA & Data Cleaning — merged the web activity files, checked and handled duplicates, nulls, and inconsistent client/visitor/visit relationships, and explored client behavior patterns.
- KPI Calculation:
- Completion Rate — the proportion of clients who reached the final
confirmstep. - Time Spent per Step — average duration between consecutive steps within the same visit.
- Error Rate — the proportion of transitions where a client moved backward to an earlier step, treated as a signal of confusion.
- Completion Rate — the proportion of clients who reached the final
- Hypothesis Testing:
- Completion rate: Test vs. Control (two-proportion z-test).
- Completion rate improvement against a 5-percentage-point cost-effectiveness threshold.
- Error rate: Test vs. Control (two-proportion z-test).
- Experiment Evaluation — assessed whether the 3-month test duration was adequate and identified additional data that could strengthen the analysis.
- Tableau Dashboard — built interactive visualizations summarizing completion rate, time per step, error rate, and demographic comparisons between groups.
| KPI | Control | Test |
|---|---|---|
| Completion Rate | 65.6% | 69.3% |
| Overall Error Rate | 9.5% | 13.3% |
| Avg. Time — Step 3 (sec) | 137.4 | 130.0 |
- The Test group had a statistically significant higher completion rate than Control, but the improvement (~3.7 percentage points) did not meet Vanguard's 5-point cost-effectiveness threshold.
- The Test group also showed a statistically significant higher overall error rate, driven mainly by increased backward navigation at Step 1 and Step 2 — suggesting the new interface introduced some confusion earlier in the process, despite a slight improvement at Step 3.
- Age and tenure distributions were similar between groups, supporting the validity of the random assignment.
- The new design, in its current form, is not cost-effective to roll out company-wide.
- Continue refining the redesign — particularly around Steps 1 and 2, where confusion increased — before a full launch.
- Run a follow-up, more targeted A/B test focused on the highest-friction steps.
vanguard-ab-test/
├── notebooks/ # Jupyter notebook(s) with the full analysis
├── tableau/ # Tableau workbook and exported CSVs used for the dashboard
├── README.md
- Python (pandas, statsmodels, matplotlib/seaborn) for data cleaning, KPI calculation, and hypothesis testing
- MySQL for supplementary data exploration
- Tableau for the interactive dashboard
- Trello for Kanban-based project management
- Kanban board: [https://trello.com/b/xvdyGffE/project-2-customer-experience-cx-team-at-vanguard]
- Data sources: [https://github.com/data-bootcamp-v4/lessons/tree/main/5_6_eda_inf_stats_tableau/project/files_for_project]
- Presentation slides: [https://www.canva.com/design/DAHQBWYdCjw/LFkDa-z2xugSTt-DrGgFlQ/edit]
- Tableau dashboard: [https://public.tableau.com/app/profile/aline.granells/viz/vanguard_tableau_17848230480820/Dashboard1?publish=yes]
- Juan Francisco Orona
- Aline De Fraia
- Melissa Sánchez