tea-tasting 0.0.2

tea-tasting is a Python package for statistical analysis of A/B tests that features:

• Student's t-test and Z-test out of the box.
• Extensible API: Define and use statistical tests of your choice.
• Delta method for ratio metrics.
• Variance reduction with CUPED/CUPAC (also in combination with delta method for ratio metrics).
• Confidence interval for both absolute and percent change.

tea-tasting calculates statistics within data backends such as BigQuery, ClickHouse, PostgreSQL, Snowflake, Spark, and other of 20+ backends supported by Ibis. This approach eliminates the need to import granular data into a Python environment, though Pandas DataFrames are also supported.

tea-tasting is still in alpha, but already includes all the features listed above. The following features are coming soon:

• Sample ratio mismatch check.
• More statistical tests:
  • Asymptotic and exact tests for frequency data.
  • Bootstrap.
  • Quantile test (using Bootstrap).
  • Mann–Whitney U test.
• Power analysis.
• A/A tests and simulations.
• Pretty output for experiment results (round etc.).
• Documentation on how to define metrics with custom statistical tests.
• Documentation with MkDocs and Material for MkDocs.
• More examples.

tea-tasting 0.0.1

tea-tasting is a Python package for statistical analysis of A/B tests that features:

• Student's t-test, Z-test, and Bootstrap out of the box.
• Extensible API: Define and use statistical tests of your choice.
• Delta method for ratio metrics.
• Variance reduction with CUPED/CUPAC (also in combination with delta method for ratio metrics).
• Fieller's confidence interval for percent change.
• Sample ratio mismatch check.
• Power analysis.
• A/A tests.

Currently, tea-tasting is in the planning stage, and I'm starting with a README that outlines the proposed API — an approach known as Readme Driven Development (RDD).

Check out my blog post where I explain the motivation for creating this package and the benefits of the RDD approach.