tea-tasting 0.4.1

What's Changed

• Fix: use 1D array if columns is a single string by @e10v in #116
• Fix: sort variants by @e10v in #117
• Add link to release notes by @e10v in #118
• Update badges by @e10v in #119

Full Changelog: v0.4.0...v0.4.1

tea-tasting 0.4.0

What's Changed

• Switch to PyArrow for internal data and remove Pandas dependency by @e10v in #114

Breaking changes

• Pandas is not automatically installed with tea-tasting anymore. Install it explicitly to export an analysis result using to_pandas method.
• The methods make_users_data and make_sessions_data now return a PyArrow Table by default. You can control the return type by using the return_type parameter. The other possible output types are a Pandas DataFrame or a Polars DataFrame. They require Pandas or Polars packages, respectively.

Enhancements

• Switching from Pandas to PyArrow for internal data can speed up calculations in some use cases.
• You can export an analysis result to a PyArrow Table using the to_arrow method.
• You can export an analysis result to a Polars DataFrame using the to_polars method. Polars is not installed automatically. Install it explicitly to use methods that return a Polars DataFrame.
• The methods make_users_data and make_sessions_data can return a Polars DataFrame. Use the return_type parameter.

Full Changelog: v0.3.1...v0.4.0

tea-tasting 0.3.1

What's Changed

• Update docs and readme by @e10v in #109
• Optimize aggregations for pandas-like dataframes by @e10v in #110
• Remove dependencies upper bounds and update versions by @e10v in #111
• Update CI workflow by @e10v in #112
• Update roadmap by @e10v in #113

Full Changelog: v0.3.0...v0.3.1

tea-tasting 0.3.0

What's Changed

Breaking changes

• Delete to_ibis parameter from make_users_data and make_sessions_data by @e10v in #103

Enhancements

• Support more dataframes with Narwhals by @e10v in #102, #104, #105, and #107

Fixes

• Return nan when dividing negative number by zero by @e10v in #106

Full Changelog: v0.2.0...v0.3.0

tea-tasting 0.2.0

What's Changed

• Multiple hypothesis testing by @e10v in #87, #91, #92, and #97
• Cast to float before aggregation by @e10v in #93
• Update docs and readme by @e10v in #85, #95, and #99
• Replace pandas backend usage with ibis.memtable by @e10v in #90
• Support pretty formatting for ExperimentResults by @e10v in #94
• Update versions by @e10v in #96 and #100
• Other minor changes by @e10v in #86

Full Changelog: v0.1.0...v0.2.0

tea-tasting 0.1.0

What's Changed

tea-tasting is currently in beta. However, I consider it ready for important tasks and use it for the analysis of switchback experiments in my work.

• Analysis of power for RatioOfMeans and Mean by @e10v in #68, #70, #71, #72, #75, and #76
• Fix: make sure that k and n for binomtest are integer by @e10v in #73
• Create a guide on how to use tea-tasting with an arbitrary data backend by @e10v in #79
• Create a guide on custom metrics by @e10v in #81
• Update user guides, docstrings, and readme by @e10v in #77, #78, #80, and #82
• Other minor changes by @e10v in #74 and #83

Full Changelog: v0.0.5...v0.1.0

tea-tasting 0.0.5

What's Changed

• Analysis of a statistic using bootstrap resampling by @e10v in #64
• Analysis of quantiles using bootstrap resampling by @e10v in #65
• Update docs by @e10v in #63 and #67

Full Changelog: v0.0.4...v0.0.5

tea-tasting 0.0.4

What's Changed

• Setup web docs with mkdocs by @e10v in #56
• Experiment result pretty formatting by @e10v in #57 and #59
• Rename is_in to in_ by @e10v in #60
• Update docs and improve naming by @e10v in #61
• Support Ibis 9 by @e10v in #62

Full Changelog: v0.0.3...v0.0.4

tea-tasting 0.0.3

What's Changed

• Sample ratio mismatch check by @e10v in #52
• Handle potential division by zero by @e10v in #53 and #54
• Multiple minor improvements by @e10v in #55 and #51

Full Changelog: v0.0.2...v0.0.3

tea-tasting 0.0.2

tea-tasting is a Python package for statistical analysis of A/B tests that features:

• Student's t-test and Z-test out of the box.
• Extensible API: Define and use statistical tests of your choice.
• Delta method for ratio metrics.
• Variance reduction with CUPED/CUPAC (also in combination with delta method for ratio metrics).
• Confidence interval for both absolute and percent change.

tea-tasting calculates statistics within data backends such as BigQuery, ClickHouse, PostgreSQL, Snowflake, Spark, and other of 20+ backends supported by Ibis. This approach eliminates the need to import granular data into a Python environment, though Pandas DataFrames are also supported.

tea-tasting is still in alpha, but already includes all the features listed above. The following features are coming soon:

• Sample ratio mismatch check.
• More statistical tests:
  • Asymptotic and exact tests for frequency data.
  • Bootstrap.
  • Quantile test (using Bootstrap).
  • Mann–Whitney U test.
• Power analysis.
• A/A tests and simulations.
• Pretty output for experiment results (round etc.).
• Documentation on how to define metrics with custom statistical tests.
• Documentation with MkDocs and Material for MkDocs.
• More examples.