Skip to content

Commit

Permalink
update sensitivity example (#100)
Browse files Browse the repository at this point in the history
  • Loading branch information
aloctavodia authored Feb 6, 2025
1 parent 4c01174 commit 26042c0
Show file tree
Hide file tree
Showing 18 changed files with 36 additions and 26 deletions.
2 changes: 1 addition & 1 deletion Chapters/Case_study_model_comparison.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ import pymc as pm
import pytensor.tensor as pt
import xarray as xr
import matplotlib.pyplot as plt
azp.style.use("arviz-clean")
azp.style.use("arviz-variat")
plt.rcParams["figure.dpi"] = 100
```

Expand Down
2 changes: 1 addition & 1 deletion Chapters/DataTree.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ import matplotlib.pyplot as plt
import xarray as xr
xr.set_options(display_expand_data=False, display_expand_attrs=False)
np.random.seed(19)
azp.style.use("arviz-clean")
azp.style.use("arviz-variat")
plt.rcParams["figure.dpi"] = 100
```

Expand Down
2 changes: 1 addition & 1 deletion Chapters/Distributions.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ import numpy as np
import preliz as pz
import matplotlib.pyplot as plt
np.random.seed(19)
azp.style.use("arviz-clean")
azp.style.use("arviz-variat")
plt.rcParams["figure.dpi"] = 100
```

Expand Down
6 changes: 3 additions & 3 deletions Chapters/Elements_of_visualization.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ import arviz_plots as azp
import numpy as np
import preliz as pz
import matplotlib.pyplot as plt
azp.style.use("arviz-clean")
azp.style.use("arviz-variat")
plt.rcParams["figure.dpi"] = 100
```
Plots occupy a central place in modern statistics, both in exploratory data analysis and inferential statistics.
Expand Down Expand Up @@ -92,8 +92,8 @@ Matplotlib allows users to easily switch between plotting styles by defining sty

```{python}
#| label: fig-style_sheets
#| fig-cap: "arviz-clean style use a color-blind friendly palette"
azp.style.use('arviz-clean')
#| fig-cap: "arviz-variat style use a color-blind friendly palette"
azp.style.use('arviz-variat')
x = np.linspace(0, 1, 100)
dist = pz.Beta(2, 5).pdf(x)
Expand Down
2 changes: 1 addition & 1 deletion Chapters/MCMC_diagnostics.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ import numpy as np
import preliz as pz
import matplotlib.pyplot as plt
rng = np.random.default_rng(793)
azp.style.use("arviz-clean")
azp.style.use("arviz-variat")
plt.rcParams["figure.dpi"] = 100
```

Expand Down
2 changes: 1 addition & 1 deletion Chapters/Model_comparison.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ import numpy as np
import preliz as pz
import pymc as pm
import matplotlib.pyplot as plt
azp.style.use("arviz-clean")
azp.style.use("arviz-variat")
plt.rcParams["figure.dpi"] = 100
SEED = 3592
```
Expand Down
2 changes: 1 addition & 1 deletion Chapters/Prior_elicitation.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ import preliz as pz
import pymc as pm
import matplotlib.pyplot as plt
rng = np.random.default_rng(793)
azp.style.use("arviz-clean")
azp.style.use("arviz-variat")
plt.rcParams["figure.dpi"] = 100
```

Expand Down
2 changes: 1 addition & 1 deletion Chapters/Prior_posterior_predictive_checks.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ import pandas as pd
import preliz as pz
import pymc as pm
import matplotlib.pyplot as plt
azp.style.use("arviz-clean")
azp.style.use("arviz-variat")
plt.rcParams["figure.dpi"] = 100
SEED = 3592
np.random.seed(SEED)
Expand Down
24 changes: 17 additions & 7 deletions Chapters/Sensitivity_checks.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ import pandas as pd
import preliz as pz
import pymc as pm
import matplotlib.pyplot as plt
azp.style.use("arviz-clean")
azp.style.use("arviz-variat")
plt.rcParams["figure.dpi"] = 100
SEED = 3592
np.random.seed(SEED)
Expand Down Expand Up @@ -205,13 +205,16 @@ azp.plot_psense_dist(idata_bf_00,
)
```

We can also use the `plot_psense_quantities` function to plot quantities of interest like the mean, median, standard deviation, etc.
We can also use the `plot_psense_quantities` function to plot quantities of interest like the mean, median, standard deviation, etc. With this plot we can evaluate how much the quantities of interest change when we perturb the prior and likelihood. As a reference the plot also shows the ±2 Monte Carlo standard errors (MCSE) of the base posterior quantity. From @fig-quantities_bf_00 we can see that power-scaling the prior and likelihood pulls the posterior mean in opposite directions. This suggests there might be a potential mismatch or conflict between the prior and the likelihood.

```{python}
## coming soon
## add plot_psense_quantities
#| label: fig-quantities_bf_00
#| fig-cap: "Posterior quantities of βwrist as a function of power-scaling of the prior (purple line and circle markers) and the likelihood (green line and square markers). Dashed lines are ± 2 Monte Carlo standard errors (MCSE) of the base posterior quantity"
azp.plot_psense_quantities(idata_bf_00, var_names=["β"], coords={"covariates": ["wrist"]})
```

#### Adjusting the prior
As there is prior sensitivity arising from prior-data conflict, which is unexpected and unintentional as our priors were chosen to be weakly informative, we consider modifying the priors. On inspecting the raw data, we see that although the predictor variables are all measured on similar scales, the variances of the variables differ substantially. For example, the variance of
wrist circumference is 0.83, while the variance of abdomen is 102.65. This leads to our chosen prior to be unintentionally
informative for some of the regression coefficients, including wrist, while being weakly informative for others
Expand Down Expand Up @@ -242,7 +245,6 @@ with pm.Model(coords={"covariates": body_fat.columns}) as model_bf_01:
```
:::


Now we can perform the sensitivity analysis again. We can see that we managed to remove the warning!

```{python}
Expand All @@ -251,7 +253,7 @@ azs.psense_summary(idata_bf_01, var_names=["~μ"])

We can see that after changing the prior, the posterior mean for `wrist` changes from -1.45 to -1.86, indicating that the base prior was indeed unintentionally informative and in conflict with the data, pulling the estimate towards zero.

As previously, we can also evaluate the effect of power-scaling visually with the function `plot_psense_dist`.
As previously, we can also evaluate the effect of power-scaling visually with the function `plot_psense_dist`. From @fig-posterior-sensitivity_bf_01 we can see a better overlap between the posterior distributions, compared to the original prior.

```{python}
#| label: fig-posterior-sensitivity_bf_01
Expand All @@ -264,7 +266,15 @@ azp.plot_psense_dist(idata_bf_01,
)
```

Overall, the power-scaling sensitivity analysis on the adjusted model shows that there is no longer prior sensitivity, and there is appropriate likelihood sensitivity.
In terms or the mean for `βwrist`, we can see that there is no longer prior or likelihood sensitivity, indicating no prior-data conflict. For the standard deviation, likelihood sensitivity remains, indicating that the likelihood is informative.
```{python}
#| label: fig-quantities_bf_01
#| fig-cap: "Posterior quantities of βwrist as a function of power-scaling of the prior (purple line and circle markers) and the likelihood (green line and square markers). Dashed lines are ± 2 Monte Carlo standard errors (MCSE) of the base posterior quantity"
azp.plot_psense_quantities(idata_bf_01, var_names=["β"], coords={"covariates": ["wrist"]})
```

Overall, the power-scaling sensitivity analysis on the adjusted prior shows that there is no longer prior sensitivity, and there is appropriate likelihood sensitivity.


### Bacteria treatment
Expand Down
2 changes: 1 addition & 1 deletion Chapters/Variable_selection.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ import pymc_bart as pmb
import matplotlib.pyplot as plt
plt.rcParams["figure.dpi"] = 100
seed = 793
azp.style.use("arviz-clean")
azp.style.use("arviz-variat")
```

Variable selection refers to the process of identifying the most relevant variables in a model from a larger set of predictors. When performing this process we usually assume that variables contribute unevenly to the outcome and we want to identify the most important ones. Sometimes we also care about the order in which variables are included in the model [@martin_2024; @piironen_2020].
Expand Down
2 changes: 1 addition & 1 deletion img/coodinates.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
"metadata": {},
"outputs": [],
"source": [
"azp.style.use(\"arviz-clean\")"
"azp.style.use(\"arviz-variat\")"
]
},
{
Expand Down
2 changes: 1 addition & 1 deletion models/crabs/crabs.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -181,7 +181,7 @@
"source": [
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"az.style.use(\"arviz-clean\")\n",
"az.style.use(\"arviz-variat\")\n",
"\n",
"def rotogram(idata, ax):\n",
" max_ = 17\n",
Expand Down
2 changes: 1 addition & 1 deletion scripts/bayes_theorem.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
import preliz as pz
import matplotlib.pyplot as plt

azp.style.use("arviz-clean")
azp.style.use("arviz-variat")

grid = np.linspace(0, 1, 500)
prior = pz.Beta(4, 8).pdf(grid)
Expand Down
2 changes: 1 addition & 1 deletion scripts/beta_binomial_ani.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
import preliz as pz
import matplotlib.pyplot as plt
from matplotlib.animation import FuncAnimation
azp.style.use("arviz-clean")
azp.style.use("arviz-variat")

np.random.seed(123)

Expand Down
2 changes: 1 addition & 1 deletion scripts/beta_bounds.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
import numpy as np
import arviz_plots as azp

azp.style.use("arviz-clean")
azp.style.use("arviz-variat")

def one_iter(lower, upper, mode, mass=0.99, plot=True):

Expand Down
2 changes: 1 addition & 1 deletion scripts/max_ent.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
import matplotlib.pyplot as plt
import arviz_plots as azp

azp.style.use("arviz-clean")
azp.style.use("arviz-variat")

cons = [[{"type": "eq", "fun": lambda x: np.sum(x) - 1}],
[{"type": "eq", "fun": lambda x: np.sum(x) - 1},
Expand Down
2 changes: 1 addition & 1 deletion scripts/prior_posterior.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
import matplotlib.pyplot as plt
from matplotlib.animation import FuncAnimation

azp.style.use("arviz-clean")
azp.style.use("arviz-variat")
fig, ax = plt.subplots(1, 1, figsize=(8, 2.5))

grid = np.linspace(0, 1, 500)
Expand Down
2 changes: 1 addition & 1 deletion scripts/prior_vagueness.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
import arviz_plots as azp


azp.style.use("arviz-clean")
azp.style.use("arviz-variat")

_, ax = plt.subplots(1, 1, figsize=(12, 4))

Expand Down

0 comments on commit 26042c0

Please sign in to comment.