From 4c0117401c303dba3f53504dd13ddb31beb64644 Mon Sep 17 00:00:00 2001 From: Osvaldo A Martin Date: Tue, 4 Feb 2025 22:16:09 +0200 Subject: [PATCH] update summary (#99) --- Chapters/Bayesian_workflow_example.qmd | 6 +++--- Chapters/MCMC_diagnostics.qmd | 10 +++++----- Chapters/Sensitivity_checks.qmd | 2 +- 3 files changed, 9 insertions(+), 9 deletions(-) diff --git a/Chapters/Bayesian_workflow_example.qmd b/Chapters/Bayesian_workflow_example.qmd index 163e610..6ddee13 100644 --- a/Chapters/Bayesian_workflow_example.qmd +++ b/Chapters/Bayesian_workflow_example.qmd @@ -274,7 +274,7 @@ This is a simple model, a small dataset. So we get a fast sampling. We did not get any warning messages after sampling, and no divergences! Still, we want to quickly check that we have high ESS and low $\hat R$. Otherwise, this will be an indication of sampling issues and to solve them we may need to run more samples or most likely go back and change something in the model specification. We can see, with relief, that the effective sample size is very high and $\hat R$ is low enough. ```{python} -az.summary(idata, kind="diagnostics") +azs.summary(idata, kind="diagnostics") ``` Doing a visual check, just to be extra careful is also a good idea. @@ -411,10 +411,10 @@ Nice, LOO agrees with the posterior predictive checks. We can see that the trunc ## Summarize results -We can summarize the results in a table. We can use the `az.summary` function to get the mean and the 94% HDI for the parameters. +We can summarize the results in a table. We can use the `azs.summary` function to get the mean and the 94% HDI for the parameters. ```{python} -az.summary(idata_t1, kind="stats") +azs.summary(idata_t1, kind="stats") ``` We can see that we got a positive slope indicating a positive correlation between the masses of predators and prey. In other words, as the mass of predators increases. This is in line with our discussion before the analysis. The magnitude of the slope is $\approx 0.3$, on the log-scale. On the original scale, this is $\approx 0.85$ on the original scale. diff --git a/Chapters/MCMC_diagnostics.qmd b/Chapters/MCMC_diagnostics.qmd index 2b7d907..c20d207 100644 --- a/Chapters/MCMC_diagnostics.qmd +++ b/Chapters/MCMC_diagnostics.qmd @@ -141,7 +141,7 @@ $\hat R$ is a numerical diagnostic that answers the question [@vehtari_2021]. Di The version implemented in ArviZ does several things under the hood, but the central idea is that it compares the variance between chains with the variance within each chain. Ideally, we should get $\hat R = 1$, in practice $\hat R \lessapprox 1.01$ are considered safe and in the first modeling phases, even higher values like $\hat R \approx 1.1$ may be fine. -Using ArviZ we can get the $\hat R$ with `azs.rhat(⋅)`, `az.summary(⋅)` and `az.plot_forest(⋅, r_hat=True)` +Using ArviZ we can get the $\hat R$ with `azs.rhat(⋅)`, `azs.summary(⋅)` and `az.plot_forest(⋅, r_hat=True)` ```{python} azs.rhat(sample) @@ -265,7 +265,7 @@ ax.plot(1, 0, ">k", transform=ax.get_yaxis_transform(), clip_on=False) ax.plot(48, 1, "^k", transform=ax.get_xaxis_transform(), clip_on=False) ``` -With ArviZ we can get `azs.ess(⋅)`, `az.summary(⋅)` and `az.plot_forest(⋅, ess=True)` +With ArviZ we can get `azs.ess(⋅)`, `azs.summary(⋅)` and `az.plot_forest(⋅, ess=True)` ```{python} azs.ess(sample) @@ -277,10 +277,10 @@ One way to use the ESS is as a minimum requirement for trustworthy MCMC samples. The ESS can also be used as a metric of the efficiency of MCMC sampling methods. For instance, we may want to measure the ESS per sample (ESS/n), a sampler that generates a ESS/n closer to 1 is more efficient than a sampler that generates values closer to 0. Other common metrics are the ESS per second, and the ESS per likelihood evaluation. ::: -We see that `az.summary(⋅)` returns two ESS values, `ess_bulk` and `ess_tail`. This is because different regions of the parameter space may have different ESS values since not all regions are sampled with the same efficiency. Intuitively, one may think that when sampling a distribution like a Gaussian it is easier to obtain better sample quality around the mean than around the tails, simply because we have more samples from that region. For some models, it could be the other way around, but the take-home message remains, not all regions are necessarily sampled with the same efficiency +We see that `azs.summary(⋅)` returns two ESS values, `ess_bulk` and `ess_tail`. This is because different regions of the parameter space may have different ESS values since not all regions are sampled with the same efficiency. Intuitively, one may think that when sampling a distribution like a Gaussian it is easier to obtain better sample quality around the mean than around the tails, simply because we have more samples from that region. For some models, it could be the other way around, but the take-home message remains, not all regions are necessarily sampled with the same efficiency ```{python} -az.summary(sample, kind="diagnostics") +azs.summary(sample, kind="diagnostics") ``` If we are going to use the MCMC samples to calculate central values such as means or medians then we have to make sure that the `ess_bulk` is sufficiently large, however, if we want to calculate intervals such as an HDI 94% we have to make sure that `ess_tail` be appropriate. @@ -311,7 +311,7 @@ azp.plot_ess_evolution(sample, An advantage of the ESS is that it is scale-free, it does not matter if one parameter varies between 0.1 and 0.2 and another between -2000 and 0, an ESS of 400 has the same meaning for both parameters. In models with many parameters, we can quickly identify which parameters are most problematic. However, when reporting results it is not very informative to know whether the ESS was 1372 or 1501. Instead, we would like to know the order of the errors we are making when approximating the posterior. This information is given by the **Monte Carlo standard error** (MCSE). Like the ESS, the MCSE takes into account the autocorrelation of the samples. This error should be below the desired precision in our results. That is, if for a parameter the MCSE is 0.1, it does not make sense to report that the mean of that parameter is 3.15. Since the correct value could easily be between 3.4 and 2.8. -With ArviZ we can get the MCSE with `azs.mcse(⋅)` or `az.summary(⋅)`. +With ArviZ we can get the MCSE with `azs.mcse(⋅)` or `azs.summary(⋅)`. ```{python} azs.mcse(sample) diff --git a/Chapters/Sensitivity_checks.qmd b/Chapters/Sensitivity_checks.qmd index 2821d25..9a3c579 100644 --- a/Chapters/Sensitivity_checks.qmd +++ b/Chapters/Sensitivity_checks.qmd @@ -270,7 +270,7 @@ Overall, the power-scaling sensitivity analysis on the adjusted model shows that ### Bacteria treatment -## Interpreting sensitivity diagnostics (Summary) +## Interpreting sensitivity diagnostics: Summary As with other diagnostics we need to interpret the results of the sensitivity analysis in the context of the model, the data and the problem we are trying to solve. Context and modelling purpose should always be part of an analysis.