Skip to content

Commit

Permalink
build based on 0c5409f
Browse files Browse the repository at this point in the history
  • Loading branch information
Documenter.jl committed May 1, 2024
1 parent d836129 commit 519e824
Show file tree
Hide file tree
Showing 11 changed files with 1,224 additions and 0 deletions.
1 change: 1 addition & 0 deletions previews/PR53/.documenter-siteinfo.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"documenter":{"julia_version":"1.10.3","generation_timestamp":"2024-05-01T14:17:43","documenter_version":"1.4.0"}}
15 changes: 15 additions & 0 deletions previews/PR53/api/index.html
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
<!DOCTYPE html>
<html lang="en"><head><meta charset="UTF-8"/><meta name="viewport" content="width=device-width, initial-scale=1.0"/><title>API · CalibrationTests.jl</title><meta name="title" content="API · CalibrationTests.jl"/><meta property="og:title" content="API · CalibrationTests.jl"/><meta property="twitter:title" content="API · CalibrationTests.jl"/><meta name="description" content="Documentation for CalibrationTests.jl."/><meta property="og:description" content="Documentation for CalibrationTests.jl."/><meta property="twitter:description" content="Documentation for CalibrationTests.jl."/><meta property="og:url" content="https://devmotion.github.io/CalibrationTests.jl/api/"/><meta property="twitter:url" content="https://devmotion.github.io/CalibrationTests.jl/api/"/><link rel="canonical" href="https://devmotion.github.io/CalibrationTests.jl/api/"/><script data-outdated-warner src="../assets/warner.js"></script><link href="https://cdnjs.cloudflare.com/ajax/libs/lato-font/3.0.0/css/lato-font.min.css" rel="stylesheet" type="text/css"/><link href="https://cdnjs.cloudflare.com/ajax/libs/juliamono/0.050/juliamono.min.css" rel="stylesheet" type="text/css"/><link href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.4.2/css/fontawesome.min.css" rel="stylesheet" type="text/css"/><link href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.4.2/css/solid.min.css" rel="stylesheet" type="text/css"/><link href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.4.2/css/brands.min.css" rel="stylesheet" type="text/css"/><link href="https://cdnjs.cloudflare.com/ajax/libs/KaTeX/0.16.8/katex.min.css" rel="stylesheet" type="text/css"/><script>documenterBaseURL=".."</script><script src="https://cdnjs.cloudflare.com/ajax/libs/require.js/2.3.6/require.min.js" data-main="../assets/documenter.js"></script><script src="../search_index.js"></script><script src="../siteinfo.js"></script><script src="../../versions.js"></script><link class="docs-theme-link" rel="stylesheet" type="text/css" href="../assets/themes/documenter-dark.css" data-theme-name="documenter-dark" data-theme-primary-dark/><link class="docs-theme-link" rel="stylesheet" type="text/css" href="../assets/themes/documenter-light.css" data-theme-name="documenter-light" data-theme-primary/><script src="../assets/themeswap.js"></script></head><body><div id="documenter"><nav class="docs-sidebar"><div class="docs-package-name"><span class="docs-autofit"><a href="../">CalibrationTests.jl</a></span></div><button class="docs-search-query input is-rounded is-small is-clickable my-2 mx-auto py-1 px-2" id="documenter-search-query">Search docs (Ctrl + /)</button><ul class="docs-menu"><li><a class="tocitem" href="../">Home</a></li><li class="is-active"><a class="tocitem" href>API</a><ul class="internal"><li><a class="tocitem" href="#General-Calibration-Error"><span>General Calibration Error</span></a></li><li><a class="tocitem" href="#Kernel-Calibration-Error"><span>Kernel Calibration Error</span></a></li></ul></li></ul><div class="docs-version-selector field has-addons"><div class="control"><span class="docs-label button is-static is-size-7">Version</span></div><div class="docs-selector control is-expanded"><div class="select is-fullwidth is-size-7"><select id="documenter-version-selector"></select></div></div></div></nav><div class="docs-main"><header class="docs-navbar"><a class="docs-sidebar-button docs-navbar-link fa-solid fa-bars is-hidden-desktop" id="documenter-sidebar-button" href="#"></a><nav class="breadcrumb"><ul class="is-hidden-mobile"><li class="is-active"><a href>API</a></li></ul><ul class="is-hidden-tablet"><li class="is-active"><a href>API</a></li></ul></nav><div class="docs-right"><a class="docs-navbar-link" href="https://github.com/devmotion/CalibrationTests.jl/blob/main/docs/src/api.md#" title="Edit source on GitHub"><span class="docs-icon fa-solid"></span></a><a class="docs-settings-button docs-navbar-link fa-solid fa-gear" id="documenter-settings-button" href="#" title="Settings"></a><a class="docs-article-toggle-button fa-solid fa-chevron-up" id="documenter-article-toggle-button" href="javascript:;" title="Collapse all docstrings"></a></div></header><article class="content" id="documenter-page"><h1 id="API"><a class="docs-heading-anchor" href="#API">API</a><a id="API-1"></a><a class="docs-heading-anchor-permalink" href="#API" title="Permalink"></a></h1><h2 id="General-Calibration-Error"><a class="docs-heading-anchor" href="#General-Calibration-Error">General Calibration Error</a><a id="General-Calibration-Error-1"></a><a class="docs-heading-anchor-permalink" href="#General-Calibration-Error" title="Permalink"></a></h2><article class="docstring"><header><a class="docstring-article-toggle-button fa-solid fa-chevron-down" href="javascript:;" title="Collapse docstring"></a><a class="docstring-binding" id="CalibrationTests.ConsistencyTest" href="#CalibrationTests.ConsistencyTest"><code>CalibrationTests.ConsistencyTest</code></a> — <span class="docstring-category">Type</span></header><section><div><pre><code class="language-julia hljs">ConsistencyTest(
estimator::CalibrationErrorEstimator,
predictions::AbstractVector,
targets::AbstractVector,
)</code></pre><p>Construct an hypothesis test of calibration based on consistency resampling with a calibration <code>estimator</code> as test statistic and <code>predictions</code> and <code>targets</code> of a model of interest.</p><p>Consistency resampling is a parametric bootstrap method for calibrated models.</p><p><strong>References</strong></p><p>Bröcker, J., &amp; Smith, L. A. (2007). <a href="https://doi.org/10.1175/WAF993.1">Increasing the reliability of reliability diagrams</a>. Weather and forecasting, 22(3), 651-661.</p></div><a class="docs-sourcelink" target="_blank" href="https://github.com/devmotion/CalibrationTests.jl/blob/0c5409f2bf13a411264a7289ed036aa799575090/src/consistency.jl#L12-L30">source</a></section></article><h2 id="Kernel-Calibration-Error"><a class="docs-heading-anchor" href="#Kernel-Calibration-Error">Kernel Calibration Error</a><a id="Kernel-Calibration-Error-1"></a><a class="docs-heading-anchor-permalink" href="#Kernel-Calibration-Error" title="Permalink"></a></h2><article class="docstring"><header><a class="docstring-article-toggle-button fa-solid fa-chevron-down" href="javascript:;" title="Collapse docstring"></a><a class="docstring-binding" id="CalibrationTests.AsymptoticSKCETest" href="#CalibrationTests.AsymptoticSKCETest"><code>CalibrationTests.AsymptoticSKCETest</code></a> — <span class="docstring-category">Type</span></header><section><div><pre><code class="language-julia hljs">AsymptoticSKCETest(kernel::Kernel, predictions, targets)</code></pre><p>Calibration hypothesis test based on the unbiased estimator of the squared kernel calibration error (SKCE) with quadratic sample complexity.</p><p><strong>Details</strong></p><p>Let <span>$\mathcal{D} = (P_{X_i}, Y_i)_{i=1,\ldots,n}$</span> be a data set of predictions and corresponding targets. Denote the null hypothesis &quot;the predictive probabilistic model is calibrated&quot; with <span>$H_0$</span>.</p><p>The hypothesis test approximates the p-value <span>$ℙ(\mathrm{SKCE}_{uq} &gt; c \,|\, H_0)$</span>, where <span>$\mathrm{SKCE}_{uq}$</span> is the unbiased estimator of the SKCE, defined as</p><p class="math-container">\[\frac{2}{n(n-1)} \sum_{1 \leq i &lt; j \leq n} h_k\big((P_{X_i}, Y_i), (P_{X_j}, Y_j)\big),\]</p><p>where</p><p class="math-container">\[\begin{aligned}
h_k\big((μ, y), (μ&#39;, y&#39;)\big) ={}&amp; k\big((μ, y), (μ&#39;, y&#39;)\big)
- 𝔼_{Z ∼ μ} k\big((μ, Z), (μ&#39;, y&#39;)\big) \\
&amp; - 𝔼_{Z&#39; ∼ μ&#39;} k\big((μ, y), (μ&#39;, Z&#39;)\big)
+ 𝔼_{Z ∼ μ, Z&#39; ∼ μ&#39;} k\big((μ, Z), (μ&#39;, Z&#39;)\big).
\end{aligned}\]</p><p>The p-value is estimated based on the asymptotically valid approximation</p><p class="math-container">\[ℙ(n\mathrm{SKCE}_{uq} &gt; c \,|\, H_0) \approx ℙ(T &gt; c \,|\, \mathcal{D}),\]</p><p>where <span>$T$</span> is the bootstrap statistic</p><p class="math-container">\[T = \frac{2}{n} \sum_{1 \leq i &lt; j \leq n} \bigg(h_k\big((P^*_{X_i}, Y^*_i), (P^*_{X_j}, Y^*_j)\big)
- \frac{1}{n} \sum_{r = 1}^n h_k\big((P^*_{X_i}, Y^*_i), (P_{X_r}, Y_r)\big)
- \frac{1}{n} \sum_{r = 1}^n h_k\big((P_{X_r}, Y_r), (P^*_{X_j}, Y^*_j)\big)
+ \frac{1}{n^2} \sum_{r, s = 1}^n h_k\big((P_{X_r}, Y_r), (P_{X_s}, Y_s)\big)\bigg)\]</p><p>for bootstrap samples <span>$(P^*_{X_i}, Y^*_i)_{i=1,\ldots,n}$</span> of <span>$\mathcal{D}$</span>. This can be reformulated to the approximation</p><p class="math-container">\[ℙ(n\mathrm{SKCE}_{uq}/(n - 1) - \mathrm{SKCE}_b &gt; c \,|\, H_0) \approx ℙ(T&#39; &gt; c \,|\, \mathcal{D}),\]</p><p>where</p><p class="math-container">\[\mathrm{SKCE}_b = \frac{1}{n^2} \sum_{i, j = 1}^n h_k\big((P_{X_i}, Y_i), (P_{X_j}, Y_j)\big)\]</p><p>and</p><p class="math-container">\[T&#39; = \frac{2}{n(n - 1)} \sum_{1 \leq i &lt; j \leq n} h_k\big((P^*_{X_i}, Y^*_i), (P^*_{X_j}, Y^*_j)\big)
- \frac{2}{n^2} \sum_{i, r=1}^n h_k\big((P^*_{X_i}, Y^*_i), (P_{X_r}, Y_r)\big).\]</p><p><strong>References</strong></p><p>Widmann, D., Lindsten, F., &amp; Zachariah, D. (2019). <a href="https://proceedings.neurips.cc/paper/2019/hash/1c336b8080f82bcc2cd2499b4c57261d-Abstract.html">Calibration tests in multi-class classification: A unifying framework</a>. In: Advances in Neural Information Processing Systems (NeurIPS 2019) (pp. 12257–12267).</p><p>Widmann, D., Lindsten, F., &amp; Zachariah, D. (2021). <a href="https://openreview.net/forum?id=-bxf89v3Nx">Calibration tests beyond classification</a>.</p></div><a class="docs-sourcelink" target="_blank" href="https://github.com/devmotion/CalibrationTests.jl/blob/0c5409f2bf13a411264a7289ed036aa799575090/src/skce/asymptotic.jl#L1-L62">source</a></section></article></article><nav class="docs-footer"><a class="docs-footer-prevpage" href="../">« Home</a><div class="flexbox-break"></div><p class="footer-message">Powered by <a href="https://github.com/JuliaDocs/Documenter.jl">Documenter.jl</a> and the <a href="https://julialang.org/">Julia Programming Language</a>.</p></nav></div><div class="modal" id="documenter-settings"><div class="modal-background"></div><div class="modal-card"><header class="modal-card-head"><p class="modal-card-title">Settings</p><button class="delete"></button></header><section class="modal-card-body"><p><label class="label">Theme</label><div class="select"><select id="documenter-themepicker"><option value="auto">Automatic (OS)</option><option value="documenter-light">documenter-light</option><option value="documenter-dark">documenter-dark</option></select></div></p><hr/><p>This document was generated with <a href="https://github.com/JuliaDocs/Documenter.jl">Documenter.jl</a> version 1.4.0 on <span class="colophon-date" title="Wednesday 1 May 2024 14:17">Wednesday 1 May 2024</span>. Using Julia version 1.10.3.</p></section><footer class="modal-card-foot"></footer></div></div></div></body></html>
Loading

0 comments on commit 519e824

Please sign in to comment.