[WIP] Embedded Laplace Approximation #3097

SteveBronder · 2024-07-22T16:43:59Z

Summary

Code for the embedded laplace approximation. The tests are all passing but there is a few things still needed

Update docs for all functions (@charlesm93 I added the basic arguments for a lot of the functions but could use your expertise to give the definitions for each argument)
Add corresponding rngs for each specialized laplace approximation (and tests)
Add tests for negative binomial

The file for laplace have been added to the mix folder since it uses higher order auto diff.

The current signature for the generalized laplace looks like the following in C++

inline auto laplace_marginal_lpdf(LFun&& L_f, LArgs&& l_args,
                                  const Theta0& theta_0, CovarFun&& K_f,
                                  std::ostream* msgs, Args&&... args)

which will translate to stan like the following

target += laplace_marginal_tol_lpdf(
  likelihood_functor,  make_tuple(data_arg1, data_arg2), 
  eta, theta_init,
  covariance_function, 
  1e-6, 10, 2, 1, 5, // tuning args 
  covar_fun_arg1, covar_fun_arg2);

Note that the first tuple used for the likelihood arguments must be data.

Instead of using a tuple for the first functor's inputs and variadic arguments for the covariance functors arguments I would rather have them both be tuples like the following

target += laplace_marginal_tol_lpdf(
  likelihood_functor, make_tuple(data_arg1, data_arg2), 
  covariance_function,  make_tuple(covar_fun_arg1, covar_fun_arg2)
  theta_init,
  eta,
  1e-6, 10, 2, 1, 5);

I think this is nice because it makes it makes the tolerance parameters always sit at the end and both functors have the same input scheme for their arguments. Does anyone have thoughts on this

Other additions related to this PR

A filter_map function that applies a conditionally applies a lambda f to each input of a tuple given a type_trait i.e. the following code would print "fp detected" twice and increment the double elements of the tuple by 1.

std::tuple<double, int, double> tup{1.0, 2, 3.0};
std::tuple<double, int, double> tup2 = filter_map<std::is_floating_point>([](auto x) { 
  std::cout << "fp detected" << std::endl;
  return x + 1;
  }, tup);

The test_ad suite now has a compile time option for only running the tests with only prim and reverse mode with a new boolean template parameter to expect_ad. This is needed to use laplace with the test framework as the laplace impl here does not work with higer order autodiff (since it needs higher order autodiff)

Tests

Since the tests all seem very related I kept them in their own folder, is that alright? Or should I distribute them across the test folders like normal? While this PR is WIP I'm going to leave them in the same folder and if we don't want that then we can move them before we merge

./runTests.py -j20 ./test/unit/math/mix/laplace/

Side Effects

Release notes

Checklist

Copyright holder: Simon's Foundation

The copyright holder is typically you or your assignee, such as a university or company. By submitting this pull request, the copyright holder is agreeing to the license the submitted work under the following licenses:
- Code: BSD 3-clause (https://opensource.org/licenses/BSD-3-Clause)
- Documentation: CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/)
the basic tests are passing
- unit tests pass (to run, use: ./runTests.py test/unit)
- header checks pass, (make test-headers)
- dependencies checks pass, (make test-math-dependencies)
- docs build, (make doxygen)
- code passes the built in C++ standards checks (make cpplint)
the code is written in idiomatic C++ and changes are documented in the doxygen
the new changes are tested

…ctor.

…ure arguments.

… into experimental/laplace

…instead of using an in-out parameter

…experimental/laplace

WardBrian · 2025-05-28T13:41:50Z

Changing the namespace of is_tuple will also require changes up in Stan. Not sure it's worth doing as part of this

SteveBronder · 2025-05-28T14:32:57Z

Oh good call I'll put that back

…s in internal namespace

WardBrian

Bunch of comments from in-person review. Should be good to go once addressed

Jenkinsfile

stan/math/mix/functor/hessian_block_diag.hpp

test/unit/math/prim/functor/iter_tuple_n_test.cpp

stan/math/prim/functor/iter_tuple_n.hpp

WardBrian · 2025-05-28T19:39:46Z

stan/math/prim/functor/iter_tuple_n.hpp

+          iter_tuple_n(f, std::forward<decltype(args_i)>(args_i)...);
+        },
+        std::forward<Types>(args)...);
+  } else if constexpr (is_vec_container) {


Verify that this check is only triggering for vectors of tuples (e.g, not vectors of matrices).

stan/math/mix/functor/laplace_marginal_density.hpp

stan/math/mix/prob/laplace_marginal.hpp

WardBrian · 2025-05-28T20:38:13Z

stan/math/mix/prob/laplace_marginal_neg_binomial_2_log_lpmf.hpp

+      std::forward<CovarArgs>(covar_args), ops, msgs);
+}
+
+struct neg_binomial_2_log_likelihood_summary {


If we want to keep this, @charlesm93 needs to add it to his doc PR and I need to add it to the compiler (both should be easy)

WardBrian · 2025-05-28T20:39:15Z

stan/math/mix/prob/laplace_marginal_poisson_log_exposure_lpmf.hpp

Should we have the filename and function match? exposure or 2? @charlesm93

I made both of these 2

See https://stackoverflow.com/questions/59647445/if-constexpr-with-static-assert-in-lambda-which-compiler-is-correct

stan-buildbot · 2025-05-31T06:32:40Z

Name	Old Result	New Result	Ratio	Performance change( 1 - new / old )
arma/arma.stan	0.32	0.31	1.04	4.27% faster
low_dim_corr_gauss/low_dim_corr_gauss.stan	0.01	0.01	1.05	4.77% faster
gp_regr/gen_gp_data.stan	0.02	0.02	1.05	4.68% faster
gp_regr/gp_regr.stan	0.09	0.09	1.06	5.98% faster
sir/sir.stan	71.35	66.13	1.08	7.32% faster
irt_2pl/irt_2pl.stan	4.13	3.96	1.04	4.18% faster
eight_schools/eight_schools.stan	0.06	0.05	1.09	8.57% faster
pkpd/sim_one_comp_mm_elim_abs.stan	0.25	0.24	1.04	4.24% faster
pkpd/one_comp_mm_elim_abs.stan	19.26	18.62	1.03	3.35% faster
garch/garch.stan	0.43	0.4	1.08	7.54% faster
low_dim_gauss_mix/low_dim_gauss_mix.stan	2.69	2.57	1.05	4.48% faster
arK/arK.stan	1.78	1.71	1.04	3.72% faster
gp_pois_regr/gp_pois_regr.stan	2.8	2.69	1.04	3.75% faster
low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan	8.7	8.3	1.05	4.57% faster
performance.compilation	183.58	176.63	1.04	3.79% faster
Mean result: 1.053061084655534

Jenkins Console Log
Blue Ocean
Commit hash: a82154c8a93e75b4af2ddaa645f722d4b5046e0d

Machine information

No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.3 LTS Release: 20.04 Codename: focal

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 48 bits virtual
CPU(s): 80
On-line CPU(s) list: 0-79
Thread(s) per core: 2
Core(s) per socket: 20
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
Stepping: 4
CPU MHz: 2400.000
CPU max MHz: 3700.0000
CPU min MHz: 1000.0000
BogoMIPS: 4800.00
Virtualization: VT-x
L1d cache: 1.3 MiB
L1i cache: 1.3 MiB
L2 cache: 40 MiB
L3 cache: 55 MiB
NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79
Vulnerability Gather data sampling: Mitigation; Microcode
Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled
Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Mitigation; IBRS
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; IBRS; IBPB conditional; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear flush_l1d arch_capabilities

G++:
g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang:
clang version 10.0.0-4ubuntu1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

stan-buildbot · 2025-05-31T21:01:25Z

Name	Old Result	New Result	Ratio	Performance change( 1 - new / old )
arma/arma.stan	0.32	0.3	1.07	6.66% faster
low_dim_corr_gauss/low_dim_corr_gauss.stan	0.01	0.01	1.08	7.12% faster
gp_regr/gen_gp_data.stan	0.02	0.02	1.04	3.4% faster
gp_regr/gp_regr.stan	0.09	0.09	1.04	4.04% faster
sir/sir.stan	70.33	67.61	1.04	3.87% faster
irt_2pl/irt_2pl.stan	4.2	4.02	1.04	4.14% faster
eight_schools/eight_schools.stan	0.06	0.05	1.06	5.92% faster
pkpd/sim_one_comp_mm_elim_abs.stan	0.25	0.25	1.03	3.06% faster
pkpd/one_comp_mm_elim_abs.stan	19.9	18.74	1.06	5.82% faster
garch/garch.stan	0.43	0.4	1.08	7.38% faster
low_dim_gauss_mix/low_dim_gauss_mix.stan	2.76	2.64	1.04	4.3% faster
arK/arK.stan	1.82	1.73	1.05	4.98% faster
gp_pois_regr/gp_pois_regr.stan	2.85	2.7	1.06	5.52% faster
low_dim_gauss_mix_collapse/low_dim_gauss_mix_collapse.stan	8.81	8.35	1.05	5.21% faster
performance.compilation	181.85	179.48	1.01	1.3% faster
Mean result: 1.0512292166113941

Jenkins Console Log
Blue Ocean
Commit hash: a82154c8a93e75b4af2ddaa645f722d4b5046e0d

Machine information

No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 20.04.3 LTS Release: 20.04 Codename: focal

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 48 bits virtual
CPU(s): 80
On-line CPU(s) list: 0-79
Thread(s) per core: 2
Core(s) per socket: 20
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
Stepping: 4
CPU MHz: 1100.608
CPU max MHz: 3700.0000
CPU min MHz: 1000.0000
BogoMIPS: 4800.00
Virtualization: VT-x
L1d cache: 1.3 MiB
L1i cache: 1.3 MiB
L2 cache: 40 MiB
L3 cache: 55 MiB
NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78
NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79
Vulnerability Gather data sampling: Mitigation; Microcode
Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled
Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Mitigation; IBRS
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; IBRS; IBPB conditional; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke md_clear flush_l1d arch_capabilities

G++:
g++ (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Clang:
clang version 10.0.0-4ubuntu1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

WardBrian · 2025-05-31T21:23:15Z

Without ASAN, the laplace tests only took 16 minutes? Does that sound right? @SteveBronder

SteveBronder · 2025-05-31T22:07:47Z

Yeah locally they take about 20 minutes

SteveBronder · 2025-05-31T22:08:19Z

Let me give this one hard look on Sunday then Monday we merge and Tuesday we release!

WardBrian · 2025-05-31T22:11:10Z

16 minutes -> 4 hours using ASAN is one of the worst slowdowns I’ve ever seen for a sanitizer run. Wow

SteveBronder · 2025-06-02T20:52:39Z

@WardBrian merged! Tuesday lets merge the stanc3 pr then we are good to release!

charlesm93 and others added 30 commits December 5, 2021 19:11

make laplace_marginal_bernoulli_logit_lpmf use general likelihood fun…

b2e4b2f

…ctor.

Delete research tests which are / will not be unit tests.

031a81b

refactor rng test for laplace_poisson_log.

ae138fe

Benchmark laplace_poisson_log_rng against multi_normal_rng.

5231604

refactor rng poisson log to use full autodiff and add tests for expos…

9e3f9ae

…ure arguments.

Merge remote-tracking branch 'origin/develop' into experimental/laplace

f218ba0

update tolerance for nearness tests

0fc47f5

update tolerance for nearness tests

7a40142

cleanup laplace but failing address sanitizer

a35ff32

update off by one error

cfd2199

Merge branch 'experimental/laplace' of https://github.com/stan-dev/math…

269b9af

… into experimental/laplace

clang format

d7e507f

Merge branch 'experimental/laplace' of https://github.com/stan-dev/math…

c78c7d1

… into experimental/laplace

update to remove some extra functions and just call them inline

b1d8312

Merge branch 'experimental/laplace' of https://github.com/stan-dev/math…

ce75b03

… into experimental/laplace

local changes for rng tests.

c7499b3

remove extra functions and have hessian_times_vector return a vector …

396278c

…instead of using an in-out parameter

Merge branch 'experimental/laplace' of github.com:stan-dev/math into …

ef7d2a4

…experimental/laplace

cleanup

c6f222a

clang format

fdcf8ca

remove the in/out parameters from hessian_block_diag and diff

a009b8f

removes in/out parameters from laplace_marginal_density_est

ae6511e

small fix in laplace_marginal eta adjoint

41a3d23

small cleanup

1f6199a

swap x and phi to covariance functor

adcddca

start working on variadic version of laplace

3b60f23

update laplace so it takes in variadic arguments for known likelihoods

26c6a14

clang format

03dd6e0

update tests

265991c

Removes diagonal_covariance option and code. Also removes do_line_search

6b97f1f

fix is_tuple namespace location

35bd764

SteveBronder and others added 4 commits May 28, 2025 10:48

move is_tuple back to stan::math namespace

82ce1d1

[Jenkins] auto-formatting by clang-format version 10.0.0-4ubuntu1

27684f8

update docs and cleanup signatures for laplace. Put internal function…

f09e934

…s in internal namespace

[Jenkins] auto-formatting by clang-format version 10.0.0-4ubuntu1

5b8cddc

WardBrian requested changes May 28, 2025

View reviewed changes

SteveBronder and others added 11 commits May 29, 2025 16:32

update laplace with review comments

d793de3

[Jenkins] auto-formatting by clang-format version 10.0.0-4ubuntu1

0b88d1c

add back accidentally deleted accumulate and count vars

886073c

add back accidentally deleted accumulate and count vars

8234959

update

586613a

[Jenkins] auto-formatting by clang-format version 10.0.0-4ubuntu1

f57cf66

fix value_of test

c5b1f40

fix static_asserts in laplace

f203d7e

update with runtime throws instead of static_asserts for laplace

7639a7d

[Jenkins] auto-formatting by clang-format version 10.0.0-4ubuntu1

1dbf607

Make static asserts depend on type local to lambda

044d2bc

See https://stackoverflow.com/questions/59647445/if-constexpr-with-static-assert-in-lambda-which-compiler-is-correct

Flip condition for asan in Jenkinsfile

41fe942

WardBrian approved these changes May 31, 2025

View reviewed changes

SteveBronder merged commit cbb74b3 into develop Jun 2, 2025
34 checks passed

WardBrian mentioned this pull request Jun 30, 2025

Include assertion that failed in STAN_OVERRIDE_EIGEN_ASSERT #3214

Merged

4 tasks

Uh oh!

[WIP] Embedded Laplace Approximation #3097

[WIP] Embedded Laplace Approximation #3097

Uh oh!

Conversation

SteveBronder commented Jul 22, 2024 • edited by charlesm93 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Tests

Side Effects

Release notes

Checklist

Uh oh!

WardBrian commented May 28, 2025

Uh oh!

SteveBronder commented May 28, 2025

Uh oh!

WardBrian left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

WardBrian May 28, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

WardBrian May 28, 2025

Choose a reason for hiding this comment

Uh oh!

WardBrian May 28, 2025

Choose a reason for hiding this comment

Uh oh!

SteveBronder May 30, 2025

Choose a reason for hiding this comment

Uh oh!

stan-buildbot commented May 31, 2025

Uh oh!

stan-buildbot commented May 31, 2025

Uh oh!

WardBrian commented May 31, 2025

Uh oh!

SteveBronder commented May 31, 2025

Uh oh!

SteveBronder commented May 31, 2025

Uh oh!

WardBrian commented May 31, 2025

Uh oh!

Uh oh!

SteveBronder commented Jun 2, 2025

Uh oh!

Uh oh!

SteveBronder commented Jul 22, 2024 •

edited by charlesm93

Loading