Benchmarks Branch [DO NOT MERGE] #25
Conversation
…nto benchmarks
…nto benchmarks
…nto benchmarks
|
I'm happy to take a look at this, but do you think it would best belong in a different repository? |
|
I'm keeping it on the `benchmarks` branch. I just figured the easiest way to look at that would be through a PR. I'll put a do-not-merge warning on it.
|
bob-carpenter
changed the title
|
Also, not sure why this would cause a failure as I didn't plug in any new builds or new tests. |
|
It looks like you added the stationarity example to cmake, but it’s missing a #include it needs |
|
Thanks---I hadn't looked into the cause, but I just added the include, and hopefully that will fix it. |
examples/eval_stan.cpp
Outdated
There was a problem hiding this comment.
Hi @bob-carpenter -- you might want to check out #26, which has a target very similar to this with a few more features already
| std::string sample_csv_file_numbered = prefix + "-walnuts-draws-" + std::to_string(trial) + ".csv"; | ||
| test_adaptive_walnuts(stan_model, sample_csv_file_numbered, trial_seed, iter_warmup, iter_sampling); | ||
| } | ||
| std::quick_exit(0); // crashes without this---not stan_model dtor, prob dlclose_deleter |
There was a problem hiding this comment.
Interesting, does the preexisting example-stan program crash at exit as well? This is probably isn't relevant for the purposes of evaluation, but could be important for later python bindings etc
There was a problem hiding this comment.
I wonder if this could be related to roualdes/bridgestan#111
Also, when stan is compiled using zig cc it also segfaults on exit, and since zig adds some flags to add traps to avoid undefined behavior, maybe that is also related?
There was a problem hiding this comment.
Yeah, seems likely. We should probably just leak the handle for now, and try to figure out roualdes/bridgestan#111. My guess is we need a bs_finalize() function that calls tbb::finalize() and maybe also some math library specific stuff.
|
|
||
| print("Sampling") | ||
| print(f"{0:5d}. min(ESS) = {0:.1e}") | ||
| for b in range(1, max_blocks + 1): |
There was a problem hiding this comment.
Note that this as currently coded isn't much different from one big run in terms of disk usage -- cmdstanpy cleans up it's temporary directory at exit not when the fit object is gc'd.
To fix, you could use the TemporaryDirectory context manager and the output_dir argument to sample
| da = xr.DataArray(a[np.newaxis, :, :], dims=("chain", "draw", "var")) | ||
| ds = az.ess(da, method="bulk") | ||
| data_var = next(iter(ds.data_vars)) | ||
| vec = ds[data_var].values | ||
| return np.asarray(np.squeeze(vec), dtype=np.float64) |
There was a problem hiding this comment.
This is pretty hard to read/verify it's doing the right thing without manually running it a few times. Assuming you're confident in it that's probably fine, but just wanted to flag it
| DynamicStanModel stan_model(model_so_file_c_str, data_json_file_c_str, seed); | ||
| for (int trial = 0; trial < trials; ++trial) { | ||
| std::cout << "trial = " << trial << std::endl; | ||
| unsigned int trial_seed = seed + static_cast<unsigned int>(17 * (trial + 1)); |
There was a problem hiding this comment.
Unsure it matters, but curious why you're doing seed + 17 * trial here but seed + trial in eval-nuts.py?
| for (Integer n = 0; n < iter_sampling; ++n) { | ||
| auto draw = sampler(); | ||
| model.constrain_draw(draw, draws.col(n)); | ||
| lp_grads[static_cast<std::size_t>(n)] = logp_grad_calls; |
There was a problem hiding this comment.
Curious why these static casts are necessary, and why you wouldn't just declare the loop as for(size_t n ... instead
| * `examples/stan-warmup.py` | ||
|
|
||
|
|
||
| ## III. Gradients until within error bound |
There was a problem hiding this comment.
I would recommend collecting these steps into one bash or python script at some point to make running an experiment a bit less manual
…nto benchmarks
I coded up something to run a Stan program until the ESS for E[theta], E[theta^2], E[theta^4] are all above a given threshold, where
thetaincludes all the parameters and the log densitylp__. It just iterates blocks of draws, and keeps a running sum of ESS values and a running average. It then dumps the results in scientific notation in JSON to 10 decimal places, which is more than we should need for our test cases. Ideally, we'd dump to within a number of standard deviations from the mean, but I think this should be OK unless we have examples with small very small standard deviations relative to their means.I largely used ChatGPT to code this, so more than happy to get feedback on the Python.
Here's the output for a 10-dimensional normal target with scales 1 to 10.
It produces this JSON output:
{ "vars": [ 9.2634264000e-05, -9.4769614000e-05, 1.0806610000e-03, -1.4658990000e-03, -1.1208311000e-03, 1.4941326000e-03, 3.6323028000e-03, -4.6460588000e-03, 5.7136589000e-03, 3.5120391000e-04, -5.0045560000e+00 ], "vars_sq": [ 1.0012219000e+00, 4.0041225000e+00, 9.0160672000e+00, 1.6015831000e+01, 2.5000648000e+01, 3.5992883000e+01, 4.9134498000e+01, 6.3976726000e+01, 8.1199190000e+01, 9.9941619000e+01, 3.0055958000e+01 ], "vars_fourth": [ 3.0057456000e+00, 4.8089912000e+01, 2.4402527000e+02, 7.7085734000e+02, 1.8764548000e+03, 3.8927089000e+03, 7.2470225000e+03, 1.2282180000e+04, 1.9775846000e+04, 3.0088670000e+04, 1.6872879000e+03 ], "ess": [ 2.5928430710e+06, 2.8875322166e+06, 3.9015117136e+06, 3.7920353054e+06, 3.3976893429e+06, 2.7514505746e+06, 2.3488147882e+06, 2.0175434890e+06, 1.7036521274e+06, 1.4553086826e+06, 1.0828808333e+06 ], "ess_sq": [ 1.1144011606e+06, 1.0380386513e+06, 1.0035197614e+06, 1.0797689318e+06, 1.2362565887e+06, 1.2381111553e+06, 1.1916465762e+06, 1.2102278759e+06, 1.2358123834e+06, 1.2461369728e+06, 1.0828808333e+06 ], "ess_fourth": [ 1.1144011606e+06, 1.0380386513e+06, 1.0035197614e+06, 1.0797689318e+06, 1.2362565887e+06, 1.2381111553e+06, 1.1916465762e+06, 1.2102278759e+06, 1.2358123834e+06, 1.2461369728e+06, 1.0828808333e+06 ] }