Minor corrections and cleanup of MCEM #1601
Open
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #1599 and other changes.
This PR provides several updates to MCEM that are (generally relatively minor) bug fixes and cleanup.
pStartvector, and if parameter boundaries were not being used (i.e. typically), thepStartwas actually not being put in the model. So in effect the algorithm was starting from the values currently in the model, which was the default behavior anyway. In addition, there was no calculation ofparamDeps(see next point) prior to the very first MCMC sampling.paramDeps, that is deterministic nodes between parameters and random effects, is updated to be the same as the recent updates to the same need inbuildAGHQ(moving to packagenimbleQuad). That is an improvement in efficiency.burninwas not applied in a way consistent with the rest of nimble's MCMC systems. It was used as a post-thinning burnin instead of a pre-thinning burnin. (That could cause a confusing error because it could result in no samples being used for maximization (when one would reasonably expect there should be some) and an error fromoptimas a result.) That is changed / fixed.thininterval as a control element tobuildMCEM, it was not properly used. (The run-time argumentthinto methodfindMLEwas properly used at this point, but see the next point.)resettoMCMC$runwas not being properly applied. Accidentally it was used as thethinargument, again potentially resulting in a providedthinbeing ignored and also in the intended use ofresetnot occurring. When the algorithm to increase the sample size is triggered, instead of appending new samples, it was generating an entirely new MCMC sample, which was not wrong but was inefficient. This is fixed.reset=FALSEand actually append samples, we needed to set the model parameters back to those used for the previous MCMC samples. This was not being done. In a sense this glitch and the previous glitch canceled each other: since we were generating entirely new samples, it was at least consistent with accidentally conditioning on different parameters, which should not have been wrong, just inefficient. It is possible this was all intended with the expectation of improving efficiency later. However, the values ofresetsuggest otherwise, and they were ending up in the wrong place.vcov(the method for obtaining the variance-covariance matrix at the MLE), ifresetSamples=TRUE, the newly generating MCMC samples were not using thethininterval. In addition theparamDepswere not being calculated. This typically would not have mattered becausevcovwould be called right after finding the MLE and the model would be up to date for the MLE parameters.nimDerivsargumentreset=TRUE(to re-tape with any new values of any constants, different than theMCMC$runresetdiscussed so far) was not being utilized. In most typical cases this would not matter because the AD tape constants will be the data values and those will not typically be changed. However, now the tape is reset upon the first use within a call tofindMLE. This should be safer.nimbleOptionto control whetherbuildMCMCemits a warning when not all nodes are being sampled, andbuildMCEMuses that option to silence said warning since it normally builds an MCMC only for random effects, not parameters. Thanks @paciorek.mcemis installed. Thanks @paciorek .test-mcem. A new version of themcemTestLog_Correct.Rout"gold file" was generated.Although this is kind of extensive, evidence that the changes aren't really that major is that none of the test outputs changed substantially. They changed in very minor ways due to different MCMC samples due to appending rather than fully resampling when
M(sample size) is increased. The worst problems were not usingpStartproperly and not respectingthinsettings in various ways. Other issues were about efficiency or were such that they would apparently only cause problems in particular use cases.