Compare against the merge-base if no vs is specified #134
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, PkgEvalJob was comparing against the git ref as specified (which is bad because you could then be comparing against a branch that's much ahead, e.g., JuliaLang/julia#47369 (comment)), and BenchmarkJob was checking out the merge commit. I'd argue that the latter isn't ideal as well: sometimes the merge commit is unavailable (in case of merge conflicts) leading to inconsistent behavior; the merge commit isn't available directly so if you request
runbenchmarks
as part of the PR opening post you'll end up comparing against a different commit; and finally the merge base is a commit that you can't find in the GitHub UI, so theversioninfo()
as reported in the benchmark reports isn't very useful.As an alternative, this PR switches to using the merge-base, i.e., the commit on
master
where the PR forked. However, as it's not clear when to look up the merge-base (e.g., only when specifyingvs=:master
, or also when specifyingvs=v1.8
orvs=SOME_SHA
), I've implemented @KristofferC's suggestion here to default to the master branch when not specifying avs
at all from a Nanosoldier invocation on a PR. This also considerably simplifies the most common Nanosoldier invocation to:@nanosoldier runbenchmarks(ALL)
or@nanosoldier runtests(ALL)
.I'll be testing this on the PkgEval bot in the coming days.
Fixes #131