Merge pull request #138 from nathanrboyer/nb/chairmarks

Replace BenchmarkTools.jl with Chairmarks.jl in Optimizing

Author: jacobusmmsmit

optimizing/index.md

@@ -36,7 +36,7 @@ With this in mind, after you're done with the current page, you should read the
 
 ## Measurements
 
-\tldr{Use BenchmarkTools.jl's `@benchmark` with a setup phase to get the most accurate idea of your code's performance. Use Chairmarks.jl as a faster alternative.}
+\tldr{Use Chairmarks.jl's `@be` with a setup phase to get the most accurate idea of your code's performance.}
 
 The simplest way to measure how fast a piece of code runs is to use the `@time` macro, which returns the result of the code and prints the measured runtime and allocations.
 Because code needs to be compiled before it can be run, you should first run a function without timing it so it can be compiled, and then time it:
@@ -53,39 +53,42 @@ using BenchmarkTools
 Using `@time` is quick but it has flaws, because your function is only measured once.
 That measurement might have been influenced by other things going on in your computer at the same time.
 In general, running the same block of code multiple times is a safer measurement method, because it diminishes the probability of only observing an outlier.
+The Chairmarks.jl package provides convenient syntax to do just that.
 
-### BenchmarkTools
+### Chairmarks
 
-[BenchmarkTools.jl](https://github.com/JuliaCI/BenchmarkTools.jl) is the most popular package for repeated measurements on function executions.
-Similarly to `@time`, BenchmarkTools offers `@btime` which can be used in exactly the same way but will run the code multiple times and provide an average.
-Additionally, by using `$` to [interpolate external values](https://juliaci.github.io/BenchmarkTools.jl/stable/manual/#Interpolating-values-into-benchmark-expressions), you remove the overhead caused by global variables.
+[Chairmarks.jl](https://github.com/LilithHafner/Chairmarks.jl) is the latest benchmarking toolkit, designed to make fast and accurate timing measurements.
+Chairmarks offers `@b` (for "benchmark") which can be used in the same way as `@time` but will run the code multiple times and provide a minimum execution time.
+Alternatively, Chairmarks also provides `@be` to run the same benchmark and output all of its statistics.
 
-```>$-example
-using BenchmarkTools
-@btime sum_abs(v);
-@btime sum_abs($v);
+```>chairmarks-example
+using Chairmarks
+@b sum_abs(v)
+@be sum_abs(v)
+```
+
+Chairmarks supports a pipeline syntax with optional `init`, `setup`, `teardown`, and `keywords` arguments for more extensive control over the benchmarking process.
+The `sum_abs` function could also be benchmarked using pipeline syntax as below.
+
+```>pipeline-example-simple
+@be v sum_abs
 ```
 
-In more complex settings, you might need to construct variables in a [setup phase](https://juliaci.github.io/BenchmarkTools.jl/stable/manual/#Setup-and-teardown-phases) that is run before each sample.
-This can be useful to generate a new random input every time, instead of always using the same input.
+For a more complicated example, you could write the following to benchmark a matrix multiplication function for one second, excluding the time spent to *setup* the arrays.
 
-```>setup-example
+```>pipeline-example-complex
 my_matmul(A, b) = A * b;
-@btime my_matmul(A, b) setup=(
-    A = rand(1000, 1000); # use semi-colons between setup lines
-    b = rand(1000)
-);
+@be (A=rand(1000,1000), b=rand(1000)) my_matmul(_.A, _.b) seconds=1
 ```
 
-For better visualization, the `@benchmark` macro shows performance histograms:
+See the [Chairmarks documentation](https://chairmarks.lilithhafner.com/) for more details on benchmarking options.
+For better visualization, [PrettyChairmarks.jl](https://github.com/astrozot/PrettyChairmarks.jl) shows performance histograms alongside the numerical results.
 
 \advanced{
-Certain computations may be [optimized away by the compiler]((https://juliaci.github.io/BenchmarkTools.jl/stable/manual/#Understanding-compiler-optimizations)) before the benchmark takes place.
+No matter the benchmarking tool used, certain computations may be [optimized away by the compiler]((https://juliaci.github.io/BenchmarkTools.jl/stable/manual/#Understanding-compiler-optimizations)) before the benchmark takes place.
 If you observe suspiciously fast performance, especially below the nanosecond scale, this is very likely to have happened.
 }
 
-[Chairmarks.jl](https://github.com/LilithHafner/Chairmarks.jl) offers an alternative to BenchmarkTools.jl, promising faster benchmarking while attempting to maintain high accuracy and using an alternative syntax based on pipelines.
-
 ### Benchmark suites
 
 While we previously discussed the importance of documenting breaking changes in packages using [semantic versioning](/sharing/index.md#versions-and-registration), regressions in performance can also be vital to track.
@@ -97,10 +100,13 @@ Several packages exist for this purpose:
 
 ### Other tools
 
-BenchmarkTools.jl works fine for relatively short and simple blocks of code (microbenchmarking).
+Chairmarks.jl works fine for relatively short and simple blocks of code (microbenchmarking).
 To find bottlenecks in a larger program, you should rather use a [profiler](#profiling) or the package [TimerOutputs.jl](https://github.com/KristofferC/TimerOutputs.jl).
 It allows you to label different sections of your code, then time them and display a table of grouped by label.
 
+[BenchmarkTools.jl](https://github.com/JuliaCI/BenchmarkTools.jl) is the older standard for benchmarking in Julia. It is still widely used today.
+However, its default parameters run benchmarks for longer than Chairmarks, and it requires interpolating variables into the benchmarked expressions with `$`.
+
 Finally, if you know a loop is slow and you'll need to wait for it to be done, you can use [ProgressMeter.jl](https://github.com/timholy/ProgressMeter.jl) or [ProgressLogging.jl](https://github.com/JuliaLogging/ProgressLogging.jl) to track its progress.
 
 ## Profiling
@@ -141,7 +147,7 @@ To integrate profile visualisations into environments like Jupyter and Pluto, us
 No matter which tool you use, if your code is too fast to collect samples, you may need to run it multiple times in a loop.
 
 \advanced{
-    To visualize memory allocation profiles, use PProf.jl or VSCode's `@profview_allocs`. 
+    To visualize memory allocation profiles, use PProf.jl or VSCode's `@profview_allocs`.
     A known issue with the allocation profiler is that it is not able to determine the type of every object allocated, instead `Profile.Allocs.UnknownType` is shown instead.
     Inspecting the call graph can help identify which types are responsible for the allocations.
 }
@@ -386,7 +392,7 @@ However, in order for all workers to know about a function or module, we have to
 using Distributed
 
 # Add additional workers then load code on the workers
-addprocs(3) 
+addprocs(3)
 @everywhere using SharedArrays
 @everywhere f(x) = 3x^2