Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ bin
# Ignore auto-generated benchmarking files and dependencies
benchmarking/jars/
benchmarking/datasets/
benchmarking/results/
benchmarking/outputs/
benchmarking/sources/
benchmarking/compiled_classes/

Expand Down
14 changes: 12 additions & 2 deletions benchmarking/benchmarking.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Benchmarking

# Stage Zero
# Stage Zero: Initiailization

The first stage of benchmarking involves downloading all the necessary files
required, i.e. Jar files and the NJR-1 Dataset.
Expand All @@ -11,7 +11,7 @@ In order for benchmarking to run, the jar files for all the following
dependencies must be located in the `JARS_DIR` directory, in their respective
sub-folder (i.e. `JARS_DIR/errorprone`, `JARS_DIR/nullaway`, and
`JARS_DIR/annotator`). If not present, they will be automaticlaly downloaded
from the Maven Repository.
+from the Maven Repository.

## Note: Different versions of the following dependencies may not be compatable. The newest versions of each project that are confirmed to work together are:

Expand Down Expand Up @@ -66,3 +66,13 @@ before NullAway can process them.
- [Checker-Qual](https://mvnrepository.com/artifact/org.checkerframework/checker-qual/)
- checker-qual contains annotations (type qualifiers) that a programmer writes
to specify Java code for type-checking by the Checker Framework.

# Stage One: Annotation

Stage One is the largest stage in terms of time consumption and computation. It
involves running NullAwayAnnotator on the entire NJR-1 dataset in order to
prepare it for refactoring, as well as to get an accurate count of the number of
NullAway errors in the original programs, refactoring every program using VGR,
and finally re-running annotator to get an updated error count. This cycle
(annotate->refactor->annotate) is completed for each program in NJR-1
sequentially.
Loading
Loading