Skip to content

GSACTech/contest

Repository files navigation

This project automatically tests and compares all participants' static analysis tools.

  • The participants should develop a software analyzer using the provided template.
  • They should accept the assignment and submit as a GitHub repository which will be visible only to admins.

To get all the requirements run the following script

sudo /path/to/contest/install_required_packages.sh

Usage

  • To evaluate their analyzers on our test suites
    • Create a JSON-formatted file, which contains list of {owner, URL} pairs.
    • Run the following python script
python3 /path/to/evaluate.py --test-suites PATH_TO_TESTS --tools PATH_TO_JSON_FILE
  • PATH_TO_TESTS shows the directory path containing test suites.
  • PATH_TO_JSON_FILE is the path of JSON-formatted file which must be in the following format
{
  "owner": "URL of GitHub repository"
}

Results

  • The evaluation score for each analyzer is calculating in the following way

    • There are three categories of test cases: memory leak, buffer overflow and use after free.
    • Each category has three groups of test cases: easy, medium and hard.
    • For each group there's a corresponding weight: 1 for easy, 2 for medium and 3 for hard ones.
    • The evaluator calculates F1 score for each test case.
    • It then multiplies the F1 score by the weight of its test group and adds the result to the final score.
  • In special results directory the following files will be stored

    • All analyzers with their results result_dir/analyzer_dir/result_sarif_files.
    • The evaluated scores on test suites in JSON and Excel formatted files.

About

This is the main repository which will automatically test and compare all participant static analysis tools.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published