Implement OOD validation pipeline (#5)

* change point detection module

* release automation

* debug docs

* debug docs

* change docs workflow

* debug workflow

* debug workflow

* debug `python-package.yml`

* debug release action

* update project files

* change list_datasets function

* restructure docs

* refactor datasets

* implement more ood detection methods

* remove cpd methods

* implement imagenet corrupted datasets

* debug ood pipeline

* update package structure

* update package structure

* major refactoring

* fix minor bugs

* fix small bug

* implement gradnorm

* implement vim

* update methods

* refactor pipelines

* debug datasets

* accelerate compute accuracy script

* multi-layer scores aggregations

* debug ood detection methods

* implemend `wilds` datasets

* debug pipelines

* implement layer aggregation methods

* implement more detection methods

* format code

* list available resources

* improve api

* method templates

* refactoring

* major methods refactoring

* aggregations api

* debugging

* debug docs

* prepare new version

* debug workflow

* update tests

* lint code

* update requirements

* implement logit_norm

* clean code

* implement Makefile

* lint code

* update docs

* improve documentation

* update docs template

* add docs theme

* setup readthedocs 1

* init leaderboard notebook

* ood validation pipeline

* debug detector template

* test pipelines

* lint code

.editorconfig

@@ -8,5 +8,11 @@ indent_size = 2
 trim_trailing_whitespace = true
 insert_final_newline = false
 
+[Makefile]
+indent_style = tab
+
 [*.py]
 indent_size = 4
+
+[*.ipynb]
+indent_size = 4

.flake8

@@ -0,0 +1,19 @@
+[flake8]
+exclude =
+    venv,
+    .venv,
+    __pycache__,
+    notebooks,
+    .git,
+    docs/source/conf.py,
+    old,
+    build,
+    dist,
+    deprecated
+max-line-length = 120
+max-complexity = 10
+extend-ignore =
+    E203,
+per-file-ignores =
+    # imported but unused
+    __init__.py: F401,F403,E402

.github/scripts/release.py

@@ -1,5 +1,3 @@
-#!/usr/bin/env python3
-import json
 import subprocess
 
 

.github/workflows/python-package.yml → .github/workflows/build.yml

@@ -1,6 +1,3 @@
-# This workflow will install Python dependencies, run tests and lint with a variety of Python versions
-# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python
-
 name: Python package
 
 on:
@@ -16,7 +13,7 @@ jobs:
       fail-fast: false
       matrix:
         os: [ubuntu-latest]
-        python-version: ["3.8"]
+        python-version: ["3.8", "3.9", "3.10", "3.11"]
 
     steps:
       - uses: actions/checkout@v3
@@ -33,13 +30,8 @@ jobs:
           if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
       - name: Format and Lint
         run: |
-          black --config pyproject.toml .
-          isort --settings pyproject.toml .
-          # stop the build if there are Python syntax errors or undefined names
-          flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
-          # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
-          flake8 . --count --exit-zero --max-complexity=10 --max-line-length=120 --statistics
+          make format
+          make lint
       - name: Test with pytest
         run: |
-          pytest -vv tests/models.py -s
-          pytest -vv tests/methods.py -s
+          make test

.github/workflows/python-publish.yml → .github/workflows/publish.yml

@@ -1,11 +1,3 @@
-# This workflow will upload a Python Package using Twine when a release is created
-# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python#publishing-to-package-registries
-
-# This workflow uses actions that are not certified by GitHub.
-# They are provided by a third-party and are governed by
-# separate terms of service, privacy policy, and support
-# documentation.
-
 name: Publish Python Package (PyPI.org)
 
 on:

.readthedocs.yml

@@ -0,0 +1,13 @@
+version: 2
+sphinx:
+  configuration: docs/source/conf.py
+formats:
+  - htmlzip
+  - pdf
+  - epub
+python:
+  version: 3.8
+  install:
+    - requirements: docs/requirements.txt
+    - method: pip
+      path: .

CONTRIBUTING.md

@@ -1,19 +1,110 @@
-Check back later, we will populate this soon
+# Contributing to Detectors 🧐
 
-## Important Resources
+Hi there, Thank you for even being interested in contributing to Detectors
+
+To contribute to this project, please follow a ["fork and pull request"](https://docs.github.com/en/get-started/quickstart/contributing-to-projects) workflow.
+
+## Contributing guidelines
+
+Our issues page is kept up to date with bugs, improvements, and feature requests. There is a taxonomy of labels to help with sorting and discovery of issues of interest. These include:
+
+- `method`
+- `pipeline`
+- `dataset`
+- `model`
+- `example`
+- `documentation`
+- `test`
+- `bug`
+
+If you start working on an issue, please assign it to yourself.
+
+If you are adding an issue, please try to keep it focused on a single modular bug/improvement/feature. If the two issues are related, or blocking, please link them rather than keep them as one single one.
+
+We will try to keep these issues as up to date as possible.
+
+### How to submit changes
+
+To contribute to this project, please follow a ["fork and pull request"](https://docs.github.com/en/get-started/quickstart/contributing-to-projects) workflow.
+
+### How to report a bug
+
+Create an issue with the `bug` label.
+
+### How to suggest a feature or enhancement
+
+Create an issue with the appropriate label or labels related to the desired feature or enhancement.
+
+### Getting help
+
+Although we try to have a developer setup to make it as easy as possible for others to contribute (see below) it is possible that some pain point may arise around environment setup, linting, documentation, or other. Should that occur, please contact a maintainer! Not only do we want to help get you unblocked, but we also want to make sure that the process is smooth for future contributors.
+
+### Release process
+
+The entire release process, including running tests, building documentation, and publishing to PyPI are automated with Github flows and will be triggered upon a successful push on the `master` branch.
+
+---
+
+## Installing dependencies
+
+To install development requirements:
+
+```bash
+pip install -e ".[dev]"
+```
+
+This will install all requirements for running the package, examples, linting, formatting, tests, and coverage.
 
 ## Testing
 
-## Environment details
+Unit tests are written using [pytest](https://docs.pytest.org/en/stable/).
+
+Code coverage (i.e. the amount of code that is covered by unit tests) helps identify areas of the code that are potentially more or less brittle.
+
+To run the tests, run:
+
+```bash
+make test
+```
+
+---
+
+## Documentation
+
+### Contribute Documentation
 
-## How to submit changes
+Docs are largely autogenerated by [sphinx](https://www.sphinx-doc.org/en/master/) from the code.
 
-## How to report a bug
+### Build Documentation Locally
 
-## How to suggest a feature or enhancement
+To build the documentation locally, run:
+
+```bash
+make docs_build
+```
+
+---
 
 ## Coding conventions
 
-## Code of Conduct
+### Code Formatting
+
+Formatting for this project is done via a combination of [Black](https://black.readthedocs.io/en/stable/) and [isort](https://pycqa.github.io/isort/).
+
+To format the code, run:
+
+```bash
+make format
+```
+
+### Linting
+
+Linting for this project is done via a combination of [Black](https://black.readthedocs.io/en/stable/), [isort](https://pycqa.github.io/isort/), and [flake8](https://flake8.pycqa.org/en/latest/).
+
+To lint the code, run:
+
+```bash
+make lint
+```
 
-## Thanks
+## Thanks! 🧐

Makefile

@@ -0,0 +1,33 @@
+package:
+	python -m pip install --upgrade pip
+	pip install build
+	python -m build
+
+test:
+	pytest -vv tests/models.py tests/methods.py -s --cov \
+		--cov-config=.coveragerc \
+		--cov-report xml \
+		--cov-report term-missing:skip-covered
+
+docs_build:
+	pip install -r docs/requirements.txt
+	cd docs
+	sphinx-apidoc -o ./source -f ../src/detectors -d 4
+	make clean
+	make html
+
+format:
+	black --config pyproject.toml .
+	isort --settings pyproject.toml .
+
+lint:
+	black . --check
+	flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
+	flake8 . --count --exit-zero --statistics
+
+help:
+	@echo 'package      - build the package'
+	@echo 'docs_build   - build the documentation'
+	@echo 'test         - run unit tests and generate coverage report'
+	@echo 'format       - run code formatters'
+	@echo 'lint         - run linters'