Implement further methods, pipelines, and datasets. (#4)

* change point detection module * release automation * debug docs * debug docs * change docs workflow * debug workflow * debug workflow * debug `python-package.yml` * debug release action * update project files * change list_datasets function * restructure docs * refactor datasets * implement more ood detection methods * remove cpd methods * implement imagenet corrupted datasets * debug ood pipeline * update package structure * update package structure * major refactoring * fix minor bugs * fix small bug * implement gradnorm * implement vim * update methods * refactor pipelines * debug datasets * accelerate compute accuracy script * multi-layer scores aggregations * debug ood detection methods * implemend `wilds` datasets * debug pipelines * implement layer aggregation methods * implement more detection methods * format code * list available resources * improve api * method templates * refactoring * major methods refactoring * aggregations api * debugging * debug docs * prepare new version * debug workflow * update tests * lint code * update requirements
edadaltocg · Apr 4, 2023 · 1c49512 · 1c49512
1 parent 682ef2f
commit 1c49512
Show file tree

Hide file tree

Showing 141 changed files with 209,413 additions and 2,582 deletions.
diff --git a/.gitattributes b/.gitattributes
@@ -1,4 +1,4 @@
-*.py	eol=lf
+*.py eol=lf
 *.rst	eol=lf
-*.md	eol=lf
-*.mdx   eol=lf
+*.md eol=lf
+*.mdx eol=lf
diff --git a/.github/workflows/python-package.yml b/.github/workflows/python-package.yml
@@ -5,13 +5,12 @@ name: Python package
 
 on:
   push:
-    branches: [ "master" ]
+    branches: ["master"]
   pull_request:
-    branches: [ "master" ]
+    branches: ["master"]
 
 jobs:
   build:
-
     runs-on: ${{ matrix.os }}
     strategy:
       fail-fast: false
@@ -20,27 +19,27 @@ jobs:
         python-version: ["3.8"]
 
     steps:
-    - uses: actions/checkout@v3
-    - name: Set up Python ${{ matrix.python-version }}
-      uses: actions/setup-python@v4
-      with:
-        python-version: ${{ matrix.python-version }}
-        cache: 'pip'
-        cache-dependency-path: setup.py
-    - name: Install dependencies
-      run: |
-        python -m pip install --upgrade pip setuptools wheel
-        python -m pip install -r requirements_dev.txt
-        python -m pip install -e .
-        if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
-    - name: Format and Lint
-      run: |
-        black --config pyproject.toml .
-        isort --settings pyproject.toml .
-        # stop the build if there are Python syntax errors or undefined names
-        flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
-        # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
-        flake8 . --count --exit-zero --max-complexity=10 --max-line-length=120 --statistics
-    - name: Test with pytest
-      run: |
-        pytest -vv tests/models.py
+      - uses: actions/checkout@v3
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v4
+        with:
+          python-version: ${{ matrix.python-version }}
+          cache: "pip"
+          cache-dependency-path: setup.py
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip setuptools wheel
+          python -m pip install -e ".[dev]"
+          if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
+      - name: Format and Lint
+        run: |
+          black --config pyproject.toml .
+          isort --settings pyproject.toml .
+          # stop the build if there are Python syntax errors or undefined names
+          flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
+          # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
+          flake8 . --count --exit-zero --max-complexity=10 --max-line-length=120 --statistics
+      - name: Test with pytest
+        run: |
+          pytest -vv tests/models.py -s
+          pytest -vv tests/methods.py -s
diff --git a/.github/workflows/python-publish.yml b/.github/workflows/python-publish.yml
@@ -17,27 +17,26 @@ permissions:
 
 jobs:
   deploy:
-
     runs-on: ubuntu-latest
 
     steps:
-    - name: Checkout
-      uses: actions/checkout@v3
-      with:
-        fetch-depth: 0
-    - name: Set up Python
-      uses: actions/setup-python@v3
-      with:
-        python-version: '3.x'
-    - name: Install dependencies
-      run: |
-        python -m pip install --upgrade pip
-        pip install build
-    - name: Build package
-      run: |
-        python -m build
-    - name: Publish package
-      uses: pypa/gh-action-pypi-publish@27b31702a0e7fc50959f5ad993c78deac1bdfc29
-      with:
-        user: __token__
-        password: ${{ secrets.PYPI_API_TOKEN }}
+      - name: Checkout
+        uses: actions/checkout@v3
+        with:
+          fetch-depth: 0
+      - name: Set up Python
+        uses: actions/setup-python@v3
+        with:
+          python-version: "3.x"
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install build
+      - name: Build package
+        run: |
+          python -m build
+      - name: Publish package
+        uses: pypa/gh-action-pypi-publish@27b31702a0e7fc50959f5ad993c78deac1bdfc29
+        with:
+          user: __token__
+          password: ${{ secrets.PYPI_API_TOKEN }}
diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
@@ -10,4 +10,4 @@ jobs:
       - name: Create new patch release
         run: python .github/scripts/release.py
         env:
-          GITHUB_TOKEN: ${{ secrets.PERSONAL_ACCESS_TOKEN }}
+          GITHUB_TOKEN: ${{ secrets.PERSONAL_ACCESS_TOKEN }}
diff --git a/.github/workflows/sphinx-docs.yml b/.github/workflows/sphinx-docs.yml
@@ -4,22 +4,21 @@ on:
   release:
     types: [published]
 
-
 jobs:
   build:
     runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@v3
-    - name: Build HTML
-      uses: ammaraskar/sphinx-action@master
-    - name: Upload artifacts
-      uses: actions/upload-artifact@v3
-      with:
-        name: html-docs
-        path: docs/build/html/
-    - name: Deploy
-      uses: peaceiris/actions-gh-pages@v3
-      if: github.ref == 'refs/heads/main'
-      with:
-        github_token: ${{ secrets.GITHUB_TOKEN }}
-        publish_dir: docs/build/html
+      - uses: actions/checkout@v3
+      - name: Build HTML
+        uses: ammaraskar/sphinx-action@master
+      - name: Upload artifacts
+        uses: actions/upload-artifact@v3
+        with:
+          name: html-docs
+          path: docs/build/html/
+      - name: Deploy
+        uses: peaceiris/actions-gh-pages@v3
+        if: github.ref == 'refs/heads/main'
+        with:
+          github_token: ${{ secrets.GITHUB_TOKEN }}
+          publish_dir: docs/build/html
diff --git a/.gitignore b/.gitignore
@@ -159,3 +159,6 @@ dmypy.json
 /scripts/bash
 
 .pypirc
+/images
+slurm/
+tmp/
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -0,0 +1,19 @@
+Check back later, we will populate this soon
+
+## Important Resources
+
+## Testing
+
+## Environment details
+
+## How to submit changes
+
+## How to report a bug
+
+## How to suggest a feature or enhancement
+
+## Coding conventions
+
+## Code of Conduct
+
+## Thanks
diff --git a/LICENSE b/LICENSE
@@ -1,21 +1,13 @@
-MIT License
+Copyright 2023 Eduardo Dadalto
 
-Copyright (c) 2022 Eduardo Dadalto
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
 
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
+    http://www.apache.org/licenses/LICENSE-2.0
 
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
diff --git a/README.md b/README.md
@@ -1,79 +1,128 @@
+# Detectors
+
 <p align="center">
     <br>
-    <img src="./face-with-monocle.svg" width="150" height="150" />
+    <img src="_static/face-with-monocle.svg" width="150" height="150" />
     <br>
 </p>
 
-# Detectors
-
-
 Package to accelerate research on out-of-distribution (OOD) detection inspired by Huggingface's transformers.
 
 Under development.
 
-## Deployment & Documentation & Stats & License
+## Stats
 
 [![Maintenance](https://img.shields.io/badge/Maintained%3F-yes-green.svg)](https://github.com/edadaltocg/detectors/graphs/commit-activity)
 [![PyPi version](https://badgen.net/pypi/v/pip/)](https://pypi.org/project/pip)
 [![build](https://github.com/edadaltocg/detectors/actions/workflows/python-package.yml/badge.svg)](https://github.com/edadaltocg/detectors/actions/workflows/python-package.yml)
 [![Documentation Status](https://readthedocs.org/projects/ansicolortags/badge/?version=latest)](http://ansicolortags.readthedocs.io/?badge=latest)
-[![GitHub stars](https://img.shields.io/github/stars/edadaltocg/detectors.svg?style=social&label=Star&maxAge=2592000)](https://GitHub.com/edadaltocg/detectors/stargazers/)
-[![GitHub forks](https://badgen.net/github/forks/edadaltocg/detectors/)](https://github.com/edadaltocg/detectors/network/)
 [![PyPI download month](https://img.shields.io/pypi/dm/ansicolortags.svg)](https://pypi.python.org/pypi/ansicolortags/)
 [![GitHub contributors](https://img.shields.io/github/contributors/Naereen/badges.svg)](https://GitHub.com/Naereen/badges/graphs/contributors/)
-[![DOI:10.1007/978-3-319-76207-4_15](https://zenodo.org/badge/DOI/10.1007/978-3-319-76207-4_15.svg)](https://doi.org/10.1007/978-3-319-76207-4_15)
+[![DOI:](https://zenodo.org/badge/DOI/.svg)](https://doi.org/)
 [![PyPi license](https://badgen.net/pypi/license/pip/)](https://pypi.org/project/pip/)
 
 -----
 
-<!--
-## Examples
-
-```python
-``` -->
-
 ## Features
 
-- Datasets with md5 checksums.
-- Models pre-trained on CIFAR and integrated in `timm`.
-- Pipeline for evaluating OOD detectors on CIFAR and ImageNet benchmarks.
-- Multiple seed simulations.
-- Diverse aggregation methods for OOD detection.
-- Efficient OOD detection metrics.
+- Pipelines for evaluating OOD detectors on MNIST, CIFAR, and ImageNet benchmarks.
+- Automatic OOD datasets download with md5 checksums.
+- Support models implemented at `timm`.
+- Models on CIFAR and integrated in `timm`.
+- Random seed support for reproducible simulations.
+- Several aggregation methods for multi-layer OOD detection.
+- OOD detection metrics.
+- More than 15 detection methods implemented.
+- Pipelines for open set recognition and covariate drift detection.
 
 ## Installation
 
 ```bash
 pip install detectors
 ```
 
-## Thanks to all our contributors
+To install the latest version from the source:
 
-<a href="https://github.com/edadaltocg/detectors/graphs/contributors">
-  <img src="https://contributors-img.web.app/image?repo=edadaltocg/detectors" />
-</a>
+```bash
+git clone https://github.com/edadaltocg/detectors.git
+cd detectors
+pip install -e .
+```
 
-### Contact
+## Examples
+
+### Running a benchmark
 
-Concerning this package, its use and bugs, use the [issue page](https://github.com/edadaltocg/detectors/issues) of the [ruptures repository](https://github.com/edadaltocg/detectors). For other inquiries, you can contact me [here](https://edadaltocg.github.io/contact/).
+```python
+```
 
+### Optional: Acceleration
 
-### Important links
+This package is compatible with the `accelerate` package to allow for parallel computing.
 
-- [Documentation]()
-- [Pypi package index](https://pypi.python.org/pypi/detectors)
+In order to use it, you need to install it:
 
-### Changelog
+```bash
+pip install accelerate
+```
 
-See the [changelog](https://github.com/edadaltocg/detectors/blob/master/CHANGELOG.md) for a history of notable changes to `detectors`.
+Then, you can configure it:
 
-<!-- ## TODO
+```bash
+accelerate config
+```
+
+And finally, you can run the benchmark with the `accelerate` command:
+
+```bash
+accelerate launch demo/ood_benchmark.py
+```
 
-- Pipeline for generating results table.
+#### Configuration example
+
+```text
+In which compute environment are you running? ([0] This machine, [1] AWS (Amazon SageMaker)): 0
+Which type of machine are you using? ([0] No distributed training, [1] multi-CPU, [2] multi-GPU, [3] TPU [4] MPS): 2
+How many different machines will you use (use more than 1 for multi-node training)? [1]: 1
+Do you want to use DeepSpeed? [yes/NO]: NO
+Do you want to use FullyShardedDataParallel? [yes/NO]: NO
+How many GPU(s) should be used for distributed training? [1]:2
+What GPU(s) (by id) should be used for training on this machine as a comma-seperated list? [all]:all
+Do you wish to use FP16 or BF16 (mixed precision)? [NO/fp16/bf16]: NO
+```
+
+## Changelog
+
+See the [changelog](https://github.com/edadaltocg/detectors/blob/master/CHANGELOG.md) for a history of notable changes to `detectors`.
 
 ## Contributing
 
+See the [contributing guidelines](https://github.com/edadaltocg/detectors/blob/master/CONTRIBUTING.md) for instructions on how to contribute to `detectors`.
+
+### Thanks to all our contributors
+
+<a href="https://github.com/edadaltocg/detectors/graphs/contributors">
+  <img src="https://contributors-img.web.app/image?repo=edadaltocg/detectors" />
+</a>
+
+### Contact
+
+Concerning this package, its use, and bugs, use the [issue page](https://github.com/edadaltocg/detectors/issues) of the [ruptures repository](https://github.com/edadaltocg/detectors). For other inquiries, you can contact me [here](https://edadaltocg.github.io/contact/).
+
+### Important links
+
+- [Documentation](http://detectors.readthedocs.io/)
+- [Pypi package index](https://pypi.python.org/pypi/detectors)
+
 ## Citing detectors
 
 ```bibtex
-``` -->
+@software{detectors,
+author = {Dadalto, Eduardo and Colombo, Pierre and Darrin, Maxime and Staerman, Guillaume and Nathan, Noiry and Alberge, Florence and Duhamel, Pierre and Piantanida, Pablo},
+month = {3},
+title = {{detectors: .}},
+url = {https://github.com/edadaltocg/detectors},
+version = {0.1.0},
+year = {2023}
+}
+```
diff --git a/face-with-monocle.png → _static/face-with-monocle.png b/face-with-monocle.png → _static/face-with-monocle.png
diff --git a/face-with-monocle.svg → _static/face-with-monocle.svg b/face-with-monocle.svg → _static/face-with-monocle.svg
diff --git a/docs/source/about.md b/docs/source/about.md
@@ -0,0 +1 @@
+# About
-Original file line number
+Diff line change
@@ Expand Up / @@ -159,3 +159,6 @@ dmypy.json @@
     /scripts/bash
     .pypirc
+    /images
+    slurm/
+    tmp/