Skip to content

Commit b110cfa

Browse files
ShettlandShettland
authored
Merge pull request #298 from BU-ISCIII/develop
Release 1.0.0
2 parents 1794568 + 3324f39 commit b110cfa

File tree

180 files changed

+51217
-2297
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

180 files changed

+51217
-2297
lines changed

‎.github/CONTRIBUTING.md

+72
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
# relecov-tools: Contributing Guidelines
2+
3+
## Contribution workflow
4+
5+
If you'd like to write or modify some code for relecov-tools, the standard workflow is as follows:
6+
7+
1. Check that there isn't already an issue about your idea in the [relecov-tools issues](https://github.com/BU-ISCIII/relecov-tools/issues) to avoid duplicating work. **If there isn't one already, please create one so that others know you're working on this**.
8+
2. [Fork](https://help.github.com/en/github/getting-started-with-github/fork-a-repo) the [relecov-tools repository](https://github.com/BU-ISCIII/relecov-tools/) to your GitHub account.
9+
3. Make the necessary changes / additions within your forked repository following the [code style guidelines](#code-style-guidelines).
10+
4. Modify the [`CHANGELOG`](../CHANGELOG.md) file according to your changes in the appropiate section ([X.X.Xdev]), you should register your changes regarding:
11+
1. Added enhancements
12+
2. New modules
13+
3. Fixes
14+
4. Removed stuff
15+
5. Requirements added or version update
16+
5. Update any documentation as needed.
17+
6. [Submit a Pull Request](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request) against the `develop` branch and send the url to the #pipelines-dev channel in slack (if you are not in the slack channel just wait fot the PR to be reviewed and rebased).
18+
19+
If you're not used to this workflow with git, you can start with:
20+
21+
- Some [docs in the bu-isciii wiki](https://github.com/BU-ISCIII/BU-ISCIII/wiki/Github--gitflow).
22+
- [some slides](https://docs.google.com/presentation/d/1PruqGxPQVxtNcuEbOd86mylXorgYIU5a/edit?pli=1#slide=id.p1) (in spanish).
23+
- some github generic docs [docs from GitHub](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests).
24+
- even their [excellent `git` resources](https://try.github.io/).
25+
26+
### relecov-tools repo branches
27+
28+
relecov-tools repo works with a two branching scheme: `main` and `develop`.
29+
30+
- `main`: stable code only for releases.
31+
- `develop`: new code development for the different modules.
32+
33+
You need to submit your PR always against `develop`. Once approbed, this changes must be **`rebased`** so we do not create empty unwanted merges.
34+
35+
## Tests
36+
37+
When you create a pull request with changes, [GitHub Actions](https://github.com/features/actions) will run automatic tests.
38+
Typically, pull-requests are only fully reviewed when these tests are passing, though of course we can help out before then.
39+
40+
There are typically two types of tests that run:
41+
42+
### Lint tests
43+
44+
We use black and flake8 linting based on PEP8 guidelines for python coding. You can check more information [here](https://github.com/BU-ISCIII/BU-ISCIII/wiki/Python#linting).
45+
46+
### Code tests
47+
48+
Download, read-lab-metadata, map and validate modules are executed using a test dataset.
49+
50+
Anyhow you should always submit locally tested code!!
51+
52+
### New version bumping and release
53+
54+
In order to create a new release you need to follow the next steps:
55+
56+
1. Set the new version according to [semantic versioning](https://semver.org/), in our particular case, changes in the `hotfix` branch will change the PATCH version (third one), and changes in develop will typicaly change the MINOR version, unless the developing team decides otherwise.
57+
2. Create a PR bumping the new version against `develop`. For bumping a new version just change [this line](https://github.com/BU-ISCIII/relecov-tools/blob/09c00c1ddd11f7489de7757841aff506ef4b7e1d/setup.py#L5) with the new version.
58+
3. Once that PR is merged, create via web another PR against `main` (origin `develop`). This PR would need 2 approvals.
59+
4. [Create a new release](https://docs.github.com/en/repositories/releasing-projects-on-github/managing-releases-in-a-repository) copying the appropiate notes from the `CHANGELOG`.
60+
5. Once the release is approved and merged, you're all set!
61+
62+
PRs from one branch to another, like in a release should be **`merged`** not rebased, so we avoid conflicts and the branch merge is correctly visualize in the commits history.
63+
64+
### Code style guidelines
65+
66+
We follow PEP8 conventions as code style guidelines, please check [here](https://github.com/BU-ISCIII/BU-ISCIII/wiki/Python#pep-8-guidelines-read-the-full-pep-8-documentation) for more detail.
67+
68+
When developing new code, we strongly recommend to implement LogSum() functions from log_summary.py instead of the classic python logging in order to keep track of all the warnings and errors that may appear during any of the processes.
69+
70+
## Getting help
71+
72+
For further information/help, please ask on the `#pipelines-dev` slack channel or write us an email! ([[email protected]](emailto:[email protected])).

‎.github/workflows/pypi_publish.yml

+88
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
name: Publish package python distribution to Pypi
2+
3+
on:
4+
push:
5+
branches: "main"
6+
7+
jobs:
8+
build:
9+
name: Build distribution
10+
runs-on: ubuntu-latest
11+
steps:
12+
- uses: actions/checkout@v4
13+
- name: Set up Python
14+
uses: actions/setup-python@v5
15+
with:
16+
python-version: "3.9.16"
17+
- name: Install pypi/build
18+
run: >-
19+
python3 -m
20+
pip install
21+
build
22+
--user
23+
- name: Build a binary wheel and a source tarball
24+
run: python3 -m build
25+
- name: Store the distribution packages
26+
uses: actions/upload-artifact@v3
27+
with:
28+
name: python-package-distributions
29+
path: dist/
30+
31+
publish-to-pypi:
32+
name: Publish dist to PyPI
33+
if: startsWith(github.ref, 'refs/tags/') # only publish to PyPI on tag pushes
34+
needs:
35+
- build
36+
runs-on: ubuntu-latest
37+
environment:
38+
name: pypi
39+
url: https://pypi.org/p/relecov-tools
40+
permissions:
41+
id-token: write
42+
steps:
43+
- name: Download all the dists
44+
uses: actions/download-artifact@v3
45+
with:
46+
name: python-package-distributions
47+
path: dist/
48+
- name: Publish to PyPI
49+
uses: pypa/gh-action-pypi-publish@release/v1
50+
51+
github-release:
52+
name: Sign dist with Sigstore and upload to GitHub Release
53+
needs:
54+
- publish-to-pypi
55+
runs-on: ubuntu-latest
56+
permissions:
57+
contents: write
58+
id-token: write
59+
steps:
60+
- name: Download all the dists
61+
uses: actions/download-artifact@v3
62+
with:
63+
name: python-package-distributions
64+
path: dist/
65+
- name: Sign the dists with Sigstore
66+
uses: sigstore/[email protected]
67+
with:
68+
inputs: >-
69+
./dist/*.tar.gz
70+
./dist/*.whl
71+
- name: Create GitHub Release
72+
env:
73+
GITHUB_TOKEN: ${{ github.token }}
74+
run: >-
75+
gh release create
76+
'${{ github.ref_name }}'
77+
--repo '${{ github.repository }}'
78+
--notes ""
79+
- name: Upload artifact signatures to GitHub Release
80+
env:
81+
GITHUB_TOKEN: ${{ github.token }}
82+
# Upload to GitHub Release using the `gh` CLI.
83+
# `dist/` contains the built packages, and the
84+
# sigstore-produced signatures and certificates.
85+
run: >-
86+
gh release upload
87+
'${{ github.ref_name }}' dist/**
88+
--repo '${{ github.repository }}'

‎.github/workflows/python_lint.yml

+35
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
name: python_lint
2+
3+
on:
4+
push:
5+
paths:
6+
- '**.py'
7+
pull_request:
8+
paths:
9+
- '**.py'
10+
11+
jobs:
12+
flake8_py3:
13+
runs-on: ubuntu-latest
14+
steps:
15+
- name: Setup Python
16+
uses: actions/setup-python@v1
17+
with:
18+
python-version: 3.9.x
19+
architecture: x64
20+
- name: Checkout PyTorch
21+
uses: actions/checkout@master
22+
- name: Install flake8
23+
run: pip install flake8
24+
- name: Run flake8
25+
run: flake8 --ignore E501,W503,E203,W605
26+
27+
black_lint:
28+
runs-on: ubuntu-latest
29+
steps:
30+
- name: Setup
31+
uses: actions/checkout@v2
32+
- name: Install black in jupyter
33+
run: pip install black[jupyter]
34+
- name: Check code lints with Black
35+
uses: psf/black@stable

‎.github/workflows/test_modules.yml

+79
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
name: test_modules
2+
3+
on:
4+
push:
5+
branches: "**"
6+
pull_request:
7+
types: [opened, reopened, synchronize, closed]
8+
branches: "**"
9+
10+
jobs:
11+
test_map:
12+
runs-on: ubuntu-latest
13+
strategy:
14+
max-parallel: 2
15+
matrix:
16+
map_args: ["-d 'ENA' -f relecov_tools/schema/ena_schema.json", "-d 'GISAID' -f relecov_tools/schema/gisaid_schema.json"]
17+
steps:
18+
- name: Set up Python 3.9.16
19+
uses: actions/setup-python@v3
20+
with:
21+
python-version: '3.9.16'
22+
- name: Checkout code
23+
uses: actions/checkout@v3
24+
with:
25+
ref: ${{ github.event.pull_request.head.sha }}
26+
fetch-depth: 0
27+
- name: Install package and dependencies
28+
run: |
29+
pip install -r requirements.txt
30+
pip install .
31+
- name: Run each module tests
32+
run: |
33+
relecov-tools map -j tests/data/map_validate/processed_metadata_lab_test.json -p relecov_tools/schema/relecov_schema.json ${{ matrix.map_args }} -o .
34+
env:
35+
OUTPUT_LOCATION: ${{ github.workspace }}/tests/
36+
- name: Upload output file
37+
uses: actions/upload-artifact@v2
38+
with:
39+
name: test-output
40+
path: output.txt
41+
42+
test_all_modules:
43+
runs-on: ubuntu-latest
44+
strategy:
45+
max-parallel: 3
46+
matrix:
47+
modules: ["read-lab-metadata", "read-bioinfo-metadata", "validate"]
48+
steps:
49+
50+
- name: Set up Python 3.9.16
51+
uses: actions/setup-python@v3
52+
with:
53+
python-version: '3.9.16'
54+
55+
- name: Checkout code
56+
uses: actions/checkout@v3
57+
with:
58+
ref: ${{ github.event.pull_request.head.sha }}
59+
fetch-depth: 0
60+
61+
- name: Install package and dependencies
62+
run: |
63+
pip install -r requirements.txt
64+
pip install .
65+
66+
- name: Run each module tests
67+
run: |
68+
if [ "${{ matrix.modules }}" == "read-lab-metadata" ]; then
69+
relecov-tools read-lab-metadata -m tests/data/read_lab_metadata/metadata_lab_test.xlsx -s tests/data/read_lab_metadata/samples_data_test.json -o $OUTPUT_LOCATION
70+
elif [ "${{ matrix.modules }}" == "validate" ]; then
71+
relecov-tools validate -j tests/data/map_validate/processed_metadata_lab_test.json -s relecov_tools/schema/relecov_schema.json -m tests/data/map_validate/metadata_lab_test.xlsx -o $OUTPUT_LOCATION
72+
fi
73+
env:
74+
OUTPUT_LOCATION: ${{ github.workspace }}/tests/
75+
- name: Upload output file
76+
uses: actions/upload-artifact@v2
77+
with:
78+
name: test-output
79+
path: output.txt

‎.github/workflows/test_sftp_handle.yml

+65
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
name: test_sftp_handle
2+
3+
on:
4+
push:
5+
branches: "**"
6+
pull_request_target:
7+
types: [opened, reopened, synchronize, closed]
8+
branches: "**"
9+
10+
jobs:
11+
security_check:
12+
runs-on: ubuntu-latest
13+
steps:
14+
- name: Get User Permission
15+
id: checkAccess
16+
uses: actions-cool/check-user-permission@v2
17+
with:
18+
require: write
19+
username: ${{ github.triggering_actor }}
20+
- name: Check User Permission
21+
if: steps.checkAccess.outputs.require-result == 'false'
22+
run: |
23+
echo "${{ github.triggering_actor }} does not have permissions on this repo."
24+
echo "Current permission level is ${{ steps.checkAccess.outputs.user-permission }}"
25+
echo "Job originally triggered by ${{ github.actor }}"
26+
exit 1
27+
28+
test_sftp_handle:
29+
needs: security_check
30+
concurrency:
31+
group: ${{ github.repository }}-test_sftp_handle
32+
cancel-in-progress: false
33+
runs-on: ubuntu-latest
34+
strategy:
35+
max-parallel: 1
36+
matrix:
37+
download_options: ["download_only", "download_clean", "delete_only"]
38+
target_folders: ["", "-t COD-test-1"]
39+
40+
steps:
41+
- name: Set up Python 3.9.16
42+
uses: actions/setup-python@v3
43+
with:
44+
python-version: '3.9.16'
45+
46+
- name: Checkout code
47+
uses: actions/checkout@v3
48+
with:
49+
ref: ${{ github.event.pull_request.head.sha }}
50+
fetch-depth: 0
51+
52+
- name: Install package and dependencies
53+
run: |
54+
pip install -r requirements.txt
55+
pip install .
56+
57+
- name: Run sftp_handle tests
58+
run: |
59+
python3 tests/test_sftp_handle.py --download_option ${{ matrix.download_options }} ${{ matrix.target_folders }}
60+
env:
61+
TEST_USER: ${{ secrets.TEST_USER }}
62+
TEST_PASSWORD: ${{ secrets.TEST_PASSWORD }}
63+
TEST_PORT: ${{ secrets.TEST_PORT }}
64+
OUTPUT_LOCATION: ${{ github.workspace }}/tests/
65+

‎.gitignore

+3
Original file line numberDiff line numberDiff line change
@@ -127,3 +127,6 @@ dmypy.json
127127

128128
# Pyre type checker
129129
.pyre/
130+
131+
# (Ad-hoc files)
132+
.nextflow*

‎CHANGELOG.md

+38
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
# relecov-tools Changelog
2+
3+
All notable changes to this project will be documented in this file.
4+
5+
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
6+
7+
## [1.1.0dev] - 2024-0X-0X : https://github.com/BU-ISCIII/relecov-tools/releases/tag/1.1.X
8+
9+
### Credits
10+
11+
Code contributions to the hotfix:
12+
13+
### Modules
14+
15+
#### Added enhancements
16+
17+
#### Fixes
18+
19+
#### Changed
20+
21+
#### Removed
22+
23+
### Requirements
24+
25+
## [1.0.0] - 2024-0X-0X : https://github.com/BU-ISCIII/relecov-tools/releases/tag/1.0.0
26+
27+
### Credits
28+
29+
Code contributions to the inital release:
30+
31+
- [Sara Monzón](https://github.com/saramonzon)
32+
- [Sarai Varona](https://github.com/svarona)
33+
- [Guillermo Gorines](https://github.com/GuilleGorines)
34+
- [Pablo Mata](https://github.com/Shettland)
35+
- [Luis Chapado](https://github.com/luissian)
36+
- [Erika Kvalem](https://github.com/ErikaKvalem)
37+
- [Alberto Lema](https://github.com/Alema91)
38+
- [Daniel Valle](https://github.com/Daniel-VM)

‎MANIFEST.in

+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
include LICENSE
2+
include README.md
3+
graft relecov_tools/assets
4+
graft relecov_tools/schema
5+
graft relecov_tools/conf
6+
include requirements.txt

0 commit comments

Comments
 (0)