Major cleanup

- Remove rules (rename Module into Pipeline) - completion is now using click - sequana_completion merged into sequana_pipetools standalone - added stats and config-to-schema application - added precommit
sequana · Dec 10, 2023 · 83c1b80 · 83c1b80
1 parent 0993333
commit 83c1b80
Show file tree

Hide file tree

Showing 23 changed files with 415 additions and 554 deletions.
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -0,0 +1,33 @@
+
+files: '\.(py|rst|sh)$'
+fail_fast: false
+
+repos:
+-   repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v3.2.0
+    hooks:
+    -   id: trailing-whitespace
+    -   id: end-of-file-fixer
+    -   id: check-yaml
+    #-   id: check-executables-have-shebangs
+    -   id: check-ast
+
+-   repo: https://github.com/pycqa/flake8
+    rev: 6.1.0
+    hooks:
+    -   id: flake8
+        args: ["-j8", "--ignore=E203,E501,W503,E722", "--max-line-length=120", "--exit-zero"]
+
+-   repo: https://github.com/psf/black
+    rev: 22.10.0
+    hooks:
+    -   id: black
+        args: ["--line-length=120"]
+        exclude: E501
+
+-   repo: https://github.com/pycqa/isort
+    rev: 5.12.0
+    hooks:
+      - id: isort
+        args: ["--profile", "black"] # solves conflicts between black and isort
+
diff --git a/README.rst b/README.rst
@@ -52,7 +52,7 @@ having 80% test coverage. Adding new pipelines also necessitated the addition of
     :scale: 40%
 
     **Figure 2** v0.8 of Sequana moved the Snakemake pipelines in indepdendent
-    repositories. A `cookie cutter <https://github.com/sequana/sequana_pipeline_template>`_ 
+    repositories. A `cookie cutter <https://github.com/sequana/sequana_pipeline_template>`_
     ease the creation of scuh pipelines
 
 
@@ -63,7 +63,7 @@ However, certain tools, such as those used for user interface and input data san
 
     **Figure 3** New Sequana framework. The library contains the core and
     bioinformatics tools and is now distinct from  the pipelines. Additionally, the
-    sequana_pipetools library supplies common tools to assist in the creaton and management of all pipelines, 
+    sequana_pipetools library supplies common tools to assist in the creaton and management of all pipelines,
     such as shared parser for options.
 
 Finally, we dropped the rules/ available in Sequana to build an independent package with a set of Snakemake
@@ -73,7 +73,7 @@ wrappers. These wrappers are available on https://github.com/sequana/sequana-wra
     :scale: 40%
 
     **Figure 3** New Sequana framework 2021. The library itself with the core, the
-    bioinformatics tools is now fully independent of the pipelines. 
+    bioinformatics tools is now fully independent of the pipelines.
 
 
 Installation
@@ -117,7 +117,7 @@ Will print a short summary report with common errors (if any).
 The library is intended to help Sequana developers to design their pipelines.
 See the `Sequana organization repository for examples <https://github.com/sequana>`_.
 
-In addition to those standalones, sequana_pipetools goal is to provide utilities to help Sequana developers. 
+In addition to those standalones, sequana_pipetools goal is to provide utilities to help Sequana developers.
 We currently provide a set of Options classes that should be used to
 design the API of your pipelines. For example, the
 sequana_pipetools.options.SlurmOptions can be used as follows inside a standard
@@ -155,7 +155,7 @@ Python module (the last two lines is where the magic happens)::
         # fills input_data, input_directory, input_readtag
         manager.fill_data_options()
 
-        # fill specific options. 
+        # fill specific options.
         # create a function for a given option (here --method)
         def fill_method():
             # any extra sanity checks
@@ -175,14 +175,14 @@ Python module (the last two lines is where the magic happens)::
 
     if __name__ == "__main__":
         main()
- 
+
 
 
 Developers should look at e.g. module sequana_pipetools.options
 for the API reference and one of the official sequana pipeline (e.g.,
 https://github.com/sequana/sequana_variant_calling) to get help from examples.
 
-The Options classes provided can be used and combined to design pipelines. 
+The Options classes provided can be used and combined to design pipelines.
 
 
 How to create skeleton of a Sequana pipeline with cookiecutter
@@ -267,7 +267,7 @@ for Apptainer setup.
 What is Sequana ?
 =================
 
-**Sequana** is a versatile tool that provides 
+**Sequana** is a versatile tool that provides
 
 #. A Python library dedicated to NGS analysis (e.g., tools to visualise standard NGS formats).
 #. A set of Pipelines dedicated to NGS in the form of Snakefiles
@@ -288,6 +288,14 @@ Changelog
 ========= ======================================================================
 Version   Description
 ========= ======================================================================
+0.16.5    * merge completion standalone into main sequana_pipetools application
+          * add application to create schema given a config file
+          * add application to get basic stats about the pipelines
+          * add precommit and applied black/isort on all files
+          * remove some useless code
+          * update completion to use click instead of argparse
+          * Rename Module into Pipeline (remove rules so Module are only made
+            of pipelines hence the renaming)
 0.16.4    * fix Trimming options (click) for the quality option
 0.16.3    * add class to handle multiplex entry for click.option (useful for
             multitax multiple databases)
@@ -298,7 +306,7 @@ Version   Description
           * --from-project not funtcional (example in multitax pipeline)
           * Click checks that input-directoyr is a directory indeed
 0.16.1    * Fix/rename error_report into onerror to be included in the Snakemake
-            onerror section. added 'slurm' in slurm output log file in the 
+            onerror section. added 'slurm' in slurm output log file in the
             profile
 0.16.0    * scripts now use click instead of argparse
           * All Options classes have now an equivalent using click.
@@ -313,17 +321,17 @@ Version   Description
           * sequana_slurm_status removed. Use manager.error_report in pipelines
             instead
 0.15.0    * remove useless code (readme, description) related to old rules
-          * requirements.txt renamed in tools.txt to store the required tools to 
-            run a pipeline. 
+          * requirements.txt renamed in tools.txt to store the required tools to
+            run a pipeline.
           * remove copy_requirements, not used in any pipelines (replaced by code
             in main.py of the pipelines)
-          * a utility function called getmetadata that returns dictionary 
+          * a utility function called getmetadata that returns dictionary
             with name, version, wrappers version)
 0.14.1    * remove a print statement
-0.14.0    * Module now returns the list of requirements. SequanaManager 
-            creates a txt file with all standalones from the requirements. 
+0.14.0    * Module now returns the list of requirements. SequanaManager
+            creates a txt file with all standalones from the requirements.
 0.13.0    * switch to pyproject and fixes #64
-0.12.5    * automatically populater 'wrappers' in PipelineManager' based on the 
+0.12.5    * automatically populater 'wrappers' in PipelineManager' based on the
             config entry 'sequana_wrappers'.
 0.12.4    * handles sequana pipeline with underscores (e.g. pacbio_qc)
 0.12.3    * fixes singularity-args in profile

diff --git a/pyproject.toml b/pyproject.toml
@@ -50,7 +50,6 @@ rich-click = "^1.7.1"
 
 
 [tool.poetry.scripts]
-sequana_completion = "sequana_pipetools.scripts.completion:main"
 sequana_pipetools = "sequana_pipetools.scripts.main:main"
 
 

diff --git a/sequana_pipetools/__init__.py b/sequana_pipetools/__init__.py
@@ -16,6 +16,10 @@
 logger = colorlog.getLogger(logger.name)
 
 from .misc import url2hash
-from .sequana_manager import SequanaManager #, get_pipeline_location
-from .snaketools import (Module, PipelineManager, PipelineManagerDirectory,
-                         SequanaConfig)
+from .sequana_manager import SequanaManager  # , get_pipeline_location
+from .snaketools import (
+    Pipeline,
+    PipelineManager,
+    PipelineManagerDirectory,
+    SequanaConfig,
+)
diff --git a/sequana_pipetools/options.py b/sequana_pipetools/options.py
@@ -18,6 +18,8 @@
 
 from deprecated import deprecated
 
+from sequana_pipetools.snaketools import Pipeline
+
 from .misc import print_version
 
 __all__ = [
@@ -38,7 +40,7 @@
     "before_pipeline",
     "init_click",
     "include_options_from",
-    "OptionEatAll"
+    "OptionEatAll",
 ]
 
 import rich_click as click
@@ -96,7 +98,7 @@ def decorator(f):
         caller_module = inspect.getmodule(f)
         if caller_module and "NAME" in caller_module.__dict__:
             NAME = caller_module.__dict__["NAME"]
-        else:
+        else:  # pragma: no cover
             print("You must define NAME as your pipeline name in the module main.py ")
             sys.exit(1)
 
@@ -115,7 +117,7 @@ def decorator(f):
 
 # This is a recipe from https://stackoverflow.com/questions/48391777/nargs-equivalent-for-options-in-click
 # to allow command line such as
-# sequana_multitax --databases 1 2 3 
+# sequana_multitax --databases 1 2 3
 class OptionEatAll(click.Option):
     def __init__(self, *args, **kwargs):
         self.save_other_options = kwargs.pop("save_other_options", True)
@@ -158,8 +160,6 @@ def parser_process(value, state):
         return retval
 
 
-
-
 @deprecated
 def before_pipeline(NAME):
     """A function to provide --version and --deps for all pipelines"""
@@ -171,9 +171,8 @@ def before_pipeline(NAME):
     if "--deps" in sys.argv:
         # Means than sequana must be installed, which we assume if someone uses
         # a pipeline. so must be here and not global import
-        from .snaketools import Module
 
-        module = Module("pipeline:" + NAME)
+        module = Pipeline(NAME)
         with open(str(module.requirements), "r") as fin:
             data = fin.read()
         print("Those software will be required for the pipeline to work correctly:\n{}".format(data))
@@ -184,7 +183,7 @@ class ClickGeneralOptions:
     group_name = "General"
     metadata = {
         "name": group_name,
-        "options": ["--deps", "--from-project",  "--help", "--level", "--version"],
+        "options": ["--deps", "--from-project", "--help", "--level", "--version"],
     }
 
     def __init__(self, caller=None):
@@ -240,9 +239,7 @@ def deps_callback(ctx, param, value):
         if not value:
             return
 
-        from .snaketools import Module
-
-        module = Module("pipeline:" + ctx.NAME)
+        module = Pipeline(ctx.NAME)
         with open(str(module.requirements), "r") as fin:
             data = fin.read()
         data = data.split()
@@ -502,7 +499,6 @@ def __init__(self, input_directory=".", input_pattern="*fastq.gz", add_input_rea
             )
 
 
-
 class InputOptions:
     def __init__(self, group_name="data", input_directory=".", input_pattern="*fastq.gz", add_input_readtag=True):
         """
@@ -805,7 +801,7 @@ class CutadaptOptions:  # pragma: no cover
     This section allows you to trim bases (--cutadapt-quality) with poor
     quality and/or remove adapters.
 
-    To remove adapters, you can provide the adapters directly as a 
+    To remove adapters, you can provide the adapters directly as a
     string (or a file) using  --cutadapt-fwd (AND --cutadapt-rev" for paired-end data).
 
     If you set the --cutadapt-adapter-choice to 'none', fwd and reverse

diff --git a/sequana_pipetools/resources/ext.py b/sequana_pipetools/resources/ext.py
@@ -3,11 +3,10 @@
 """
 from pykwalify.errors import SchemaError
 
-
 STRING_TYPE = {"text", "str"}
 
 
-def check_optionnal(value, rule_obj):
+def check_optional(value, rule_obj):
     for key, rule in rule_obj.mapping.items():
         if rule.type in STRING_TYPE and rule.required and not rule.nullable:
             try:
@@ -18,10 +17,10 @@ def check_optionnal(value, rule_obj):
             yield key, True
 
 
-def ext_map_optionnal(value, rule_obj, path):
+def ext_map_optional(value, rule_obj, path):
     todo = value.get("do")
     if todo:
-        for key, value_check in check_optionnal(value, rule_obj):
+        for key, value_check in check_optional(value, rule_obj):
             if not value_check:
                 raise SchemaError(f"Mandatory value is missing at {path}/{key}")
     return True
Original file line number	Diff line number	Diff line change
Expand Up		@@ -50,7 +50,6 @@ rich-click = "^1.7.1"


		[tool.poetry.scripts]
		sequana_completion = "sequana_pipetools.scripts.completion:main"
		sequana_pipetools = "sequana_pipetools.scripts.main:main"


Expand Down