v0.2.4 (#44)

* [viewer] : Fix saving erorr when bad_segments is None.

* [doc] : Fix link to Python functions

* docstring

* docstring

* docstring

* [viewer] Refactor Viewer and Editor in a way that allows the Editor to be used as a standalone.

* [viwer] : Working directly from the BIDS folder

* [docs] More doc on the Editor and Viewer

* [docs]

* [docs]

* v0.2.4

* logo

* logo

Author: Nicolas Legrand

.pre-commit-config.yaml

@@ -5,7 +5,7 @@ repos:
     - id: isort
       files: ^systole/
 -   repo: https://github.com/ambv/black
-    rev: 22.6.0
+    rev: 22.8.0
     hooks:
     - id: black
       language_version: python3

docs/source/changelog.rst

@@ -24,6 +24,8 @@ This page contains information about what has changed in each new version of ``s
 
    <div class="col-md-9">
 
+.. include:: releases/v0.2.4.txt
+
 .. include:: releases/v0.2.3.txt
 
 .. include:: releases/v0.2.2.txt

docs/source/conf.py

@@ -108,21 +108,18 @@
             icon="fas fa-box",
         ),
     ],
-    "logo_link": "https://embodied-computation-group.github.io/systole/#",
-}
+    "logo": {
+        "text": "Systole",
+    },}
 
-html_sidebars = {
-  "api": [],
-  "changelog": [],
-  "notebooks/*": []
-}
+html_sidebars = {"**": []}
 
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
 html_static_path = ['_static']
 html_css_files = ["https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/css/font-awesome.min.css"]
-html_logo = "images/logo.svg"
+html_logo = "images/logo_small.svg"
 html_favicon = "images/logo_small.svg"
 
 def setup(app):

docs/source/notebooks/3-DetectingAndCorrectingArtefacts.ipynb

@@ -1161,7 +1161,7 @@
    "id": "dd627eb6-aa7f-4d66-a763-594d4a03611d",
    "metadata": {},
    "source": [
-    "Because most heart rate variability metrics require continuous interval time series, we cannot simply remove the invalid intervals and concatenate everything, we have to extract valid segments and compute the HRD indices from there. Systole comes with the py:func`systole.utils.get_valid_segments()` function that will return the valid portion of the physiological recording, sorted according to the signal length."
+    "Because most heart rate variability metrics require continuous interval time series, we cannot simply remove the invalid intervals and concatenate everything, we have to extract valid segments and compute the HRD indices from there. Systole comes with the:py:func`systole.utils.get_valid_segments()` function that will return the valid portion of the physiological recording, sorted according to the signal length."
    ]
   },
   {

docs/source/notebooks/3-DetectingAndCorrectingArtefacts.md

@@ -272,7 +272,7 @@ plot_raw(signal=signal, show_heart_rate=True, figsize=(18, 6), bad_segments=[(60
 plt.tight_layout()
 ```
 
-Because most heart rate variability metrics require continuous interval time series, we cannot simply remove the invalid intervals and concatenate everything, we have to extract valid segments and compute the HRD indices from there. Systole comes with the py:func`systole.utils.get_valid_segments()` function that will return the valid portion of the physiological recording, sorted according to the signal length.
+Because most heart rate variability metrics require continuous interval time series, we cannot simply remove the invalid intervals and concatenate everything, we have to extract valid segments and compute the HRD indices from there. Systole comes with the:py:func`systole.utils.get_valid_segments()` function that will return the valid portion of the physiological recording, sorted according to the signal length.
 
 ```{code-cell} ipython3
 from systole.utils import get_valid_segments

docs/source/notebooks/6-WorkingWithBIDSFolders.ipynb

@@ -25,11 +25,11 @@ if 'google.colab' in sys.modules:
     ! pip install systole
 ```
 
-Starting in version `0.2.3`, Systole provides tools to interact efficiently with large datasets containing physiological recordings. Most of the functionalities interface with folders structured following the [BIDS standards](https://bids-specification.readthedocs.io/en/stable/) and this is the format we recommend using if you are following this tutorial.
+Starting in version `0.2.3`, Systole provides tools to interact with large datasets of physiological recordings. The functionalities interface with folders that are structured following the [BIDS standards](https://bids-specification.readthedocs.io/en/stable/) and this is the format we recommend using if you are following this tutorial.
 
-Under BIDS standards, physiological recordings, sometimes associated with behavioural tasks or neural recordings, are stored with a filename ending with `*_physio.tsv.gz` and are always accompanied with sidecar a `*_physio.json` file containing metadata like the recording modality or the sampling frequency. Accessing both the times series and its accompanying metadata will help Systole automate the preprocessing by finding the correct parameters for peaks detection and reports.
+Following the BIDS specifications, physiological recordings, sometimes associated with behavioural tasks or neural recordings, are stored with a filename ending with `*_physio.tsv.gz` and are always accompanied with sidecar a `*_physio.json` file containing metadata like the recording modality or the sampling frequency. Accessing both the times series and its accompanying metadata will help Systole automate the preprocessing by finding the correct parameters for peaks detection and reports.
 
-Once you have organized your folder, you should have a structure resembling this one:
+A valid BIDS folder should be structured like the following:
 
 ```
 └─ BIDS/
@@ -46,17 +46,22 @@ Once you have organized your folder, you should have a structure resembling this
 
 Here, we have physiological recordings associated with a behavioural task for `n` participants in the folder.
 
+```{tip}
+We recommend using tools like [BIDS validator](https://bids-standard.github.io/bids-validator/) to ensure that your folder complies with BIDS specification before trying to preprocess your data, or to use the editor.
+```
+
 +++
 
-## Signal preprocessing and creation of subject and group-level reports
+(preprocessing)=
+## Preprocessing
 
 +++
 
 The first step will be to preprocess the raw data and store the signal and peaks detection in a new derivative folder. During this step, we can also decide to create HTML reports for each participants, so we can visualize the signal quality and peaks detection.
 
-### Preprocessing the physiological recording from one participant
+### Preprocessing the physiological data from one participant
 
-The py:func:`systole.reports` sub-module contains tools to directly interact with BIDS formatted folders, preprocess and save individual reports in a BIDS consistent way. Those functionalities are built on the top of the py:func:`systole.reports.subject_level_report` function. This function will simply take a signal as input and will save as output the preprocessed signal with peaks detection (`_physio.tsv.gz` with the `_physio.json`), an `.html` reports adapted to the kind of signal that was provided, and a `features.tsv` file containing heart rate or respiratory rate variability features.
+The :py:func:`systole.reports` sub-module contains tools to directly interact with BIDS formatted folders, preprocess and save individual reports in a BIDS consistent way. Those functionalities are built on the top of the:py:func:`systole.reports.subject_level_report` function. This function will simply take a signal as input and will save as output the preprocessed signal with peaks detection (`_physio.tsv.gz` with the `_physio.json`), an `.html` reports adapted to the kind of signal that was provided, and a `features.tsv` file containing heart rate or respiratory rate variability features.
 
 For example, running the following code:
 
@@ -88,7 +93,7 @@ will save these four new files in the file folder.
 
 ### Preprocessing the entire BIDS folder
 
-The previous function call can be automated for each participant and each file of a given BIDS folder and to extract the physiological features using the information provided in the `json` metadata automatically. This can be done using the py:func:`systole.reports.wrapper` function, or directly from the command line. For example, the following command:
+The previous function call can be automated for each participant and each file of a given BIDS folder and to extract the physiological features using the information provided in the `json` metadata automatically. This can be done using the:py:func:`systole.reports.wrapper` function, or directly from the command line. For example, the following command:
 
 ```bash
 systole --bids_folder="/path/to/BIDS/folder/" \
@@ -99,7 +104,31 @@ systole --bids_folder="/path/to/BIDS/folder/" \
         --html_reports==False
 ```
 
-will preprocess the data for all participants with a physiological recording in the session `ses-session1` (default), for the behavioural modality (`beh`) and the task `mytask`. We set `n_jobs=10`, meaning that we will run 40 processes in parallel, and `overwrite=True` to overwrite previous data with the same ID in the derivative folder. Note that we also set `html_reports` to `False` as these files can be quite large, it is often preferable to only create it for the participant we want to review, or to use the {ref}`viewer`. 
+will preprocess the data for all participants with a physiological recording in the session `ses-session1` (default), for the behavioural modality (`beh`) and the task `mytask`. We set `n_jobs=10`, meaning that we will run 40 processes in parallel, and `overwrite=True` to overwrite previous data with the same ID in the derivative folder. Note that we also set `html_reports` to `False` as these files can be quite large, it is often preferable to only create it for the participant we want to review, or to use the {ref}`viewer`. The possible arguments are:
+
+```{list-table} Command line arguments
+:header-rows: 1
+:name: label-to-reference
+
+* - Argument
+  - Description
+* - --bids_folder (-i)
+  - Path to the BIDS folder containing the physiological recordings.
+* - --participant_id (-p)
+  - The id of the participant that should be preprocessed. If this argument is not provided, all the participants will be preprocessed.
+* - --patern (-t)
+  - Only the files that contains the pattern string will be preprocessed. If the number of files matching is not exactly 1, the files are not processed.
+* - --html_reports (-r)
+  - Create subject-level HTML reports if `True`.
+* - --result_folder (-o)
+  - Path to the result folder. If not provided, the default will be ./derivatives/systole/.
+* - --n_jobs (-n)
+  - The number of jobs to run concurrently.
+* - --modality (-d)
+  - The modality of the recording (i.e. `"beh"`, `"func"`...).
+* - --overwrite (-w)
+  - If `True`, overwrite preexisting files in the result folder (DOES NOT include the corrected files).
+```
 
 +++
 
@@ -136,59 +165,124 @@ Once the preprocessing is completed, and if you did not asked for an external re
 +++
 
 (viewer)=
-## Manual edition of peaks vector and labelling bad segments using the Viewer
+## Manual edition of peaks vector and bad segments labelling
 
 While we hope that the peaks detection function used by [Systole](https://embodied-computation-group.github.io/systole/#) is sufficiently robust to extract peak vectors without errors for most of the uses cases, you might still encounter noisy or invalid recording that you will want to manually inspect and sometimes edit.
 
-The py:func:`systole.viewer` sub-module is built on the top of Matplotlib widgets and can help for manual peaks edition or bad segments labelling. For example, running the following cells in a Jupyter notebook:
+The :py:mod:`systole.interact` sub-module provides two classes (:py:class:`systole.interact.Editor` and :py:class:`systole.interact.Viewer`) built on the top of Matplotlib widgets that can help for manual edition, and interactive visualization of BIDS fodlers directly in the notebook.
+
+### Using the Editor to inspect raw signal
+
+The :py:mod:`systole.interact.Editor` can be use alone (apart from a BISD structured folder) to edit peaks detection from a raw ECG, PPG or respiratory signal.
 
 ```python
+from systole import import_dataset1
+from systole.interact import Viewer, Editor
 from IPython.display import display
-from systole.viewer import Viewer
 
 %matplotlib ipympl
 ```
 
 ```python
-view = Viewer(
+# Load a ray ECG time series
+ecg = import_dataset1(modalities=['ECG'], disable=True).ecg.to_numpy()
+```
+
+```python
+editor = Editor(
+    signal=ecg,
+    sfreq=1000,
+    corrected_json="./corrected.json",
     figsize=(15, 5),
-    input_folder="/BIDS_folder/derivatives/systole/",
-    pattern="task-hrd", # A string long enough to disambiguate in case of mmultiple recordings
-    modality="beh",
     signal_type="ECG"
 )
+display(editor.commands_box)
 ```
 
-```python
-display(view.io_box, view.commands_box, view.output)
++++
+
+```{note}
+Note that we are using the package [ipympl](https://matplotlib.org/ipympl/), and activating it using the magic cell `%matplotlib ipympl` so we can render Matplotlib interactive widgets in the Notebook. If you are working in another IDE, you can also render the same windows using a different backend like PyQt.
 ```
 
-will create an interactive windows from which all the preprocessed recordings and peaks detection can be inspected.
+This windows will automatically apply peaks detection given the `signal_type` parameter, and plot the raw signal with the instantaneous heart / respiration rate to check for artefacts. The class embed a `command_box` that can be used for edition. 
 
-<p align='center'><img src='https://github.com/embodied-computation-group/systole/raw/dev/docs/source/images/editor.gif'/></p>
+* When using the **Correction** mode:
+    * Use the *left* mouse button to select segment where all the peaks should be removed.
+    * Use the *right* mouse button to select segment where peak will be added at the local maximum.
+    <p align='center'><img src='https://github.com/embodied-computation-group/systole/raw/dev/docs/source/images/peaks.gif'/></p>
+
+* When using the **Rejection** mode:
+    * Use the *right* mouse button to select a segment that should be marked as bad.
+    <p align='center'><img src='https://github.com/embodied-computation-group/systole/raw/dev/docs/source/images/segments.gif'/></p>
+
+* By deselecting the check box, you can mark the entire signal as **invalid**.
+
+* Once that the signal has been edited, you can **save** the modification using the `Save modification` button, or directly use the method from the class.
 
 +++
 
-### Inserting / deleting peaks
+This function will create a JSON file (using the path specified in the `corrected_json` parameter) with all the information about bad segments labelling, peaks deletion and peaks insertion. The JSON file contains the following entries for each modality (ECG, PPG and respiration)
+
+* `valid` : is the recording valid or should it be discared (`True` unless otherwise stated).
+* `corrected_peaks` : the peaks indexes after correction.
+* `bad_segments` : a list of `start` and `end` indexed of bad segments.
 
-Peaks can be inserted (using the local maxima of the selected range) or deleted.
++++ {"tags": []}
 
-* Left mouse button : remove all the peaks in the selected interval.
-* Right mouse button : add one new peaks where the signal local maximum is found.
+### Using the Viewer to navigate preprocessed folder
 
-<p align='center'><img src='https://github.com/embodied-computation-group/systole/raw/dev/docs/source/images/editor_peaks.gif'/></p>
+The :py:mod:`systole.interact.Viewer` class wrap the Editor and allows to easily navigate and edit folder that contains large number of recoring. You can for example simply read the results generated by the command line (see {ref}`preprocessing`). Considering that the files were create in the folder `"/path/to/BIDS/folder/derivatives/systole/"` (which is the default behavior if `--result _folder is not provided`), the Viewer can be called using:
+
+```python
+from IPython.display import display
+from systole.interact import Viewer
+
+%matplotlib ipympl
+```
+
+```python
+view = Viewer(
+    figsize=(15, 5),
+    preprocessed_folder="/path/to/BIDS/folder/derivatives/systole/",
+    pattern="task-my_task", # A string long enough to disambiguate in case of mmultiple recordings
+    modality="beh",
+    signal_type="ECG"
+)
+```
+
+```python
+display(view.io_box, view.editor.commands_box, view.output)
+```
+
+This will create an interactive windows where all the preprocessed ECG recordings from the behavioral task `my_task` can be inspected and further edited.
+
+<p align='center'><img src='https://github.com/embodied-computation-group/systole/raw/dev/docs/source/images/editor.gif'/></p>
+
+```{note}
+If the signal was previously edited, the Viewer will automatically load the edited version and display bad segment (if any).
+```
 
 +++
 
-### Bad segments labelling
+### Using the Viewer to navigate BIDS folder
+
+Using the same logic, the :py:mod:`systole.interact.Viewer` can also work with the raw BIDS folder, instead of the derivatives, and preprocess data on the fly. This mode is more appropriate if you want to quickly inspect the data and do not want to generate subject or group level reports. The only different is that the input path should be parameter using the `bids_folder` argument, instead of `preprocessed_folder`. This will make the viewer aware that the signal are located in this folder, but that previously edited signal might also be located in `./derivatives/systole/corrected/`.
 
-In case a whole segment of the recording contain noise and should be entirely removed from future analysis, it can be labelled by swiching the edition mode from `Correction` to `Rejection`.
 
-<p align='center'><img src='https://github.com/embodied-computation-group/systole/raw/dev/docs/source/images/editor_peaks.gif'/></p>
+```python
+view = Viewer(
+    figsize=(15, 5),
+    bids_folder="/path/to/BIDS/folder/",
+    pattern="task-my_task", # A string long enough to disambiguate in case of mmultiple recordings
+    modality="beh",
+    signal_type="ECG"
+)
+```
 
 +++
 
-### Working with corrected signals
+### Importing signals after manual edition
 
 After manual peaks correction and segments labelling, a new `corrected` subfolder will be appended to the systole derivatives:
 
@@ -218,9 +312,3 @@ After manual peaks correction and segments labelling, a new `corrected` subfolde
    ├─ sub-0003/
    └─ ... 
 ```
-
-The logs of artefacts correction are located in the new `_physio.json` file and contains all information about bad segments labelling, peaks deletion and peaks insertion. The JSON file contains the following entries for each modality (ECG, PPG and respiration)
-
-* `valid` : is the recording valid or should it be discared (`True` unless otherwise stated).
-* `corrected_peaks` : the peaks indexes after correction.
-* `bad_segments` : a list of `start` and `end` indexed of bad segments.

docs/source/notebooks/6-WorkingWithBIDSFolders.md

@@ -1,5 +1,5 @@
 
-v0.0.1 (January 2020)
----------------------
+v0.0.1
+------
 
 Alpha release.

docs/source/releases/v0.0.1.txt

@@ -1,6 +1,6 @@
 
-v0.1.0 (January 2020)
----------------------
+v0.1.0
+------
 
 Initial release.
 

docs/source/releases/v0.1.0.txt

@@ -1,5 +1,5 @@
-v0.1.1 (June 2020)
-------------------
+v0.1.1
+------
 
 **New functions**
 

docs/source/releases/v0.1.1.txt

@@ -1,5 +1,5 @@
-v0.1.2 (September 2020)
------------------------
+v0.1.2
+------
 
  **New functions**
 

docs/source/releases/v0.1.2.txt

@@ -1,5 +1,5 @@
-v0.1.3 (April 2021)
--------------------
+v0.1.3
+------
 
 - |Enhancement| :py:func:`systole.plotly.plot_raw()`: add `ecg_method` parameter to control the ECG peak detection method used.
 - |Enhancement| Download dataset directly from GitHub instead of copying the files at install.

docs/source/releases/v0.1.3.txt

@@ -1,5 +1,5 @@
-v0.2.0 (October 2021)
------------------------
+v0.2.0
+------
 
 Highlights
 ++++++++++

docs/source/releases/v0.2.0.txt

@@ -1,5 +1,5 @@
-v0.2.1 - December 2021
-----------------------
+v0.2.1
+------
 
 Highlights
 ++++++++++

docs/source/releases/v0.2.1.txt

@@ -1,5 +1,5 @@
-v0.2.2 (December 2021)
-----------------------
+v0.2.2
+------
 
 Highlights
 ++++++++++