Incremental BIDS import

[maximemulder]

BIDS dataset import script rewrite

List of new features

• Fully incremental BIDS import. That is, a BIDS dataset can be imported in several parts, and only the files that have not been imported yet will be added to LORIS.
• Better logging and error reporting. Prints and errors are now logged in the LORIS-MRI logging system. Moreover, additional information is given during the script run, and previously silent errors (such as unidentified scan types) are now signaled to the user.
• Better support for sessionful and sessionless BIDS datasets. Before, some code paths were only implemented for one of those case, now the system is generic over the presence or absence of BIDS session directories, ensuring better support for both these cases.
• The --create_candidate or --create_session options are now transactional. Either all candidates and sessions are created without error, or none is created if an error occurs (due to incorrect input data).
• The --create_candidate option now looks for sites and projects using both names and aliases, in that order.

List of breaking changes

• The script has been renamed from bids_import.py to import_bids_dataset.py. This should better follow the LORIS-MRI scripts naming conventions.
• An error is now returned if the dataset_description.json file is absent. This file is a required BIDS file, and was already needed unless the --nocopy option was used. It is also trivial to write anyway.
• An error is now returned for MRI files if the --type=derivatives or --no_copy options are used. It seems to me that the current code was not tested and probably not working, so it is better to have a hard error rather than an untested and potentially wrong implementation IMO.
• The --idsvalidation option has been removed. This option allowed to validate the BIDS directory name according to a pscid_candid_blablabla format. This is obviously esoteric and overly-specific as BIDS dataset can contain several participants, and their identifiers should be retrieved from the subject directory names or the participants.tsv file instead.
• The --create_candidate option no longer tries to find the site by matching site aliases against participants IDs. This approach can give false positives if a similar (and not even necessarily equal) alias as a project. The site participants.tsv column is now mandatory to create candidates.

List of major architectural changes

Note that these changes only apply to the generic BIDS import code (subjects and sessions), and the MRI import code. It does not apply to EEG import code.

• The code has been fully rewritten.
• The code is now fully typed 😌.
• The code now uses the new ORM database abstraction instead of the old database abstraction or inline queries.
• The code now uses the LORIS-MRI log abstraction instead of print.
• The BIDS dataset import code has been moved and isolated in the new lib.import_bids_dataset module.
• Created the new lib.imaging_lib module to replace the old lib.imaging module. Unlike lib.imaging, the new module is fully typed, uses the new ORM database abstraction, and is divided into submodules related to different kinds of imaging data (BIDS, DICOM, file parameters...).
• Candidates and sessions are obtained at the subject and session levels. This was previously done at the data type level like before and introduced significant duplication between MRI and EEG code and nonsensical dependencies.

Future works

• Migrate the EEG code.
• Add an overwrite option to replace the existing files by the new ones if present.
• Remove the default visit label configuration option and make it an argument (along with default site, default cohort...) ?

Other comments

This PR is very large so it will likely be divided in smaller PRs to be reviewed and merged.

[laemtl]

Hello, a few comments:

• instead of removing --idsvalidation, I think it is better to migrate the logic to use participants.tsv instead.
• ideally, we need a way for the script to work without dataset_description.json, or to use the one in a specific location, depending on how the incremental ingestion works.

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 20 days.

[MaximeBICMTL]

Replaced by #1325