Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file modified bins/mgikit
Binary file not shown.
41 changes: 28 additions & 13 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,20 +20,20 @@ demultiplexing:
1. Fastq files (single/paired-end).
2. Sample sheet which contains sample indexes and their templates (will be explained in detail).

Simply, the tool reads the barcodes at the end of R2 (reveres) reads for paired-end reads input or the end of
Simply, the tool reads the barcodes at the end of R2 (reverse) reads for paired-end reads input, or the end of
R1 (forward) reads for single read input. Based on the barcode, it assigns the read to the relevant
sample allowing for mismatches less than a specific threshold. The tool outputs fastq files for each sample
sample, allowing for mismatches less than a specific threshold. The tool outputs fastq files for each sample
as well as some summary reports that can be visualised through the MultiQC tool and mgikit plugin.

mgikit reports can be parsed by [mgikit-multiqc plugin](https://github.com/sagc-bioinformatics/mgikit-multiqc) to generate an html report using [multiqc](https://multiqc.info/) tool summarising the results of the demultiplexing and the quality of the output data as described [here](/mgikit/mgikit-multiqc).
mgikit reports can be parsed by [mgikit-multiqc plugin](https://github.com/sagc-bioinformatics/mgikit-multiqc) to generate an HTML report using [multiqc](https://multiqc.info/) tool summarising the results of the demultiplexing and the quality of the output data as described [here](/mgikit/mgikit-multiqc).

<hr/>

### template

This command is used to detect the location and form of the indexes within the read barcode. It simply goes through a small number of the reads and investigates the number of matches with the indexes in the sample sheet within each possible location in the read barcode and considers the indexes as is and their reverse complementary.
This command is used to detect the location and form of the indexes within the read barcode. It simply goes through a small number of the reads and investigates the number of matches with the indexes in the sample sheet within each possible location in the read barcode, and considers the indexes as is and their reverse complement.

It reports matches for all possible combinations and uses the read template that has the maximum number of matches. This process happens for each sample individually and therefore, the best matching template for each sample will be reported.
It reports matches for all possible combinations and uses the read template that has the maximum number of matches. This process happens for each sample individually, and therefore, the best-matching template for each sample will be reported.

Using this comprehensive scan, the tool can detect the templates for mixed libraries.

Expand All @@ -53,36 +53,51 @@ This command is to reformat fastq files generated by `splitBarcode` into Illumin

## Important notes

- The tools only accepts unix line breakers `\n'. If your data has other than unix line breakers, consder reformating it (using dos2unix or other tools).
- The tools only accept Unix line breakers `\n'. If your data has other than Unix line breakers, consider reformating it (using dos2unix or other tools).

- The `--flexible` parameters can handle input reads with variable length as long as there is no read longer than double the length of the shortest read. (We don't expect the user to have such case, but just in case the user is expermenting with somthing.)
- The `--flexible` parameters can handle input reads with variable length as long as there is no read longer than double the length of the shortest read. (We don't expect the user to have such a case, but just in case the user is experimenting with something.)

- The `template` functionality checks the top reads in the file, if the input data is sorted, the functionality might not find all samples. Make sure your data is not sorted or wait for our future improvements.
- The `template` functionality checks the top reads in the file; if the input data is sorted, the functionality might not find all samples. Make sure your data is not sorted, or wait for our future improvements.

<hr/>

## Installation

You can use the static binary under bins directly, however, if you like to build it from the source code:
You can use the static binary under bins directly; however, if you would like to build it from the source code:

You need to have `Rust` and `Cargo` installed first, check rust [documenation](https://doc.rust-lang.org/cargo/getting-started/installation.html)
You need to have `Rust` and `Cargo` installed first, check Rust [documentation](https://doc.rust-lang.org/cargo/getting-started/installation.html)

```bash
git clone https://github.com/sagc-bioinformatics/mgikit.git
cd mgikit
cargo build --release
```

## User Guide Table of Content
You can also install mgikit through conda, as it is available on Bioconda.

```bash
conda install bioconda::mgikit
```

Or create a conda environment with mgikit

```bash
conda create -n mgikit_env bioconda::mgikit
conda activate mgikit_env
```

Additionally, Biocontainers has [docker image](https://quay.io/repository/biocontainers/mgikit) and [singularity image](https://depot.galaxyproject.org/singularity/) for mgikit.

## User Guide Table of Contents

{% include section-navigation-tiles.html type="guides" %}

## Commerical Use
## Commercial Use

Please contact us if you want to use the software for commercial purposes.

## Citation

If you use mgikit in your resear ch, please cite the following publication:
If you use mgikit in your research, please cite the following publication:

> Al Bkhetan, Ziad, and Sen Wang. ["mgikit: demultiplexing toolkit for MGI fastq files."](https://academic.oup.com/bioinformatics/article/40/9/btae554/7755041) Bioinformatics 40.9 (2024): btae554.
2 changes: 1 addition & 1 deletion docs/pages/contact_us.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,4 @@ title: Support and Contact us
toc: false
---

For any help or inquiries, please contact: ziad.albkhetan@gmail.com or sagc@sahmri.com
For any help or inquiries, please contact: <ziad.albkhetan@gmail.com> or <sagc@sahmri.com>
7 changes: 3 additions & 4 deletions docs/pages/demultiplex.md
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,7 @@ The input fastq files can be provided to the tool in two ways:

2. Using `-i` or `--input` parameter which refers to the path to the lane subdirectory in the sequencing output directory (or the directory that contains the fastq files if the data is obtained from somewhere else). In this case, the tool will search for the file that ends with `_read_1.fq.gz` and `_read_2.fq.gz` as forward and reverse reads respectively and if no reverse read file is found, the tool considers the run as a single end run. These suffixes can be also customised using the parameters (`--r1-file-suf` and `--r2-file-suf`).

### Sample sheet format and preparation.
### Sample sheet format and preparation

For the tool to perform demultiplexing, it needs to know the indexes of each sample to match them with the barcodes at the end of the read sequence as well as where to look for each index in the barcode. We refer to the location of the indexes within the barcode by the barcode template. For example
This information can be provided to the tool in two different ways:
Expand Down Expand Up @@ -400,7 +400,7 @@ If either `--writer-threads` or `--reader-threads` is `0` (or not set), The `--t

#### Default Thread Allocation

mgikit will only use the minimim number of requested threads and available CPU core. The default behavoir is:
mgikit will only use the minimim number of requested threads and available CPU core. The default behavoir is:

1. If you only have 1 CPU core, all work happens in a single processing thread.

Expand All @@ -420,13 +420,12 @@ mgikit will only use the minimim number of requested threads and available CPU c

This default configuration showed the best performance in testing datasets.

#### Example:
#### Example

threads = 8 → 2 reader threads, 6 processing threads.

`reader-threads` = 4 and `writer-threads` = 2 → Uses exactly 4 readers and 2 writer threads; threads is ignored.


### Performance evaluation

Performance time (in minutes) evaluation and comparison on different datasets.
Expand Down
41 changes: 17 additions & 24 deletions src/hardware_resources.rs
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
use sysinfo::{ SystemExt, System };
use log::{ warn, info };
use log::{info, warn};
use sysinfo::{System, SystemExt};

pub fn get_available_memory(memory: f64) -> f64 {
let mut system = System::new_all();
Expand All @@ -10,16 +10,15 @@ pub fn get_available_memory(memory: f64) -> f64 {
panic!("Available memory should be greater than 0.5 GB!");
}

let available_memory =
(if memory == 0.0 {
sys_memory
} else {
info!("Requested memory by the user is {} GigaByte", memory);
if sys_memory < memory * 1000_000_000.0 {
panic!("Requested memory is greater than the available memory!");
}
memory * 1000_000_000.0
}) - 500_000_000.0;
let available_memory = (if memory == 0.0 {
sys_memory
} else {
info!("Requested memory by the user is {} GigaByte", memory);
if sys_memory < memory * 1000_000_000.0 {
panic!("Requested memory is greater than the available memory!");
}
memory * 1000_000_000.0
}) - 500_000_000.0;
available_memory
}

Expand All @@ -32,24 +31,21 @@ pub fn get_cpus(requested_threads: usize, paired_read_input: bool) -> (usize, us
};
info!(
"Available CPUs: {}. Requested CPUs {}. Used CPUs: {}.",
available_cpus,
requested_threads,
used_cpus
available_cpus, requested_threads, used_cpus
);
if requested_threads > available_cpus {
warn!(
"Used CPUs are redueced to the available CPUs as the requested number is not available."
);
}

let reader_threads = if used_cpus == 1
{

let reader_threads = if used_cpus == 1 {
0
}else{
} else {
if paired_read_input {
if used_cpus < 5 {
1
}else if used_cpus < 9 {
} else if used_cpus < 9 {
2
} else {
4
Expand All @@ -63,14 +59,11 @@ pub fn get_cpus(requested_threads: usize, paired_read_input: bool) -> (usize, us
}
};



let processing_threads = used_cpus - reader_threads;
if used_cpus > 1 {
info!(
"Multi-threading mode is enabled. Readers: {}, Processors {}.",
reader_threads,
processing_threads
reader_threads, processing_threads
);
}
if (reader_threads == 2 && !paired_read_input) || reader_threads > 2 {
Expand Down
Loading