diff --git a/bins/mgikit b/bins/mgikit
index 2fc9a3e..462a49d 100755
Binary files a/bins/mgikit and b/bins/mgikit differ
diff --git a/docs/index.md b/docs/index.md
index d6cdeb9..353e020 100755
--- a/docs/index.md
+++ b/docs/index.md
@@ -20,20 +20,20 @@ demultiplexing:
1. Fastq files (single/paired-end).
2. Sample sheet which contains sample indexes and their templates (will be explained in detail).
-Simply, the tool reads the barcodes at the end of R2 (reveres) reads for paired-end reads input or the end of
+Simply, the tool reads the barcodes at the end of R2 (reverse) reads for paired-end reads input, or the end of
R1 (forward) reads for single read input. Based on the barcode, it assigns the read to the relevant
-sample allowing for mismatches less than a specific threshold. The tool outputs fastq files for each sample
+sample, allowing for mismatches less than a specific threshold. The tool outputs fastq files for each sample
as well as some summary reports that can be visualised through the MultiQC tool and mgikit plugin.
-mgikit reports can be parsed by [mgikit-multiqc plugin](https://github.com/sagc-bioinformatics/mgikit-multiqc) to generate an html report using [multiqc](https://multiqc.info/) tool summarising the results of the demultiplexing and the quality of the output data as described [here](/mgikit/mgikit-multiqc).
+mgikit reports can be parsed by [mgikit-multiqc plugin](https://github.com/sagc-bioinformatics/mgikit-multiqc) to generate an HTML report using [multiqc](https://multiqc.info/) tool summarising the results of the demultiplexing and the quality of the output data as described [here](/mgikit/mgikit-multiqc).
### template
-This command is used to detect the location and form of the indexes within the read barcode. It simply goes through a small number of the reads and investigates the number of matches with the indexes in the sample sheet within each possible location in the read barcode and considers the indexes as is and their reverse complementary.
+This command is used to detect the location and form of the indexes within the read barcode. It simply goes through a small number of the reads and investigates the number of matches with the indexes in the sample sheet within each possible location in the read barcode, and considers the indexes as is and their reverse complement.
-It reports matches for all possible combinations and uses the read template that has the maximum number of matches. This process happens for each sample individually and therefore, the best matching template for each sample will be reported.
+It reports matches for all possible combinations and uses the read template that has the maximum number of matches. This process happens for each sample individually, and therefore, the best-matching template for each sample will be reported.
Using this comprehensive scan, the tool can detect the templates for mixed libraries.
@@ -53,19 +53,19 @@ This command is to reformat fastq files generated by `splitBarcode` into Illumin
## Important notes
-- The tools only accepts unix line breakers `\n'. If your data has other than unix line breakers, consder reformating it (using dos2unix or other tools).
+- The tools only accept Unix line breakers `\n'. If your data has other than Unix line breakers, consider reformating it (using dos2unix or other tools).
-- The `--flexible` parameters can handle input reads with variable length as long as there is no read longer than double the length of the shortest read. (We don't expect the user to have such case, but just in case the user is expermenting with somthing.)
+- The `--flexible` parameters can handle input reads with variable length as long as there is no read longer than double the length of the shortest read. (We don't expect the user to have such a case, but just in case the user is experimenting with something.)
-- The `template` functionality checks the top reads in the file, if the input data is sorted, the functionality might not find all samples. Make sure your data is not sorted or wait for our future improvements.
+- The `template` functionality checks the top reads in the file; if the input data is sorted, the functionality might not find all samples. Make sure your data is not sorted, or wait for our future improvements.
## Installation
-You can use the static binary under bins directly, however, if you like to build it from the source code:
+You can use the static binary under bins directly; however, if you would like to build it from the source code:
-You need to have `Rust` and `Cargo` installed first, check rust [documenation](https://doc.rust-lang.org/cargo/getting-started/installation.html)
+You need to have `Rust` and `Cargo` installed first, check Rust [documentation](https://doc.rust-lang.org/cargo/getting-started/installation.html)
```bash
git clone https://github.com/sagc-bioinformatics/mgikit.git
@@ -73,16 +73,31 @@ cd mgikit
cargo build --release
```
-## User Guide Table of Content
+You can also install mgikit through conda, as it is available on Bioconda.
+
+```bash
+conda install bioconda::mgikit
+```
+
+Or create a conda environment with mgikit
+
+```bash
+conda create -n mgikit_env bioconda::mgikit
+conda activate mgikit_env
+```
+
+Additionally, Biocontainers has [docker image](https://quay.io/repository/biocontainers/mgikit) and [singularity image](https://depot.galaxyproject.org/singularity/) for mgikit.
+
+## User Guide Table of Contents
{% include section-navigation-tiles.html type="guides" %}
-## Commerical Use
+## Commercial Use
Please contact us if you want to use the software for commercial purposes.
## Citation
-If you use mgikit in your resear ch, please cite the following publication:
+If you use mgikit in your research, please cite the following publication:
> Al Bkhetan, Ziad, and Sen Wang. ["mgikit: demultiplexing toolkit for MGI fastq files."](https://academic.oup.com/bioinformatics/article/40/9/btae554/7755041) Bioinformatics 40.9 (2024): btae554.
diff --git a/docs/pages/contact_us.md b/docs/pages/contact_us.md
index 0b921c4..ddb3766 100755
--- a/docs/pages/contact_us.md
+++ b/docs/pages/contact_us.md
@@ -3,4 +3,4 @@ title: Support and Contact us
toc: false
---
-For any help or inquiries, please contact: ziad.albkhetan@gmail.com or sagc@sahmri.com
+For any help or inquiries, please contact: or
diff --git a/docs/pages/demultiplex.md b/docs/pages/demultiplex.md
index cf7edd5..4160bfc 100755
--- a/docs/pages/demultiplex.md
+++ b/docs/pages/demultiplex.md
@@ -152,7 +152,7 @@ The input fastq files can be provided to the tool in two ways:
2. Using `-i` or `--input` parameter which refers to the path to the lane subdirectory in the sequencing output directory (or the directory that contains the fastq files if the data is obtained from somewhere else). In this case, the tool will search for the file that ends with `_read_1.fq.gz` and `_read_2.fq.gz` as forward and reverse reads respectively and if no reverse read file is found, the tool considers the run as a single end run. These suffixes can be also customised using the parameters (`--r1-file-suf` and `--r2-file-suf`).
-### Sample sheet format and preparation.
+### Sample sheet format and preparation
For the tool to perform demultiplexing, it needs to know the indexes of each sample to match them with the barcodes at the end of the read sequence as well as where to look for each index in the barcode. We refer to the location of the indexes within the barcode by the barcode template. For example
This information can be provided to the tool in two different ways:
@@ -400,7 +400,7 @@ If either `--writer-threads` or `--reader-threads` is `0` (or not set), The `--t
#### Default Thread Allocation
-mgikit will only use the minimim number of requested threads and available CPU core. The default behavoir is:
+mgikit will only use the minimim number of requested threads and available CPU core. The default behavoir is:
1. If you only have 1 CPU core, all work happens in a single processing thread.
@@ -420,13 +420,12 @@ mgikit will only use the minimim number of requested threads and available CPU c
This default configuration showed the best performance in testing datasets.
-#### Example:
+#### Example
threads = 8 → 2 reader threads, 6 processing threads.
`reader-threads` = 4 and `writer-threads` = 2 → Uses exactly 4 readers and 2 writer threads; threads is ignored.
-
### Performance evaluation
Performance time (in minutes) evaluation and comparison on different datasets.
diff --git a/src/hardware_resources.rs b/src/hardware_resources.rs
index 3ee36ad..54e56d1 100644
--- a/src/hardware_resources.rs
+++ b/src/hardware_resources.rs
@@ -1,5 +1,5 @@
-use sysinfo::{ SystemExt, System };
-use log::{ warn, info };
+use log::{info, warn};
+use sysinfo::{System, SystemExt};
pub fn get_available_memory(memory: f64) -> f64 {
let mut system = System::new_all();
@@ -10,16 +10,15 @@ pub fn get_available_memory(memory: f64) -> f64 {
panic!("Available memory should be greater than 0.5 GB!");
}
- let available_memory =
- (if memory == 0.0 {
- sys_memory
- } else {
- info!("Requested memory by the user is {} GigaByte", memory);
- if sys_memory < memory * 1000_000_000.0 {
- panic!("Requested memory is greater than the available memory!");
- }
- memory * 1000_000_000.0
- }) - 500_000_000.0;
+ let available_memory = (if memory == 0.0 {
+ sys_memory
+ } else {
+ info!("Requested memory by the user is {} GigaByte", memory);
+ if sys_memory < memory * 1000_000_000.0 {
+ panic!("Requested memory is greater than the available memory!");
+ }
+ memory * 1000_000_000.0
+ }) - 500_000_000.0;
available_memory
}
@@ -32,24 +31,21 @@ pub fn get_cpus(requested_threads: usize, paired_read_input: bool) -> (usize, us
};
info!(
"Available CPUs: {}. Requested CPUs {}. Used CPUs: {}.",
- available_cpus,
- requested_threads,
- used_cpus
+ available_cpus, requested_threads, used_cpus
);
if requested_threads > available_cpus {
warn!(
"Used CPUs are redueced to the available CPUs as the requested number is not available."
);
}
-
- let reader_threads = if used_cpus == 1
- {
+
+ let reader_threads = if used_cpus == 1 {
0
- }else{
+ } else {
if paired_read_input {
if used_cpus < 5 {
1
- }else if used_cpus < 9 {
+ } else if used_cpus < 9 {
2
} else {
4
@@ -63,14 +59,11 @@ pub fn get_cpus(requested_threads: usize, paired_read_input: bool) -> (usize, us
}
};
-
-
let processing_threads = used_cpus - reader_threads;
if used_cpus > 1 {
info!(
"Multi-threading mode is enabled. Readers: {}, Processors {}.",
- reader_threads,
- processing_threads
+ reader_threads, processing_threads
);
}
if (reader_threads == 2 && !paired_read_input) || reader_threads > 2 {
diff --git a/src/lib.rs b/src/lib.rs
index 292a916..7485b8b 100755
--- a/src/lib.rs
+++ b/src/lib.rs
@@ -1,44 +1,40 @@
#[global_allocator]
static GLOBAL: mimalloc::MiMalloc = mimalloc::MiMalloc;
+use clap::ArgMatches;
+use crossbeam_channel::{bounded, Receiver, Sender};
use file_utils::{
- get_buf_reader,
- get_gzip_reader,
- parallel_reader_decompressor_thread,
- parallel_reader_thread,
- read_buffers,
- write_file,
+ get_buf_reader, get_gzip_reader, parallel_reader_decompressor_thread, parallel_reader_thread,
+ read_buffers, write_file,
};
-use variables::{ I5_COLUMN, I7_COLUMN, PROJECT_ID_COLUMN, SAMPLE_COLUMN };
-use std::collections::{ HashMap, HashSet };
+use log::{debug, info, warn};
+use memchr::{memchr, memchr_iter};
+use std::collections::{HashMap, HashSet};
use std::error::Error;
+use std::fs;
+use std::io::BufRead;
+use std::path::Path;
use std::path::PathBuf;
-use std::time::Instant;
-use memchr::{ memchr_iter, memchr };
-use log::{ info, warn, debug };
+use std::sync::{Arc, Mutex};
use std::thread;
-use crossbeam_channel::{ Receiver, Sender, bounded };
-use std::sync::{ Arc, Mutex };
-use clap::ArgMatches;
-use std::path::Path;
-use std::io::BufRead;
-use std::fs;
+use std::time::Instant;
+use variables::{I5_COLUMN, I7_COLUMN, PROJECT_ID_COLUMN, SAMPLE_COLUMN};
// my modules
-mod variables;
mod file_utils;
-mod sample_manager;
+mod formater;
+mod hardware_resources;
+mod report_manager;
mod run_manager;
mod sample_data;
-mod report_manager;
-mod hardware_resources;
-mod formater;
+mod sample_manager;
+mod variables;
+pub use crate::hardware_resources::{get_available_memory, get_cpus};
pub use crate::sample_data::*;
-pub use crate::hardware_resources::{ get_available_memory, get_cpus };
-pub use sample_manager::{ SampleManager, get_all_mismatches, reverse_complement };
-pub use run_manager::RunManager;
+pub use formater::{parse_sb_file_name, ReformatedSample};
pub use report_manager::ReportManager;
-pub use formater::{ ReformatedSample, parse_sb_file_name };
+pub use run_manager::RunManager;
+pub use sample_manager::{get_all_mismatches, reverse_complement, SampleManager};
const BUFFER_SIZE: usize = 1 << 22;
const RAW_BUFFER_SIZE: usize = 1 << 23;
@@ -48,20 +44,18 @@ fn hamming_distance(bytes1: &[u8], bytes2: &[u8]) -> Result
return Err("Byte slices must be of equal length");
}
- Ok(
- bytes1
- .iter()
- .zip(bytes2.iter())
- .filter(|(b1, b2)| b1 != b2)
- .count()
- )
+ Ok(bytes1
+ .iter()
+ .zip(bytes2.iter())
+ .filter(|(b1, b2)| b1 != b2)
+ .count())
}
pub fn add_match(
curr_template: String,
sample_itr: usize,
matches_stat: &mut HashMap>,
- sample_list_ln: usize
+ sample_list_ln: usize,
) {
match matches_stat.get_mut(&curr_template) {
Some(sample_reads) => {
@@ -83,18 +77,17 @@ fn find_matches(
sample_first_indx: &String,
sample_first_indx_rc: &String,
sample_second_indx: &String,
- sample_second_indx_rc: &String
+ sample_second_indx_rc: &String,
) {
//println!("{} - {} - {} - {} -> {}", sample_first_indx, sample_first_indx_rc, sample_second_indx, sample_second_indx_rc, read_barcode_seq);
let indexes = match sample_second_indx.len() < 2 {
true => vec![sample_first_indx, sample_first_indx_rc],
- false =>
- vec![
- sample_first_indx,
- sample_first_indx_rc,
- sample_second_indx,
- sample_second_indx_rc
- ],
+ false => vec![
+ sample_first_indx,
+ sample_first_indx_rc,
+ sample_second_indx,
+ sample_second_indx_rc,
+ ],
};
//println!("{:?}", indexes);
for i in 0..2 {
@@ -116,15 +109,14 @@ fn find_matches(
for first_match in &first_matches {
//println!("sample - {}: comparing {} -> {} {}:{}",sample_itr, first_match, second_match, i, j);
- if
- *second_match >= first_match + sample_first_indx.len() ||
- *first_match >= second_match + sample_second_indx.len()
+ if *second_match >= first_match + sample_first_indx.len()
+ || *first_match >= second_match + sample_second_indx.len()
{
add_match(
format!("{}:{}_{}:{}", first_match, second_match, i, j - 2),
sample_itr,
matches_stat,
- sample_list_ln
+ sample_list_ln,
);
}
}
@@ -137,7 +129,7 @@ fn find_matches(
format!("{}_{}:.", first_match, i),
sample_itr,
matches_stat,
- sample_list_ln
+ sample_list_ln,
);
}
}
@@ -150,7 +142,7 @@ pub fn get_mgikit_template(
umi: bool,
barcode_length: usize,
i7_len: usize,
- i5_len: usize
+ i5_len: usize,
) -> (String, String, String) {
let mut final_template: Vec = Vec::new();
let template_elements: Vec = initial_template.split('_').map(String::from).collect();
@@ -221,21 +213,23 @@ pub fn get_mgikit_template(
}
//println!("5- {:?}", final_template);
- (final_template.join(":"), rc_ls[0].to_owned(), rc_ls[1].to_owned())
+ (
+ final_template.join(":"),
+ rc_ls[0].to_owned(),
+ rc_ls[1].to_owned(),
+ )
}
pub fn find_matching_sample(
- all_template_data: &Vec<
- (
- u32,
- HashSet,
- HashSet,
- String,
- HashMap)>,
- bool,
- [usize; 10],
- )
- >,
+ all_template_data: &Vec<(
+ u32,
+ HashSet,
+ HashSet,
+ String,
+ HashMap)>,
+ bool,
+ [usize; 10],
+ )>,
mismatches_dic_i7: &Vec, (Vec<&String>, usize)>>,
mismatches_dic_i5: &Vec, (Vec<&String>, usize)>>,
read_barcode: &[u8],
@@ -243,7 +237,7 @@ pub fn find_matching_sample(
all_index_allowed_mismatches: usize,
total_samples: usize,
comprehensive_scan: bool,
- i7_rc: bool
+ i7_rc: bool,
) -> (usize, usize, String, String) {
let mut template_itr = 0;
let undetermined_label_id = total_samples - 2;
@@ -265,80 +259,64 @@ pub fn find_matching_sample(
//println!("{}", template_details.5);
let extract_umi = if indexes_info[6] == 1 { true } else { false };
- match
- all_i7_mismatches.get(
- &read_barcode
- [
- read_barcode.len() - indexes_info[2]..read_barcode.len() -
- indexes_info[2] +
- indexes_info[1]
- ]
- )
- {
+ match all_i7_mismatches.get(
+ &read_barcode[read_barcode.len() - indexes_info[2]
+ ..read_barcode.len() - indexes_info[2] + indexes_info[1]],
+ ) {
Some(i7_matches) => {
//info!("{:?}", i7_matches);
if i7_matches.1 <= allowed_mismatches {
if template_details.5 {
- match
- all_i5_mismatches.get(
- &read_barcode
- [
- read_barcode.len() - indexes_info[5]..read_barcode.len() -
- indexes_info[5] +
- indexes_info[4]
- ]
- )
- {
+ match all_i5_mismatches.get(
+ &read_barcode[read_barcode.len() - indexes_info[5]
+ ..read_barcode.len() - indexes_info[5] + indexes_info[4]],
+ ) {
Some(i5_matches) => {
//info!("{:?} and {:?}", i7_matches, i5_matches);
for i7_match_itr in 0..i7_matches.0.len() {
for i5_match_itr in 0..i5_matches.0.len() {
- if
- i5_matches.1 > allowed_mismatches ||
- latest_mismatch < i7_matches.1 + i5_matches.1
+ if i5_matches.1 > allowed_mismatches
+ || latest_mismatch < i7_matches.1 + i5_matches.1
{
continue;
}
curr_mismatch = i7_matches.1 + i5_matches.1;
if curr_mismatch <= all_index_allowed_mismatches {
//println!("{}", i7_matches.0[i7_match_itr]);
- match
- sample_info
- .get(i7_matches.0[i7_match_itr])
- .unwrap()
- .1.get(i5_matches.0[i5_match_itr])
+ match sample_info
+ .get(i7_matches.0[i7_match_itr])
+ .unwrap()
+ .1
+ .get(i5_matches.0[i5_match_itr])
{
Some(i5_info) => {
- if
- sample_id >= total_samples ||
- latest_mismatch > curr_mismatch
+ if sample_id >= total_samples
+ || latest_mismatch > curr_mismatch
{
sample_id = *i5_info;
latest_mismatch = curr_mismatch;
curr_barcode = unsafe {
String::from_utf8_unchecked(
- read_barcode[
- read_barcode.len() -
- indexes_info
- [2]..read_barcode.len() -
- indexes_info[2] +
- indexes_info[1]
- ].to_vec()
+ read_barcode[read_barcode.len()
+ - indexes_info[2]
+ ..read_barcode.len()
+ - indexes_info[2]
+ + indexes_info[1]]
+ .to_vec(),
)
};
curr_barcode.push('+');
curr_barcode.push_str(
&(unsafe {
String::from_utf8_unchecked(
- read_barcode[
- read_barcode.len() -
- indexes_info
- [5]..read_barcode.len() -
- indexes_info[5] +
- indexes_info[4]
- ].to_vec()
+ read_barcode[read_barcode.len()
+ - indexes_info[5]
+ ..read_barcode.len()
+ - indexes_info[5]
+ + indexes_info[4]]
+ .to_vec(),
)
- })
+ }),
);
if extract_umi {
@@ -408,11 +386,9 @@ pub fn find_matching_sample(
//barcode_length = indexes_info[9];
curr_barcode = unsafe {
String::from_utf8_unchecked(
- read_barcode[
- read_barcode.len() - indexes_info[2]..read_barcode.len() -
- indexes_info[2] +
- indexes_info[1]
- ].to_vec()
+ read_barcode[read_barcode.len() - indexes_info[2]
+ ..read_barcode.len() - indexes_info[2] + indexes_info[1]]
+ .to_vec(),
)
};
if extract_umi {
@@ -420,14 +396,12 @@ pub fn find_matching_sample(
curr_umi.push_str(
&(unsafe {
String::from_utf8_unchecked(
- read_barcode[
- read_barcode.len() -
- indexes_info[8]..read_barcode.len() -
- indexes_info[8] +
- indexes_info[7]
- ].to_vec()
+ read_barcode[read_barcode.len() - indexes_info[8]
+ ..read_barcode.len() - indexes_info[8]
+ + indexes_info[7]]
+ .to_vec(),
)
- })
+ }),
);
/*if i7_rc {
curr_umi.push_str(
@@ -507,24 +481,29 @@ pub fn check_read_content(read_seq: &[u8], seq_start: usize, plus_start: usize,
);
}
for &nec in read_seq[seq_start..plus_start - 1].iter() {
- if
- nec != b'A' &&
- nec != b'C' &&
- nec != b'G' &&
- nec != b'T' &&
- nec != b'N' &&
- nec != b'a' &&
- nec != b'c' &&
- nec != b'g' &&
- nec != b't' &&
- nec != b'n'
+ if nec != b'A'
+ && nec != b'C'
+ && nec != b'G'
+ && nec != b'T'
+ && nec != b'N'
+ && nec != b'a'
+ && nec != b'c'
+ && nec != b'g'
+ && nec != b't'
+ && nec != b'n'
{
- panic!("Seqeunce bases need to be in [A, C, G, T, a, c, g, t, N, n]! Found {}.", nec);
+ panic!(
+ "Seqeunce bases need to be in [A, C, G, T, a, c, g, t, N, n]! Found {}.",
+ nec
+ );
}
}
for &qs in read_seq[qc_start..read_seq.len()].iter() {
if qs > 73 || qs < 33 {
- panic!("Quality score must be between [0 and 40]! Found {}!", qs - 33);
+ panic!(
+ "Quality score must be between [0 and 40]! Found {}!",
+ qs - 33
+ );
}
}
}
@@ -537,13 +516,19 @@ fn demultiplex(
reporting_level: usize,
all_index_error: bool,
reader_threads: usize,
- processing_threads: usize
+ processing_threads: usize,
) -> Result> {
let start = Instant::now();
//let dur;
- info!("Comprehensive scan mode: {}", run_manager.comprehensive_scan());
- info!("Allowed mismatches when finding the index are: {}", allowed_mismatches);
+ info!(
+ "Comprehensive scan mode: {}",
+ run_manager.comprehensive_scan()
+ );
+ info!(
+ "Allowed mismatches when finding the index are: {}",
+ allowed_mismatches
+ );
if all_index_error {
info!(
@@ -570,7 +555,7 @@ fn demultiplex(
run_manager,
buffer_info,
run_manager.read2_has_sequence(),
- run_manager.illumina_format()
+ run_manager.illumina_format(),
);
}
@@ -586,7 +571,7 @@ fn demultiplex(
full_receiver_rb/full_receiver_rp
full_sender
-
+
workers
full_receiver
empty_sender_rb
@@ -595,7 +580,7 @@ fn demultiplex(
empty_raw_receiver_rp/empty_raw_receiver_rb
full_raw_sender_rp/full_raw_sender_rb
-
+
decoder
full_raw_receiver_rp/full_raw_receiver_rb
empty_receiver_rb/empty_receiver_rp
@@ -618,9 +603,15 @@ fn demultiplex(
barcode_process_master = true;
((BUFFER_SIZE as f32) / (run_manager.paired_read_len() as f32)) * 0.95
}) as usize;
- debug!("Barceode data reader is master reader: {}", barcode_process_master);
+ debug!(
+ "Barceode data reader is master reader: {}",
+ barcode_process_master
+ );
let reader_handler = if reader_threads > 0 {
- info!("Parallel readers, processing batches of {} reads.", batch_size);
+ info!(
+ "Parallel readers, processing batches of {} reads.",
+ batch_size
+ );
for _ in 0..processing_threads * 2 {
match empty_sender_rb.send((0, vec![0u8; BUFFER_SIZE])) {
Ok(_) => {}
@@ -641,53 +632,48 @@ fn demultiplex(
}
}
}
- if
- (reader_threads == 4 && run_manager.paired_read_input()) ||
- (reader_threads == 2 && !run_manager.paired_read_input())
+ if (reader_threads == 4 && run_manager.paired_read_input())
+ || (reader_threads == 2 && !run_manager.paired_read_input())
{
- Some(
- parallel_reader_decompressor_thread(
- run_manager.barcode_reads().clone(),
- batch_size,
- empty_raw_sender_rb,
- empty_raw_receiver_rb,
- full_raw_sender_rb,
- full_raw_receiver_rb,
- full_receiver_rb.clone(),
- full_receiver_rp.clone(),
- full_sender_rb.clone(),
- empty_receiver_rb.clone(),
- empty_sender_rb.clone(),
- full_sender.clone(),
- processing_threads,
- BUFFER_SIZE,
- run_manager.paired_read_input(),
- barcode_process_master
- )
- )
+ Some(parallel_reader_decompressor_thread(
+ run_manager.barcode_reads().clone(),
+ batch_size,
+ empty_raw_sender_rb,
+ empty_raw_receiver_rb,
+ full_raw_sender_rb,
+ full_raw_receiver_rb,
+ full_receiver_rb.clone(),
+ full_receiver_rp.clone(),
+ full_sender_rb.clone(),
+ empty_receiver_rb.clone(),
+ empty_sender_rb.clone(),
+ full_sender.clone(),
+ processing_threads,
+ BUFFER_SIZE,
+ run_manager.paired_read_input(),
+ barcode_process_master,
+ ))
} else {
- Some(
- parallel_reader_thread(
- run_manager.paired_reads().clone(),
- run_manager.barcode_reads().clone(),
- batch_size,
- true,
- reader_threads == 1 && run_manager.paired_read_input(),
- full_sender_rb.clone(),
- full_sender_rp.clone(),
- empty_sender_rb.clone(),
- empty_sender_rp.clone(),
- full_receiver_rb.clone(),
- full_receiver_rp.clone(),
- empty_receiver_rb.clone(),
- empty_receiver_rp.clone(),
- full_sender.clone(),
- processing_threads,
- BUFFER_SIZE,
- run_manager.paired_read_input(),
- barcode_process_master
- )
- )
+ Some(parallel_reader_thread(
+ run_manager.paired_reads().clone(),
+ run_manager.barcode_reads().clone(),
+ batch_size,
+ true,
+ reader_threads == 1 && run_manager.paired_read_input(),
+ full_sender_rb.clone(),
+ full_sender_rp.clone(),
+ empty_sender_rb.clone(),
+ empty_sender_rp.clone(),
+ full_receiver_rb.clone(),
+ full_receiver_rp.clone(),
+ empty_receiver_rb.clone(),
+ empty_receiver_rp.clone(),
+ full_sender.clone(),
+ processing_threads,
+ BUFFER_SIZE,
+ run_manager.paired_read_input(),
+ barcode_process_master,
+ ))
}
} else {
None
@@ -695,57 +681,54 @@ fn demultiplex(
let reader_handler_secondary = if run_manager.paired_read_input() && reader_threads > 1 {
if reader_threads == 4 {
- Some(
- parallel_reader_decompressor_thread(
- run_manager.paired_reads().clone(),
- batch_size,
- empty_raw_sender_rp,
- empty_raw_receiver_rp,
- full_raw_sender_rp,
- full_raw_receiver_rp,
- full_receiver_rb.clone(),
- full_receiver_rp.clone(),
- full_sender_rp.clone(),
- empty_receiver_rp.clone(),
- empty_sender_rp.clone(),
- full_sender.clone(),
- processing_threads,
- BUFFER_SIZE,
- true,
- !barcode_process_master
- )
- )
+ Some(parallel_reader_decompressor_thread(
+ run_manager.paired_reads().clone(),
+ batch_size,
+ empty_raw_sender_rp,
+ empty_raw_receiver_rp,
+ full_raw_sender_rp,
+ full_raw_receiver_rp,
+ full_receiver_rb.clone(),
+ full_receiver_rp.clone(),
+ full_sender_rp.clone(),
+ empty_receiver_rp.clone(),
+ empty_sender_rp.clone(),
+ full_sender.clone(),
+ processing_threads,
+ BUFFER_SIZE,
+ true,
+ !barcode_process_master,
+ ))
} else {
- Some(
- parallel_reader_thread(
- run_manager.paired_reads().clone(),
- run_manager.barcode_reads().clone(),
- batch_size,
- false,
- true,
- full_sender_rb.clone(),
- full_sender_rp.clone(),
- empty_sender_rb.clone(),
- empty_sender_rp.clone(),
- full_receiver_rb.clone(),
- full_receiver_rp.clone(),
- empty_receiver_rb.clone(),
- empty_receiver_rp.clone(),
- full_sender.clone(),
- processing_threads,
- BUFFER_SIZE,
- true,
- !barcode_process_master
- )
- )
+ Some(parallel_reader_thread(
+ run_manager.paired_reads().clone(),
+ run_manager.barcode_reads().clone(),
+ batch_size,
+ false,
+ true,
+ full_sender_rb.clone(),
+ full_sender_rp.clone(),
+ empty_sender_rb.clone(),
+ empty_sender_rp.clone(),
+ full_receiver_rb.clone(),
+ full_receiver_rp.clone(),
+ empty_receiver_rb.clone(),
+ empty_receiver_rp.clone(),
+ full_sender.clone(),
+ processing_threads,
+ BUFFER_SIZE,
+ true,
+ !barcode_process_master,
+ ))
}
} else {
None
};
- let report_manager_arc = Arc::new(
- Mutex::new(ReportManager::new(total_samples, allowed_mismatches))
- );
+ let report_manager_arc = Arc::new(Mutex::new(ReportManager::new(
+ total_samples,
+ allowed_mismatches,
+ )));
let mut processor_pool = Vec::new();
let mut sample_lockes = Vec::new();
for _ in 0..total_samples {
@@ -765,8 +748,7 @@ fn demultiplex(
let report_manager_arc = report_manager_arc.clone();
let samples_locks_arc: Arc>> = samples_locks_arc.clone();
processor_pool.push(
- thread::Builder
- ::new()
+ thread::Builder::new()
.name(thread_name.clone())
.spawn(move || {
let curr_report_manager = analyse_fastq(
@@ -785,12 +767,12 @@ fn demultiplex(
samples_locks_arc,
true,
&ReformatedSample::default(),
- vec![false; 10]
+ vec![false; 10],
);
let mut report_manager = report_manager_arc.lock().unwrap();
report_manager.update(&curr_report_manager);
- })
+ }),
);
}
@@ -810,18 +792,22 @@ fn demultiplex(
samples_locks_arc,
true,
&ReformatedSample::default(),
- vec![true; 10]
+ vec![true; 10],
);
if reader_threads > 0 {
let _ = match reader_handler {
Some(handler) => handler.join().unwrap(),
- None => { panic!("there must be a reader thread here!") }
+ None => {
+ panic!("there must be a reader thread here!")
+ }
};
if reader_threads == 2 {
let _ = match reader_handler_secondary {
Some(handler) => handler.join().unwrap(),
- None => { panic!("there must be a secondary reader thread here!") }
+ None => {
+ panic!("there must be a secondary reader thread here!")
+ }
};
}
@@ -831,11 +817,10 @@ fn demultiplex(
}
let mut report_manager = match Arc::try_unwrap(report_manager_arc) {
- Ok(mutex) =>
- match mutex.into_inner() {
- Ok(manager) => manager,
- Err(_) => panic!("Report Manager Mutex poisoned"),
- }
+ Ok(mutex) => match mutex.into_inner() {
+ Ok(manager) => manager,
+ Err(_) => panic!("Report Manager Mutex poisoned"),
+ },
Err(_) => panic!("Report Manager Arc still has multiple owners"),
};
@@ -843,7 +828,11 @@ fn demultiplex(
let dur = start.elapsed();
- info!("{} reads were processed in {} secs.", report_manager.get_total_reads(), dur.as_secs());
+ info!(
+ "{} reads were processed in {} secs.",
+ report_manager.get_total_reads(),
+ dur.as_secs()
+ );
Ok(report_manager)
}
@@ -854,17 +843,15 @@ fn process_buffer(
report_manager: &mut ReportManager,
all_index_allowed_mismatches: usize,
sample_manager: &SampleManager,
- all_template_data: &Vec<
- (
- u32,
- HashSet,
- HashSet,
- String,
- HashMap)>,
- bool,
- [usize; 10],
- )
- >,
+ all_template_data: &Vec<(
+ u32,
+ HashSet,
+ HashSet,
+ String,
+ HashMap)>,
+ bool,
+ [usize; 10],
+ )>,
samples_reads: &mut Vec,
reporting_level: usize,
allowed_mismatches: usize,
@@ -877,13 +864,17 @@ fn process_buffer(
lines_rp: Vec,
demultiplex: bool,
reformated_sample: &ReformatedSample,
- warning_ls: &mut Vec
+ warning_ls: &mut Vec,
) -> (usize, usize) {
let l_position: usize = run_manager.l_position();
let total_samples: usize = sample_manager.get_sample_count();
let undetermined_label_id = total_samples - 2;
let ambiguous_label_id = total_samples - 1;
- let shift: usize = if run_manager.paired_read_input() { 1 } else { 0 };
+ let shift: usize = if run_manager.paired_read_input() {
+ 1
+ } else {
+ 0
+ };
let barcode_length: usize = if demultiplex {
all_template_data[0].6[9]
} else {
@@ -998,8 +989,16 @@ fn process_buffer(
}
if reached_an_end {
- debug!("reached end - buffer 2 - bytes: {} - header start: {}", buffer_2.len(), header_start);
- debug!("reached end - buffer 1 - bytes: {} - header start: {}", buffer_1.len(), header_start_pr);
+ debug!(
+ "reached end - buffer 2 - bytes: {} - header start: {}",
+ buffer_2.len(),
+ header_start
+ );
+ debug!(
+ "reached end - buffer 1 - bytes: {} - header start: {}",
+ buffer_1.len(),
+ header_start_pr
+ );
break;
} else {
i7_rc = false;
@@ -1015,7 +1014,7 @@ fn process_buffer(
all_index_allowed_mismatches,
total_samples,
comprehensive_scan,
- i7_rc
+ i7_rc,
);
tail_offset = curr_barcode.len() + 6;
if sample_id >= total_samples {
@@ -1023,11 +1022,10 @@ fn process_buffer(
}
if sample_id == undetermined_label_id || sample_id == ambiguous_label_id {
- if
- warning_ls[0] &&
- read_cntr > undertmined_threshold_check &&
- report_manager.get_sample_reads(sample_id) >
- ((0.75 * (read_cntr as f64)) as u64)
+ if warning_ls[0]
+ && read_cntr > undertmined_threshold_check
+ && report_manager.get_sample_reads(sample_id)
+ > ((0.75 * (read_cntr as f64)) as u64)
{
if run_manager.ignore_undetermined() {
warn!(
@@ -1051,12 +1049,9 @@ fn process_buffer(
let indexes_info = all_template_data[0].6;
curr_barcode = unsafe {
String::from_utf8_unchecked(
- buffer_2[
- plus_start - indexes_info[2] - 1..plus_start -
- indexes_info[2] +
- indexes_info[1] -
- 1
- ].to_vec()
+ buffer_2[plus_start - indexes_info[2] - 1
+ ..plus_start - indexes_info[2] + indexes_info[1] - 1]
+ .to_vec(),
)
};
@@ -1066,20 +1061,17 @@ fn process_buffer(
curr_barcode.push_str(
&(unsafe {
String::from_utf8_unchecked(
- buffer_2[
- plus_start - indexes_info[5] - 1..plus_start -
- indexes_info[5] +
- indexes_info[4] -
- 1
- ].to_vec()
+ buffer_2[plus_start - indexes_info[5] - 1
+ ..plus_start - indexes_info[5] + indexes_info[4] - 1]
+ .to_vec(),
)
- })
+ }),
);
}
} else {
curr_barcode = unsafe {
String::from_utf8_unchecked(
- buffer_2[plus_start - barcode_length - 1..plus_start - 1].to_vec()
+ buffer_2[plus_start - barcode_length - 1..plus_start - 1].to_vec(),
)
};
}
@@ -1106,18 +1098,18 @@ fn process_buffer(
raw_shift = 2;
seq_start - header_start - 3
}
- Some(loc) => { loc }
+ Some(loc) => loc,
};
- curr_mismatch = if
- reformated_sample.barcode().len() == 0 ||
- sep_position == seq_start - header_start - 3
+ curr_mismatch = if reformated_sample.barcode().len() == 0
+ || sep_position == seq_start - header_start - 3
{
0
} else {
hamming_distance(
reformated_sample.barcode().as_bytes(),
- &buffer_2[sep_position + header_start + 7..seq_start - 1]
- ).unwrap()
+ &buffer_2[sep_position + header_start + 7..seq_start - 1],
+ )
+ .unwrap()
};
sample_id = 0; //reformated_sample.sample_index();
@@ -1125,9 +1117,9 @@ fn process_buffer(
if reformated_sample.umi_length() > 0 {
unsafe {
curr_umi = String::from_utf8_unchecked(
- buffer_2[
- plus_start - 1 - reformated_sample.umi_length()..plus_start - 1
- ].to_vec()
+ buffer_2
+ [plus_start - 1 - reformated_sample.umi_length()..plus_start - 1]
+ .to_vec(),
);
}
}
@@ -1137,9 +1129,8 @@ fn process_buffer(
header_shift = seq_start - sep_position - header_start - 3;
tail_offset = header_shift + 1;
- if
- reformated_sample.barcode().len() > 0 &&
- sep_position == seq_start - header_start - 3
+ if reformated_sample.barcode().len() > 0
+ && sep_position == seq_start - header_start - 3
{
header_shift = 0;
tail_offset = reformated_sample.barcode().len() + 6;
@@ -1166,24 +1157,23 @@ fn process_buffer(
&buffer_1[header_start_pr..read_end_pr],
seq_start_pr - header_start_pr,
plus_start_pr - header_start_pr,
- qual_start_pr - header_start_pr
+ qual_start_pr - header_start_pr,
);
- if
- run_manager.mgi_data() &&
- hamming_distance(
+ if run_manager.mgi_data()
+ && hamming_distance(
&buffer_1[header_start_pr..seq_start_pr - 2],
- &buffer_2[header_start..seq_start - 2]
- ).unwrap() > 0
+ &buffer_2[header_start..seq_start - 2],
+ )
+ .unwrap()
+ > 0
{
panic!(
"Headers seem to be in differnet orders: {} - {}",
- String::from_utf8(
- buffer_1[header_start_pr..seq_start_pr - 2].to_vec()
- ).unwrap(),
- String::from_utf8(
- buffer_2[header_start..seq_start - 2].to_vec()
- ).unwrap()
+ String::from_utf8(buffer_1[header_start_pr..seq_start_pr - 2].to_vec())
+ .unwrap(),
+ String::from_utf8(buffer_2[header_start..seq_start - 2].to_vec())
+ .unwrap()
);
}
}
@@ -1191,31 +1181,29 @@ fn process_buffer(
&buffer_2[header_start..read_end],
seq_start - header_start,
plus_start - header_start,
- qual_start - header_start
+ qual_start - header_start,
);
}
if reporting_level > 0 {
if run_manager.paired_read_input() {
// this is for r1 only if paired end
- let (qc_total, high_qc) = sum_qc(
- &buffer_1[qual_start_pr..read_end_pr],
- check_content
- );
+ let (qc_total, high_qc) =
+ sum_qc(&buffer_1[qual_start_pr..read_end_pr], check_content);
report_manager.update_stats(sample_id, 0, high_qc);
report_manager.update_stats(sample_id, 6, qc_total);
}
let (qc_total, high_qc) = sum_qc(
&buffer_2[read_end - barcode_length..read_end],
- check_content
+ check_content,
);
report_manager.update_stats(sample_id, 2, high_qc);
report_manager.update_stats(sample_id, 8, qc_total);
let (qc_total, high_qc) = sum_qc(
&buffer_2[qual_start..read_end - barcode_length],
- check_content
+ check_content,
);
report_manager.update_stats(sample_id, shift, high_qc);
report_manager.update_stats(sample_id, 6 + shift, qc_total);
@@ -1239,9 +1227,8 @@ fn process_buffer(
Some(curr_sample) => {
curr_sample.add_barcode_reads(&buffer_2[header_start..read_end + 1]);
if run_manager.paired_read_input() {
- curr_sample.add_paired_reads(
- &buffer_1[header_start_pr..read_end_pr + 1]
- );
+ curr_sample
+ .add_paired_reads(&buffer_1[header_start_pr..read_end_pr + 1]);
}
}
None => {}
@@ -1262,13 +1249,12 @@ fn process_buffer(
sep_position,
!demultiplex,
true,
- barcode_read_illumina_header_start == usize::MAX
+ barcode_read_illumina_header_start == usize::MAX,
);
if demultiplex {
curr_sample.add_barcode_reads(&curr_barcode.as_bytes());
- } else if
- sep_position == seq_start - header_start - 3 &&
- reformated_sample.barcode().len() > 0
+ } else if sep_position == seq_start - header_start - 3
+ && reformated_sample.barcode().len() > 0
{
let mut new_barcode = String::from(" ");
new_barcode.push(buffer_2[seq_start - 2] as char);
@@ -1277,18 +1263,16 @@ fn process_buffer(
curr_sample.add_barcode_reads(&new_barcode.as_bytes());
}
if barcode_read_illumina_header_start < usize::MAX {
- curr_sample.copy_header_2_paired(
- barcode_read_illumina_header_start
- );
+ curr_sample
+ .copy_header_2_paired(barcode_read_illumina_header_start);
if run_manager.paired_read_input() {
- if
- demultiplex ||
- reformated_sample.barcode().len() > 0 ||
- sep_position < seq_start - header_start - 3
+ if demultiplex
+ || reformated_sample.barcode().len() > 0
+ || sep_position < seq_start - header_start - 3
{
curr_sample.fix_paired_read_header(
tail_offset,
- buffer_1[seq_start_pr - 2 - header_shift]
+ buffer_1[seq_start_pr - 2 - header_shift],
);
}
@@ -1301,16 +1285,15 @@ fn process_buffer(
if run_manager.paired_read_input() {
curr_sample.fix_paired_read_header(
tail_offset,
- buffer_1[seq_start_pr - 2 - header_shift]
+ buffer_1[seq_start_pr - 2 - header_shift],
);
header_start_pr = seq_start_pr - 1;
}
}
header_start = seq_start - 1;
} else if run_manager.mgi_full_header() {
- curr_sample.add_barcode_reads(
- &buffer_2[header_start..seq_start - 1]
- );
+ curr_sample
+ .add_barcode_reads(&buffer_2[header_start..seq_start - 1]);
header_info = String::from(" ");
if curr_umi.len() > 0 {
header_info.push_str(&curr_umi[1..curr_umi.len()]);
@@ -1324,12 +1307,11 @@ fn process_buffer(
if run_manager.paired_read_input() {
curr_sample.add_paired_reads(
- &buffer_1[header_start_pr..seq_start_pr - 1]
+ &buffer_1[header_start_pr..seq_start_pr - 1],
);
unsafe {
- header_info.as_bytes_mut()[curr_umi.len() + 1] = buffer_1[
- seq_start_pr - 2
- ];
+ header_info.as_bytes_mut()[curr_umi.len() + 1] =
+ buffer_1[seq_start_pr - 2];
}
curr_sample.add_paired_reads(header_info.as_bytes());
header_start_pr = seq_start_pr - 1;
@@ -1337,16 +1319,15 @@ fn process_buffer(
}
curr_sample.add_barcode_reads(
- &buffer_2[header_start..plus_start - writen_barcode_length - 1]
+ &buffer_2[header_start..plus_start - writen_barcode_length - 1],
);
curr_sample.add_barcode_reads(
- &buffer_2[plus_start - 1..read_end - writen_barcode_length]
+ &buffer_2[plus_start - 1..read_end - writen_barcode_length],
);
curr_sample.add_barcode_reads(&[b'\n']);
if run_manager.paired_read_input() {
- curr_sample.add_paired_reads(
- &buffer_1[header_start_pr..read_end_pr + 1]
- );
+ curr_sample
+ .add_paired_reads(&buffer_1[header_start_pr..read_end_pr + 1]);
}
}
None => {
@@ -1370,17 +1351,15 @@ fn analyse_fastq(
sample_manager: &SampleManager,
run_manager: &RunManager,
buffer_info: &BufferInfo,
- all_template_data: Vec<
- (
- u32,
- HashSet,
- HashSet,
- String,
- HashMap)>,
- bool,
- [usize; 10],
- )
- >,
+ all_template_data: Vec<(
+ u32,
+ HashSet,
+ HashSet,
+ String,
+ HashMap)>,
+ bool,
+ [usize; 10],
+ )>,
reporting_level: usize,
allowed_mismatches: usize,
all_index_error: bool,
@@ -1392,7 +1371,7 @@ fn analyse_fastq(
samples_locks: Arc>>,
demultiplex: bool,
reformated_sample: &ReformatedSample,
- mut warnings_ls: Vec
+ mut warnings_ls: Vec,
) -> ReportManager {
let curr_thread = thread::current().name().unwrap_or("Unnamed").to_string();
info!("Thread ({}) has started.", curr_thread);
@@ -1402,7 +1381,7 @@ fn analyse_fastq(
run_manager,
buffer_info,
run_manager.read2_has_sequence(),
- run_manager.illumina_format()
+ run_manager.illumina_format(),
)
} else {
let mut sample = SampleData::new(
@@ -1410,16 +1389,20 @@ fn analyse_fastq(
run_manager.paired_read_input(),
run_manager.read2_has_sequence(),
buffer_info.clone(),
- run_manager.create_illumina_header_prefix()
+ run_manager.create_illumina_header_prefix(),
);
sample.create_files(
run_manager.lane(),
reformated_sample.sample_index(),
run_manager.illumina_format(),
run_manager.output_dir(),
- run_manager.paired_read_input()
+ run_manager.paired_read_input(),
);
- vec![sample, SampleData::minimal_sample(), SampleData::minimal_sample()]
+ vec![
+ sample,
+ SampleData::minimal_sample(),
+ SampleData::minimal_sample(),
+ ]
};
//let project_samples = get_project_samples(sample_information).unwrap();
@@ -1489,17 +1472,11 @@ fn analyse_fastq(
}
false => {
//let start = Instant::now();
- read_bytes_2 = read_buffers(
- read_bytes_2,
- &mut main_buffer_2,
- &mut reader_barcode_read
- );
+ read_bytes_2 =
+ read_buffers(read_bytes_2, &mut main_buffer_2, &mut reader_barcode_read);
if paired_input {
- read_bytes_1 = read_buffers(
- read_bytes_1,
- &mut main_buffer_1,
- &mut reader_paired_read
- );
+ read_bytes_1 =
+ read_buffers(read_bytes_1, &mut main_buffer_1, &mut reader_paired_read);
}
//reading_time += start.elapsed();
(Vec::new(), Vec::new(), Vec::new(), Vec::new())
@@ -1510,9 +1487,8 @@ fn analyse_fastq(
if read_bytes_2 == 0 && read_bytes_1 == 0 {
info!("Thread ({}) has finished.", curr_thread);
break;
- } else if
- (read_bytes_2 == 0 && read_bytes_1 != 0) ||
- (read_bytes_2 != 0 && read_bytes_1 == 0 && run_manager.paired_read_input())
+ } else if (read_bytes_2 == 0 && read_bytes_1 != 0)
+ || (read_bytes_2 != 0 && read_bytes_1 == 0 && run_manager.paired_read_input())
{
panic!("Something wrong in the input files!");
}
@@ -1556,7 +1532,7 @@ fn analyse_fastq(
},
demultiplex,
reformated_sample,
- &mut warnings_ls
+ &mut warnings_ls,
);
//read_leftover_leng_rp
@@ -1588,7 +1564,7 @@ fn analyse_fastq(
&mut main_buffer_2,
header_start,
0,
- read_bytes_2 - header_start
+ read_bytes_2 - header_start,
);
read_bytes_2 -= header_start;
header_start = 0;
@@ -1603,7 +1579,7 @@ fn analyse_fastq(
&mut main_buffer_1,
header_start_pr,
0,
- read_bytes_1 - header_start_pr
+ read_bytes_1 - header_start_pr,
);
read_bytes_1 -= header_start_pr;
header_start_pr = 0;
@@ -1642,34 +1618,97 @@ pub fn initiate_demultiplexing(demultiplex_command: &ArgMatches) {
//let arg_read2_file_name_suf: &String = demultiplex_command.get_one::("arg_read2_file_name_suf").unwrap();
//let arg_info_file: &String = demultiplex_command.get_one::("arg_info_file").unwrap();
let sample_manager = SampleManager::new(
- demultiplex_command.get_one::("arg_sample_sheet_file_path").unwrap().to_string(),
- demultiplex_command.get_one::("arg_template").unwrap().to_string(),
- demultiplex_command.get_one::("arg_i7_rc").unwrap().clone(),
- demultiplex_command.get_one::("arg_i5_rc").unwrap().clone(),
- demultiplex_command.get_one::("arg_undetermined_label").unwrap().to_string(),
- demultiplex_command.get_one::("arg_ambiguous_label").unwrap().to_string()
+ demultiplex_command
+ .get_one::("arg_sample_sheet_file_path")
+ .unwrap()
+ .to_string(),
+ demultiplex_command
+ .get_one::("arg_template")
+ .unwrap()
+ .to_string(),
+ demultiplex_command
+ .get_one::("arg_i7_rc")
+ .unwrap()
+ .clone(),
+ demultiplex_command
+ .get_one::("arg_i5_rc")
+ .unwrap()
+ .clone(),
+ demultiplex_command
+ .get_one::("arg_undetermined_label")
+ .unwrap()
+ .to_string(),
+ demultiplex_command
+ .get_one::("arg_ambiguous_label")
+ .unwrap()
+ .to_string(),
);
let mut run_manager = RunManager::new(
- demultiplex_command.get_one::("arg_input_folder_path").unwrap().to_string(),
- demultiplex_command.get_one::("arg_read1_file_path").unwrap().to_string(),
- demultiplex_command.get_one::("arg_read2_file_path").unwrap().to_string(),
- demultiplex_command.get_one::("arg_ouput_dir").unwrap().to_string(),
- demultiplex_command.get_one::("arg_report_dir").unwrap().to_string(),
- demultiplex_command.get_one::("arg_lane").unwrap().to_string(),
- demultiplex_command.get_one::("arg_instrument").unwrap().to_string(),
- demultiplex_command.get_one::("arg_run").unwrap().to_string(),
+ demultiplex_command
+ .get_one::("arg_input_folder_path")
+ .unwrap()
+ .to_string(),
+ demultiplex_command
+ .get_one::("arg_read1_file_path")
+ .unwrap()
+ .to_string(),
+ demultiplex_command
+ .get_one::("arg_read2_file_path")
+ .unwrap()
+ .to_string(),
+ demultiplex_command
+ .get_one::("arg_ouput_dir")
+ .unwrap()
+ .to_string(),
+ demultiplex_command
+ .get_one::("arg_report_dir")
+ .unwrap()
+ .to_string(),
+ demultiplex_command
+ .get_one::("arg_lane")
+ .unwrap()
+ .to_string(),
+ demultiplex_command
+ .get_one::("arg_instrument")
+ .unwrap()
+ .to_string(),
+ demultiplex_command
+ .get_one::("arg_run")
+ .unwrap()
+ .to_string(),
!*demultiplex_command.get_one::("arg_not_mgi").unwrap(),
*demultiplex_command.get_one::("arg_force").unwrap(),
- demultiplex_command.get_one::("arg_read1_file_name_suf").unwrap().to_string(),
- demultiplex_command.get_one::("arg_read2_file_name_suf").unwrap().to_string(),
- demultiplex_command.get_one::("arg_info_file").unwrap().to_string(),
- !*demultiplex_command.get_one::("arg_disable_illumina_format").unwrap(),
- *demultiplex_command.get_one::("arg_keep_barcode").unwrap(),
- *demultiplex_command.get_one::("arg_comprehensive_scan").unwrap(),
- *demultiplex_command.get_one::("arg_ignore_undetermined").unwrap(),
- *demultiplex_command.get_one::("arg_check_content").unwrap(),
- *demultiplex_command.get_one::("arg_mgi_full_header").unwrap()
+ demultiplex_command
+ .get_one::("arg_read1_file_name_suf")
+ .unwrap()
+ .to_string(),
+ demultiplex_command
+ .get_one::("arg_read2_file_name_suf")
+ .unwrap()
+ .to_string(),
+ demultiplex_command
+ .get_one::("arg_info_file")
+ .unwrap()
+ .to_string(),
+ !*demultiplex_command
+ .get_one::("arg_disable_illumina_format")
+ .unwrap(),
+ *demultiplex_command
+ .get_one::("arg_keep_barcode")
+ .unwrap(),
+ *demultiplex_command
+ .get_one::("arg_comprehensive_scan")
+ .unwrap(),
+ *demultiplex_command
+ .get_one::("arg_ignore_undetermined")
+ .unwrap(),
+ *demultiplex_command
+ .get_one::("arg_check_content")
+ .unwrap(),
+ *demultiplex_command
+ .get_one::("arg_mgi_full_header")
+ .unwrap(),
);
let (barcode_read_info, paired_read_info) = run_manager.get_read_information();
if run_manager.lane().len() == 0 {
@@ -1701,54 +1740,63 @@ pub fn initiate_demultiplexing(demultiplex_command: &ArgMatches) {
let mut buffer_info = BufferInfo::new(
writing_buffer_size,
compression_buffer_size,
- *demultiplex_command.get_one::("arg_compression_level").unwrap(),
+ *demultiplex_command
+ .get_one::("arg_compression_level")
+ .unwrap(),
compression_buffer_size - barcode_read_info.read_length() - paired_read_info.read_length(),
- writing_buffer_size
+ writing_buffer_size,
);
let arg_memory: f64 = *demultiplex_command.get_one::("arg_memory").unwrap();
if 0.0 < arg_memory && arg_memory <= 0.5 {
panic!("Requested memory should be greater than 0.5 GB!");
}
-
- let mut tmp_reader_threads = *demultiplex_command.get_one::("arg_threads_r").unwrap();
- let mut tmp_processing_threads = *demultiplex_command.get_one::("arg_threads_w").unwrap();
-
- let (reader_threads, processing_threads) = if tmp_processing_threads > 0 && tmp_reader_threads > 0 {
- if run_manager.paired_read_input(){
+
+ let mut tmp_reader_threads = *demultiplex_command
+ .get_one::("arg_threads_r")
+ .unwrap();
+
+ let mut tmp_processing_threads = *demultiplex_command
+ .get_one::("arg_threads_w")
+ .unwrap();
+
+ let (reader_threads, processing_threads) = if tmp_processing_threads > 0
+ && tmp_reader_threads > 0
+ {
+ if run_manager.paired_read_input() {
if tmp_reader_threads > 4 {
- warn!("Reader threads can not be more than 4!");
+ warn!("Reader threads can not be more than 4! extra threads will be used for processing!");
+ tmp_processing_threads += tmp_reader_threads - 4;
tmp_reader_threads = 4;
- }else if tmp_reader_threads == 3 {
- warn!("Reader threads shoyuld be either 0, 1, 2, or 4!");
+ } else if tmp_reader_threads == 3 {
+ warn!("Reader threads shoyuld be either 0, 1, 2, or 4! extra threads will be used for processing!");
tmp_reader_threads = 2;
+ tmp_processing_threads += 1;
}
- }else{
- if tmp_reader_threads > 2{
- warn!("Reader threads can not be more than 2 for single end input!");
+ } else {
+ if tmp_reader_threads > 2 {
+ warn!("Reader threads can not be more than 2 for single end input!, extra threads will be used for processing!");
+ tmp_processing_threads += tmp_reader_threads - 2;
tmp_reader_threads = 2;
}
}
- info!("Reader threads ({}) and processing threads ({}) will be used!",
- tmp_reader_threads,
- tmp_processing_threads
+ info!(
+ "Reader threads ({}) and processing threads ({}) will be used!",
+ tmp_reader_threads, tmp_processing_threads
);
(tmp_reader_threads, tmp_processing_threads)
- }else
- {
+ } else {
get_cpus(
- *demultiplex_command.get_one::("arg_threads").unwrap(),
- run_manager.paired_read_input()
+ *demultiplex_command.get_one::("arg_threads").unwrap(),
+ run_manager.paired_read_input(),
)
};
-
-
let max_buffer_size = calculate_largest_buffer_size(
get_available_memory(arg_memory),
sample_manager.get_sample_count(),
buffer_info.compression_buffer_size(),
!run_manager.paired_read_input(),
- processing_threads
+ processing_threads,
);
buffer_info.calculate_final_writing_buffer_size(max_buffer_size);
@@ -1766,31 +1814,33 @@ pub fn initiate_demultiplexing(demultiplex_command: &ArgMatches) {
.get_one::("arg_all_index_error")
.unwrap();
- match
- demultiplex(
- &sample_manager,
- &run_manager,
- &buffer_info,
- *arg_allowed_mismatches,
- *arg_report_level,
- *arg_all_index_error,
- reader_threads,
- processing_threads
- )
- {
+ match demultiplex(
+ &sample_manager,
+ &run_manager,
+ &buffer_info,
+ *arg_allowed_mismatches,
+ *arg_report_level,
+ *arg_all_index_error,
+ reader_threads,
+ processing_threads,
+ ) {
Ok(mut report_manager) => {
let max_mismatches = if *arg_all_index_error {
*arg_allowed_mismatches + 1
} else {
*arg_allowed_mismatches * 2 + 1
};
- let shift = if run_manager.paired_read_input() { 1 } else { 0 };
+ let shift = if run_manager.paired_read_input() {
+ 1
+ } else {
+ 0
+ };
report_manager.prepare_final_data(
max_mismatches,
shift,
&barcode_read_info,
&paired_read_info,
- barcode_length
+ barcode_length,
);
report_manager.write_reports(
&run_manager,
@@ -1798,7 +1848,7 @@ pub fn initiate_demultiplexing(demultiplex_command: &ArgMatches) {
*arg_report_level,
*arg_report_limit,
max_mismatches,
- usize::MAX
+ usize::MAX,
);
}
Err(err) => eprintln!("Error: {}", err),
@@ -1809,7 +1859,7 @@ pub fn merge_qc_reports(
qc_report_paths: &[String],
output_dir: &String,
lane: &String,
- project: &String
+ project: &String,
) {
if qc_report_paths.len() == 0 {
panic!("report directories are not provided!");
@@ -1818,7 +1868,11 @@ pub fn merge_qc_reports(
if output_dir.len() == 0 {
info!("output will be written to the current work directory!");
}
- let used_lane = if lane.len() == 0 { String::from("all") } else { lane.to_string() };
+ let used_lane = if lane.len() == 0 {
+ String::from("all")
+ } else {
+ lane.to_string()
+ };
let mut project_id: String;
let mut sample_id: String;
@@ -1855,10 +1909,7 @@ pub fn merge_qc_reports(
continue;
}
//println!("{}", line);
- vals = line
- .split("\t")
- .map(|x| x.to_string())
- .collect();
+ vals = line.split("\t").map(|x| x.to_string()).collect();
project_id = vals.remove(0);
sample_id = vals.remove(0);
sample_index = sample_manager.get_sample_index(&sample_id);
@@ -1882,7 +1933,14 @@ pub fn merge_qc_reports(
}
report_manager.truncate_mismatches(max_mismatches);
- report_manager.write_reports(&run_manager, &sample_manager, 2, 0, max_mismatches, usize::MAX);
+ report_manager.write_reports(
+ &run_manager,
+ &sample_manager,
+ 2,
+ 0,
+ max_mismatches,
+ usize::MAX,
+ );
//for (sample_id, val) in map.iter_mut() { }
}
@@ -1890,7 +1948,10 @@ pub fn merge_qc_reports(
pub fn detect_template(template_command: &ArgMatches) {
let start = Instant::now();
let sample_manager = SampleManager::from_simple_sheet(
- template_command.get_one::("arg_sample_sheet_file_path").unwrap().to_string()
+ template_command
+ .get_one::("arg_sample_sheet_file_path")
+ .unwrap()
+ .to_string(),
);
let path = Path::new(template_command.get_one::("arg_ouput_dir").unwrap());
@@ -1901,15 +1962,21 @@ pub fn detect_template(template_command: &ArgMatches) {
.unwrap_or("")
.to_string();
- let parent_path = path
- .parent()
- .and_then(|p| p.to_str())
- .unwrap_or("");
+ let parent_path = path.parent().and_then(|p| p.to_str()).unwrap_or("");
let mut run_manager = RunManager::new(
- template_command.get_one::("arg_input_folder_path").unwrap().to_string(),
- template_command.get_one::("arg_read1_file_path").unwrap().to_string(),
- template_command.get_one::("arg_read2_file_path").unwrap().to_string(),
+ template_command
+ .get_one::("arg_input_folder_path")
+ .unwrap()
+ .to_string(),
+ template_command
+ .get_one::("arg_read1_file_path")
+ .unwrap()
+ .to_string(),
+ template_command
+ .get_one::("arg_read2_file_path")
+ .unwrap()
+ .to_string(),
parent_path.to_string(),
String::new(),
String::new(),
@@ -1917,22 +1984,35 @@ pub fn detect_template(template_command: &ArgMatches) {
String::new(),
true,
true,
- template_command.get_one::("arg_read1_file_name_suf").unwrap().to_string(),
- template_command.get_one::("arg_read2_file_name_suf").unwrap().to_string(),
- template_command.get_one::("arg_info_file").unwrap().to_string(),
+ template_command
+ .get_one::("arg_read1_file_name_suf")
+ .unwrap()
+ .to_string(),
+ template_command
+ .get_one::("arg_read2_file_name_suf")
+ .unwrap()
+ .to_string(),
+ template_command
+ .get_one::("arg_info_file")
+ .unwrap()
+ .to_string(),
true,
false,
true,
true,
true,
- false
+ false,
);
let input_barcode_length: usize = *template_command
.get_one::("arg_barcode_length")
.unwrap();
- let testing_reads: usize = *template_command.get_one::("arg_testing_reads").unwrap();
- let max_umi_length: usize = *template_command.get_one::("arg_max_umi_length").unwrap();
+ let testing_reads: usize = *template_command
+ .get_one::("arg_testing_reads")
+ .unwrap();
+ let max_umi_length: usize = *template_command
+ .get_one::("arg_max_umi_length")
+ .unwrap();
let use_popular_template: bool = *template_command
.get_one::("arg_popular_template")
.unwrap();
@@ -1947,11 +2027,10 @@ pub fn detect_template(template_command: &ArgMatches) {
} else {
barcode_length = barcode_read_info.sequence_length() - paired_read_info.sequence_length();
info!("Barcode length is calculated as the difference between R2 length and R1.");
- let max_barcode_length: usize = match
- sample_indexes
- .iter()
- .map(|it| it[0].len() + it[2].len())
- .max()
+ let max_barcode_length: usize = match sample_indexes
+ .iter()
+ .map(|it| it[0].len() + it[2].len())
+ .max()
{
Some(tmp_max) => tmp_max,
None => panic!("Sample sheet should have samples!"),
@@ -1987,10 +2066,9 @@ pub fn detect_template(template_command: &ArgMatches) {
}
read_barcode_seq = String::new();
barcode_reader.read_line(&mut read_barcode_seq).unwrap();
- read_barcode_seq =
- read_barcode_seq[
- read_barcode_seq.len() - barcode_length - 1..read_barcode_seq.len() - 1
- ].to_string();
+ read_barcode_seq = read_barcode_seq
+ [read_barcode_seq.len() - barcode_length - 1..read_barcode_seq.len() - 1]
+ .to_string();
//println!(" {} Barcode seq: {}", read_cntr, read_barcode_seq);
for sample_itr in 0..sample_indexes.len() {
@@ -2007,7 +2085,7 @@ pub fn detect_template(template_command: &ArgMatches) {
&sample_i7,
&sample_i7_rc,
&sample_i5,
- &sample_i5_rc
+ &sample_i5_rc,
);
}
@@ -2023,10 +2101,8 @@ pub fn detect_template(template_command: &ArgMatches) {
sample_reads_final.push(Vec::new());
for (template_str, sample_reads) in &matches_stat {
if sample_reads[sample_itr] > 0 {
- sample_reads_final[sample_itr].push((
- template_str.to_string(),
- sample_reads[sample_itr],
- ));
+ sample_reads_final[sample_itr]
+ .push((template_str.to_string(), sample_reads[sample_itr]));
no_matches = false;
}
}
@@ -2061,7 +2137,7 @@ pub fn detect_template(template_command: &ArgMatches) {
add_umi,
barcode_length,
sample_indexes[0][0].len(),
- sample_indexes[0][2].len()
+ sample_indexes[0][2].len(),
);
info!(
"The most frequent template is {} - Appeared {} times!",
@@ -2088,9 +2164,8 @@ pub fn detect_template(template_command: &ArgMatches) {
out_str.push('\t');
if use_popular_template {
- if
- sample_reads_final[sample_itr].len() > 0 &&
- popular_template != sample_reads_final[sample_itr][0].0
+ if sample_reads_final[sample_itr].len() > 0
+ && popular_template != sample_reads_final[sample_itr][0].0
{
info!(
"Using popular template for sample {} instead of its detected template {}!",
@@ -2109,7 +2184,7 @@ pub fn detect_template(template_command: &ArgMatches) {
add_umi,
barcode_length,
sample_information[sample_itr][I7_COLUMN].len(),
- sample_information[sample_itr][I5_COLUMN].len()
+ sample_information[sample_itr][I5_COLUMN].len(),
);
out_str.push_str(&template_info.0);
out_str.push('\t');
@@ -2118,20 +2193,18 @@ pub fn detect_template(template_command: &ArgMatches) {
out_str.push_str(&template_info.2);
} else {
let template_info = match sample_reads_final[sample_itr].len() > 0 {
- true =>
- get_mgikit_template(
- &sample_reads_final[sample_itr][0].0,
- add_umi,
- barcode_length,
- sample_indexes[sample_itr][0].len(),
- sample_indexes[sample_itr][2].len()
- ),
- false =>
- (
- String::from("No matches with this sample"),
- String::from("."),
- String::from("."),
- ),
+ true => get_mgikit_template(
+ &sample_reads_final[sample_itr][0].0,
+ add_umi,
+ barcode_length,
+ sample_indexes[sample_itr][0].len(),
+ sample_indexes[sample_itr][2].len(),
+ ),
+ false => (
+ String::from("No matches with this sample"),
+ String::from("."),
+ String::from("."),
+ ),
};
out_str.push_str(&template_info.0);
out_str.push('\t');
@@ -2165,45 +2238,87 @@ pub fn detect_template(template_command: &ArgMatches) {
}
write_file(
- &format!("{}/{}_template.tsv", run_manager.output_dir().display(), output_file),
- &out_str
+ &format!(
+ "{}/{}_template.tsv",
+ run_manager.output_dir().display(),
+ output_file
+ ),
+ &out_str,
);
write_file(
- &format!("{}/{}_details.tsv", run_manager.output_dir().display(), output_file),
- &out_str_full
+ &format!(
+ "{}/{}_details.tsv",
+ run_manager.output_dir().display(),
+ output_file
+ ),
+ &out_str_full,
);
let dur = start.elapsed();
- info!("{} reads were processed in {} secs.", read_cntr, dur.as_secs());
+ info!(
+ "{} reads were processed in {} secs.",
+ read_cntr,
+ dur.as_secs()
+ );
}
pub fn reformat(reformat_command: &ArgMatches) -> Result<(), Box> {
- let reporting_level: usize = *reformat_command.get_one::("arg_report_level").unwrap();
+ let reporting_level: usize = *reformat_command
+ .get_one::("arg_report_level")
+ .unwrap();
let start = Instant::now();
let dur: std::time::Duration;
let mut run_manager = RunManager::new(
String::new(),
- reformat_command.get_one::("arg_read1_file_path").unwrap().to_string(),
- reformat_command.get_one::("arg_read2_file_path").unwrap().to_string(),
- reformat_command.get_one::("arg_ouput_dir").unwrap().to_string(),
- reformat_command.get_one::("arg_report_dir").unwrap().to_string(),
- reformat_command.get_one::("arg_lane").unwrap().to_string(),
- reformat_command.get_one::("arg_instrument").unwrap().to_string(),
- reformat_command.get_one::("arg_run").unwrap().to_string(),
+ reformat_command
+ .get_one::("arg_read1_file_path")
+ .unwrap()
+ .to_string(),
+ reformat_command
+ .get_one::("arg_read2_file_path")
+ .unwrap()
+ .to_string(),
+ reformat_command
+ .get_one::("arg_ouput_dir")
+ .unwrap()
+ .to_string(),
+ reformat_command
+ .get_one::("arg_report_dir")
+ .unwrap()
+ .to_string(),
+ reformat_command
+ .get_one::("arg_lane")
+ .unwrap()
+ .to_string(),
+ reformat_command
+ .get_one::("arg_instrument")
+ .unwrap()
+ .to_string(),
+ reformat_command
+ .get_one::("arg_run")
+ .unwrap()
+ .to_string(),
true,
*reformat_command.get_one::("arg_force").unwrap(),
String::new(),
String::new(),
- reformat_command.get_one::("arg_info_file").unwrap().to_string(),
- !*reformat_command.get_one::("arg_disable_illumina_format").unwrap(),
+ reformat_command
+ .get_one::("arg_info_file")
+ .unwrap()
+ .to_string(),
+ !*reformat_command
+ .get_one::("arg_disable_illumina_format")
+ .unwrap(),
+ false,
false,
false,
+ *reformat_command
+ .get_one::("arg_check_content")
+ .unwrap(),
false,
- *reformat_command.get_one::("arg_check_content").unwrap(),
- false
);
let (barcode_read_info, paired_read_info) = run_manager.get_read_information();
if run_manager.lane().len() == 0 {
@@ -2220,7 +2335,12 @@ pub fn reformat(reformat_command: &ArgMatches) -> Result<(), Box> {
.unwrap();
let (sample_label_sb, _, sb_lane, _) = parse_sb_file_name(
- &run_manager.barcode_reads().file_stem().unwrap().to_string_lossy().to_string()
+ &run_manager
+ .barcode_reads()
+ .file_stem()
+ .unwrap()
+ .to_string_lossy()
+ .to_string(),
);
let mut sample_label = reformat_command
@@ -2244,17 +2364,24 @@ pub fn reformat(reformat_command: &ArgMatches) -> Result<(), Box> {
}
let reformated_sample = ReformatedSample::new(
sample_label.clone(),
- *reformat_command.get_one::("arg_sample_index").unwrap(),
+ *reformat_command
+ .get_one::("arg_sample_index")
+ .unwrap(),
*reformat_command.get_one::("arg_umi_length").unwrap(),
- reformat_command.get_one::("arg_barcode").unwrap().to_string()
+ reformat_command
+ .get_one::("arg_barcode")
+ .unwrap()
+ .to_string(),
);
let mut buffer_info = BufferInfo::new(
writing_buffer_size,
compression_buffer_size,
- *reformat_command.get_one::("arg_compression_level").unwrap(),
+ *reformat_command
+ .get_one::("arg_compression_level")
+ .unwrap(),
compression_buffer_size - barcode_read_info.read_length() - paired_read_info.read_length(),
- writing_buffer_size
+ writing_buffer_size,
);
let arg_memory: f64 = *reformat_command.get_one::("arg_memory").unwrap();
@@ -2266,7 +2393,7 @@ pub fn reformat(reformat_command: &ArgMatches) -> Result<(), Box> {
1,
buffer_info.compression_buffer_size(),
!run_manager.paired_read_input(),
- 1
+ 1,
);
buffer_info.calculate_final_writing_buffer_size(max_buffer_size);
@@ -2276,14 +2403,14 @@ pub fn reformat(reformat_command: &ArgMatches) -> Result<(), Box> {
run_manager.paired_read_input(),
run_manager.read2_has_sequence(),
buffer_info.clone(),
- run_manager.create_illumina_header_prefix()
+ run_manager.create_illumina_header_prefix(),
);
sample.create_files(
run_manager.lane(),
reformated_sample.sample_index(),
run_manager.illumina_format(),
run_manager.output_dir(),
- run_manager.paired_read_input()
+ run_manager.paired_read_input(),
);
sample.delete_sample_files();
}
@@ -2310,7 +2437,7 @@ pub fn reformat(reformat_command: &ArgMatches) -> Result<(), Box> {
samples_locks_arc,
false,
&reformated_sample,
- vec![true; 10]
+ vec![true; 10],
);
if reporting_level > 0 {
@@ -2323,11 +2450,15 @@ pub fn reformat(reformat_command: &ArgMatches) -> Result<(), Box> {
},
&barcode_read_info,
&paired_read_info,
- reformated_sample.umi_length()
+ reformated_sample.umi_length(),
);
}
report_manager.write_reports(&run_manager, &sample_manager, reporting_level, 0, 5, 0);
dur = start.elapsed();
- info!("{} reads were processed in {} secs.", report_manager.get_total_reads(), dur.as_secs());
+ info!(
+ "{} reads were processed in {} secs.",
+ report_manager.get_total_reads(),
+ dur.as_secs()
+ );
Ok(())
}
diff --git a/src/main.rs b/src/main.rs
index 1828148..94c2769 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -2,24 +2,24 @@
#![doc(html_logo_url = "assets/SAGC-logo-hover.png")]
#![doc(issue_tracker_base_url = "https://github.com/sagc-bioinformatics/mgikit/issues")]
-use std::time::Instant;
-use std::str;
use chrono;
+use clap::{Arg, ArgAction, Command};
+use env_logger::{Builder, Target};
+use log::{error, info, warn, LevelFilter};
use mgikit::*;
-use termion::terminal_size;
-use clap::{ ArgAction, Command, Arg };
-use log::{ warn, info, error, LevelFilter };
use std::env;
-use env_logger::{ Builder, Target };
+use std::str;
use std::str::FromStr;
+use std::time::Instant;
+use termion::terminal_size;
const VERSION: &str = env!("CARGO_PKG_VERSION");
fn print_logo() {
//let (width, _) = terminal_size().unwrap();
let (width, _) = match terminal_size() {
- Ok(value) => { value }
- Err(_) => { (80, 0) }
+ Ok(value) => value,
+ Err(_) => (80, 0),
};
//let width = 80;
if width > 43 {
@@ -737,9 +737,8 @@ fn main() {
initiate_demultiplexing(demultiplex_command);
}
Some(("report", report_command)) => {
- let arg_ouput_dir: &String = report_command
- .get_one::("arg_ouput_dir")
- .unwrap();
+ let arg_ouput_dir: &String =
+ report_command.get_one::("arg_ouput_dir").unwrap();
let arg_qc_report_path: Vec = report_command
.get_many::("arg_qc_report_path")
.unwrap()
diff --git a/src/report_manager.rs b/src/report_manager.rs
index e9ee187..48f47ff 100644
--- a/src/report_manager.rs
+++ b/src/report_manager.rs
@@ -1,13 +1,13 @@
+use crate::file_utils::{create_output_file, get_buf_writer};
+use crate::{run_manager::ReadInfo, variables::*, RunManager, SampleManager};
+use log::{error, info};
use std::collections::HashMap;
-use log::{ info, error };
-use std::time::Instant;
use std::collections::HashSet;
-use crate::{ run_manager::ReadInfo, RunManager, SampleManager, variables::* };
-use crate::file_utils::{ get_buf_writer, create_output_file };
-use std::path::PathBuf;
-use std::io::Write;
-use std::io::BufWriter;
use std::fs::File;
+use std::io::BufWriter;
+use std::io::Write;
+use std::path::PathBuf;
+use std::time::Instant;
pub fn write_general_info_report(
sample_information: &Vec>,
@@ -16,7 +16,7 @@ pub fn write_general_info_report(
run: &String,
lane: &String,
output_file: &PathBuf,
- execluded_samples: &Vec
+ execluded_samples: &Vec,
) {
//Mb Total Yield: total bases as cnt of r1 + r2
//M Total Clusters: number of reads
@@ -25,22 +25,22 @@ pub fn write_general_info_report(
//% Perfect Index: index with 0 mismatches / all reads.
/*
- sample_statistics[sample_id]:
- 0: r1 count of qs > 30
- 1: r2 count os qs > 30
- 2: r3 count of (barcode) with qs > 30
-
- 3: r1 count of bases
- 4: r2 count of bases
- 5: r3 count of bases
-
- 6: qs of r1
- 7: qs of r2
- 8: qs of r3
- 9: read count
- from 10 to whatever mismatches allowed: is the number oif reads with the iteration mismatch
-
- */
+ sample_statistics[sample_id]:
+ 0: r1 count of qs > 30
+ 1: r2 count os qs > 30
+ 2: r3 count of (barcode) with qs > 30
+
+ 3: r1 count of bases
+ 4: r2 count of bases
+ 5: r3 count of bases
+
+ 6: qs of r1
+ 7: qs of r2
+ 8: qs of r3
+ 9: read count
+ from 10 to whatever mismatches allowed: is the number oif reads with the iteration mismatch
+
+ */
let mut out_str = String::from(
"#sample general info\nSample ID\tM Clusters\tMb Yield ≥ Q30\t% R1 Yield ≥ Q30\t% R2 Yield ≥ Q30\t% R3 Yield ≥ Q30\t% Perfect Index\n"
);
@@ -64,13 +64,17 @@ pub fn write_general_info_report(
}
if unique_sample_ids.len() == sample_information.len() {
- sample_statistics_copy.sort_by(|a, b|
- b[9].cmp(&a[9]).then_with(|| a.last().unwrap().cmp(&b.last().unwrap()))
- );
+ sample_statistics_copy.sort_by(|a, b| {
+ b[9].cmp(&a[9])
+ .then_with(|| a.last().unwrap().cmp(&b.last().unwrap()))
+ });
}
for sample_id_itr in 0..sample_statistics_copy.len() {
- sample_id = sample_statistics_copy[sample_id_itr].last().unwrap().clone() as usize;
+ sample_id = sample_statistics_copy[sample_id_itr]
+ .last()
+ .unwrap()
+ .clone() as usize;
if kept_samples.len() > 0 && !kept_samples.contains(&sample_id) {
continue;
@@ -169,13 +173,17 @@ pub fn write_general_info_report(
final_out_str.push_str(&format!("{}", lane_statistics[1] / 1000000.0));
final_out_str.push('\t');
- final_out_str.push_str(&format!("{:.3?}", (lane_statistics[2] / lane_statistics[0]) * 100.0));
+ final_out_str.push_str(&format!(
+ "{:.3?}",
+ (lane_statistics[2] / lane_statistics[0]) * 100.0
+ ));
final_out_str.push('\t');
final_out_str.push_str(&format!("{:.3?}", lane_statistics[3] / lane_statistics[0]));
final_out_str.push('\t');
- final_out_str.push_str(
- &format!("{:.3?}", (lane_statistics[4] / (lane_statistics[1] - execluded_reads)) * 100.0)
- );
+ final_out_str.push_str(&format!(
+ "{:.3?}",
+ (lane_statistics[4] / (lane_statistics[1] - execluded_reads)) * 100.0
+ ));
final_out_str.push('\n');
final_out_str.push_str(&out_str);
@@ -190,7 +198,7 @@ pub fn write_index_info_report(
kept_samples: &Vec,
max_mismatches: usize,
output_file: &PathBuf,
- execluded_samples: &Vec
+ execluded_samples: &Vec,
) {
let mut report_str = String::from("sample");
@@ -212,9 +220,10 @@ pub fn write_index_info_report(
}
if unique_sample_ids.len() == sample_information.len() {
- sample_mismatches_copy.sort_by(|a, b|
- b[0].cmp(&a[0]).then_with(|| a.last().unwrap().cmp(&b.last().unwrap()))
- );
+ sample_mismatches_copy.sort_by(|a, b| {
+ b[0].cmp(&a[0])
+ .then_with(|| a.last().unwrap().cmp(&b.last().unwrap()))
+ });
}
//sample_mismatches_copy.sort_by(|a, b| b[0].cmp(&a[0]));
for sample_id_itr in 0..sample_mismatches_copy.len() {
@@ -263,17 +272,17 @@ impl ReportManager {
0: r1 count of qs > 30
1: r2 count os qs > 30
2: r3 count of (barcode) with qs > 30
-
+
3: r1 count of bases
4: r2 count of bases
5: r3 count of bases
-
+
6: qs of r1
7: qs of r2
8: qs of r3
9: read count
from 10 to whatever mismatches allowed: is the number oif reads with the iteration mismatch
-
+
*/
}
Self {
@@ -315,7 +324,8 @@ impl ReportManager {
increment
);
debug!("update mismatches: {:?}", &self.sample_mismatches[sample_id]);
- */ self.sample_mismatches[sample_id][index] += increment;
+ */
+ self.sample_mismatches[sample_id][index] += increment;
}
pub fn update_undetermined(&mut self, curr_barcode: String, increment: u64) {
@@ -430,7 +440,7 @@ impl ReportManager {
shift: usize,
barcode_read_info: &ReadInfo,
paired_read_info: &ReadInfo,
- barcode_length: usize
+ barcode_length: usize,
) {
//let max_mismatches = if all_index_error {allowed_mismatches + 1} else {allowed_mismatches * 2 + 1};
for sample_id in 0..self.total_samples {
@@ -438,8 +448,8 @@ impl ReportManager {
(*paired_read_info.sequence_length() as u64) * self.sample_mismatches[sample_id][0];
self.sample_statistics[sample_id][6] -= self.sample_statistics[sample_id][3] * 33;
self.sample_statistics[sample_id][shift + 3] =
- ((barcode_read_info.sequence_length() - barcode_length) as u64) *
- self.sample_mismatches[sample_id][0];
+ ((barcode_read_info.sequence_length() - barcode_length) as u64)
+ * self.sample_mismatches[sample_id][0];
self.sample_statistics[sample_id][5] =
(barcode_length as u64) * self.sample_mismatches[sample_id][0];
self.sample_statistics[sample_id][6 + shift] -=
@@ -459,7 +469,7 @@ impl ReportManager {
reporting_level: usize,
report_limit: usize,
mut max_mismatches: usize,
- individual_sample: usize
+ individual_sample: usize,
) {
let start_logs = Instant::now();
let mut sample_stats_width: usize = self.sample_statistics[0].len();
@@ -531,7 +541,7 @@ impl ReportManager {
samples,
max_mismatches,
&out_file,
- &execluded_samples
+ &execluded_samples,
);
//Finish writing info report
@@ -548,7 +558,7 @@ impl ReportManager {
&run_manager.flowcell(),
&run_manager.lane(),
&out_file,
- &execluded_samples
+ &execluded_samples,
);
}
//start writing general report
@@ -579,7 +589,10 @@ impl ReportManager {
}
}
outfile = create_output_file(
- &run_manager.report_dir().clone().join(format!("{}sample_stats", &report_path_main))
+ &run_manager
+ .report_dir()
+ .clone()
+ .join(format!("{}sample_stats", &report_path_main)),
);
outfile.write_all(&out_str.as_bytes()).unwrap();
}
@@ -593,7 +606,7 @@ impl ReportManager {
&run_manager
.report_dir()
.clone()
- .join(format!("{}ambiguous_barcode", &report_path_main))
+ .join(format!("{}ambiguous_barcode", &report_path_main)),
);
ambiguous_barcodes_out.sort_by(|a, b| (a.1, a.0).cmp(&(b.1, b.0)).reverse());
rep_itr = 0;
@@ -611,7 +624,7 @@ impl ReportManager {
&run_manager
.report_dir()
.clone()
- .join(format!("{}ambiguous_barcode.complete", &report_path_main))
+ .join(format!("{}ambiguous_barcode.complete", &report_path_main)),
);
for barcode in &ambiguous_barcodes_out {
outfile
@@ -628,7 +641,7 @@ impl ReportManager {
&run_manager
.report_dir()
.clone()
- .join(&format!("{}undetermined_barcode", report_path_main))
+ .join(&format!("{}undetermined_barcode", report_path_main)),
);
undetermined_barcodes_out.sort_by(|a, b| (a.1, a.0).cmp(&(b.1, b.0)).reverse());
for barcode in &undetermined_barcodes_out {
@@ -641,12 +654,10 @@ impl ReportManager {
}
}
- outfile = get_buf_writer(
- &run_manager
- .report_dir()
- .clone()
- .join(&format!("{}undetermined_barcode.complete", report_path_main))
- );
+ outfile = get_buf_writer(&run_manager.report_dir().clone().join(&format!(
+ "{}undetermined_barcode.complete",
+ report_path_main
+ )));
undetermined_barcodes_out.sort_by(|a, b| (a.1, a.0).cmp(&(b.1, b.0)).reverse());
for barcode in &undetermined_barcodes_out {
outfile
@@ -658,6 +669,9 @@ impl ReportManager {
let log_dur = start_logs.elapsed();
- info!("Writing all logs and reports took {} secs.", log_dur.as_secs());
+ info!(
+ "Writing all logs and reports took {} secs.",
+ log_dur.as_secs()
+ );
}
}
diff --git a/src/sample_data.rs b/src/sample_data.rs
index 673e9ed..44e78d1 100644
--- a/src/sample_data.rs
+++ b/src/sample_data.rs
@@ -1,11 +1,11 @@
+use crate::file_utils::delete_file;
+use crate::{RunManager, SampleManager};
use getset::CopyGetters;
-use std::{ path::PathBuf, sync::Mutex };
+use libdeflater::{CompressionLvl, Compressor};
+use log::{debug, info, warn};
use std::fs::OpenOptions;
-use crate::{ RunManager, SampleManager };
-use libdeflater::{ Compressor, CompressionLvl };
-use crate::file_utils::delete_file;
-use log::{ info, warn, debug };
use std::io::Write;
+use std::{path::PathBuf, sync::Mutex};
use crate::variables::*;
@@ -28,38 +28,32 @@ impl SampleData {
paired_end: bool,
read2_has_sequence: bool,
buffer_info: BufferInfo,
- illumina_header_prefix: String
+ illumina_header_prefix: String,
) -> Self {
Self {
label,
barcode_reads: match read2_has_sequence {
false => None,
- true =>
- Some(
- SampleReads::new(
- PathBuf::new(),
- vec![0; buffer_info.reqiured_output_buffer_size()],
- vec![0; buffer_info.compression_buffer_size()],
- 0,
- 0
- )
- ),
+ true => Some(SampleReads::new(
+ PathBuf::new(),
+ vec![0; buffer_info.reqiured_output_buffer_size()],
+ vec![0; buffer_info.compression_buffer_size()],
+ 0,
+ 0,
+ )),
},
paired_reads: match paired_end {
false => None,
- true =>
- Some(
- SampleReads::new(
- PathBuf::new(),
- vec![0; buffer_info.reqiured_output_buffer_size()],
- vec![0; buffer_info.compression_buffer_size()],
- 0,
- 0
- )
- ),
+ true => Some(SampleReads::new(
+ PathBuf::new(),
+ vec![0; buffer_info.reqiured_output_buffer_size()],
+ vec![0; buffer_info.compression_buffer_size()],
+ 0,
+ 0,
+ )),
},
compressor: Compressor::new(
- CompressionLvl::new(buffer_info.compression_level() as i32).unwrap()
+ CompressionLvl::new(buffer_info.compression_level() as i32).unwrap(),
),
buffer_info,
illumina_header_prefix,
@@ -97,14 +91,14 @@ impl SampleData {
sample_indx: usize,
illumina_format: bool,
output_dir: &PathBuf,
- paired_read_input: bool
+ paired_read_input: bool,
) {
let (output_file_r1, output_file_r2) = create_output_file_name(
&self.label,
&lane,
sample_indx,
illumina_format,
- paired_read_input
+ paired_read_input,
);
//debug!("create files for sample {}: {}, {} with path {}", sample_indx, lane, illumina_format, output_dir.display());
match self.paired_reads {
@@ -140,27 +134,27 @@ impl SampleData {
pub fn barcode_read_buffer_end(&self) -> usize {
match &self.barcode_reads {
- Some(sr) => { sr.out_buffer_last }
- None => { 0 }
+ Some(sr) => sr.out_buffer_last,
+ None => 0,
}
}
pub fn paired_read_buffer_end(&self) -> usize {
match &self.paired_reads {
- Some(sr) => { sr.out_buffer_last }
- None => { 0 }
+ Some(sr) => sr.out_buffer_last,
+ None => 0,
}
}
pub fn barcode_read_compression_end(&self) -> usize {
match &self.barcode_reads {
- Some(sr) => { sr.compression_buffer_last }
- None => { 0 }
+ Some(sr) => sr.compression_buffer_last,
+ None => 0,
}
}
pub fn paired_read_compression_end(&self) -> usize {
match &self.paired_reads {
- Some(sr) => { sr.compression_buffer_last }
- None => { 0 }
+ Some(sr) => sr.compression_buffer_last,
+ None => 0,
}
}
@@ -185,10 +179,9 @@ impl SampleData {
pub fn compress_and_write(&mut self, force: bool, lock: &Mutex) {
//debug!("br_comp_end: {}, pr_comp_end: {}, br_out_end: {}, pr_out_end: {}", self.barcode_read_compression_end(), self.paired_read_compression_end(), self.barcode_read_buffer_end(), self.paired_read_buffer_end());
- if
- self.barcode_read_compression_end() >= self.buffer_info.compression_threshold() ||
- self.paired_read_compression_end() >= self.buffer_info.compression_threshold() ||
- force
+ if self.barcode_read_compression_end() >= self.buffer_info.compression_threshold()
+ || self.paired_read_compression_end() >= self.buffer_info.compression_threshold()
+ || force
{
match self.barcode_reads {
Some(ref mut sr) => {
@@ -204,10 +197,9 @@ impl SampleData {
};
}
- if
- self.barcode_read_buffer_end() >= self.buffer_info.writing_threshold() ||
- self.paired_read_buffer_end() >= self.buffer_info.writing_threshold() ||
- force
+ if self.barcode_read_buffer_end() >= self.buffer_info.writing_threshold()
+ || self.paired_read_buffer_end() >= self.buffer_info.writing_threshold()
+ || force
{
let _l = lock.lock().unwrap();
self.write();
@@ -255,7 +247,7 @@ impl SampleData {
sep_position: usize,
sb_header: bool,
barcode_read: bool,
- paired_read: bool
+ paired_read: bool,
) {
if barcode_read {
match self.barcode_reads {
@@ -269,7 +261,7 @@ impl SampleData {
umi,
l_position,
sep_position,
- sb_header
+ sb_header,
);
}
None => {}
@@ -287,7 +279,7 @@ impl SampleData {
umi,
l_position,
sep_position,
- sb_header
+ sb_header,
);
}
None => {}
@@ -297,14 +289,14 @@ impl SampleData {
pub fn get_barcode_read_compression_end(&self) -> usize {
match &self.barcode_reads {
- Some(sr) => { sr.compression_buffer_last }
- None => { usize::MAX }
+ Some(sr) => sr.compression_buffer_last,
+ None => usize::MAX,
}
}
pub fn get_paired_read_compression_end(&self) -> usize {
match &self.paired_reads {
- Some(sr) => { sr.compression_buffer_last }
- None => { usize::MAX }
+ Some(sr) => sr.compression_buffer_last,
+ None => usize::MAX,
}
}
pub fn fix_paired_read_header(&mut self, tail_offset: usize, val: u8) {
@@ -321,13 +313,11 @@ impl SampleData {
Some(ref mut pr) => {
match self.barcode_reads {
Some(ref mut br) => {
- pr.compression_buffer[
- pr.compression_buffer_last..pr.compression_buffer_last +
- br.compression_buffer_last -
- start
- ].copy_from_slice(
- &br.compression_buffer[start..br.compression_buffer_last]
- );
+ pr.compression_buffer[pr.compression_buffer_last
+ ..pr.compression_buffer_last + br.compression_buffer_last - start]
+ .copy_from_slice(
+ &br.compression_buffer[start..br.compression_buffer_last],
+ );
pr.compression_buffer_last += br.compression_buffer_last - start;
}
None => {}
@@ -352,7 +342,7 @@ impl SampleReads {
out_buffer: Vec,
compression_buffer: Vec,
out_buffer_last: usize,
- compression_buffer_last: usize
+ compression_buffer_last: usize,
) -> Self {
Self {
output_file,
@@ -363,7 +353,7 @@ impl SampleReads {
}
}
- /*
+ /*
pub fn compress_and_write(&mut self, compressor: &mut Compressor, compression_threshold: usize, writing_threshold: usize, force: bool){
if self.compression_buffer_last >= compression_threshold || force {
self.out_buffer_last = compress_buffer(&self.compression_buffer, self.compression_buffer_last, &mut self.out_buffer, self.out_buffer_last, compressor);
@@ -372,7 +362,7 @@ impl SampleReads {
if self.out_buffer_last >= writing_threshold || force {
write_data(&self.out_buffer, self.out_buffer_last, &self.output_file);
self.out_buffer_last = 0;
- }
+ }
}
*/
@@ -389,7 +379,7 @@ impl SampleReads {
self.compression_buffer_last,
&mut self.out_buffer,
self.out_buffer_last,
- compressor
+ compressor,
);
self.compression_buffer_last = 0;
}
@@ -401,9 +391,9 @@ impl SampleReads {
pub fn add_reads(&mut self, data: &[u8]) {
//debug!("adding read with length {}", data.len());
- self.compression_buffer[
- self.compression_buffer_last..self.compression_buffer_last + data.len()
- ].copy_from_slice(data);
+ self.compression_buffer
+ [self.compression_buffer_last..self.compression_buffer_last + data.len()]
+ .copy_from_slice(data);
self.compression_buffer_last += data.len();
}
}
@@ -425,14 +415,16 @@ impl BufferInfo {
compression_buffer_size: usize,
compression_level: u32,
compression_threshold: usize,
- writing_threshold: usize
+ writing_threshold: usize,
) -> Self {
info!("Output buffer size: {}", writing_buffer_size);
info!("Compression buffer size: {}", compression_buffer_size);
- info!("Compression level: {}. (0 no compression but fast, 12 best compression but slow.)", compression_level);
- let mut compressor = Compressor::new(
- CompressionLvl::new(compression_level as i32).unwrap()
+ info!(
+ "Compression level: {}. (0 no compression but fast, 12 best compression but slow.)",
+ compression_level
);
+ let mut compressor =
+ Compressor::new(CompressionLvl::new(compression_level as i32).unwrap());
if compression_buffer_size > writing_buffer_size {
panic!(
"Compression buffer size '--compression-buffer-size' should be less than Writing buffer size ('--writing-buffer-size')."
@@ -446,7 +438,10 @@ impl BufferInfo {
let reqiured_output_buffer_size =
writing_buffer_size + compressor.gzip_compress_bound(compression_buffer_size);
- debug!("Compression buffer flush threshold: {}", compression_threshold);
+ debug!(
+ "Compression buffer flush threshold: {}",
+ compression_threshold
+ );
debug!("Output buffer flush threshold: {}", writing_threshold);
Self {
@@ -461,12 +456,13 @@ impl BufferInfo {
pub fn calculate_final_writing_buffer_size(&mut self, max_buffer_size: usize) {
if self.writing_buffer_size > max_buffer_size {
- warn!("Writing buffer size has been reduced to {} as there is no enough memory!", max_buffer_size);
- self.reqiured_output_buffer_size =
- max_buffer_size +
- Compressor::new(
- CompressionLvl::new(self.compression_level as i32).unwrap()
- ).gzip_compress_bound(self.compression_buffer_size);
+ warn!(
+ "Writing buffer size has been reduced to {} as there is no enough memory!",
+ max_buffer_size
+ );
+ self.reqiured_output_buffer_size = max_buffer_size
+ + Compressor::new(CompressionLvl::new(self.compression_level as i32).unwrap())
+ .gzip_compress_bound(self.compression_buffer_size);
self.writing_threshold = max_buffer_size;
}
@@ -478,14 +474,16 @@ fn compress_buffer(
source_size: usize,
destination: &mut Vec,
destination_size: usize,
- compressor: &mut Compressor
+ compressor: &mut Compressor,
) -> usize {
//debug!("Compressing {} bytes, on {}", source_size, destination_size);
if source_size > 0 {
- return
- destination_size +
- compressor
- .gzip_compress(&source[0..source_size], &mut destination[destination_size..])
+ return destination_size
+ + compressor
+ .gzip_compress(
+ &source[0..source_size],
+ &mut destination[destination_size..],
+ )
.unwrap();
}
return destination_size;
@@ -508,7 +506,7 @@ pub fn calculate_reqiured_memory(
total_samples: usize,
writing_buffer_size: usize,
compression_buffer_size: usize,
- single_read_input: bool
+ single_read_input: bool,
) -> f64 {
if single_read_input {
(total_samples * (writing_buffer_size + 2 * compression_buffer_size)) as f64
@@ -522,18 +520,16 @@ pub fn calculate_largest_buffer_size(
mut total_samples: usize,
compression_buffer_size: usize,
single_end: bool,
- processors: usize
+ processors: usize,
) -> usize {
if single_end {
total_samples *= 2;
}
(2_usize).pow(
- (
- available_memory / ((total_samples * processors) as f64) -
- 2_f64 * (compression_buffer_size as f64)
- )
+ (available_memory / ((total_samples * processors) as f64)
+ - 2_f64 * (compression_buffer_size as f64))
.log2()
- .floor() as u32
+ .floor() as u32,
)
}
@@ -542,10 +538,9 @@ fn create_output_file_name(
lane: &String,
sample_index: usize,
illumina_format: bool,
- paired_read_input: bool
+ paired_read_input: bool,
) -> (String, String) {
-
- let br_suff = if paired_read_input{"R2"}else{"R1"};
+ let br_suff = if paired_read_input { "R2" } else { "R1" };
if illumina_format {
if sample_index == usize::MAX {
@@ -556,7 +551,10 @@ fn create_output_file_name(
} else {
return (
format!("{}_S{}_{}_R1_001.fastq.gz", sample_name, sample_index, lane),
- format!("{}_S{}_{}_{}_001.fastq.gz", sample_name, sample_index, lane, br_suff),
+ format!(
+ "{}_S{}_{}_{}_001.fastq.gz",
+ sample_name, sample_index, lane, br_suff
+ ),
);
}
} else {
@@ -575,7 +573,7 @@ fn write_illumina_header(
umi: &[u8],
l_position: usize,
sep_position: usize,
- sb_header: bool
+ sb_header: bool,
) -> usize {
output_buffer[buffer_end] = mgi_read_header[l_position + 1];
buffer_end += 1;
@@ -585,9 +583,8 @@ fn write_illumina_header(
for i in l_position + 10..sep_position {
if mgi_read_header[i] != b'0' {
- output_buffer[buffer_end..buffer_end + sep_position - i].copy_from_slice(
- &mgi_read_header[i..sep_position]
- );
+ output_buffer[buffer_end..buffer_end + sep_position - i]
+ .copy_from_slice(&mgi_read_header[i..sep_position]);
buffer_end += sep_position - i;
break;
}
@@ -635,9 +632,8 @@ fn write_illumina_header(
output_buffer[buffer_end..buffer_end + umi.len()].copy_from_slice(&umi[..]);
buffer_end += umi.len();
}
- output_buffer[
- buffer_end..buffer_end + mgi_read_header.len() - sep_position - 1
- ].copy_from_slice(&mgi_read_header[sep_position..mgi_read_header.len() - 1]);
+ output_buffer[buffer_end..buffer_end + mgi_read_header.len() - sep_position - 1]
+ .copy_from_slice(&mgi_read_header[sep_position..mgi_read_header.len() - 1]);
buffer_end += mgi_read_header.len() - sep_position - 1;
} else {
@@ -662,7 +658,7 @@ pub fn create_sample_data_list(
run_manager: &RunManager,
buffer_info: &BufferInfo,
read2_has_sequence: bool,
- illumina_format: bool
+ illumina_format: bool,
) -> Vec {
let mut sample_data_list: Vec = Vec::new();
let writing_samples = sample_manager.writing_samples();
@@ -686,7 +682,7 @@ pub fn create_sample_data_list(
run_manager.paired_read_input(),
read2_has_sequence || i >= undetermined_label_id,
buffer_info.clone(),
- illumina_header.clone()
+ illumina_header.clone(),
);
if i < undetermined_label_id && illumina_format {
sample_data.create_files(
@@ -694,7 +690,7 @@ pub fn create_sample_data_list(
unique_samples_ids[i] + 1,
true,
run_manager.output_dir(),
- run_manager.paired_read_input()
+ run_manager.paired_read_input(),
);
} else if i >= undetermined_label_id && illumina_format {
sample_data.create_files(
@@ -702,7 +698,7 @@ pub fn create_sample_data_list(
usize::MAX,
true,
run_manager.output_dir(),
- run_manager.paired_read_input()
+ run_manager.paired_read_input(),
);
} else {
sample_data.create_files(
@@ -710,7 +706,7 @@ pub fn create_sample_data_list(
unique_samples_ids[i] + 1,
false,
run_manager.output_dir(),
- run_manager.paired_read_input()
+ run_manager.paired_read_input(),
);
}
} else {
@@ -720,7 +716,7 @@ pub fn create_sample_data_list(
false,
false,
buffer_info.clone(),
- illumina_header.clone()
+ illumina_header.clone(),
);
}
sample_data_list.push(sample_data);
@@ -734,7 +730,7 @@ pub fn clean_output_directory(
run_manager: &RunManager,
buffer_info: &BufferInfo,
read2_has_sequence: bool,
- illumina_format: bool
+ illumina_format: bool,
) {
let writing_samples = sample_manager.writing_samples();
let unique_samples_ids = sample_manager.unique_samples_ids();
@@ -755,7 +751,7 @@ pub fn clean_output_directory(
run_manager.paired_read_input(),
read2_has_sequence || i >= undetermined_label_id,
buffer_info.clone(),
- illumina_header.clone()
+ illumina_header.clone(),
);
if i < undetermined_label_id && illumina_format {
sample_data.create_files(
@@ -763,7 +759,7 @@ pub fn clean_output_directory(
unique_samples_ids[i] + 1,
true,
run_manager.output_dir(),
- run_manager.paired_read_input()
+ run_manager.paired_read_input(),
);
sample_data.delete_sample_files();
} else if i >= undetermined_label_id && illumina_format {
@@ -772,7 +768,7 @@ pub fn clean_output_directory(
usize::MAX,
true,
run_manager.output_dir(),
- run_manager.paired_read_input()
+ run_manager.paired_read_input(),
);
sample_data.delete_sample_files();
} else {
@@ -781,7 +777,7 @@ pub fn clean_output_directory(
unique_samples_ids[i] + 1,
false,
run_manager.output_dir(),
- run_manager.paired_read_input()
+ run_manager.paired_read_input(),
);
sample_data.delete_sample_files();
}
diff --git a/tst.sh b/tst.sh
deleted file mode 100755
index 078818f..0000000
--- a/tst.sh
+++ /dev/null
@@ -1,36 +0,0 @@
-!/bin/bash
-set -euo pipefail
-
-if [ "$#" -eq 0 ]; then
- echo "Usage:"
- echo " $0 /path/to/files/*.gz # pass a glob (UNQUOTED)"
- echo " $0 file1.gz file2.gz ... # or list files explicitly"
- echo " $0 /path/to/dir # or a directory (processes *.gz inside)"
- exit 1
-fi
-
-process_file() {
- local gz="$1"
- [[ -f "$gz" ]] || { echo "Skip (not a file): $gz"; return; }
-
- echo "Processing: $gz"
- gunzip -f -- "$gz" # decompress; leaves file without .gz
- local decompressed="${gz%.gz}"
- local new_name="${decompressed/_R2/_R1}" # rename _R2 -> _R1
- if [[ "$decompressed" != "$new_name" ]]; then
- mv -- "$decompressed" "$new_name"
- fi
- gzip -f -- "$new_name" # recompress
-}
-
-# If a single directory is given, process all *.gz inside it
-if [[ "$#" -eq 1 && -d "$1" ]]; then
- while IFS= read -r -d '' f; do
- process_file "$f"
- done < <(find "$1" -type f -name '*.gz' -print0)
-else
- # Otherwise, treat all arguments as files (supports glob expansion)
- for f in "$@"; do
- process_file "$f"
- done
-fi