Skip to content

Replace limitGenomeGenerateRAM parameter by required memory #2680

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
entzian wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Replace limitGenomeGenerateRAM parameter by required memory #2680

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 1 addition & 8 deletions source/Genome.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -219,14 +219,7 @@ void Genome::chrBinFill() {
void Genome::genomeSequenceAllocate(uint64 nGenomeIn, uint64 &nG1allocOut, char*& Gout, char*& G1out)
{
nG1allocOut=(nGenomeIn + 100)*2; //extra 100 bytes at the beginning, just in case

if (P.limitGenomeGenerateRAM < (nG1allocOut+nG1allocOut/3)) {//allocate nG1alloc/3 for SA generation
ostringstream errOut;
errOut <<"EXITING because of FATAL PARAMETER ERROR: limitGenomeGenerateRAM="<< (P.limitGenomeGenerateRAM) <<"is too small for your genome\n";
errOut <<"SOLUTION: please specify --limitGenomeGenerateRAM not less than "<< nG1allocOut+nG1allocOut/3 <<" and make that much RAM available \n";
exitWithError(errOut.str(),std::cerr, P.inOut->logMain, EXIT_CODE_INPUT_FILES, P);
};


G1out=new char[nG1allocOut];
Gout=G1out+100;

Expand Down
3 changes: 2 additions & 1 deletion source/Genome_genomeGenerate.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -230,7 +230,8 @@ void Genome::genomeGenerate() {
};
};

uint saChunkSize=(P.limitGenomeGenerateRAM-nG1alloc)/8/P.runThreadN; //number of SA indexes per chunk
uint required_memory_sa = nG1alloc/3;
uint saChunkSize = required_memory_sa/8/P.runThreadN; //number of SA indexes per chunk
saChunkSize=saChunkSize*6/10; //allow extra space for qsort
//uint saChunkN=((nSA/saChunkSize+1)/P.runThreadN+1)*P.runThreadN;//ensure saChunkN is divisible by P.runThreadN
//saChunkSize=nSA/saChunkN+100000;//final chunk size
Expand Down
9 changes: 0 additions & 9 deletions source/Parameters.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,6 @@ Parameters::Parameters() {//initalize parameters info
parArray.push_back(new ParameterInfoScalar <uint> (-1, -1, "bamRemoveDuplicatesMate2basesN", &removeDuplicates.mate2basesN));

//limits
parArray.push_back(new ParameterInfoScalar <uint> (-1, -1, "limitGenomeGenerateRAM", &limitGenomeGenerateRAM));
parArray.push_back(new ParameterInfoVector <uint64> (-1, -1, "limitIObufferSize", &limitIObufferSize));
parArray.push_back(new ParameterInfoScalar <uint> (-1, -1, "limitOutSAMoneReadBytes", &limitOutSAMoneReadBytes));
parArray.push_back(new ParameterInfoScalar <uint> (-1, -1, "limitOutSJcollapsed", &limitOutSJcollapsed));
Expand Down Expand Up @@ -729,14 +728,6 @@ void Parameters::inputParameters (int argInN, char* argIn[]) {//input parameters
if (alignSJstitchMismatchNmax.at(ii)<0) alignSJstitchMismatchNmax.at(ii)=numeric_limits<int32>::max();
};

if (limitGenomeGenerateRAM==0) {//must be >0
inOut->logMain <<"EXITING because of FATAL PARAMETER ERROR: limitGenomeGenerateRAM=0\n";
inOut->logMain <<"SOLUTION: please specify a >0 value for limitGenomeGenerateRAM\n"<<flush;
exit(1);
} else if (limitGenomeGenerateRAM>1000000000000) {//
inOut->logMain <<"WARNING: specified limitGenomeGenerateRAM="<<limitGenomeGenerateRAM<<" bytes appears to be too large, if you do not have enough memory the code will crash!\n"<<flush;
};

outSAMfilter.KeepOnlyAddedReferences=false;
outSAMfilter.KeepAllAddedReferences=false;
outSAMfilter.yes=true;
Expand Down
1 change: 0 additions & 1 deletion source/Parameters.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -274,7 +274,6 @@ class Parameters {
} sjdbInsert;

//storage limits
uint64 limitGenomeGenerateRAM;
vector<uint64> limitIObufferSize; //max size of the in/out buffer, bytes
uint64 limitOutSAMoneReadBytes;
uint64 limitOutSJoneRead, limitOutSJcollapsed;
Expand Down
3 changes: 0 additions & 3 deletions source/parametersDefault
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -226,9 +226,6 @@ clip5pAfterAdapterNbases 0
#####UnderDevelopment_end

### Limits
limitGenomeGenerateRAM 31000000000
int>0: maximum available RAM (bytes) for genome generation

limitIObufferSize 30000000 50000000
int(s)>0: max available buffers size (bytes) for input/output, per thread

Expand Down