Compleasm: use symlink instead of copying busco data #6679
Conversation
| cp -r '${busco_database.fields.path}/lineages/${lineage_dataset}/' 'galaxy_db/' && | ||
| mkdir -p 'galaxy_db/' && | ||
| ln -s '${busco_database.fields.path}/lineages/${lineage_dataset}/' 'galaxy_db/${lineage_dataset}' && | ||
| touch 'galaxy_db/${lineage_dataset}.done' && |
There was a problem hiding this comment.
Is this an empty file also existing in the DB folder? Then I would prefer a symlink.
Anyway a small explaining comment would be great.
There was a problem hiding this comment.
It's just an empty file specific to compleasm, to make it understand that it should not redownload it, I'm gonna add a comment
| @@ -2,5 +2,5 @@ | |||
| # - value | |||
| # - name | |||
| # - version | |||
| # - /path/to/data | |||
| eukaryota_odb10 eukaryota 5.4.6 ${__HERE__}/test-db/busco_downloads | |||
There was a problem hiding this comment.
May I ask why you updated the test (data)?
There was a problem hiding this comment.
to keep the test-data dir as small as possible as entomoplasmatales_odb10 is a much smaller lineage than eukaryota_odb10
There was a problem hiding this comment.
True, but changing it will increase the size of the repo. The "problem" with git repos is that everything that is in in will be there forever.
There was a problem hiding this comment.
But changing it might still be a good idea if the runtime is reduced significantly.
FOR CONTRIBUTOR:
I looked at the compleasm code, and found this way to avoid copying the busco dataset (which can take time and make compleasm jobs longer than busco jobs)