'egapx:gnomon_plane:gnomon_training_iterations:gnomon_training_iteration:chainer:run_chainer' run with long time.

[meiyang12]

When I annotated another insect genome (genome size: 110MB) with 5 chromosomes based on 13 Illumina RNA-seq data, I met this problem:

[35/9fdcbd] ega…on:chainer:run_chainer (8) | 0 of 16, retries: 8
[0e/137ae8] ega…chainer:run_align_sort (1) | 1 of 1, cached: 1 ✔
[c6/4bf968] ega…ion2:chainer:generate_jobs | 1 of 1, cached: 1 ✔
[8f/f2c71f] ega…chainer:run_align_sort (1) | 1 of 1, cached: 1 ✔
[dd/e56f3d] ega…ion3:chainer:generate_jobs | 1 of 1, cached: 1 ✔
[ae/d3ee04] ega…chainer:run_align_sort (1) | 1 of 1, cached: 1 ✔
[f4/132d5f] ega…ion4:chainer:generate_jobs | 1 of 1, cached: 1 ✔
[5b/e0441d] ega…chainer:run_align_sort (1) | 1 of 1, cached: 1 ✔
[00/d64956] ega…lane:chainer:generate_jobs | 1 of 1, cached: 1 ✔
[27/84b62e] ega…plane:fetch_swiss_prot_asn | 1 of 1, cached: 1 ✔
[b9/3adbe6] ega…n_plane:get_swiss_prot_ids | 1 of 1, cached: 1 ✔
[9e/980a68] ega…_plane:print_fake_lxr_data | 1 of 1, cached: 1 ✔
[f0/b94ff1] ega…:fetch_ortholog_references | 1 of 1, cached: 1 ✔
[71/d9f68b] ega…ext_genome:get_genome_info | 1 of 1, cached: 1 ✔
[40/42188c] ega…_proteins:convert_proteins | 1 of 1, cached: 1 ✔
[2b/126493] ega…ogy_plane:get_prot_ref_ids | 1 of 1, cached: 1 ✔
Plus 52 more processes waiting for tasks…
[34/63265f] NOTE: Process `egapx:gnomon_plane:gnomon_training_iterations:gnomon_training_iteration:chainer:run_chainer (9)` failed -- Execution is retried (1)
[f4/4a53aa] NOTE: Process `egapx:gnomon_plane:gnomon_training_iterations:gnomon_training_iteration:chainer:run_chainer (10)` failed -- Execution is retried (1)
[44/7780ee] NOTE: Process `egapx:gnomon_plane:gnomon_training_iterations:gnomon_training_iteration:chainer:run_chainer (11)` failed -- Execution is retried (1)
[b4/c29bed] NOTE: Process `egapx:gnomon_plane:gnomon_training_iterations:gnomon_training_iteration:chainer:run_chainer (2)` failed -- Execution is retried (1)
[ef/42e48d] NOTE: Process `egapx:gnomon_plane:gnomon_training_iterations:gnomon_training_iteration:chainer:run_chainer (16)` failed -- Execution is retried (1)
[e8/cfe727] NOTE: Process `egapx:gnomon_plane:gnomon_training_iterations:gnomon_training_iteration:chainer:run_chainer (15)` failed -- Execution is retried (1)
[4e/7c7dfd] NOTE: Process `egapx:gnomon_plane:gnomon_training_iterations:gnomon_training_iteration:chainer:run_chainer (14)` failed -- Execution is retried (1)
[19/f5b079] NOTE: Process `egapx:gnomon_plane:gnomon_training_iterations:gnomon_training_iteration:chainer:run_chainer (5)` failed -- Execution is retried (1)

The egapx:gnomon_plane:gnomon_training_iterations:gnomon_training_iteration:chainer:run_chainer has been running with a long time. And the process did not report an error to exit.

And the nextflow.log looks like:

9月-12 00:47:32.542 [Task submitter] DEBUG n.processor.TaskPollingMonitor - %% executor local > tasks in the submission queue: 8 -- tasks to be submitted are shown below
~> TaskHandler[id: 288; name: egapx:gnomon_plane:gnomon_training_iterations:gnomon_training_iteration:chainer:run_chainer (11); status: NEW; exit: -; error: -; workDir: /teacher/meiyang/project/09_Lsat_genome/04_egapx/anno/1a/22c45f57d24348c09304303139d940]
~> TaskHandler[id: 291; name: egapx:gnomon_plane:gnomon_training_iterations:gnomon_training_iteration:chainer:run_chainer (15); status: NEW; exit: -; error: -; workDir: /teacher/meiyang/project/09_Lsat_genome/04_egapx/anno/49/7b5985984941249efd8003ee561b6e]
~> TaskHandler[id: 289; name: egapx:gnomon_plane:gnomon_training_iterations:gnomon_training_iteration:chainer:run_chainer (2); status: NEW; exit: -; error: -; workDir: /teacher/meiyang/project/09_Lsat_genome/04_egapx/anno/73/18da1aa0369e99793e17b0b31efae4]
~> TaskHandler[id: 292; name: egapx:gnomon_plane:gnomon_training_iterations:gnomon_training_iteration:chainer:run_chainer (14); status: NEW; exit: -; error: -; workDir: /teacher/meiyang/project/09_Lsat_genome/04_egapx/anno/8c/94ae0ab1134781a723182d7e0cc5bb]
~> TaskHandler[id: 286; name: egapx:gnomon_plane:gnomon_training_iterations:gnomon_training_iteration:chainer:run_chainer (9); status: NEW; exit: -; error: -; workDir: /teacher/meiyang/project/09_Lsat_genome/04_egapx/anno/f6/c7de4700648f917b1fd0adea0829c9]
~> TaskHandler[id: 290; name: egapx:gnomon_plane:gnomon_training_iterations:gnomon_training_iteration:chainer:run_chainer (16); status: NEW; exit: -; error: -; workDir: /teacher/meiyang/project/09_Lsat_genome/04_egapx/anno/5c/5ddd5e412702f541add5548a2e0cba]
~> TaskHandler[id: 287; name: egapx:gnomon_plane:gnomon_training_iterations:gnomon_training_iteration:chainer:run_chainer (10); status: NEW; exit: -; error: -; workDir: /teacher/meiyang/project/09_Lsat_genome/04_egapx/anno/45/67a47ae4e0541346dc50806d60fed6]
~> TaskHandler[id: 293; name: egapx:gnomon_plane:gnomon_training_iterations:gnomon_training_iteration:chainer:run_chainer (5); status: NEW; exit: -; error: -; workDir: /teacher/meiyang/project/09_Lsat_genome/04_egapx/anno/cf/d933c5cc8ad79b0ac295fbfdfe8e00]
9月-12 00:52:16.987 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 8 -- submitted tasks are shown below
~> TaskHandler[id: 276; name: egapx:gnomon_plane:gnomon_training_iterations:gnomon_training_iteration:chainer:run_chainer (7); status: RUNNING; exit: -; error: -; workDir: /teacher/meiyang/project/09_Lsat_genome/04_egapx/anno/41/0a4c76790003fcadfc6adefbd5e939]
~> TaskHandler[id: 270; name: egapx:gnomon_plane:gnomon_training_iterations:gnomon_training_iteration:chainer:run_chainer (1); status: RUNNING; exit: -; error: -; workDir: /teacher/meiyang/project/09_Lsat_genome/04_egapx/anno/af/89ab2801d602c512816aeb5b0bf89e]
~> TaskHandler[id: 273; name: egapx:gnomon_plane:gnomon_training_iterations:gnomon_training_iteration:chainer:run_chainer (4); status: RUNNING; exit: -; error: -; workDir: /teacher/meiyang/project/09_Lsat_genome/04_egapx/anno/c0/6d453918aa7a81f09ffa0fb23c02f7]
~> TaskHandler[id: 275; name: egapx:gnomon_plane:gnomon_training_iterations:gnomon_training_iteration:chainer:run_chainer (6); status: RUNNING; exit: -; error: -; workDir: /teacher/meiyang/project/09_Lsat_genome/04_egapx/anno/f1/22eb5eae4a1e8aae683bd805e407b7]
~> TaskHandler[id: 282; name: egapx:gnomon_plane:gnomon_training_iterations:gnomon_training_iteration:chainer:run_chainer (13); status: RUNNING; exit: -; error: -; workDir: /teacher/meiyang/project/09_Lsat_genome/04_egapx/anno/48/b79dfa49945ecde358fbdac5ba5b3e]
~> TaskHandler[id: 272; name: egapx:gnomon_plane:gnomon_training_iterations:gnomon_training_iteration:chainer:run_chainer (3); status: RUNNING; exit: -; error: -; workDir: /teacher/meiyang/project/09_Lsat_genome/04_egapx/anno/fc/9e3396e70a7d694847cef452ffa3a8]
~> TaskHandler[id: 281; name: egapx:gnomon_plane:gnomon_training_iterations:gnomon_training_iteration:chainer:run_chainer (12); status: RUNNING; exit: -; error: -; workDir: /teacher/meiyang/project/09_Lsat_genome/04_egapx/anno/c5/7871e85d179641dae59b6106152bdf]
~> TaskHandler[id: 277; name: egapx:gnomon_plane:gnomon_training_iterations:gnomon_training_iteration:chainer:run_chainer (8); status: RUNNING; exit: -; error: -; workDir: /teacher/meiyang/project/09_Lsat_genome/04_egapx/anno/35/9fdcbdcd3353260851dc0641c507ee]
9月-12 00:52:32.603 [Task submitter] DEBUG n.processor.TaskPollingMonitor - %% executor local > tasks in the submission queue: 8 -- tasks to be submitted are shown below
~> TaskHandler[id: 288; name: egapx:gnomon_plane:gnomon_training_iterations:gnomon_training_iteration:chainer:run_chainer (11); status: NEW; exit: -; error: -; workDir: /teacher/meiyang/project/09_Lsat_genome/04_egapx/anno/1a/22c45f57d24348c09304303139d940]
~> TaskHandler[id: 291; name: egapx:gnomon_plane:gnomon_training_iterations:gnomon_training_iteration:chainer:run_chainer (15); status: NEW; exit: -; error: -; workDir: /teacher/meiyang/project/09_Lsat_genome/04_egapx/anno/49/7b5985984941249efd8003ee561b6e]
~> TaskHandler[id: 289; name: egapx:gnomon_plane:gnomon_training_iterations:gnomon_training_iteration:chainer:run_chainer (2); status: NEW; exit: -; error: -; workDir: /teacher/meiyang/project/09_Lsat_genome/04_egapx/anno/73/18da1aa0369e99793e17b0b31efae4]
~> TaskHandler[id: 292; name: egapx:gnomon_plane:gnomon_training_iterations:gnomon_training_iteration:chainer:run_chainer (14); status: NEW; exit: -; error: -; workDir: /teacher/meiyang/project/09_Lsat_genome/04_egapx/anno/8c/94ae0ab1134781a723182d7e0cc5bb]
~> TaskHandler[id: 286; name: egapx:gnomon_plane:gnomon_training_iterations:gnomon_training_iteration:chainer:run_chainer (9); status: NEW; exit: -; error: -; workDir: /teacher/meiyang/project/09_Lsat_genome/04_egapx/anno/f6/c7de4700648f917b1fd0adea0829c9]
~> TaskHandler[id: 290; name: egapx:gnomon_plane:gnomon_training_iterations:gnomon_training_iteration:chainer:run_chainer (16); status: NEW; exit: -; error: -; workDir: /teacher/meiyang/project/09_Lsat_genome/04_egapx/anno/5c/5ddd5e412702f541add5548a2e0cba]
~> TaskHandler[id: 287; name: egapx:gnomon_plane:gnomon_training_iterations:gnomon_training_iteration:chainer:run_chainer (10); status: NEW; exit: -; error: -; workDir: /teacher/meiyang/project/09_Lsat_genome/04_egapx/anno/45/67a47ae4e0541346dc50806d60fed6]
~> TaskHandler[id: 293; name: egapx:gnomon_plane:gnomon_training_iterations:gnomon_training_iteration:chainer:run_chainer (5); status: NEW; exit: -; error: -; workDir: /teacher/meiyang/project/09_Lsat_genome/04_egapx/anno/cf/d933c5cc8ad79b0ac295fbfdfe8e00]

run.trace.txt looks like:

278     34/63265f       3888962 egapx:gnomon_plane:gnomon_training_iterations:gnomon_training_iteration:chainer:run_chainer (9) FAILED  -       2025-09-11 08:07:20.759 16h     16h     -       -       -       -       -       /teacher/meiyang/project/09_Lsat_genome/04_egapx/anno/34/63265f05e926daf3b6ed23e2a20267 -
279     f4/4a53aa       3888970 egapx:gnomon_plane:gnomon_training_iterations:gnomon_training_iteration:chainer:run_chainer (10)        FAILED  -       2025-09-11 08:07:20.766 16h     16h     -       -       -       -       -       /teacher/meiyang/project/09_Lsat_genome/04_egapx/anno/f4/4a53aa8a9842bdbc7b7db0a523651a      -
280     44/7780ee       3889056 egapx:gnomon_plane:gnomon_training_iterations:gnomon_training_iteration:chainer:run_chainer (11)        FAILED  -       2025-09-11 08:07:20.797 16h     16h     -       -       -       -       -       /teacher/meiyang/project/09_Lsat_genome/04_egapx/anno/44/7780ee7f392dc5f2d57782ea6716b1      -
271     b4/c29bed       3889004 egapx:gnomon_plane:gnomon_training_iterations:gnomon_training_iteration:chainer:run_chainer (2) FAILED  -       2025-09-11 08:07:20.781 16h     16h     -       -       -       -       -       /teacher/meiyang/project/09_Lsat_genome/04_egapx/anno/b4/c29bed434277e7e40bb7a222a62922 -
285     ef/42e48d       3889020 egapx:gnomon_plane:gnomon_training_iterations:gnomon_training_iteration:chainer:run_chainer (16)        FAILED  -       2025-09-11 08:07:20.787 16h     16h     -       -       -       -       -       /teacher/meiyang/project/09_Lsat_genome/04_egapx/anno/ef/42e48db92f388070a32d9c7ded8a4d      -
284     e8/cfe727       3888957 egapx:gnomon_plane:gnomon_training_iterations:gnomon_training_iteration:chainer:run_chainer (15)        FAILED  -       2025-09-11 08:07:20.750 16h     16h     -       -       -       -       -       /teacher/meiyang/project/09_Lsat_genome/04_egapx/anno/e8/cfe7273e69ca2b9a9eca0f0758b3fa      -
283     4e/7c7dfd       3889037 egapx:gnomon_plane:gnomon_training_iterations:gnomon_training_iteration:chainer:run_chainer (14)        FAILED  -       2025-09-11 08:07:20.791 16h     16h     -       -       -       -       -       /teacher/meiyang/project/09_Lsat_genome/04_egapx/anno/4e/7c7dfdce17140c5b053c1ef3a7bf9a      -
274     19/f5b079       3888990 egapx:gnomon_plane:gnomon_training_iterations:gnomon_training_iteration:chainer:run_chainer (5) FAILED  -       2025-09-11 08:07:20.774 16h     16h     -       -       -       -       -       /teacher/meiyang/project/09_Lsat_genome/04_egapx/anno/19/f5b079f1c19cd9c854e06a08218a5b -

And I check the work dir, there is nothing in it.

[pstrope]

Hi,
Could you do ls -la work_dir and see if there is a .command.err and other files and see if they have any error message?

[meiyang12]

the last 100 lines of .command.err looks like:

+++ sum[i]=157020
+++ for i in {1..7}
+++ '[' 664 -lt 664 ']'
+++ cpu_peak[i]=664
+++ for i in {1..7}
+++ '[' 3 -lt 15 ']'
+++ sum[i]=15
+++ '[' 0 = 1 ']'
+++ nxf_stat_ret=(${sum[*]})
+++ '[' 2014 -lt 10 ']'
+++ '[' 2014 -lt 120 ']'
+++ timeout=30
+++ read -t 30 -r DONE
00452/001/19204/RB F1D001C48C445411 38430/38371 2025-09-13T08:06:55.848380 LENOVO-LA0X1859 UNK_CLIENT      UNK_SESSION              chainer_wnode request-start internal_thread=monitor&cpu_count=128&normalized_load=20&total_ram=540182609920&effective_ram=538035126272&pct_ram_used=0&mem_total=150630400&mem_peak=150630400&mem_self=150630400&mem_available=537884495872&ncbi_app_version=0.0.1881&ncbi_app_sc_version=29&ncbi_app_vcs_revision=698963&ncbi_app_revision=698963
00452/001/19204/RE F1D001C48C445411 38431/38372 2025-09-13T08:06:55.848458 LENOVO-LA0X1859 UNK_CLIENT      UNK_SESSION              chainer_wnode request-stop  200 0.000106096 0 0
00452/001/19205/RB F1D001C48C445411 38432/38373 2025-09-13T08:06:58.848992 LENOVO-LA0X1859 UNK_CLIENT      UNK_SESSION              chainer_wnode request-start internal_thread=monitor&cpu_count=128&normalized_load=20&total_ram=540182609920&effective_ram=538035126272&pct_ram_used=0&mem_total=150630400&mem_peak=150630400&mem_self=150630400&mem_available=537884495872&ncbi_app_version=0.0.1881&ncbi_app_sc_version=29&ncbi_app_vcs_revision=698963&ncbi_app_revision=698963
00452/001/19205/RE F1D001C48C445411 38433/38374 2025-09-13T08:06:58.849091 LENOVO-LA0X1859 UNK_CLIENT      UNK_SESSION              chainer_wnode request-stop  200 0.000129938 0 0
00452/001/19206/RB F1D001C48C445411 38434/38375 2025-09-13T08:07:01.849572 LENOVO-LA0X1859 UNK_CLIENT      UNK_SESSION              chainer_wnode request-start internal_thread=monitor&cpu_count=128&normalized_load=20&total_ram=540182609920&effective_ram=538035126272&pct_ram_used=0&mem_total=150630400&mem_peak=150630400&mem_self=150630400&mem_available=537884495872&ncbi_app_version=0.0.1881&ncbi_app_sc_version=29&ncbi_app_vcs_revision=698963&ncbi_app_revision=698963
00452/001/19206/RE F1D001C48C445411 38435/38376 2025-09-13T08:07:01.849650 LENOVO-LA0X1859 UNK_CLIENT      UNK_SESSION              chainer_wnode request-stop  200 0.000107049 0 0
00452/001/19207/RB F1D001C48C445411 38436/38377 2025-09-13T08:07:04.850185 LENOVO-LA0X1859 UNK_CLIENT      UNK_SESSION              chainer_wnode request-start internal_thread=monitor&cpu_count=128&normalized_load=20&total_ram=540182609920&effective_ram=538035126272&pct_ram_used=0&mem_total=150630400&mem_peak=150630400&mem_self=150630400&mem_available=537884495872&ncbi_app_version=0.0.1881&ncbi_app_sc_version=29&ncbi_app_vcs_revision=698963&ncbi_app_revision=698963
00452/001/19207/RE F1D001C48C445411 38437/38378 2025-09-13T08:07:04.850265 LENOVO-LA0X1859 UNK_CLIENT      UNK_SESSION              chainer_wnode request-stop  200 0.000110149 0 0
00452/001/19208/RB F1D001C48C445411 38438/38379 2025-09-13T08:07:07.850762 LENOVO-LA0X1859 UNK_CLIENT      UNK_SESSION              chainer_wnode request-start internal_thread=monitor&cpu_count=128&normalized_load=20&total_ram=540182609920&effective_ram=538035126272&pct_ram_used=0&mem_total=150630400&mem_peak=150630400&mem_self=150630400&mem_available=537884495872&ncbi_app_version=0.0.1881&ncbi_app_sc_version=29&ncbi_app_vcs_revision=698963&ncbi_app_revision=698963
00452/001/19208/RE F1D001C48C445411 38439/38380 2025-09-13T08:07:07.850839 LENOVO-LA0X1859 UNK_CLIENT      UNK_SESSION              chainer_wnode request-stop  200 0.000105857 0 0
00452/001/19209/RB F1D001C48C445411 38440/38381 2025-09-13T08:07:10.851356 LENOVO-LA0X1859 UNK_CLIENT      UNK_SESSION              chainer_wnode request-start internal_thread=monitor&cpu_count=128&normalized_load=20&total_ram=540182609920&effective_ram=538035126272&pct_ram_used=0&mem_total=150630400&mem_peak=150630400&mem_self=150630400&mem_available=537884495872&ncbi_app_version=0.0.1881&ncbi_app_sc_version=29&ncbi_app_vcs_revision=698963&ncbi_app_revision=698963
00452/001/19209/RE F1D001C48C445411 38441/38382 2025-09-13T08:07:10.851434 LENOVO-LA0X1859 UNK_CLIENT      UNK_SESSION              chainer_wnode request-stop  200 0.000108957 0 0
00452/001/19210/RB F1D001C48C445411 38442/38383 2025-09-13T08:07:13.851943 LENOVO-LA0X1859 UNK_CLIENT      UNK_SESSION              chainer_wnode request-start internal_thread=monitor&cpu_count=128&normalized_load=20&total_ram=540182609920&effective_ram=538035126272&pct_ram_used=0&mem_total=150630400&mem_peak=150630400&mem_self=150630400&mem_available=537884495872&ncbi_app_version=0.0.1881&ncbi_app_sc_version=29&ncbi_app_vcs_revision=698963&ncbi_app_revision=698963
00452/001/19210/RE F1D001C48C445411 38443/38384 2025-09-13T08:07:13.852022 LENOVO-LA0X1859 UNK_CLIENT      UNK_SESSION              chainer_wnode request-stop  200 0.000112056 0 0
00452/001/19211/RB F1D001C48C445411 38444/38385 2025-09-13T08:07:16.852551 LENOVO-LA0X1859 UNK_CLIENT      UNK_SESSION              chainer_wnode request-start internal_thread=monitor&cpu_count=128&normalized_load=20&total_ram=540182609920&effective_ram=538035126272&pct_ram_used=0&mem_total=150630400&mem_peak=150630400&mem_self=150630400&mem_available=537884495872&ncbi_app_version=0.0.1881&ncbi_app_sc_version=29&ncbi_app_vcs_revision=698963&ncbi_app_revision=698963
00452/001/19211/RE F1D001C48C445411 38445/38386 2025-09-13T08:07:16.852628 LENOVO-LA0X1859 UNK_CLIENT      UNK_SESSION              chainer_wnode request-stop  200 0.000106096 0 0
00452/001/19212/RB F1D001C48C445411 38446/38387 2025-09-13T08:07:19.853180 LENOVO-LA0X1859 UNK_CLIENT      UNK_SESSION              chainer_wnode request-start internal_thread=monitor&cpu_count=128&normalized_load=20&total_ram=540182609920&effective_ram=538035126272&pct_ram_used=0&mem_total=150630400&mem_peak=150630400&mem_self=150630400&mem_available=537884495872&ncbi_app_version=0.0.1881&ncbi_app_sc_version=29&ncbi_app_vcs_revision=698963&ncbi_app_revision=698963
00452/001/19212/RE F1D001C48C445411 38447/38388 2025-09-13T08:07:19.853259 LENOVO-LA0X1859 UNK_CLIENT      UNK_SESSION              chainer_wnode request-stop  200 0.000109910 0 0

[pstrope]

Hi, is your genome and RNA-seq data public? What taxid?

[etvedte]

Please retry this with our newest release v0.5.0 to see if the issue persists.