millironx/nf-configs
1
0
Fork 0
mirror of https://github.com/MillironX/nf-configs.git synced 2024-11-25 01:19:54 +00:00
Code Issues Releases Packages Activity

cbe: send SIGUSR2 upon job termination

Browse source 
Previously, if a process hit the walltime limit and received SIGKILL
from the slurm scheduler, singularity did not properly propagate such
(soft) kill signal. This prevented the exit code to be caught, e.g for
resubmission purposes.

This commit introduces a workaround using slurms --signal directive
to send SIGUSR2 to the singularity process itself (instead of
container child processes, which presumably was happening before).
Effectively, once a job reaches walltime limit, this will result in
exitcode 140 which is typically caught by the errorStrategy in nf-core
pipelines

See also:
https://slurm.schedmd.com/sbatch.html#OPT_signal
https://github.com/nextflow-io/nextflow/issues/2163
https://github.com/nextflow-io/nextflow/issues/1561
This commit is contained in:
phue 2021-06-16 16:36:39 +02:00
parent 9cca0eaf85
commit f5c36dde3e
1 changed files with 1 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
conf/cbe.config
View file

@ -8,7 +8,7 @@ params {
process {
executor = 'slurm'
queue = { task.memory <= 170.GB ? 'c' : 'm' }
clusterOptions = { task.time <= 1.h ? '--qos rapid' : task.time <= 8.h ? '--qos short': task.time <= 48.h ? '--qos medium' : '--qos long' }
clusterOptions = { def qos = task.time <= 1.h ? '--qos rapid' : { task.time <= 8.h ? '--qos short': { task.time <= 48.h ? '--qos medium' : '--qos long' } }; qos << ' --signal B:USR2' }
module = 'anaconda3/2019.10'
}