Previously, if a process hit the walltime limit and received SIGKILL
from the slurm scheduler, singularity did not properly propagate such
(soft) kill signal. This prevented the exit code to be caught, e.g for
resubmission purposes.
This commit introduces a workaround using slurms --signal directive
to send SIGUSR2 to the singularity process itself (instead of
container child processes, which presumably was happening before).
Effectively, once a job reaches walltime limit, this will result in
exitcode 140 which is typically caught by the errorStrategy in nf-core
pipelines
See also:
https://slurm.schedmd.com/sbatch.html#OPT_signalhttps://github.com/nextflow-io/nextflow/issues/2163https://github.com/nextflow-io/nextflow/issues/1561
