From beae329b29ba2d4a510f8a08140cb21c84538d37 Mon Sep 17 00:00:00 2001 From: "James A. Fellows Yates" Date: Wed, 7 Apr 2021 20:24:55 +0200 Subject: [PATCH 01/12] Add docs as to what exactly you have to do when adding new config --- README.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 7637725..de48a77 100644 --- a/README.md +++ b/README.md @@ -67,7 +67,15 @@ If you decide to upload your custom config file to `nf-core/configs` then this w You will simply have to specify `-profile ` in the command used to run the pipeline. See [`nf-core/configs`](https://github.com/nf-core/configs/tree/master/conf) for examples. -Please also make sure to add an extra `params` section with `params.config_profile_description`, `params.config_profile_contact` and `params.config_profile_url` set to reasonable values. +When adding a new config you will need to either add and/or update the following files: + +* **Add** your institutional config itself under the `conf/` directory accordingly +* **Add** documentation for your institutional config under the `docs/` directory +* **Update** the file `nfcore_custom.config` to include your config in list of nf-core institutional configs +* **Update** the file `README.md` to include your config documentation within the list of configs docs +* **Update** the `.github/workflows/main/yml` to include your config in the profile strategy matrix for the GitHub actions CI tests + +N.B. In your config file under `confg/`, please also make sure to add an extra `params` section with `params.config_profile_description`, `params.config_profile_contact` and `params.config_profile_url` set to reasonable values. Users will get information on who wrote the configuration profile then when executing a nf-core pipeline and can report back if there are things missing for example. ### Checking user hostnames @@ -156,6 +164,8 @@ Sometimes it may be desirable to have configuration options for an institute tha Such options should not be added to the main institutional config, as this will be applied to all pipelines. Instead, we can create a pipeline-specific institutional config file. +> The following steps are similar to the instructions for standard institutional profilers, however using `pipeline` variants of folders e.g., `conf/pipeline/` or under `pipeline/` + :warning: Remember to replace the `` and `` placeholders with the pipeline name and profile name in the following examples Institutional configs work because the pipeline `nextflow.config` file loads the [`nf-core/configs/nfcore_custom.config` config file](https://github.com/nf-core/configs/blob/master/nfcore_custom.config), which in turn loads the institutional configuration file based on the profile `` supplied on the command line. From 033f81734a980cde8a6588281c3989011e968f7a Mon Sep 17 00:00:00 2001 From: "James A. Fellows Yates" Date: Wed, 7 Apr 2021 20:32:44 +0200 Subject: [PATCH 02/12] Make a bit more explicitly exactly what you have to add/update --- README.md | 23 ++++++++++------------- 1 file changed, 10 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index de48a77..34d7fbf 100644 --- a/README.md +++ b/README.md @@ -67,15 +67,9 @@ If you decide to upload your custom config file to `nf-core/configs` then this w You will simply have to specify `-profile ` in the command used to run the pipeline. See [`nf-core/configs`](https://github.com/nf-core/configs/tree/master/conf) for examples. -When adding a new config you will need to either add and/or update the following files: +Before adding your config file to nf-core/configs, we highly recommend writing and testing your own custom config file (as described [above](Using an existing config)), and then continuing with the next steps. -* **Add** your institutional config itself under the `conf/` directory accordingly -* **Add** documentation for your institutional config under the `docs/` directory -* **Update** the file `nfcore_custom.config` to include your config in list of nf-core institutional configs -* **Update** the file `README.md` to include your config documentation within the list of configs docs -* **Update** the `.github/workflows/main/yml` to include your config in the profile strategy matrix for the GitHub actions CI tests - -N.B. In your config file under `confg/`, please also make sure to add an extra `params` section with `params.config_profile_description`, `params.config_profile_contact` and `params.config_profile_url` set to reasonable values. +N.B. In your config file, please also make sure to add an extra `params` section with `params.config_profile_description`, `params.config_profile_contact` and `params.config_profile_url` set to reasonable values. Users will get information on who wrote the configuration profile then when executing a nf-core pipeline and can report back if there are things missing for example. ### Checking user hostnames @@ -141,11 +135,14 @@ Currently documentation is available for the following systems: ### Uploading to `nf-core/configs` [Fork](https://help.github.com/articles/fork-a-repo/) the [`nf-core/configs`](https://github.com/nf-core/configs/) repository to your own GitHub account. -Within the local clone of your fork add the custom config file to the [`conf/`](https://github.com/nf-core/configs/tree/master/conf) directory, and the documentation file to the [`docs/`](https://github.com/nf-core/configs/tree/master/docs) directory. -You will also need to edit and add your custom profile to the [`nfcore_custom.config`](https://github.com/nf-core/configs/blob/master/nfcore_custom.config) file in the top-level directory of the clone. -You will also need to edit and add your custom profile to the [`README.md`](https://github.com/nf-core/configs/blob/master/README.md) file in the top-level directory of the clone. +Within the local clone of your fork: -In order to ensure that the config file is tested automatically with GitHub Actions please add your profile name to the `profile:` scope in [`.github/workflows/main.yml`](.github/workflows/main.yml). If you forget to do this the tests will fail with the error: +* **add** the custom config file to the [`conf/`](https://github.com/nf-core/configs/tree/master/conf) directory, +* **add** the documentation file to the [`docs/`](https://github.com/nf-core/configs/tree/master/docs) directory. +* **edit** and add your custom profile to the [`nfcore_custom.config`](https://github.com/nf-core/configs/blob/master/nfcore_custom.config) file in the top-level directory of the clone. +* **edit** and add your custom profile to the [`README.md`](https://github.com/nf-core/configs/blob/master/README.md) file in the top-level directory of the clone. + +In order to ensure that the config file is tested automatically with GitHub Actions please add your profile name to the `profile:` scope (under strategy matrix) in [`.github/workflows/main.yml`](.github/workflows/main.yml). If you forget to do this the tests will fail with the error: ```bash Run python ${GITHUB_WORKSPACE}/bin/cchecker.py ${GITHUB_WORKSPACE}/nfcore_custom.config ${GITHUB_WORKSPACE}/.github/workflows/main.yml @@ -164,7 +161,7 @@ Sometimes it may be desirable to have configuration options for an institute tha Such options should not be added to the main institutional config, as this will be applied to all pipelines. Instead, we can create a pipeline-specific institutional config file. -> The following steps are similar to the instructions for standard institutional profilers, however using `pipeline` variants of folders e.g., `conf/pipeline/` or under `pipeline/` +> The following steps are similar to the instructions for standard institutional config, however using `pipeline` variants of folders e.g., `conf/pipeline/` or under `pipeline/` :warning: Remember to replace the `` and `` placeholders with the pipeline name and profile name in the following examples From 095cd483e6339ce59cdc719ca3d948d459de1908 Mon Sep 17 00:00:00 2001 From: "James A. Fellows Yates" Date: Wed, 7 Apr 2021 20:43:31 +0200 Subject: [PATCH 03/12] Update README.md --- README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 34d7fbf..b95d766 100644 --- a/README.md +++ b/README.md @@ -137,10 +137,10 @@ Currently documentation is available for the following systems: [Fork](https://help.github.com/articles/fork-a-repo/) the [`nf-core/configs`](https://github.com/nf-core/configs/) repository to your own GitHub account. Within the local clone of your fork: -* **add** the custom config file to the [`conf/`](https://github.com/nf-core/configs/tree/master/conf) directory, -* **add** the documentation file to the [`docs/`](https://github.com/nf-core/configs/tree/master/docs) directory. -* **edit** and add your custom profile to the [`nfcore_custom.config`](https://github.com/nf-core/configs/blob/master/nfcore_custom.config) file in the top-level directory of the clone. -* **edit** and add your custom profile to the [`README.md`](https://github.com/nf-core/configs/blob/master/README.md) file in the top-level directory of the clone. +* **add** the custom config file to the [`conf/`](https://github.com/nf-core/configs/tree/master/conf) directory +* **add** the documentation file to the [`docs/`](https://github.com/nf-core/configs/tree/master/docs) directory +* **edit** and add your custom profile to the [`nfcore_custom.config`](https://github.com/nf-core/configs/blob/master/nfcore_custom.config) file in the top-level directory of the clone +* **edit** and add your custom profile to the [`README.md`](https://github.com/nf-core/configs/blob/master/README.md) file in the top-level directory of the clone In order to ensure that the config file is tested automatically with GitHub Actions please add your profile name to the `profile:` scope (under strategy matrix) in [`.github/workflows/main.yml`](.github/workflows/main.yml). If you forget to do this the tests will fail with the error: From fb3cede5d31be34df5defde57b6ae7ff8b99084a Mon Sep 17 00:00:00 2001 From: ikeller Date: Wed, 19 May 2021 08:41:55 +0200 Subject: [PATCH 04/12] add config file --- conf/unibe_ibu.config | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 conf/unibe_ibu.config diff --git a/conf/unibe_ibu.config b/conf/unibe_ibu.config new file mode 100644 index 0000000..e69de29 From af1e57027d17dd09c416ae94c7e5dbb2db1f0e2b Mon Sep 17 00:00:00 2001 From: ikeller Date: Wed, 19 May 2021 09:46:07 +0200 Subject: [PATCH 05/12] edit config and readme --- .github/workflows/main.yml | 2 +- README.md | 1 + conf/unibe_ibu.config | 23 +++++++++++++++++++++++ docs/unibe_ibu.md | 8 ++++++++ nfcore_custom.config | 1 + 5 files changed, 34 insertions(+), 1 deletion(-) create mode 100644 docs/unibe_ibu.md diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 65cabe5..25a583b 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -16,7 +16,7 @@ jobs: needs: test_all_profiles strategy: matrix: - profile: ['abims', 'awsbatch', 'bi','bigpurple', 'binac', 'biohpc_gen', 'cbe', 'ccga_dx', 'ccga_med', 'cfc', 'cfc_dev', 'crick', 'denbi_qbic', 'ebc', 'eddie', 'eva', 'genotoul', 'genouest', 'gis', 'google', 'hebbe', 'icr_davros', 'ifb_core', 'imperial', 'imperial_mb', 'jax', 'kraken', 'mpcdf', 'munin', 'oist', 'pasteur', 'phoenix', 'prince', 'sanger', 'seg_globe', 'shh', 'uct_hpc', 'uppmax', 'utd_ganymede', 'uzh'] + profile: ['abims', 'awsbatch', 'bi','bigpurple', 'binac', 'biohpc_gen', 'cbe', 'ccga_dx', 'ccga_med', 'cfc', 'cfc_dev', 'crick', 'denbi_qbic', 'ebc', 'eddie', 'eva', 'genotoul', 'genouest', 'gis', 'google', 'hebbe', 'icr_davros', 'ifb_core', 'imperial', 'imperial_mb', 'jax', 'kraken', 'mpcdf', 'munin', 'oist', 'pasteur', 'phoenix', 'prince', 'sanger', 'seg_globe', 'shh', 'uct_hpc', 'unibe_ibu', 'uppmax', 'utd_ganymede', 'uzh'] steps: - uses: actions/checkout@v1 - name: Install Nextflow diff --git a/README.md b/README.md index 6daff2a..2d0ca9f 100644 --- a/README.md +++ b/README.md @@ -128,6 +128,7 @@ Currently documentation is available for the following systems: * [SEG_GLOBE](docs/seg_globe.md) * [SHH](docs/shh.md) * [UCT_HPC](docs/uct_hpc.md) +* [UNIBE_IBU](docs/unibe_ibu.md) * [UPPMAX](docs/uppmax.md) * [UTD_GANYMEDE](docs/utd_ganymede.md) * [UZH](docs/uzh.md) diff --git a/conf/unibe_ibu.config b/conf/unibe_ibu.config index e69de29..cda6ace 100644 --- a/conf/unibe_ibu.config +++ b/conf/unibe_ibu.config @@ -0,0 +1,23 @@ +params { + config_profile_description = "University of Bern, Interfaculty Bioinformatics Unit cluster profile" + config_profile_contact = "irene.keller@dbmr.unibe.ch; info@bioinformatics.unibe.ch" + config_profile_url = "https://www.bioinformatics.unibe.ch/" + max_memory = 500.GB + max_cpus = 128 + max_time = 28.d +} + +process { + executor = "slurm" + maxRetries = 2 + beforeScript = 'mkdir -p ./tmp/ && export TMPDIR=./tmp/' +} + +executor { + queueSize = 30 +} + +singularity { + enable = true + autoMounts = true +} \ No newline at end of file diff --git a/docs/unibe_ibu.md b/docs/unibe_ibu.md new file mode 100644 index 0000000..8070dc5 --- /dev/null +++ b/docs/unibe_ibu.md @@ -0,0 +1,8 @@ +# nf-core/configs: UNIBE_IBU Configuration + +Configuration file to run nf-core pipelines on the cluster of the [Interfaculty Bioinformatics Unit](https://www.bioinformatics.unibe.ch/) of the University of Bern. + +To use, run the pipeline with `-profile unibe_ibu`. This will download and launch the profile.config which has been pre-configured with a setup suitable for the IBU cluster. Using this profile, a docker image containing all of the required software will be downloaded, and converted to a Singularity image before execution of the pipeline. **This requires a local installation of singularity**. It is easiest to submit the pipeline from a compute node. Once the image is cached, you can also submit from the login node. + + + diff --git a/nfcore_custom.config b/nfcore_custom.config index 9b4fbda..8781322 100644 --- a/nfcore_custom.config +++ b/nfcore_custom.config @@ -47,6 +47,7 @@ profiles { seg_globe { includeConfig "${params.custom_config_base}/conf/seg_globe.config"} shh { includeConfig "${params.custom_config_base}/conf/shh.config" } uct_hpc { includeConfig "${params.custom_config_base}/conf/uct_hpc.config" } + unibe_ibu { includeConfig "${params.custom_config_base}/conf/unibe_ibu.config" } uppmax { includeConfig "${params.custom_config_base}/conf/uppmax.config" } utd_ganymede { includeConfig "${params.custom_config_base}/conf/utd_ganymede.config" } uzh { includeConfig "${params.custom_config_base}/conf/uzh.config" } From 82fadc70611e8aa27930049830c9f2070612c17a Mon Sep 17 00:00:00 2001 From: Asaf Peer Date: Wed, 19 May 2021 13:48:49 -0400 Subject: [PATCH 06/12] replaced owner in JAX --- conf/jax.config | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/conf/jax.config b/conf/jax.config index 56c0f12..a56df71 100644 --- a/conf/jax.config +++ b/conf/jax.config @@ -1,6 +1,6 @@ params { config_profile_description = 'The Jackson Laboratory Sumner HPC profile provided by nf-core/configs.' - config_profile_contact = 'Asaf Peer (@peera)' + config_profile_contact = 'Bill Flynn (@flynnb)' config_profile_url = 'https://jacksonlaboratory.sharepoint.com/sites/ResearchIT/SitePages/Welcome-to-Sumner.aspx' singularity_cache_dir = '/fastscratch/singularity_cache_nfcore' } From 0e3898eabff95652d76ce297a0d1d7ba24c413c6 Mon Sep 17 00:00:00 2001 From: ikeller Date: Thu, 20 May 2021 10:53:35 +0200 Subject: [PATCH 07/12] modify singularity settings --- conf/unibe_ibu.config | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/conf/unibe_ibu.config b/conf/unibe_ibu.config index cda6ace..fa6cf97 100644 --- a/conf/unibe_ibu.config +++ b/conf/unibe_ibu.config @@ -1,5 +1,7 @@ +singularity.autoMounts = true + params { - config_profile_description = "University of Bern, Interfaculty Bioinformatics Unit cluster profile" + config_profile_description = "University of Bern, Interfaculty Bioinformatics Unit cluster profile" config_profile_contact = "irene.keller@dbmr.unibe.ch; info@bioinformatics.unibe.ch" config_profile_url = "https://www.bioinformatics.unibe.ch/" max_memory = 500.GB @@ -8,16 +10,16 @@ params { } process { - executor = "slurm" + executor = "slurm" maxRetries = 2 beforeScript = 'mkdir -p ./tmp/ && export TMPDIR=./tmp/' } executor { - queueSize = 30 + queueSize = 30 } singularity { - enable = true - autoMounts = true + docker.enabled = false + singularity.enabled = true } \ No newline at end of file From 6def686ab57d4e93560e043b60afe6d68dbf2dcc Mon Sep 17 00:00:00 2001 From: ikeller Date: Thu, 20 May 2021 13:44:59 +0200 Subject: [PATCH 08/12] edit config --- conf/unibe_ibu.config | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/conf/unibe_ibu.config b/conf/unibe_ibu.config index fa6cf97..33ff956 100644 --- a/conf/unibe_ibu.config +++ b/conf/unibe_ibu.config @@ -1,12 +1,10 @@ -singularity.autoMounts = true - params { config_profile_description = "University of Bern, Interfaculty Bioinformatics Unit cluster profile" config_profile_contact = "irene.keller@dbmr.unibe.ch; info@bioinformatics.unibe.ch" config_profile_url = "https://www.bioinformatics.unibe.ch/" max_memory = 500.GB max_cpus = 128 - max_time = 28.d + max_time = 200.h } process { @@ -20,6 +18,6 @@ executor { } singularity { - docker.enabled = false - singularity.enabled = true + enabled = true + autoMounts = true } \ No newline at end of file From 5a0ecbd81d1752da385a0a7a6d6e2c984401317c Mon Sep 17 00:00:00 2001 From: ikeller Date: Thu, 20 May 2021 13:48:59 +0200 Subject: [PATCH 09/12] edit max_time --- conf/unibe_ibu.config | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/conf/unibe_ibu.config b/conf/unibe_ibu.config index 33ff956..d41896e 100644 --- a/conf/unibe_ibu.config +++ b/conf/unibe_ibu.config @@ -4,7 +4,7 @@ params { config_profile_url = "https://www.bioinformatics.unibe.ch/" max_memory = 500.GB max_cpus = 128 - max_time = 200.h + max_time = '7.d' } process { From ae114dd142084c999fd8ed55576be116568404dc Mon Sep 17 00:00:00 2001 From: ikeller Date: Thu, 20 May 2021 13:52:13 +0200 Subject: [PATCH 10/12] edit max_time --- conf/unibe_ibu.config | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/conf/unibe_ibu.config b/conf/unibe_ibu.config index d41896e..6ebce5a 100644 --- a/conf/unibe_ibu.config +++ b/conf/unibe_ibu.config @@ -4,7 +4,7 @@ params { config_profile_url = "https://www.bioinformatics.unibe.ch/" max_memory = 500.GB max_cpus = 128 - max_time = '7.d' + max_time = 240.h } process { From 7469e20b4e79404097693ab83498106c94609166 Mon Sep 17 00:00:00 2001 From: Anthony Underwood Date: Sat, 25 Sep 2021 09:23:24 +0100 Subject: [PATCH 11/12] change max_time diretcive in the correct place --- conf/sanger.config | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/conf/sanger.config b/conf/sanger.config index 5611762..1955ca9 100644 --- a/conf/sanger.config +++ b/conf/sanger.config @@ -17,7 +17,6 @@ process{ process.maxRetries = 5 withLabel:process_long { queue = 'long' - max_time = 48.h } } @@ -32,5 +31,5 @@ executor{ params { max_memory = 128.GB max_cpus = 64 - max_time = 12.h + max_time = 48.h } From 286fa6753a347bdf189039ca534d453d87588a0f Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 19 Nov 2021 00:07:33 +0100 Subject: [PATCH 12/12] Apply suggestions from code review --- docs/unibe_ibu.md | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/docs/unibe_ibu.md b/docs/unibe_ibu.md index 8070dc5..1adc407 100644 --- a/docs/unibe_ibu.md +++ b/docs/unibe_ibu.md @@ -1,8 +1,5 @@ # nf-core/configs: UNIBE_IBU Configuration -Configuration file to run nf-core pipelines on the cluster of the [Interfaculty Bioinformatics Unit](https://www.bioinformatics.unibe.ch/) of the University of Bern. +Configuration file to run nf-core pipelines on the cluster of the [Interfaculty Bioinformatics Unit](https://www.bioinformatics.unibe.ch/) of the University of Bern. To use, run the pipeline with `-profile unibe_ibu`. This will download and launch the profile.config which has been pre-configured with a setup suitable for the IBU cluster. Using this profile, a docker image containing all of the required software will be downloaded, and converted to a Singularity image before execution of the pipeline. **This requires a local installation of singularity**. It is easiest to submit the pipeline from a compute node. Once the image is cached, you can also submit from the login node. - - -