Merge pull request #85 from MaxUlysse/sarek

Adding a pipeline specific config for a specific system

README.md

@@ -2,7 +2,7 @@
 
 [![Lint Status](https://github.com/nf-core/configs/workflows/Configs%20tests/badge.svg)](https://github.com/nf-core/configs/workflows/Configs%20tests/badge.svg)
 
-A repository for hosting nextflow config files containing custom parameters required to run nf-core pipelines at different Institutions.
+A repository for hosting Nextflow configuration files containing custom parameters required to run nf-core pipelines at different Institutions.
 
 ## Table of contents
 
@@ -11,28 +11,36 @@ A repository for hosting nextflow config files containing custom parameters requ
   * [Configuration and parameters](#configuration-and-parameters)
   * [Offline usage](#offline-usage)
 * [Adding a new config](#adding-a-new-config)
-* [Checking user hostnames](#checking-user-hostnames)
+  * [Checking user hostnames](#checking-user-hostnames)
   * [Testing](#testing)
   * [Documentation](#documentation)
   * [Uploading to `nf-core/configs`](#uploading-to-nf-coreconfigs)
+* [Adding a new pipeline-specific config](#adding-a-new-pipeline-specific-config)
+  * [Pipeline-specific documentation](#pipeline-specific-documentation)
+  * [Enabling pipeline-specific configs within a pipeline](#enabling-pipeline-specific-configs-within-a-pipeline)
+  * [Create the pipeline-specific `nf-core/configs` files](#create-the-pipeline-specific-nf-coreconfigs-files)
 * [Help](#help)
 
 ## Using an existing config
 
-The Nextflow [`-c`](https://www.nextflow.io/docs/latest/config.html) parameter can be used with nf-core pipelines in order to load custom config files that you have available locally. However, if you or other people within your organisation are likely to be running nf-core pipelines regularly it may be a good idea to use/create a custom config file that defines some generic settings unique to the computing environment within your organisation.
+The Nextflow [`-c`](https://www.nextflow.io/docs/latest/config.html) parameter can be used with nf-core pipelines in order to load custom config files that you have available locally.
+However, if you or other people within your organisation are likely to be running nf-core pipelines regularly it may be a good idea to use/create a custom config file that defines some generic settings unique to the computing environment within your organisation.
 
 ### Configuration and parameters
 
 The config files hosted in this repository define a set of parameters which are specific to compute environments at different Institutions but generic enough to be used with all nf-core pipelines.
 
-All nf-core pipelines inherit the functionality provided by Nextflow, and as such custom config files can contain parameters/definitions that are available to both. For example, if you have the ability to use [Singularity](https://singularity.lbl.gov/) on your HPC you can add and customize the Nextflow [`singularity`](https://www.nextflow.io/docs/latest/config.html#scope-singularity) scope in your config file. Similarly, you can define a Nextflow [`executor`](https://www.nextflow.io/docs/latest/executor.html) depending on the job submission process available on your cluster. In contrast, the `params` section in your custom config file will typically define parameters that are specific to nf-core pipelines.
+All nf-core pipelines inherit the functionality provided by Nextflow, and as such custom config files can contain parameters/definitions that are available to both.
+For example, if you have the ability to use [Singularity](https://sylabs.io/singularity/) on your HPC you can add and customize the Nextflow [`singularity`](https://www.nextflow.io/docs/latest/config.html#scope-singularity) scope in your config file.
+Similarly, you can define a Nextflow [`executor`](https://www.nextflow.io/docs/latest/executor.html) depending on the job submission process available on your cluster.
+In contrast, the `params` section in your custom config file will typically define parameters that are specific to nf-core pipelines.
 
 You should be able to get a good idea as to how other people are customising the execution of their nf-core pipelines by looking at some of the config files in [`nf-core/configs`](https://github.com/nf-core/configs/tree/master/conf).
 
 ### Offline usage
 
-If you want to use an existing config available in `nf-core/configs`, and you're running on a system that has no internet connection, you'll need to download the config file and place it in a location that is visible to the file system on which you are running the pipeline. Then run the pipeline with `--custom_config_base`
+If you want to use an existing config available in `nf-core/configs`, and you're running on a system that has no internet connection, you'll need to download the config file and place it in a location that is visible to the file system on which you are running the pipeline.
-or `params.custom_config_base` set to the location of the directory containing the repository files:
+Then run the pipeline with `--custom_config_base` or `params.custom_config_base` set to the location of the directory containing the repository files:
 
 ```bash
 ## Download and unzip the config files
@@ -45,28 +53,28 @@ cd /path/to/my/data
 nextflow run /path/to/pipeline/ --custom_config_base /path/to/my/configs/configs-master/
 ```
 
-Alternatively, instead of using the configuration profiles from this repository, you can run your
+Alternatively, instead of using the configuration profiles from this repository, you can run your pipeline directly calling the single institutional config file that you need with the `-c` parameter.
-pipeline directly calling the single institutional config file that you need with the `-c` parameter.
 
 ```bash
 nextflow run /path/to/pipeline/ -c /path/to/my/configs/configs-master/conf/my_config.config
 ```
 
-> Note that the nf-core/tools helper package has a `download` command to download all required pipeline
+> Note that the nf-core/tools helper package has a `download` command to download all required pipeline files + singularity containers + institutional configs in one go for you, to make this process easier.
-> files + singularity containers + institutional configs in one go for you, to make this process easier.
 
 ## Adding a new config
 
-If you decide to upload your custom config file to `nf-core/configs` then this will ensure that your custom config file will be automatically downloaded, and available at run-time to all nf-core pipelines, and to everyone within your organisation. You will simply have to specify `-profile <config_name>` in the command used to run the pipeline. See [`nf-core/configs`](https://github.com/nf-core/configs/tree/master/conf) for examples.
+If you decide to upload your custom config file to `nf-core/configs` then this will ensure that your custom config file will be automatically downloaded, and available at run-time to all nf-core pipelines, and to everyone within your organisation.
+You will simply have to specify `-profile <config_name>` in the command used to run the pipeline.
+See [`nf-core/configs`](https://github.com/nf-core/configs/tree/master/conf) for examples.
 
-Please also make sure to add an extra `params` section with `params.config_profile_description`, `params.config_profile_contact` and `params.config_profile_url` set to reasonable values. Users will get information on who wrote the configuration profile then when executing a nf-core pipeline and can report back if there are things missing for example.
+Please also make sure to add an extra `params` section with `params.config_profile_description`, `params.config_profile_contact` and `params.config_profile_url` set to reasonable values.
+Users will get information on who wrote the configuration profile then when executing a nf-core pipeline and can report back if there are things missing for example.
 
-## Checking user hostnames
+### Checking user hostnames
 
 If your cluster has a set of consistent hostnames, nf-core pipelines can check that users are using your profile.
 Add one or more hostname substrings to `params.hostnames` under a key that matches the profile name.
-If the user's hostname contains this string at the start of a run or when a run fails and their profile
+If the user's hostname contains this string at the start of a run or when a run fails and their profile does not contain the profile name, a warning message will be printed.
-does not contain the profile name, a warning message will be printed.
 
 ### Testing
 
@@ -74,12 +82,13 @@ If you want to add a new custom config file to `nf-core/configs` please test tha
 
 ```bash
 ## Example command for nf-core/rnaseq
-nextflow run nf-core/rnaseq --reads '*_R{1,2}.fastq.gz' --genome GRCh37 -c '[path to custom config]'
+nextflow run nf-core/rnaseq --reads '*_R{1,2}.fastq.gz' --genome GRCh37 -c '/path/to/custom.config'
 ```
 
 ### Documentation
 
-You will have to create a [Markdown document](https://www.markdownguide.org/getting-started/) outlining the details required to use the custom config file within your organisation. You might orientate yourself using the [Template](docs/template.md) that we provide and filling out the information for your cluster there.
+You will have to create a [Markdown document](https://www.markdownguide.org/getting-started/) outlining the details required to use the custom config file within your organisation.
+You might orientate yourself using the [Template](docs/template.md) that we provide and filling out the information for your cluster there.
 
 See [`nf-core/configs/docs`](https://github.com/nf-core/configs/tree/master/docs) for examples.
 
@@ -113,7 +122,10 @@ Currently documentation is available for the following systems:
 
 ### Uploading to `nf-core/configs`
 
-[Fork](https://help.github.com/articles/fork-a-repo/) the `nf-core/configs` repository to your own GitHub account. Within the local clone of your fork add the custom config file to the [`conf/`](https://github.com/nf-core/configs/tree/master/conf) directory, and the documentation file to the [`docs/`](https://github.com/nf-core/configs/tree/master/docs) directory. You will also need to edit and add your custom profile to the [`nfcore_custom.config`](https://github.com/nf-core/configs/blob/master/nfcore_custom.config) file in the top-level directory of the clone.
+[Fork](https://help.github.com/articles/fork-a-repo/) the [`nf-core/configs`](https://github.com/nf-core/configs/) repository to your own GitHub account.
+Within the local clone of your fork add the custom config file to the [`conf/`](https://github.com/nf-core/configs/tree/master/conf) directory, and the documentation file to the [`docs/`](https://github.com/nf-core/configs/tree/master/docs) directory.
+You will also need to edit and add your custom profile to the [`nfcore_custom.config`](https://github.com/nf-core/configs/blob/master/nfcore_custom.config) file in the top-level directory of the clone.
+You will also need to edit and add your custom profile to the [`README.md`](https://github.com/nf-core/configs/blob/master/README.md) file in the top-level directory of the clone.
 
 Afterwards, make sure to edit the `.github/main.yml` file and add your profile name to the alphabetically sorted `profile:` scope. This way, it will be tested automatically using GitHub Actions. If you forget to do this, tests will fail and complain about that.
 
@@ -121,6 +133,87 @@ Commit and push these changes to your local clone on GitHub, and then [create a
 
 We will be notified automatically when you have created your pull request, and providing that everything adheres to nf-core guidelines we will endeavour to approve your pull request as soon as possible.
 
+## Adding a new pipeline-specific config
+
+Sometimes it may be desirable to have configuration options for an institute that are specific to a single nf-core pipeline.
+Such options should not be added to the main institutional config, as this will be applied to all pipelines.
+Instead, we can create a pipeline-specific institutional config file.
+
+:warning: Remember to replace the `<PIPELINE>` and `<PROFILE>` placeholders with the pipeline name and profile name in the following examples
+
+Institutional configs work because the pipeline `nextflow.config` file loads the [`nf-core/configs/nfcore_custom.config` config file](https://github.com/nf-core/configs/blob/master/nfcore_custom.config), which in turn loads the institutional configuration file based on the profile `<PROFILE>` supplied on the command line.
+
+To add in pipeline-specific institutional configs, we add a second `includeConfig` call in the pipeline `nextflow.config` file, which loads the `pipeline/<PIPELINE>.config` file from the [`nf-core/configs`](https://github.com/nf-core/configs) repo.
+This file has `<PIPELINE>` specific institution configuration again with different profiles `<PROFILE>`.
+
+The pipeline `nextflow.config` file should first load the generic institutional configuration file and then the pipeline-specific institutional configuration file.
+Each configuration file will add new params and overwrite the params already existing.
+
+Note that pipeline-specific configs are not required and should only be added if needed.
+
+### Pipeline-specific documentation
+
+Currently documentation is available for the following pipeline within the specific profile:
+
+* sarek
+  * [MUNIN](docs/pipeline/sarek/munin.md)
+
+### Enabling pipeline-specific configs within a pipeline
+
+:warning: **This has to be done on a fork of the `nf-core/<PIPELINE>` repository.**
+
+[Fork](https://help.github.com/articles/fork-a-repo/) the `nf-core/<PIPELINE>` repository to your own GitHub account.
+Within the local clone of your fork, if not already present, add the following to `nextflow.config` **after** the code that loads the generic nf-core/configs config file:
+
+```nextflow
+// Load nf-core/<PIPELINE> custom profiles from different Institutions
+try {
+  includeConfig "${params.custom_config_base}/pipeline/<PIPELINE>.config"
+} catch (Exception e) {
+  System.err.println("WARNING: Could not load nf-core/config/<PIPELINE> profiles: ${params.custom_config_base}/pipeline/<PIPELINE>.config")
+}
+```
+
+Commit and push these changes to your local clone on GitHub, and then [create a pull request](https://help.github.com/articles/creating-a-pull-request-from-a-fork/) on the `nf-core/<PIPELINE>` GitHub repo with the appropriate information.
+
+We will be notified automatically when you have created your pull request, and providing that everything adheres to nf-core guidelines we will endeavour to approve your pull request as soon as possible.
+
+### Create the pipeline-specific `nf-core/configs` files
+
+:warning: This has to be done on a fork of the [`nf-core/configs`](https://github.com/nf-core/configs/) repository.
+
+[Fork](https://help.github.com/articles/fork-a-repo/) the [`nf-core/configs`](https://github.com/nf-core/configs/) repository to your own GitHub account.
+And add or edit the following files in the local clone of your fork.
+
+* `pipeline/<PIPELINE>.config`
+
+If not already created, create the `pipeline/<PIPELINE>.config` file, and add your custom profile to the profile scope
+
+```Groovy
+profiles {
+  <PROFILE> { includeConfig "${params.custom_config_base}/conf/pipeline/<PIPELINE>/<PROFILE>.config" }
+}
+```
+
+* `conf/pipeline/<PIPELINE>/<PROFILE>.config`
+
+Add the custom configuration file to the `conf/pipeline/<PIPELINE>/` directory.
+Make sure to add an extra `params` section with `params.config_profile_description`, `params.config_profile_contact` to the top of `pipeline/<PIPELINE>.config` and set to reasonable values.
+Users will get information on who wrote the pipeline-specific configuration profile then when executing the nf-core pipeline and can report back if there are things missing for example.
+
+* `docs/pipeline/<PIPELINE>/<PROFILE>.md`
+
+Add the documentation file to the `docs/pipeline/<PIPELINE>/` directory.
+You will also need to edit and add your custom profile to the [`README.md`](https://github.com/nf-core/configs/blob/master/README.md) file in the top-level directory of the clone.
+
+* `README.md`
+
+Edit this file, and add the new pipeline-specific institutional profile to the list in the section Pipeline specific documentation
+
+Commit and push these changes to your local clone on GitHub, and then [create a pull request](https://help.github.com/articles/creating-a-pull-request-from-a-fork/) on the `nf-core/configs` GitHub repo with the appropriate information.
+In the pull-request description, add a link to the repository specific pull-request(s) that use this new code.
+Both PRs will need to be merged at the approximately the same time.
+
 ## Help
 
-If you have any questions or issues please send us a message on [Slack](https://nf-co.re/join/slack).
+If you have any questions or issues please send us a message on [Slack](https://nfcore.slack.com/channels/configs).

conf/munin.config

@@ -1,8 +1,18 @@
-//Profile config names for nf-core/configs
+// Profile config names for nf-core/configs
+
 params {
+  // Specific nf-core/configs params
+  config_profile_contact = 'Szilveszter Juhos (@szilvajuhos)'
   config_profile_description = 'MUNIN profile provided by nf-core/configs.'
-  config_profile_contact = 'Szilveszter Juhos (@szilva)'
+  config_profile_url = 'https://ki.se/forskning/barntumorbanken'
-  config_profile_url = ''
+
+  // Local AWS iGenomes reference file paths on munin
+  igenomes_base = '/data1/references/igenomes/'
+
+  // General cpus/memory/time requirements
+  max_cpus = 46
+  max_memory = 752.GB
+  max_time = 72.h
 }
 
 process {
@@ -10,9 +20,9 @@ process {
   maxForks = 46
 }
 
-// To use singularity, use nextflow run -profile munin,singularity
 singularity {
   enabled = true
+  cacheDir = '/data1/containers/'
 }
 
 // To use docker, use nextflow run -profile munin,docker
@@ -21,13 +31,3 @@ docker {
   mountFlags = 'z'
   fixOwnership = true
 }
-
-params {
-  // general params
-  max_memory = 752.GB
-  max_cpus = 46
-  max_time = 72.h
-
-  // Local AWS iGenomes reference file paths on munin
-  igenomes_base = '/data1/references/igenomes/'
-}

conf/pipeline/sarek/munin.config

@@ -0,0 +1,24 @@
+// Profile config names for nf-core/configs
+
+params {
+  // Specific nf-core/configs params
+  config_profile_contact = 'Maxime Garcia (@MaxUlysse)'
+  config_profile_description = 'nf-core/sarek MUNIN profile provided by nf-core/configs'
+
+  // Specific nf-core/sarek params
+  annotation_cache = true
+  pon = '/data1/PON/vcfs/BTB.PON.vcf.gz'
+  pon_index = '/data1/PON/vcfs/BTB.PON.vcf.gz.tbi'
+  snpEff_cache = '/data1/cache/snpEff/'
+  vep_cache = '/data1/cache/VEP/'
+}
+
+// Specific nf-core/sarek process configuration
+process {
+  withLabel:sentieon {
+    if (params.sentieon) {
+      module = 'sentieon/201808.05'
+      container = null
+    }
+  }
+}

docs/pipeline/sarek/munin.md

@@ -0,0 +1,22 @@
+# nf-core/configs: MUNIN sarek specific configuration
+
+Extra specific configuration for sarek pipeline
+
+## Usage
+
+To use, run the pipeline with `-profile munin`.
+
+This will download and launch the sarek specific [`munin.config`](../conf/pipeline/sarek/munin.config) which has been pre-configured with a setup suitable for the MUNIN cluster.
+
+Example: `nextflow run nf-core/sarek -profile munin`
+
+## Sarek specific configurations for MUNIN
+
+Specific configurations for MUNIN has been made for sarek.
+
+* Params `annotation_cache` set to `true`
+* Path to `snpEff_cache`: `/data1/cache/snpEff/`
+* Path to `vep_cache`: `/data1/cache/VEP/`
+* Path to `pon`: `/data1/PON/vcfs/BTB.PON.vcf.gz`
+* Path to `pon_index`: `/data1/PON/vcfs/BTB.PON.vcf.gz.tbi`
+* Load module `Sentieon` for Processes with `sentieon` labels

pipeline/sarek.config

@@ -0,0 +1,13 @@
+/*
+ * -------------------------------------------------
+ *  nfcore/sarek custom profile Nextflow config file
+ * -------------------------------------------------
+ * Config options for custom environments.
+ * Cluster-specific config options should be saved
+ * in the conf/pipeline/sarek folder and imported
+ * under a profile name here.
+ */
+
+profiles {
+  munin { includeConfig "${params.custom_config_base}/conf/pipeline/sarek/munin.config" }
+}