- [Enabling pipeline-specific configs within a pipeline](#enabling-pipeline-specific-configs-within-a-pipeline)
- [Create the pipeline-specific `nf-core/configs` files](#create-the-pipeline-specific-nf-coreconfigs-files)
- [Help](#help)
## Using an existing config
@ -86,68 +86,68 @@ See [`nf-core/configs/docs`](https://github.com/nf-core/configs/tree/master/docs
Currently documentation is available for the following systems:
* [ABIMS](docs/abims.md)
* [ALICE](docs/alice.md)
* [AWSBATCH](docs/awsbatch.md)
* [AWS_TOWER](docs/aws_tower.md)
* [AZUREBATCH](docs/azurebatch.md)
* [BIGPURPLE](docs/bigpurple.md)
* [BI](docs/bi.md)
* [BINAC](docs/binac.md)
* [BIOHPC_GEN](docs/biohpc_gen.md)
* [CAMBRIDGE](docs/cambridge.md)
* [CBE](docs/cbe.md)
* [CCGA_DX](docs/ccga_dx.md)
* [CCGA_MED](docs/ccga_med.md)
* [CFC](docs/cfc.md)
* [CHEAHA](docs/cheaha.md)
* [Computerome](docs/computerome.md)
* [CRICK](docs/crick.md)
* [CZBIOHUB_AWS](docs/czbiohub.md)
* [DENBI_QBIC](docs/denbi_qbic.md)
* [EBC](docs/ebc.md)
* [EVA](docs/eva.md)
* [FGCZ](docs/fgcz.md)
* [GENOTOUL](docs/genotoul.md)
* [GENOUEST](docs/genouest.md)
* [GIS](docs/gis.md)
* [GOOGLE](docs/google.md)
* [HASTA](docs/hasta.md)
* [HEBBE](docs/hebbe.md)
* [ICR_DAVROS](docs/icr_davros.md)
* [IMPERIAL](docs/imperial.md)
* [JAX](docs/jax.md)
* [LUGH](docs/lugh.md)
* [MAESTRO](docs/maestro.md)
* [MARVIN](docs/marvin.md)
* [MPCDF](docs/mpcdf.md)
* [MUNIN](docs/munin.md)
* [NU_GENOMICS](docs/nu_genomics.md)
* [NIHBIOWULF](docs/nihbiowulf.md)
* [OIST](docs/oist.md)
* [PASTEUR](docs/pasteur.md)
* [PHOENIX](docs/phoenix.md)
* [PRINCE](docs/prince.md)
* [ROSALIND](docs/rosalind.md)
* [SANGER](docs/sanger.md)
* [SEG_GLOBE](docs/seg_globe.md)
* [UCT_HPC](docs/uct_hpc.md)
* [UNIBE_IBU](docs/unibe_ibu.md)
* [UPPMAX](docs/uppmax.md)
* [UTD_GANYMEDE](docs/utd_ganymede.md)
* [UTD_SYSBIO](docs/utd_sysbio.md)
* [UZH](docs/uzh.md)
* [VAI](docs/vai.md)
- [ABIMS](docs/abims.md)
- [ALICE](docs/alice.md)
- [AWSBATCH](docs/awsbatch.md)
- [AWS_TOWER](docs/aws_tower.md)
- [AZUREBATCH](docs/azurebatch.md)
- [BIGPURPLE](docs/bigpurple.md)
- [BI](docs/bi.md)
- [BINAC](docs/binac.md)
- [BIOHPC_GEN](docs/biohpc_gen.md)
- [CAMBRIDGE](docs/cambridge.md)
- [CBE](docs/cbe.md)
- [CCGA_DX](docs/ccga_dx.md)
- [CCGA_MED](docs/ccga_med.md)
- [CFC](docs/cfc.md)
- [CHEAHA](docs/cheaha.md)
- [Computerome](docs/computerome.md)
- [CRICK](docs/crick.md)
- [CZBIOHUB_AWS](docs/czbiohub.md)
- [DENBI_QBIC](docs/denbi_qbic.md)
- [EBC](docs/ebc.md)
- [EVA](docs/eva.md)
- [FGCZ](docs/fgcz.md)
- [GENOTOUL](docs/genotoul.md)
- [GENOUEST](docs/genouest.md)
- [GIS](docs/gis.md)
- [GOOGLE](docs/google.md)
- [HASTA](docs/hasta.md)
- [HEBBE](docs/hebbe.md)
- [ICR_DAVROS](docs/icr_davros.md)
- [IMPERIAL](docs/imperial.md)
- [JAX](docs/jax.md)
- [LUGH](docs/lugh.md)
- [MAESTRO](docs/maestro.md)
- [MARVIN](docs/marvin.md)
- [MPCDF](docs/mpcdf.md)
- [MUNIN](docs/munin.md)
- [NU_GENOMICS](docs/nu_genomics.md)
- [NIHBIOWULF](docs/nihbiowulf.md)
- [OIST](docs/oist.md)
- [PASTEUR](docs/pasteur.md)
- [PHOENIX](docs/phoenix.md)
- [PRINCE](docs/prince.md)
- [ROSALIND](docs/rosalind.md)
- [SANGER](docs/sanger.md)
- [SEG_GLOBE](docs/seg_globe.md)
- [UCT_HPC](docs/uct_hpc.md)
- [UNIBE_IBU](docs/unibe_ibu.md)
- [UPPMAX](docs/uppmax.md)
- [UTD_GANYMEDE](docs/utd_ganymede.md)
- [UTD_SYSBIO](docs/utd_sysbio.md)
- [UZH](docs/uzh.md)
- [VAI](docs/vai.md)
### Uploading to `nf-core/configs`
[Fork](https://help.github.com/articles/fork-a-repo/) the [`nf-core/configs`](https://github.com/nf-core/configs/) repository to your own GitHub account.
Within the local clone of your fork:
***add** the custom config file to the [`conf/`](https://github.com/nf-core/configs/tree/master/conf) directory
***add** the documentation file to the [`docs/`](https://github.com/nf-core/configs/tree/master/docs) directory
***edit** and add your custom profile to the [`nfcore_custom.config`](https://github.com/nf-core/configs/blob/master/nfcore_custom.config) file in the top-level directory of the clone
***edit** and add your custom profile to the [`README.md`](https://github.com/nf-core/configs/blob/master/README.md) file in the top-level directory of the clone
-**add** the custom config file to the [`conf/`](https://github.com/nf-core/configs/tree/master/conf) directory
-**add** the documentation file to the [`docs/`](https://github.com/nf-core/configs/tree/master/docs) directory
-**edit** and add your custom profile to the [`nfcore_custom.config`](https://github.com/nf-core/configs/blob/master/nfcore_custom.config) file in the top-level directory of the clone
-**edit** and add your custom profile to the [`README.md`](https://github.com/nf-core/configs/blob/master/README.md) file in the top-level directory of the clone
In order to ensure that the config file is tested automatically with GitHub Actions please add your profile name to the `profile:` scope (under strategy matrix) in [`.github/workflows/main.yml`](.github/workflows/main.yml). If you forget to do this the tests will fail with the error:
@ -186,25 +186,25 @@ Note that pipeline-specific configs are not required and should only be added if
Currently documentation is available for the following pipelines within specific profiles:
* ampliseq
* [BINAC](docs/pipeline/ampliseq/binac.md)
* [UPPMAX](docs/pipeline/ampliseq/uppmax.md)
* eager
* [EVA](docs/pipeline/eager/eva.md)
* rnafusion
* [MUNIN](docs/pipeline/rnafusion/munin.md)
* sarek
* [MUNIN](docs/pipeline/sarek/munin.md)
* [UPPMAX](docs/pipeline/sarek/uppmax.md)
* rnavar
* [MUNIN](docs/pipeline/rnavar/munin.md)
- ampliseq
- [BINAC](docs/pipeline/ampliseq/binac.md)
- [UPPMAX](docs/pipeline/ampliseq/uppmax.md)
- eager
- [EVA](docs/pipeline/eager/eva.md)
- rnafusion
- [MUNIN](docs/pipeline/rnafusion/munin.md)
- sarek
- [MUNIN](docs/pipeline/sarek/munin.md)
- [UPPMAX](docs/pipeline/sarek/uppmax.md)
- rnavar
- [MUNIN](docs/pipeline/rnavar/munin.md)
### Pipeline-specific documentation
Currently documentation is available for the following pipeline:
* viralrecon
* [genomes](docs/pipeline/viralrecon/genomes.md)
- viralrecon
- [genomes](docs/pipeline/viralrecon/genomes.md)
### Enabling pipeline-specific configs within a pipeline
@ -233,7 +233,7 @@ We will be notified automatically when you have created your pull request, and p
[Fork](https://help.github.com/articles/fork-a-repo/) the [`nf-core/configs`](https://github.com/nf-core/configs/) repository to your own GitHub account.
And add or edit the following files in the local clone of your fork.
*`pipeline/<PIPELINE>.config`
-`pipeline/<PIPELINE>.config`
If not already created, create the `pipeline/<PIPELINE>.config` file, and add your custom profile to the profile scope
@ -243,18 +243,18 @@ profiles {
}
```
*`conf/pipeline/<PIPELINE>/<PROFILE>.config`
-`conf/pipeline/<PIPELINE>/<PROFILE>.config`
Add the custom configuration file to the `conf/pipeline/<PIPELINE>/` directory.
Make sure to add an extra `params` section with `params.config_profile_description`, `params.config_profile_contact` to the top of `pipeline/<PIPELINE>.config` and set to reasonable values.
Users will get information on who wrote the pipeline-specific configuration profile then when executing the nf-core pipeline and can report back if there are things missing for example.
*`docs/pipeline/<PIPELINE>/<PROFILE>.md`
-`docs/pipeline/<PIPELINE>/<PROFILE>.md`
Add the documentation file to the `docs/pipeline/<PIPELINE>/` directory.
You will also need to edit and add your custom profile to the [`README.md`](https://github.com/nf-core/configs/blob/master/README.md) file in the top-level directory of the clone.
*`README.md`
-`README.md`
Edit this file, and add the new pipeline-specific institutional profile to the list in the section Pipeline specific documentation
@ -15,4 +15,4 @@ has finished successfully because it can get quite large, and all of the main ou
> NB: You will need an account to use the Cambridge HPC cluster in order to run the pipeline. If in doubt contact IT.
> NB: Nextflow will need to submit the jobs via SLURM to the Cambridge HPC cluster and as such the commands above will have to be executed on one of the login
@ -5,4 +5,5 @@ Deployment and testing of nf-core pipelines at the CCGA DX cluster is on-going.
To use, run the pipeline with `-profile ccga_dx`. This will download and launch the [`ccga_dx.config`](../conf/ccga_dx.config) which has been pre-configured with a setup suitable for the CCGA cluster. Using this profile, a docker image containing all of the required software will be downloaded, and converted to a Singularity image before execution of the pipeline.
Before running the pipeline you will need to have Nextflow installed.
> NB: Access to the CCGA DX cluster is restricted to IKMB/CCGA employes. Please talk to Marc Hoeppner to get access (@marchoeppner).
@ -17,7 +17,7 @@ All of the intermediate files required to run the pipeline will be stored in the
> NB: You will need an account to use the HPC cluster on Cheaha in order to run the pipeline. If in doubt contact UAB IT Research Computing.</br></br>
> NB: Nextflow will need to submit the jobs via SLURM to the HPC cluster and as such the commands above will have to be executed on one of the login nodes (or alternatively in an interactive partition, but be aware of time limit). If in doubt contact UAB IT Research Computing.</br></br>
>NB: Instead of using `module load Nextflow`, you may instead create a conda environment (e.g: `conda create -p $USER_DATA/nf-core_nextflow_env nf-core nextflow`) if you would like to have a more personalized environment of Nextflow (versions which may not be modules yet) and nf-core tools. This __requires__ you to instead do the following:
>NB: Instead of using `module load Nextflow`, you may instead create a conda environment (e.g: `conda create -p $USER_DATA/nf-core_nextflow_env nf-core nextflow`) if you would like to have a more personalized environment of Nextflow (versions which may not be modules yet) and nf-core tools. This **requires** you to instead do the following:
@ -22,7 +22,7 @@ Now you can run pipelines with abandon!
### 2. Make a GitHub repo for your workflows (optional :)
To make sharing your pipelines and commands easy between your teammates, it's best to share code in a GitHub repository. One way is to store the commands in a Makefile ([example](https://github.com/czbiohub/kh-workflows/blob/master/nf-kmer-similarity/Makefile)) which can contain multiple `nextflow run` commands so that you don't need to remember the S3 bucket or output directory for every single one. [Makefiles](https://kbroman.org/minimal_make/) are broadly used in the software community for running many complex commands. Makefiles can have a lot of dependencies and be confusing, so we're only going to write *simple* Makefiles.
To make sharing your pipelines and commands easy between your teammates, it's best to share code in a GitHub repository. One way is to store the commands in a Makefile ([example](https://github.com/czbiohub/kh-workflows/blob/master/nf-kmer-similarity/Makefile)) which can contain multiple `nextflow run` commands so that you don't need to remember the S3 bucket or output directory for every single one. [Makefiles](https://kbroman.org/minimal_make/) are broadly used in the software community for running many complex commands. Makefiles can have a lot of dependencies and be confusing, so we're only going to write _simple_ Makefiles.
```bash
rnaseq:
@ -125,7 +125,7 @@ For Human and Mouse, we use [GENCODE](https://www.gencodegenes.org/) gene annota
## High Priority Queue
If you would like to run with the *High Priority* queue, specify the `highpriority` config profile after `czbiohub_aws`. When applied after the main `czbiohub_aws` config, it overwrites the process `queue` identifier.
If you would like to run with the _High Priority_ queue, specify the `highpriority` config profile after `czbiohub_aws`. When applied after the main `czbiohub_aws` config, it overwrites the process `queue` identifier.
To use it, submit your run with with `-profile czbiohub_aws,highpriority`.
The configuration profile design is very simple. If your process exceeds 64GB memory or 16 cpus, it is sent to the `highmem` queue. If not, it is sent to the `normal` queue. Please do not use the `MSC` queue, this is reserved for Masters students.
@ -16,34 +16,34 @@ Specific configurations for `MUNIN` has been made for rnavar.
Genome references
* Path to `fasta`: `/data1/references/CTAT_GenomeLib_v37_Mar012021/GRCh38_gencode_v37_CTAT_lib_Mar012021.plug-n-play/ctat_genome_lib_build_dir/ref_genome.fa`
* Path to `fasta_fai`: `/data1/references/CTAT_GenomeLib_v37_Mar012021/GRCh38_gencode_v37_CTAT_lib_Mar012021.plug-n-play/ctat_genome_lib_build_dir/ref_genome.fa.fai`
* Path to `gtf`: `/data1/references/CTAT_GenomeLib_v37_Mar012021/GRCh38_gencode_v37_CTAT_lib_Mar012021.plug-n-play/ctat_genome_lib_build_dir/ref_annot.gtf`
* Path to `gene_bed`: `/data1/references/CTAT_GenomeLib_v37_Mar012021/GRCh38_gencode_v37_CTAT_lib_Mar012021.plug-n-play/ctat_genome_lib_build_dir/ref_annot.bed`
- Path to `fasta`: `/data1/references/CTAT_GenomeLib_v37_Mar012021/GRCh38_gencode_v37_CTAT_lib_Mar012021.plug-n-play/ctat_genome_lib_build_dir/ref_genome.fa`
- Path to `fasta_fai`: `/data1/references/CTAT_GenomeLib_v37_Mar012021/GRCh38_gencode_v37_CTAT_lib_Mar012021.plug-n-play/ctat_genome_lib_build_dir/ref_genome.fa.fai`
- Path to `gtf`: `/data1/references/CTAT_GenomeLib_v37_Mar012021/GRCh38_gencode_v37_CTAT_lib_Mar012021.plug-n-play/ctat_genome_lib_build_dir/ref_annot.gtf`
- Path to `gene_bed`: `/data1/references/CTAT_GenomeLib_v37_Mar012021/GRCh38_gencode_v37_CTAT_lib_Mar012021.plug-n-play/ctat_genome_lib_build_dir/ref_annot.bed`
Known genome resources
* Path to `dbsnp`: `/data1/references/annotations/GATK_bundle/dbsnp_146.hg38.vcf.gz`
* Path to `dbsnp_tbi`: `/data1/references/annotations/GATK_bundle/dbsnp_146.hg38.vcf.gz.tbi`
* Path to `known_indels`: `/data1/references/annotations/GATK_bundle/Mills_and_1000G_gold_standard.indels.hg38.vcf.gz`
* Path to `known_indels_tbi`: `/data1/references/annotations/GATK_bundle/Mills_and_1000G_gold_standard.indels.hg38.vcf.gz.tbi`
- Path to `dbsnp`: `/data1/references/annotations/GATK_bundle/dbsnp_146.hg38.vcf.gz`
- Path to `dbsnp_tbi`: `/data1/references/annotations/GATK_bundle/dbsnp_146.hg38.vcf.gz.tbi`
- Path to `known_indels`: `/data1/references/annotations/GATK_bundle/Mills_and_1000G_gold_standard.indels.hg38.vcf.gz`
- Path to `known_indels_tbi`: `/data1/references/annotations/GATK_bundle/Mills_and_1000G_gold_standard.indels.hg38.vcf.gz.tbi`
STAR index
* Path to `star_index`: `/data1/references/CTAT_GenomeLib_v37_Mar012021/GRCh38_gencode_v37_CTAT_lib_Mar012021.plug-n-play/ctat_genome_lib_build_dir/STAR.2.7.9a_2x151bp/`
* Params `read_length` set to `151`
- Path to `star_index`: `/data1/references/CTAT_GenomeLib_v37_Mar012021/GRCh38_gencode_v37_CTAT_lib_Mar012021.plug-n-play/ctat_genome_lib_build_dir/STAR.2.7.9a_2x151bp/`
- Params `read_length` set to `151`
Variant annotation configurations
* Params `annotation_cache` and `cadd_cache` set to `true`
* Params `snpeff_db` set to `GRCh38.99`
* Params `vep_cache_version` set to `99`
* Params `vep_genome` set to `GRCh38`
* Path to `snpeff_cache`: `/data1/cache/snpEff/`
* Path to `vep_cache`: `/data1/cache/VEP/`
* Path to `pon`: `/data1/PON/vcfs/BTB.PON.vcf.gz`
* Path to `pon_index`: `/data1/PON/vcfs/BTB.PON.vcf.gz.tbi`
* Path to `cadd_indels`: `/data1/cache/CADD/v1.4/InDels.tsv.gz`
* Path to `cadd_indels_tbi`: `/data1/cache/CADD/v1.4/InDels.tsv.gz.tbi`
* Path to `cadd_wg_snvs`: `/data1/cache/CADD/v1.4/whole_genome_SNVs.tsv.gz`
* Path to `cadd_wg_snvs_tbi`: `/data1/cache/CADD/v1.4/whole_genome_SNVs.tsv.gz.tbi`
- Params `annotation_cache` and `cadd_cache` set to `true`
- Params `snpeff_db` set to `GRCh38.99`
- Params `vep_cache_version` set to `99`
- Params `vep_genome` set to `GRCh38`
- Path to `snpeff_cache`: `/data1/cache/snpEff/`
- Path to `vep_cache`: `/data1/cache/VEP/`
- Path to `pon`: `/data1/PON/vcfs/BTB.PON.vcf.gz`
- Path to `pon_index`: `/data1/PON/vcfs/BTB.PON.vcf.gz.tbi`
- Path to `cadd_indels`: `/data1/cache/CADD/v1.4/InDels.tsv.gz`
- Path to `cadd_indels_tbi`: `/data1/cache/CADD/v1.4/InDels.tsv.gz.tbi`
- Path to `cadd_wg_snvs`: `/data1/cache/CADD/v1.4/whole_genome_SNVs.tsv.gz`
- Path to `cadd_wg_snvs_tbi`: `/data1/cache/CADD/v1.4/whole_genome_SNVs.tsv.gz.tbi`
For security reasons, there is no internet access on `bianca` so you can't download from or upload files to the cluster directly.
Before running a nf-core pipeline on `bianca` you will first have to download the pipeline and singularity images needed elsewhere and transfer them via the `wharf` area to your own `bianca` project.
Edmund Miller
2 years ago