1
0
Fork 0
mirror of https://github.com/MillironX/nf-configs.git synced 2024-11-24 17:19:54 +00:00

Merge pull request #100 from MaxUlysse/UPPMAX

Improve UPPMAX configs
This commit is contained in:
Maxime Garcia 2019-12-06 10:52:44 +01:00 committed by GitHub
commit 9497b78149
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
6 changed files with 49 additions and 42 deletions

View file

@ -16,7 +16,7 @@ jobs:
needs: test_all_profiles
strategy:
matrix:
profile: ['awsbatch', 'bigpurple', 'binac', 'cbe', 'ccga_dx', 'ccga', 'cfc', 'crick', 'denbi_qbic', 'genotoul', 'genouest', 'gis', 'hebbe', 'kraken', 'munin', 'pasteur', 'phoenix', 'prince', 'shh', 'uct_hex', 'uppmax_devel', 'uppmax', 'uzh']
profile: ['awsbatch', 'bigpurple', 'binac', 'cbe', 'ccga_dx', 'ccga', 'cfc', 'crick', 'denbi_qbic', 'genotoul', 'genouest', 'gis', 'hebbe', 'kraken', 'munin', 'pasteur', 'phoenix', 'prince', 'shh', 'uct_hex', 'uppmax', 'uzh']
steps:
- uses: actions/checkout@v1
- name: Install Nextflow

View file

@ -117,7 +117,6 @@ Currently documentation is available for the following systems:
* [SHH](docs/shh.md)
* [UCT_HEX](docs/uct_hex.md)
* [UPPMAX](docs/uppmax.md)
* [UPPMAX_DEVEL](docs/uppmax.md)
* [UZH](docs/uzh.md)
### Uploading to `nf-core/configs`

View file

@ -17,9 +17,36 @@ process {
params {
saveReference = true
max_memory = 128.GB
max_memory = 125.GB
max_cpus = 16
max_time = 240.h
// illumina iGenomes reference file paths on UPPMAX
igenomes_base = '/sw/data/uppnex/igenomes/'
}
def hostname = "hostname".execute().text.trim()
if (hostname ==~ "b.*") {
params.max_memory = 109.GB
}
if (hostname ==~ "i.*") {
params.max_memory = 250.GB
}
if (hostname ==~ "r.*") {
params.max_cpus = 20
}
profiles {
devel {
params {
config_profile_description = 'Testing & development profile for UPPMAX, provided by nf-core/configs.'
// Max resources to be requested by a devel job
max_memory = 120.GB
max_time = 1.h
}
executor.queueSize = 1
process.queue = 'devel'
}
}

View file

@ -1,21 +0,0 @@
// Profile config names for nf-core/configs
params {
config_profile_description = 'Testing & development profile for UPPMAX, provided by nf-core/configs.'
config_profile_contact = 'Phil Ewels (@ewels)'
config_profile_url = 'https://www.uppmax.uu.se/'
}
executor {
name = 'slurm'
queueSize = 1
}
process {
queue = 'devel'
}
params {
// Max resources to be requested by a devel job
max_memory = 120.GB
max_cpus = 16
max_time = 1.h
}

View file

@ -4,7 +4,9 @@ All nf-core pipelines have been successfully configured for use on the Swedish U
## Using the UPPMAX config profile
To use, run the pipeline with `-profile uppmax` (one hyphen). This will download and launch the [`uppmax.config`](../conf/uppmax.config) which has been pre-configured with a setup suitable for the UPPMAX servers. Using this profile, a docker image containing all of the required software will be downloaded, and converted to a Singularity image before execution of the pipeline.
To use, run the pipeline with `-profile uppmax` (one hyphen).
This will download and launch the [`uppmax.config`](../conf/uppmax.config) which has been pre-configured with a setup suitable for the UPPMAX servers.
Using this profile, a docker image containing all of the required software will be downloaded, and converted to a Singularity image before execution of the pipeline.
In addition to this config profile, you will also need to specify an UPPMAX project id.
You can do this with the `--project` flag (two hyphens) when launching nextflow. For example:
@ -23,18 +25,6 @@ Just run Nextflow on a login node and it will handle everything else.
A local copy of the iGenomes resource has been made available on all UPPMAX clusters so you should be able to run the pipeline against any reference available in the `igenomes.config`.
You can do this by simply using the `--genome <GENOME_ID>` parameter.
## Running offline with Bianca
If running on Bianca, you will have no internet connection and these configs will not be loaded.
Please use the nf-core helper tool on a different system to download the required pipeline files, and transfer them to bianca.
This helper tool bundles the config files in this repo together with the pipeline files, so the profile will still be available.
Note that Bianca only allocates 7 GB memory per core so the max memory needs to be limited:
```bash
--max_memory "112GB"
```
## Getting more memory
If your nf-core pipeline run is running out of memory, you can run on a fat node with more memory using the following nextflow flags:
@ -50,13 +40,27 @@ Note that each job will still start with the same request as normal, but restart
All jobs will be submitted to fat nodes using this method, so it's only for use in extreme circumstances.
## Uppmax-devel config
## How to specify a UPPMAX cluster
If doing pipeline development work on Uppmax, this profile allows for faster testing.
The UPPMAX nf-core configuration profile uses the `hostname` of the active environment to automatically apply the following resource limits:
* `bianca`
* cpus available: 16 cpus
* memory available: 109 GB
* `irma`
* cpus available: 16 cpus
* memory available: 250 GB
* `rackham`
* cpus available: 20 cpus
* memory available: 125 GB
## Development config
If doing pipeline development work on UPPMAX, the `devel` profile allows for faster testing.
Applied after main UPPMAX config, it overwrites certain parts of the config and submits jobs to the `devcore` queue, which has much faster queue times.
All jobs are limited to 1 hour to be eligible for this queue and only one job allowed at a time.
It is not suitable for use with real data.
To use it, submit with `-profile uppmax-devel`.
To use it, submit with `-profile uppmax,devel`.

View file

@ -38,8 +38,6 @@ profiles {
shh { includeConfig "${params.custom_config_base}/conf/shh.config" }
uct_hex { includeConfig "${params.custom_config_base}/conf/uct_hex.config" }
uppmax { includeConfig "${params.custom_config_base}/conf/uppmax.config" }
uppmax_devel { includeConfig "${params.custom_config_base}/conf/uppmax.config";
includeConfig "${params.custom_config_base}/conf/uppmax_devel.config" }
uzh { includeConfig "${params.custom_config_base}/conf/uzh.config" }
}