Merge pull request #424 from nf-core/demux-awstower

feat(demultiplex): Add optimized transfers for bases2fastq

README.md

@@ -206,6 +206,8 @@ Currently documentation is available for the following pipelines within specific
   - [SBC_SHARC](docs/pipeline/atacseq/sbc_sharc.md)
 - chipseq
   - [SBC_SHARC](docs/pipeline/chipseq/sbc_sharc.md)
+- demultiplex
+  - [AWS_TOWER](docs/pipeline/demultiplex/aws_tower.md)
 - eager
   - [EVA](docs/pipeline/eager/eva.md)
 - mag

conf/pipeline/demultiplex/aws_tower.config

@@ -0,0 +1,29 @@
+// Profile config names for nf-core/configs
+
+params {
+  // Specific nf-core/configs params
+  config_profile_contact = 'Edmund Miller(@emiller88)'
+  config_profile_description = 'nf-core/demultiplex AWS Tower profile provided by nf-core/configs'
+}
+
+aws {
+   batch {
+      maxParallelTransfers = 24
+      maxTransferAttempts = 3
+   }
+   client {
+      maxConnections = 24
+      uploadMaxThreads = 24
+      maxErrorRetry = 3
+      socketTimeout = 3600000
+      uploadRetrySleep = 1000
+      uploadChunkSize = 32.MB
+   }
+}
+
+process {
+    withName: BASES2FASTQ {
+        cpus = 16
+        memory = 48.GB
+    }
+}

docs/pipeline/demultiplex/aws_tower.md

@@ -0,0 +1,19 @@
+# nf-core/configs: AWS Tower Demultiplex specific configuration
+
+Extra specific configuration for demultiplex pipeline
+
+## Usage
+
+To use, run the pipeline with `-profile aws_tower`.
+
+This will download and launch the demultiplex specific [`aws_tower.config`](../../../conf/pipeline/demultiplex/aws_tower.config) which has been pre-configured with a setup suitable for AWS batch through tower.
+
+Example: `nextflow run nf-core/demultiplex -profile aws_tower`
+
+## eager specific configurations for eva
+
+Specific configurations for AWS has been made for demultiplex.
+
+### General profiles
+
+- The general AWS Tower profile runs with default nf-core/demultiplex parameters, but with modifications to account file transfer speed and optimized bases2fastq resources.

pipeline/demultiplex.config

@@ -0,0 +1,13 @@
+/*
+ * -------------------------------------------------
+ *  nfcore/demultiplex custom profile Nextflow config file
+ * -------------------------------------------------
+ * Config options for custom environments.
+ * Cluster-specific config options should be saved
+ * in the conf/pipeline/demultiplex folder and imported
+ * under a profile name here.
+ */
+
+profiles {
+  aws_tower { includeConfig "${params.custom_config_base}/conf/pipeline/demultiplex/aws_tower.config" }
+}