Module/bcl2fastq (#1883)

* add bcl2fastq

* test fixes

* fixed tests

* add tests to workflow

* change container source

modules/bcl2fastq/.gitignore

@@ -0,0 +1,2 @@
+bcl-convert
+*.rpm

modules/bcl2fastq/Dockerfile

@@ -0,0 +1,17 @@
+# Dockerfile to create container with bcl2fastq
+# Push to nfcore/bcl2fastq:<VER>
+
+FROM debian:bullseye-slim
+LABEL authors="Matthias De Smet <matthias.desmet@ugent.be>" \
+    description="Docker image containing bcl2fastq"
+# Disclaimer: this container is not provided nor supported by Illumina
+# 'ps' command is needed by some nextflow executions to collect system stats
+# Install procps and clean apt cache
+RUN apt-get update \
+    && apt-get install -y \
+    procps \
+    && apt-get clean -y && rm -rf /var/lib/apt/lists/*
+# Link hostname cmd to fix hardcoded path
+RUN ln -s /bin/hostname /usr/bin/hostname
+COPY bcl2fastq /usr/local/bin/bcl2fastq
+RUN chmod +x /usr/local/bin/bcl2fastq

modules/bcl2fastq/LICENSE

@@ -0,0 +1,30 @@
+ILLUMINA END-USER SOFTWARE LICENSE AGREEMENT
+
+IMPORTANT-READ CAREFULLY. THIS IS A LICENSE AGREEMENT THAT YOU ARE REQUIRED TO ACCEPT BEFORE, DOWNLOADING, INSTALLING AND USING ANY SOFTWARE MADE AVAILABLE FROM THE ILLUMINA SUPPORT CENTER (https://support.illumina.com).
+
+CAREFULLY READ ALL THE TERMS AND CONDITIONS OF THIS LICENSE AGREEMENT BEFORE PROCEEDING WITH DOWNLOADING, INSTALLING, AND/OR USING THE SOFTWARE. YOU ARE NOT PERMITTED TO DOWNLOAD, INSTALL, AND/OR USE THE SOFTWARE UNTIL YOU HAVE AGREED TO BE BOUND BY ALL OF THE TERMS AND CONDITIONS OF THIS LICENSE AGREEMENT.  YOU REPRESENT AND WARRANT THAT YOU ARE DULY AUTHORIZED TO ACCEPT THE TERMS AND CONDITIONS OF THIS LICENSE AGREEMENT ON BEHALF OF YOUR EMPLOYER.
+
+Software made available through the Illumina Support Center is licensed, not sold, to you. Your license to each software program made available through the Illumina Support Center is subject to your prior acceptance of either this Illumina End-User Software License Agreement (“Agreement”), or a custom end user license agreement (“Custom EULA”), if one is provided with the software.  Any software that is subject to this Agreement is referred to herein as the “Software.” By accepting this Agreement, you agree the terms and conditions of this Agreement will apply to and govern any and all of your downloads, installations, and uses of each Illumina software program made available through the Illumina Support Center, except that your download, installation, and use of any software provided with a Custom EULA will be governed by the terms and conditions of the Custom EULA.
+
+This Agreement is made and entered into by and between Illumina, Inc., a Delaware corporation, having offices at 5200 Illumina Way, San Diego, CA 92122 (“Illumina”) and you as the end-user of the Software (hereinafter, “Licensee” or “you”).  All software, firmware, and associated media, printed materials, and online and electronic documentation, including any updates or upgrades thereof, made available through the Illumina Support Center (collectively, “Software”) provided to Licensee are for use solely by Licensee and the provisions herein WILL apply with respect to such Software.
+
+License Grant.  Subject to the terms and conditions of this Agreement, Illumina grants to Licensee, under the following terms and conditions, a personal, non-exclusive, revocable, non-transferable, non-sublicensable license, for its internal end-use purposes only, in the ordinary course of Licensee’s business to use the Software in executable object code form only, solely at the Licensee’s facility to, install and use the Software on a single computer accessible only by Licensee (and not on any public network or server), where the single computer is owned, leased, or otherwise substantially controlled by Licensee, for the purpose of processing and analyzing data generated from an Illumina genetic sequencing instrument owned and operated solely by Licensee (the “Product”). In the case of Software provided by Illumina in non-compiled form, Illumina grants Licensee a personal, non-exclusive,  non-sublicenseable, restricted right to compile, install, and use one copy of the Software solely for processing and analyzing data generated from the Product.
+License Restrictions. Except as expressly permitted in Section 1, Licensee may not make, have made, import, use, copy, reproduce, distribute, display, publish, sell, re-sell, lease, or sub-license the Software, in whole or in part, except as expressly provided for in this Agreement. Licensee may not modify, improve, translate, reverse engineer, decompile, disassemble, or create derivative works of the Software or otherwise attempt to (a) defeat, avoid, by-pass, remove, deactivate, or otherwise circumvent any software protection mechanisms in the Software including, without limitation, any such mechanism used to restrict or control the functionality of the Software, or (b) derive the source code or the underlying ideas, algorithms, structure, or organization form of the Software.  Licensee will not allow, at any time, including during and after the term of the license, the Software or any portions or copies thereof in any form to become available to any third parties.  Licensee may use the Software solely with genomic data that is generated using the Product; Licensee may not use the Software with any data generated from other products or instruments.  Licensee may not use the Software to perform any data analysis services for any third party.
+Ownership. The Software is protected by United States and international intellectual property laws. All right, title, and interest in and to the Software (including associated intellectual property rights) are and will remain vested in Illumina or Illumina’s affiliated companies or licensors. Licensee acknowledges that no rights, license or interest to any Illumina trademarks are granted hereunder. Licensee acknowledges that unauthorized reproduction or distribution of the Software, or any portion of it, may result in severe civil and criminal penalties.  Illumina reserves all rights in and to the Software not expressly granted to Licensee under this Agreement.
+Upgrades/Updates. Illumina may, at its sole discretion, provide updates or upgrades to the Software. In that case, Licensee WILL have the same rights and obligations under such updates or upgrades as it has for the versions of the Software initially provided to Licensee hereunder.  Licensee recognizes that Illumina is not obligated to provide any upgrades or updates to, or support for, the Software.
+Data Integrity/Loss. Licensee is responsible for the integrity and availability, including preventing the loss of data that Licensee generates, uses, analyzes, manages, or stores in connection with or through its use of the Software, including without limitation, investigating and implementing industry appropriate policies and procedures regarding the provision of access to Licensee’s data, monitoring access and use of Licensee’s data, conducting routine backups and archiving of Licensee’s data, and ensuring the adequacy of anti-virus software. Accordingly, Licensee agrees that Illumina is not responsible for any inability to access, loss or corruption of data as a result of Licensee’s use of the Software, and Illumina has no liability to Licensee in connection with such inability to access, loss or corruption of data.
+Term of License. This Agreement will be in effect from the time Licensee expressly accepts the terms and conditions of this license, or otherwise installs the Software, thereby accepting the terms and conditions contained herein, and will remain in effect until terminated. This license will otherwise terminate upon the conditions set forth in this Agreement, if revoked by Illumina, or if Licensee fails to comply with any term or condition of this Agreement including failure to pay any applicable license fee. Licensee agrees upon termination of this Agreement for any reason to immediately discontinue use of and un-install the Software and destroy all copies of the Software in its possession and/or under its control, and return or destroy, at Illumina’s option, any compact disks, floppy disks or other media provided by Illumina storing the Software thereon (together with any authorized copies thereof), as well as any documentation associated therewith
+Limited Warranty. Illumina warrants that, for a period of 6 months from the date of download or installation of the Software by Licensee, the Software will perform in all material respects in accordance with the accompanying documentation available on the Illumina Support Center. EXCEPT AND TO THE EXTENT EXPRESSLY PROVIDED IN THE FOREGOING, AND TO THE FULLEST EXTENT PERMITTED BY APPLICABLE LAW, THE SOFTWARE IS PROVIDED “AS IS” AND ILLUMINA EXPRESSLY DISCLAIMS ALL WARRANTIES AND CONDITIONS REGARDING THE SOFTWARE AND RESULTS GENERATED BY THE SOFTWARE, INCLUDING WITHOUT LIMITATION, TO THE FULLEST EXTENT PERMITTED BY APPLICABLE LAW, ALL OTHER EXPRESS OR IMPLIED WARRANTIES OR CONDITIONS OF MERCHANTABLE QUALITY, NON-INFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE, AND THOSE ARISING BY STATUTE OR OTHERWISE IN LAW OR FROM A COURSE OF DEALING OR USAGE OF TRADE.  ILLUMINA DOES NOT WARRANT THAT THE FUNCTIONS CONTAINED IN THE SOFTWARE WILL MEET LICENSEE"S REQUIREMENTS, OR THAT THE OPERATION OF THE SOFTWARE WILL BE ERROR FREE OR UNINTERRUPTED.
+Limitation of Liability.
+(a) ILLUMINA’S ENTIRE LIABILITY AND LICENSEE"S EXCLUSIVE REMEDY UNDER THE LIMITED WARRANTY PROVISION  OF SECTION 7 ABOVE WILL BE, AT ILLUMINA’S OPTION, EITHER (i) RETURN OF THE PRICE PAID FOR THE SOFTWARE, OR (ii) REPAIR OR REPLACEMENT OF THE PORTIONS OF THE SOFTWARE THAT DO NOT COMPLY WITH ILLUMINA’S LIMITED WARRANTY. THIS LIMITED WARRANTY IS VOID AND ILLUMINA WILL HAVE NO LIABILITY AT ALL IF FAILURE OF THE SOFTWARE TO COMPLY WITH ILLUMINA LIMITED WARRANTY HAS RESULTED FROM: (w) FAILURE TO USE THE SOFTWARE IN ACCORDANCE WITH ILLUMINA’S THEN CURRENT USER MANUAL OR THIS AGREEMENT; (x) ACCIDENT, ABUSE, OR MISAPPLICATION; (y) PRODUCTS OR EQUIPMENT NOT SPECIFIED BY ILLUMINA AS BEING COMPATIBLE WITH THE SOFTWARE; OR (z) IF LICENSEE HAS NOT NOTIFIED ILLUMINA IN WRITING OF THE DEFECT WITHIN THE ABOVE WARRANTY PERIOD.
+
+(b) TO THE FULLEST EXTENT PERMITTED BY APPLICABLE LAW, IN NO EVENT WILL ILLUMINA BE LIABLE UNDER ANY THEORY OF CONTRACT, TORT, STRICT LIABILITY OR OTHER LEGAL OR EQUITABLE THEORY FOR ANY PERSONAL INJURY OR ANY INDIRECT, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, EVEN IF ILLUMINA HAS BEEN ADVISED OF THE POSSIBILITY THEREOF INCLUDING, WITHOUT LIMITATION, LOST PROFITS, LOST DATA, INTERRUPTION OF BUSINESS, LOST BUSINESS REVENUE, OTHER ECONOMIC LOSS, OR ANY LOSS OF RECORDED DATA ARISING OUT OF THE USE OF OR INABILITY TO USE THE SOFTWARE. EXCEPT AND TO THE EXTENT EXPRESSLY PROVIDED IN SECTION 7 AND 8(a) ABOVE OR AS OTHERWISE PERMITTED BY LAW, IN NO EVENT WILL ILLUMINA’S TOTAL LIABILITY TO LICENSEE FOR ALL DAMAGES (OTHER THAN AS MAY BE REQUIRED BY APPLICABLE LAW IN CASES INVOLVING PERSONAL INJURY) EXCEED THE AMOUNT OF $500 USD.  THE FOREGOING LIMITATIONS WILL APPLY EVEN IF THE ABOVE STATED REMEDY FAILS OF ITS ESSENTIAL PURPOSE.
+
+Survival. The limitations of liability and ownership rights of Illumina contained herein and Licensee’s obligations following termination of this Agreement WILL survive the termination of this Agreement for any reason.
+Research Use Only. The Software is labeled with a For Research Use Only or similar labeling statement and the performance characteristics of the Software have not been established and the Software is not for use in diagnostic procedures.  Licensee acknowledges and agrees that (i) the Software has not been approved, cleared, or licensed by the United States Food and Drug Administration or any other regulatory entity whether foreign or domestic for any specific intended use, whether research, commercial, diagnostic, or otherwise, and (ii) Licensee must ensure it has any regulatory approvals that are necessary for Licensee’s intended uses of the Software.  Licensee will comply with all applicable laws and regulations when using and maintaining the Software.
+General. Licensee may not sublicense, assign, share, pledge, rent or transfer any of its rights under this Agreement in relation to the Software or any portion thereof including documentation. Illumina reserves the right to change this Agreement at any time. When Illumina makes any changes, Illumina will provide the updated Agreement, or a link to it, on Illumina’s website (www.illumina.com) and such updated Agreement WILL become effective immediately. Licensee’s continued access to or use of the Software represents Licensee’s agreement to any revised Agreement. If one or more provisions of this Agreement are found to be invalid or unenforceable, this Agreement WILL not be rendered inoperative but the remaining provisions WILL continue in full force and effect.  This Agreement constitutes the entire agreement between the parties with respect to the subject matter of this Agreement and merges all prior communications except that a “hard-copy” form of licensing agreement relating to the Software previously agreed to in writing by Illumina and Licensee WILL supersede and govern in the event of any conflicting provisions.
+Governing Law. This Agreement WILL be governed by and construed in accordance with the laws of the state of California, USA, without regard to its conflicts of laws principles, and independent of where a suit or action hereunder may be filed.
+U.S. Government End Users.  If Licensee is a branch agency or instrumentality of the United States Government, the following provision applies.  The Software is a “commercial item” as that term is defined at 48 C.F.R. 2.101, consisting of “commercial computer software” and “commercial computer software documentation,” as such terms are used in 48 C.F.R. 12.212 or 48 C.F.R. 227.7202 (as applicable).  Consistent with 48 C.F.R. 12.212 and 48 C.F.R. 227.7202-1 through 227.7202-4, all United States Government end users acquire the Software with only those rights set forth herein.
+Contact. Any questions regarding legal rights, duties, obligations, or restrictions associated with the software hereunder should be directed to Illumina, Inc., 5200 Illumina Way, San Diego, CA 92122, Attention: Legal Department, Phone: (858) 202-4500, Fax: (858) 202-4599, web site: www.illumina.com <http://www.illumina.com>.
+Third Party Components. The Software may include third party software (“Third Party Programs”). Some of the Third Party Programs are available under open source or free software licenses. The License Agreement accompanying the Licensed Software does not alter any rights or obligations Licensee may have under those open source or free software licenses.  The licenses that govern the terms and conditions of use of the Third Party Programs included in the Licensed Software are provided in the READ ME provided with the Software.  The READ ME also contains copyright statements for the various open source software components (or portions thereof) that are distributed with the Licensed Software.
+END OF END-USER SOFTWARE LICENSE AGREEMENT.

modules/bcl2fastq/README.md

@@ -0,0 +1,17 @@
+# Updating the docker container and making a new module release
+
+bcl2fastq is a commercial tool from Illumina. The container provided for the bcl2fastq nf-core module is not provided nor supported by Illumina. Updating the bcl2fastq versions in the container and pushing the update to Dockerhub needs to be done manually.
+
+1. Navigate to the appropriate download page. - [bcl2fastq](https://support.illumina.com/sequencing/sequencing_software/bcl2fastq-conversion-software/downloads.html): download the rpm of the desired bcl2fastq version with `curl` or `wget`.
+2. Unpack the RPM package using `rpm2cpio bcl2fastq2-*.rpm | cpio -i --make-directories`. Place the executable located in `<unpack_dir>/usr/bin/bcl2fastq` in the same folder where the Dockerfile lies.
+3. Create and test the container:
+
+   ```bash
+   docker build . -t nfcore/bcl2fastq:<VERSION>
+   ```
+
+4. Access rights are needed to push the container to the Dockerhub nfcore organization, please ask a core team member to do so.
+
+   ```bash
+   docker push nfcore/bcl2fastq:<VERSION>
+   ```

modules/bcl2fastq/main.nf

@@ -0,0 +1,42 @@
+process BCL2FASTQ {
+    tag {"$meta.lane" ? "$meta.id"+"."+"$meta.lane" : "$meta.id" }
+    label 'process_high'
+
+    if (params.enable_conda) {
+        exit 1, "Conda environments cannot be used when using bcl2fastq. Please use docker or singularity containers."
+    }
+    container "nfcore/bcl2fastq:2.20.0.422"
+
+    input:
+    tuple val(meta), path(samplesheet), path(run_dir)
+
+    output:
+    tuple val(meta), path("**[!Undetermined]_S*_L00?_R?_00?.fastq.gz")  ,emit: fastq
+    tuple val(meta), path("**_S*_L00?_I?_00?.fastq.gz")                 ,optional:true ,emit: fastq_idx
+    tuple val(meta), path("Undetermined_S0_L00?_R?_00?.fastq.gz")       ,optional:true ,emit: undetermined
+    tuple val(meta), path("Undetermined_S0_L00?_I?_00?.fastq.gz")       ,optional:true, emit: undetermined_idx
+    tuple val(meta), path("Reports")                                    ,emit: reports
+    tuple val(meta), path("Stats")                                      ,emit: stats
+    tuple val(meta), path("**/InterOp/*.bin")                           ,emit: interop
+    path("versions.yml")                                                ,emit: versions
+
+    when:
+    task.ext.when == null || task.ext.when
+
+    script:
+    def args = task.ext.args ?: ''
+
+    """
+    bcl2fastq \\
+        $args \\
+        --output-dir . \\
+        --runfolder-dir ${run_dir} \\
+        --sample-sheet ${samplesheet} \\
+        --processing-threads ${task.cpus}
+
+    cat <<-END_VERSIONS > versions.yml
+    "${task.process}":
+        bcl2fastq: \$(bcl2fastq -V 2>&1 | grep -m 1 bcl2fastq | sed 's/^.*bcl2fastq v//')
+    END_VERSIONS
+    """
+}

modules/bcl2fastq/meta.yml

@@ -0,0 +1,61 @@
+name: "bcl2fastq"
+description: Demultiplex Illumina BCL files
+keywords:
+  - demultiplex
+  - illumina
+  - fastq
+tools:
+  - "bcl2fastq":
+      description: "Demultiplex Illumina BCL files"
+      homepage: "https://support.illumina.com/sequencing/sequencing_software/bcl2fastq-conversion-software"
+      documentation: "https://support.illumina.com/content/dam/illumina-support/documents/documentation/software_documentation/bcl2fastq/bcl2fastq2-v2-20-software-guide-15051736-03.pdf"
+      licence: "ILLUMINA"
+
+input:
+  - meta:
+      description: |
+        Groovy Map containing sample information
+        e.g. [ id:'test', single_end:false ]
+  - samplesheet:
+      type: file
+      description: "Input samplesheet"
+      pattern: "*.{csv}"
+  - run_dir:
+      type: directory
+      description: "Input run directory containing RunInfo.xml and BCL data"
+
+output:
+  - versions:
+      type: file
+      description: File containing software versions
+      pattern: "versions.yml"
+  - fastq:
+      type: file
+      description: Demultiplexed sample FASTQ files
+      pattern: "**_S*_L00?_R?_00?.fastq.gz"
+  - fastq_idx:
+      type: file
+      description: Optional demultiplexed index FASTQ files
+      pattern: "**_S*_L00?_I?_00?.fastq.gz"
+  - undetermined:
+      type: file
+      description: Optional undetermined sample FASTQ files
+      pattern: "Undetermined_S0_L00?_R?_00?.fastq.gz"
+  - undetermined_idx:
+      type: file
+      description: Optional undetermined index FASTQ files
+      pattern: "Undetermined_S0_L00?_I?_00?.fastq.gz"
+  - reports:
+      type: file
+      description: Demultiplexing Reports
+      pattern: "Reports/*"
+  - stats:
+      type: file
+      description: Statistics files
+      pattern: "Stats/*"
+  - interop:
+      type: file
+      description: Interop files
+      pattern: "*.{bin}"
+authors:
+  - "@matthdsm"

tests/config/pytest_modules.yml

@@ -210,6 +210,10 @@ bcftools/view:
   - modules/bcftools/view/**
   - tests/modules/bcftools/view/**
 
+bcl2fastq:
+  - modules/bcl2fastq/**
+  - tests/modules/bcl2fastq/**
+
 bclconvert:
   - modules/bclconvert/**
   - tests/modules/bclconvert/**

tests/modules/bcl2fastq/main.nf

@@ -0,0 +1,23 @@
+#!/usr/bin/env nextflow
+
+nextflow.enable.dsl = 2
+
+include { BCL2FASTQ } from '../../../modules/bcl2fastq/main.nf'
+include { UNTAR      } from '../../../modules/untar/main.nf'
+
+workflow test_bcl2fastq {
+    ch_flowcell = Channel.value([
+            [id:'test', lane:1 ], // meta map
+            file(params.test_data['homo_sapiens']['illumina']['test_flowcell_samplesheet'], checkIfExists: true),
+            file(params.test_data['homo_sapiens']['illumina']['test_flowcell'], checkIfExists: true)])
+
+    ch_flowcell
+    .multiMap { meta, ss, run ->
+        samplesheet: [meta, ss]
+        tar: [meta, run]
+    }.set{ ch_fc_split }
+
+    ch_flowcell_untar = ch_fc_split.samplesheet.join( UNTAR ( ch_fc_split.tar ).untar )
+
+    BCL2FASTQ (ch_flowcell_untar)
+}

tests/modules/bcl2fastq/nextflow.config

@@ -0,0 +1,12 @@
+process {
+
+    publishDir = { "${params.outdir}/${task.process.tokenize(':')[-1].tokenize('_')[0].toLowerCase()}" }
+    withName: BCL2FASTQ {
+        ext.args = {[
+            "--tiles s_1_1101"
+        ].join(" ").trim()}
+    }
+    withName: UNTAR {
+        publishDir = [ enabled: false ]
+    }
+}

tests/modules/bcl2fastq/test.yml

@@ -0,0 +1,33 @@
+- name: bcl2fastq test_bcl2fastq
+  command: nextflow run ./tests/modules/bcl2fastq -entry test_bcl2fastq -c ./tests/config/nextflow.config  -c ./tests/modules/bcl2fastq/nextflow.config
+  tags:
+    - bcl2fastq
+  files:
+    - path: output/bcl2fastq/Sample1_S1_L001_R1_001.fastq.gz
+      md5sum: e92fce7b86c6447b053d72c5cb4be89c
+    - path: output/bcl2fastq/Stats/AdapterTrimming.txt
+      md5sum: 48ed2b914b1246c0b5d8667525550946
+    - path: output/bcl2fastq/Stats/ConversionStats.xml
+      md5sum: 8fe0f57f3f5d256a0762dba75ac62d05
+    - path: output/bcl2fastq/Stats/DemultiplexingStats.xml
+      md5sum: 2047ff18f5b9107c084de06e9ff943ad
+    - path: output/bcl2fastq/Stats/DemuxSummaryF1L1.txt
+      md5sum: 03e5fd0c1e3079c5f8c7b4d0501b37ff
+    - path: output/bcl2fastq/Stats/FastqSummaryF1L1.txt
+      md5sum: 0c6f2d87ee183b84d1051cde9a5643d1
+    - path: output/bcl2fastq/Stats/Stats.json
+      md5sum: 8e5f038b8aa9e465599d3575f930e604
+    - path: output/bcl2fastq/flowcell/InterOp/ControlMetricsOut.bin
+      md5sum: 6d77b38d0793a6e1ce1e85706e488953
+    - path: output/bcl2fastq/flowcell/InterOp/CorrectedIntMetricsOut.bin
+      md5sum: 2bbf84d3be72734addaa2fe794711434
+    - path: output/bcl2fastq/flowcell/InterOp/ErrorMetricsOut.bin
+      md5sum: 38c88def138e9bb832539911affdb286
+    - path: output/bcl2fastq/flowcell/InterOp/ExtractionMetricsOut.bin
+      md5sum: 7497c3178837eea8f09350b5cd252e99
+    - path: output/bcl2fastq/flowcell/InterOp/IndexMetricsOut.bin
+      md5sum: 9e688c58a5487b8eaf69c9e1005ad0bf
+    - path: output/bcl2fastq/flowcell/InterOp/QMetricsOut.bin
+      md5sum: 7e9f198d53ebdfbb699a5f94cf1ed51c
+    - path: output/bcl2fastq/flowcell/InterOp/TileMetricsOut.bin
+      md5sum: 83891751ec1c91a425a524b476b6ca3c