Chromap Module (#659)

* Initialise chromap module

* Revert "Initialise chromap module"

This reverts commit 47c67ae231a6f221ef5b9b7b444b583b5406852b.

* Remake chromap base files with new layout

* Copy chromap

* Copy index

* Add compression

* Update padding

* Update container

* Update chromap input test data

* Add chromap chromap tests

* Add padding

* Update comment

* update yaml file

* Remove TODOs

* Add fasta input to yaml

* Update YAML

* Remove comment, update container

* Remove comments

* Import Chromap index

* Update test.yml

* Fix read input

* Update test.yml

* Add bcftools/concat module. (#641)

* draft for bcftools modules [ci skip]

* initial test for bcftools concat

* Update the params for testing

* fix tests

* Accomodate code review [ci skip]

Co-authored-by: James A. Fellows Yates <jfy133@gmail.com>

* Update the meta file and open PR for review

* Update the keyword

* Update the tags for module [ci skip[

* add threads

Co-authored-by: James A. Fellows Yates <jfy133@gmail.com>

* add module for dragonflye (#633)

* add module for dragonflye

* fix tests for dragonflye

* Update test.yml

* Update meta.yml

* Update main.nf

* Update main.nf

* Update modules/dragonflye/meta.yml

Co-authored-by: Gregor Sturm <mail@gregor-sturm.de>

* update typos. change quote from ' to ". (#652)

* Add bcftools/norm module (#655)

* Initial draft [ci skip]

* trigger first test

* update output file path

* Tests passing

* finishing touches for meta.yml and update checksum

* tweak checksum

* add threads to the module

* skip version info for matching test md5sum [ci skip]

* Add ref fasta and finalize the module

Co-authored-by: Gregor Sturm <mail@gregor-sturm.de>

* Expansionhunter (#666)


Please enter the commit message for your changes. Lines starting

* adds expansionhunter module

Co-authored-by: Maxime U. Garcia <maxime.garcia@scilifelab.se>

* Update test.yml (#668)

* Specify in guidelines one should split CPUs when module has n > 1 tool (#660)

* Specify more guidelines on input channels

* Linting

* Updates based on code review

* Update README.md

* Fix broken sentence

* Describe CPU splitting

* Update README.md

Co-authored-by: Gregor Sturm <mail@gregor-sturm.de>

* More CPU examples

Co-authored-by: Gregor Sturm <mail@gregor-sturm.de>

* Add dsh-bio export-segments module (#631)

Co-authored-by: Gregor Sturm <mail@gregor-sturm.de>

* update: `BWA/ALN` (#653)

* Specify more guidelines on input channels

* Linting

* Updates based on code review

* Update README.md

* Fix broken sentence

* Remove reads from output channel following module guidelines. Should do a .join() based on $meta, to reassociate.

Co-authored-by: Gregor Sturm <mail@gregor-sturm.de>

* Update seqwish reported version to match bioconda version. (#678)

* Bbmap index (#683)

BBMap index module

* Initialise chromap module

* Revert "Initialise chromap module"

This reverts commit 47c67ae231a6f221ef5b9b7b444b583b5406852b.

* Remove unnecessary files

* Remove unnecessary files

* Update modules/chromap/index/main.nf

Co-authored-by: Harshil Patel <drpatelh@users.noreply.github.com>

* Update modules/chromap/index/main.nf

Co-authored-by: Harshil Patel <drpatelh@users.noreply.github.com>

* Update modules/chromap/chromap/main.nf

Co-authored-by: Harshil Patel <drpatelh@users.noreply.github.com>

* Update tests/modules/chromap/chromap/main.nf

Co-authored-by: Harshil Patel <drpatelh@users.noreply.github.com>

* Update tests/modules/chromap/chromap/main.nf

Co-authored-by: Harshil Patel <drpatelh@users.noreply.github.com>

* Update tests/modules/chromap/chromap/main.nf

Co-authored-by: Harshil Patel <drpatelh@users.noreply.github.com>

* Update modules/chromap/index/main.nf

Co-authored-by: Harshil Patel <drpatelh@users.noreply.github.com>

* Remove pytest_software.yml

* Apply suggestions from code review

Co-authored-by: Abhinav Sharma <abhi18av@users.noreply.github.com>
Co-authored-by: James A. Fellows Yates <jfy133@gmail.com>
Co-authored-by: Robert A. Petit III <robbie.petit@gmail.com>
Co-authored-by: Gregor Sturm <mail@gregor-sturm.de>
Co-authored-by: JIANHONG OU <jianhong@users.noreply.github.com>
Co-authored-by: Anders Jemt <jemten@users.noreply.github.com>
Co-authored-by: Maxime U. Garcia <maxime.garcia@scilifelab.se>
Co-authored-by: Michael L Heuer <heuermh@acm.org>
Co-authored-by: Daniel Lundin <daniel.lundin@lnu.se>
Co-authored-by: Harshil Patel <drpatelh@users.noreply.github.com>

modules/chromap/chromap/functions.nf

@@ -0,0 +1,68 @@
+//
+//  Utility functions used in nf-core DSL2 module files
+//
+
+//
+// Extract name of software tool from process name using $task.process
+//
+def getSoftwareName(task_process) {
+    return task_process.tokenize(':')[-1].tokenize('_')[0].toLowerCase()
+}
+
+//
+// Function to initialise default values and to generate a Groovy Map of available options for nf-core modules
+//
+def initOptions(Map args) {
+    def Map options = [:]
+    options.args            = args.args ?: ''
+    options.args2           = args.args2 ?: ''
+    options.args3           = args.args3 ?: ''
+    options.publish_by_meta = args.publish_by_meta ?: []
+    options.publish_dir     = args.publish_dir ?: ''
+    options.publish_files   = args.publish_files
+    options.suffix          = args.suffix ?: ''
+    return options
+}
+
+//
+// Tidy up and join elements of a list to return a path string
+//
+def getPathFromList(path_list) {
+    def paths = path_list.findAll { item -> !item?.trim().isEmpty() }      // Remove empty entries
+    paths     = paths.collect { it.trim().replaceAll("^[/]+|[/]+\$", "") } // Trim whitespace and trailing slashes
+    return paths.join('/')
+}
+
+//
+// Function to save/publish module results
+//
+def saveFiles(Map args) {
+    if (!args.filename.endsWith('.version.txt')) {
+        def ioptions  = initOptions(args.options)
+        def path_list = [ ioptions.publish_dir ?: args.publish_dir ]
+        if (ioptions.publish_by_meta) {
+            def key_list = ioptions.publish_by_meta instanceof List ? ioptions.publish_by_meta : args.publish_by_meta
+            for (key in key_list) {
+                if (args.meta && key instanceof String) {
+                    def path = key
+                    if (args.meta.containsKey(key)) {
+                        path = args.meta[key] instanceof Boolean ? "${key}_${args.meta[key]}".toString() : args.meta[key]
+                    }
+                    path = path instanceof String ? path : ''
+                    path_list.add(path)
+                }
+            }
+        }
+        if (ioptions.publish_files instanceof Map) {
+            for (ext in ioptions.publish_files) {
+                if (args.filename.endsWith(ext.key)) {
+                    def ext_list = path_list.collect()
+                    ext_list.add(ext.value)
+                    return "${getPathFromList(ext_list)}/$args.filename"
+                }
+            }
+        } else if (ioptions.publish_files == null) {
+            return "${getPathFromList(path_list)}/$args.filename"
+        }
+    }
+}

modules/chromap/chromap/main.nf

@@ -0,0 +1,93 @@
+// Import generic module functions
+include { initOptions; saveFiles; getSoftwareName } from './functions'
+
+params.options = [:]
+options        = initOptions(params.options)
+
+def VERSION = 0.1 // No version information printed
+
+process CHROMAP_CHROMAP {
+    tag "$meta.id"
+    label 'process_medium'
+    publishDir "${params.outdir}",
+        mode: params.publish_dir_mode,
+        saveAs: { filename -> saveFiles(filename:filename, options:params.options, publish_dir:getSoftwareName(task.process), meta:meta, publish_by_meta:['id']) }
+
+    conda (params.enable_conda ? "bioconda::chromap=0.1 bioconda::samtools=1.13" : null)
+    if (workflow.containerEngine == 'singularity' && !params.singularity_pull_docker_container) {
+        container "https://depot.galaxyproject.org/singularity/mulled-v2-1f09f39f20b1c4ee36581dc81cc323c70e661633:2cad7c5aa775241887eff8714259714a39baf016-0"
+    } else {
+        container "quay.io/biocontainers/mulled-v2-1f09f39f20b1c4ee36581dc81cc323c70e661633:2cad7c5aa775241887eff8714259714a39baf016-0"
+    }
+
+    input:
+    tuple val(meta), path(reads)
+    path fasta
+    path index
+    path barcodes
+    path whitelist
+    path chr_order
+    path pairs_chr_order
+
+    output:
+    tuple val(meta), path("*.bed.gz")     , optional:true, emit: bed
+    tuple val(meta), path("*.bam")        , optional:true, emit: bam
+    tuple val(meta), path("*.tagAlign.gz"), optional:true, emit: tagAlign
+    tuple val(meta), path("*.pairs.gz")   , optional:true, emit: pairs
+    path "*.version.txt"                                 , emit: version
+
+    script:
+    def software = getSoftwareName(task.process)
+    def prefix   = options.suffix ? "${meta.id}${options.suffix}" : "${meta.id}"
+    def args     = options.args.tokenize()
+
+    def file_extension = options.args.contains("--SAM")? 'sam' :
+                        options.args.contains("--TagAlign")? 'tagAlign' :
+                        options.args.contains("--pairs")? 'pairs' : 'bed'
+    if (barcodes) {
+        args << "-b ${barcodes.join(',')}"
+        if (whitelist) {
+            args << "--barcode-whitelist $whitelist"
+        }
+    }
+    if (chr_order) {
+        args << "--chr-order $chr_order"
+    }
+    if (pairs_chr_order){
+        args << "--pairs-natural-chr-order $pairs_chr_order"
+    }
+    def compression_cmds = """
+    gzip ${prefix}.${file_extension}
+    """
+    if (options.args.contains("--SAM")) {
+        compression_cmds = """
+        samtools view $options.args2 -@ ${task.cpus} -bh \\
+            -o ${prefix}.bam ${prefix}.${file_extension}
+        rm ${prefix}.${file_extension}
+
+        samtools --version 2>&1 | sed 's/^.*samtools //; s/Using.*\$//' > ${software}.version.txt
+        """
+    }
+    if (meta.single_end) {
+        """
+        chromap ${args.join(' ')} \\
+            -t $task.cpus \\
+            -x $index \\
+            -r $fasta \\
+            -1 ${reads.join(',')} \\
+            -o ${prefix}.${file_extension}
+        echo "$VERSION" > ${software}.version.txt
+        """ + compression_cmds
+    } else {
+        """
+        chromap ${args.join(' ')} \\
+            -t $task.cpus \\
+            -x $index \\
+            -r $fasta \\
+            -1 ${reads[0]} \\
+            -2 ${reads[1]} \\
+            -o ${prefix}.${file_extension}
+        echo "$VERSION" > ${software}.version.txt
+        """ + compression_cmds
+    }
+}

modules/chromap/chromap/meta.yml

@@ -0,0 +1,88 @@
+name: chromap_chromap
+description: |
+    Performs preprocessing and alignment of chromatin fastq files to
+    fasta reference files using chromap.
+keywords:
+    - chromap
+    - alignment
+    - map
+    - fastq
+    - bam
+    - sam
+    - hi-c
+    - atac-seq
+    - chip-seq
+    - trimming
+    - duplicate removal
+tools:
+  - chromap:
+      description: Fast alignment and preprocessing of chromatin profiles
+      homepage: https://github.com/haowenz/chromap
+      documentation: https://github.com/haowenz/chromap
+      tool_dev_url: https://github.com/haowenz/chromap
+      doi: ""
+      licence: ['GPL v3']
+input:
+  - meta:
+      type: map
+      description: |
+        Groovy Map containing sample information
+        e.g. [ id:'test', single_end:false ]
+  - reads:
+      type: file
+      description: |
+        List of input FastQ files of size 1 and 2 for single-end and paired-end data,
+        respectively.
+  - fasta:
+      type: file
+      description: |
+        The fasta reference file.
+  - index:
+      type: file
+      description: |
+        Chromap genome index files (*.index)
+  - barcodes:
+      type: file
+      description: |
+        Cell barcode files
+  - whitelist:
+      type: file
+      description: |
+        Cell barcode whitelist file
+  - chr_order:
+      type: file
+      description: |
+        Custom chromosome order
+  - pairs_chr_order:
+      type: file
+      description: |
+        Natural chromosome order for pairs flipping
+output:
+  - meta:
+      type: map
+      description: |
+        Groovy Map containing sample information
+        e.g. [ id:'test', single_end:false ]
+  - version:
+      type: file
+      description: File containing software version
+      pattern: "*.{version.txt}"
+  - bed:
+      type: file
+      description: BED file
+      pattern: "*.bed.gz"
+  - bam:
+      type: file
+      description: BAM file
+      pattern: "*.bam"
+  - tagAlign:
+      type: file
+      description: tagAlign file
+      pattern: "*.tagAlign.gz"
+  - pairs:
+      type: file
+      description: pairs file
+      pattern: "*.pairs.gz"
+
+authors:
+  - "@mahesh-panchal"

modules/chromap/index/functions.nf

@@ -0,0 +1,68 @@
+//
+//  Utility functions used in nf-core DSL2 module files
+//
+
+//
+// Extract name of software tool from process name using $task.process
+//
+def getSoftwareName(task_process) {
+    return task_process.tokenize(':')[-1].tokenize('_')[0].toLowerCase()
+}
+
+//
+// Function to initialise default values and to generate a Groovy Map of available options for nf-core modules
+//
+def initOptions(Map args) {
+    def Map options = [:]
+    options.args            = args.args ?: ''
+    options.args2           = args.args2 ?: ''
+    options.args3           = args.args3 ?: ''
+    options.publish_by_meta = args.publish_by_meta ?: []
+    options.publish_dir     = args.publish_dir ?: ''
+    options.publish_files   = args.publish_files
+    options.suffix          = args.suffix ?: ''
+    return options
+}
+
+//
+// Tidy up and join elements of a list to return a path string
+//
+def getPathFromList(path_list) {
+    def paths = path_list.findAll { item -> !item?.trim().isEmpty() }      // Remove empty entries
+    paths     = paths.collect { it.trim().replaceAll("^[/]+|[/]+\$", "") } // Trim whitespace and trailing slashes
+    return paths.join('/')
+}
+
+//
+// Function to save/publish module results
+//
+def saveFiles(Map args) {
+    if (!args.filename.endsWith('.version.txt')) {
+        def ioptions  = initOptions(args.options)
+        def path_list = [ ioptions.publish_dir ?: args.publish_dir ]
+        if (ioptions.publish_by_meta) {
+            def key_list = ioptions.publish_by_meta instanceof List ? ioptions.publish_by_meta : args.publish_by_meta
+            for (key in key_list) {
+                if (args.meta && key instanceof String) {
+                    def path = key
+                    if (args.meta.containsKey(key)) {
+                        path = args.meta[key] instanceof Boolean ? "${key}_${args.meta[key]}".toString() : args.meta[key]
+                    }
+                    path = path instanceof String ? path : ''
+                    path_list.add(path)
+                }
+            }
+        }
+        if (ioptions.publish_files instanceof Map) {
+            for (ext in ioptions.publish_files) {
+                if (args.filename.endsWith(ext.key)) {
+                    def ext_list = path_list.collect()
+                    ext_list.add(ext.value)
+                    return "${getPathFromList(ext_list)}/$args.filename"
+                }
+            }
+        } else if (ioptions.publish_files == null) {
+            return "${getPathFromList(path_list)}/$args.filename"
+        }
+    }
+}

modules/chromap/index/main.nf

@@ -0,0 +1,40 @@
+// Import generic module functions
+include { initOptions; saveFiles; getSoftwareName } from './functions'
+
+params.options = [:]
+options        = initOptions(params.options)
+
+def VERSION = 0.1 // No version information printed
+
+process CHROMAP_INDEX {
+    tag '$fasta'
+    label 'process_medium'
+    publishDir "${params.outdir}",
+        mode: params.publish_dir_mode,
+        saveAs: { filename -> saveFiles(filename:filename, options:params.options, publish_dir:getSoftwareName(task.process), meta:[:], publish_by_meta:[]) }
+
+    conda (params.enable_conda ? "bioconda::chromap=0.1 bioconda::samtools=1.13" : null)
+    if (workflow.containerEngine == 'singularity' && !params.singularity_pull_docker_container) {
+        container "https://depot.galaxyproject.org/singularity/mulled-v2-1f09f39f20b1c4ee36581dc81cc323c70e661633:2cad7c5aa775241887eff8714259714a39baf016-0"
+    } else {
+        container "quay.io/biocontainers/mulled-v2-1f09f39f20b1c4ee36581dc81cc323c70e661633:2cad7c5aa775241887eff8714259714a39baf016-0"
+    }
+
+    input:
+    path fasta
+
+    output:
+    path "*.index"      , emit: index
+    path "*.version.txt", emit: version
+
+    script:
+    def software = getSoftwareName(task.process)
+    def prefix   = fasta.baseName
+    """
+    chromap -i $options.args \\
+        -t $task.cpus \\
+        -r $fasta \\
+        -o ${prefix}.index
+    echo "$VERSION" > ${software}.version.txt
+    """
+}

modules/chromap/index/meta.yml

@@ -0,0 +1,33 @@
+name: chromap_index
+description: Indexes a fasta reference genome ready for chromatin profiling.
+keywords:
+    - index
+    - fasta
+    - genome
+    - reference
+tools:
+  - chromap:
+      description: Fast alignment and preprocessing of chromatin profiles
+      homepage: https://github.com/haowenz/chromap
+      documentation: https://github.com/haowenz/chromap
+      tool_dev_url: https://github.com/haowenz/chromap
+      doi: ""
+      licence: ['GPL v3']
+
+input:
+  - fasta:
+      type: file
+      description: Fasta reference file.
+
+output:
+  - version:
+      type: file
+      description: File containing software version
+      pattern: "*.{version.txt}"
+  - index:
+      type: file
+      description: Index file of the reference genome
+      pattern: "*.{index}"
+
+authors:
+  - "@mahesh-panchal"

tests/config/pytest_modules.yml

@@ -226,6 +226,14 @@ cat/fastq:
   - modules/cat/fastq/**
   - tests/modules/cat/fastq/**
 
+chromap/chromap:
+  - modules/chromap/chromap/**
+  - tests/modules/chromap/chromap/**
+
+chromap/index:
+  - modules/chromap/index/**
+  - tests/modules/chromap/index/**
+
 cnvkit:
   - modules/cnvkit/**
   - tests/modules/cnvkit/**

tests/modules/chromap/chromap/main.nf

@@ -0,0 +1,79 @@
+#!/usr/bin/env nextflow
+
+nextflow.enable.dsl = 2
+
+include { CHROMAP_INDEX                           } from '../../../../modules/chromap/index/main.nf'   addParams( options: [:] )
+include { CHROMAP_CHROMAP as CHROMAP_CHROMAP_BASE } from '../../../../modules/chromap/chromap/main.nf' addParams( options: [:] )
+include { CHROMAP_CHROMAP as CHROMAP_CHROMAP_SAM  } from '../../../../modules/chromap/chromap/main.nf' addParams( options: ['args': '--SAM'] )
+
+workflow test_chromap_chromap_single_end {
+
+    // Test single-end and gz compressed output
+
+    fasta = file(params.test_data['sarscov2']['genome']['genome_fasta'], checkIfExists: true)
+    input = [
+        [ id:'test', single_end:true ], // meta map
+        [ file(params.test_data['sarscov2']['illumina']['test_1_fastq_gz'], checkIfExists: true) ]
+    ]
+
+    CHROMAP_INDEX ( fasta )
+    CHROMAP_CHROMAP_BASE (
+        input,                      // meta + read data
+        fasta,                      // reference genome
+        CHROMAP_INDEX.out.index,    // reference index
+        [],                         // barcode file
+        [],                         // barcode whitelist
+        [],                         // chromosome order file
+        []                          // pairs chromosome order file
+    )
+}
+
+workflow test_chromap_chromap_paired_end {
+
+    // Test paired-end and gz compressed output
+
+    fasta = file(params.test_data['sarscov2']['genome']['genome_fasta'], checkIfExists: true)
+    input = [
+        [ id:'test', single_end:false ], // meta map
+        [
+            file(params.test_data['sarscov2']['illumina']['test_1_fastq_gz'], checkIfExists: true),
+            file(params.test_data['sarscov2']['illumina']['test_2_fastq_gz'], checkIfExists: true)
+        ]
+    ]
+
+    CHROMAP_INDEX ( fasta )
+    CHROMAP_CHROMAP_BASE (
+        input,                      // meta + read data
+        fasta,                      // reference genome
+        CHROMAP_INDEX.out.index,    // reference index
+        [],                         // barcode file
+        [],                         // barcode whitelist
+        [],                         // chromosome order file
+        []                          // pairs chromosome order file
+    )
+}
+
+workflow test_chromap_chromap_paired_bam {
+
+    // Test paired-end and bam output
+
+    fasta = file(params.test_data['sarscov2']['genome']['genome_fasta'], checkIfExists: true)
+    input = [
+        [ id:'test', single_end:false ], // meta map
+        [
+            file(params.test_data['sarscov2']['illumina']['test_1_fastq_gz'], checkIfExists: true),
+            file(params.test_data['sarscov2']['illumina']['test_2_fastq_gz'], checkIfExists: true)
+        ]
+    ]
+
+    CHROMAP_INDEX ( fasta )
+    CHROMAP_CHROMAP_SAM (
+        input,                      // meta + read data
+        fasta,                      // reference genome
+        CHROMAP_INDEX.out.index,    // reference index
+        [],                         // barcode file
+        [],                         // barcode whitelist
+        [],                         // chromosome order file
+        []                          // pairs chromosome order file
+    )
+}

tests/modules/chromap/chromap/test.yml

@@ -0,0 +1,32 @@
+- name: chromap chromap test_chromap_chromap_single_end
+  command: nextflow run tests/modules/chromap/chromap -entry test_chromap_chromap_single_end -c tests/config/nextflow.config
+  tags:
+    - chromap/chromap
+    - chromap
+  files:
+    - path: output/chromap/genome.index
+      md5sum: f889d5f61d80823766af33277d27d386
+    - path: output/chromap/test.bed.gz
+      md5sum: 7029066c27ac6f5ef18d660d5741979a
+
+- name: chromap chromap test_chromap_chromap_paired_end
+  command: nextflow run tests/modules/chromap/chromap -entry test_chromap_chromap_paired_end -c tests/config/nextflow.config
+  tags:
+    - chromap/chromap
+    - chromap
+  files:
+    - path: output/chromap/genome.index
+      md5sum: f889d5f61d80823766af33277d27d386
+    - path: output/chromap/test.bed.gz
+      md5sum: cafd8fb21977f5ae69e9008b220ab169
+
+- name: chromap chromap test_chromap_chromap_paired_bam
+  command: nextflow run tests/modules/chromap/chromap -entry test_chromap_chromap_paired_bam -c tests/config/nextflow.config
+  tags:
+    - chromap/chromap
+    - chromap
+  files:
+    - path: output/chromap/genome.index
+      md5sum: f889d5f61d80823766af33277d27d386
+    - path: output/chromap/test.bam
+      md5sum: bd1e3fe0f3abd1430ae191754f16a3ed

tests/modules/chromap/index/main.nf

@@ -0,0 +1,12 @@
+#!/usr/bin/env nextflow
+
+nextflow.enable.dsl = 2
+
+include { CHROMAP_INDEX } from '../../../../modules/chromap/index/main.nf' addParams( options: [:] )
+
+workflow test_chromap_index {
+
+    input = file(params.test_data['sarscov2']['genome']['genome_fasta'], checkIfExists: true)
+
+    CHROMAP_INDEX ( input )
+}

tests/modules/chromap/index/test.yml

@@ -0,0 +1,8 @@
+- name: chromap index test_chromap_index
+  command: nextflow run tests/modules/chromap/index -entry test_chromap_index -c tests/config/nextflow.config
+  tags:
+    - chromap/index
+    - chromap
+  files:
+    - path: output/chromap/genome.index
+      md5sum: f889d5f61d80823766af33277d27d386