Commit graph

14 commits

Author SHA1 Message Date
Matthias Hörtenhuber
e745e167c1
Fix formatting in yaml files, add yamllint config (#1279)
* fix yml formatting

* allow fastq.gz and fq.gz as file input, add meta.yml and test

* fix yaml files

* Revert "allow fastq.gz and fq.gz as file input, add meta.yml and test"

This reverts commit 34002d7a7a8c7f7bb4600c3377f35c87849f71a4.

* prettier magic!

* fix comments for yamllint

* remove node version number

* fix linting errors

Co-authored-by: Harshil Patel <drpatelh@users.noreply.github.com>
2022-02-15 11:15:27 +00:00
Francesco L
a68c563e54
Added UMI sub-workflow (#1098)
* added code for subworkflow fgbio call umi consensus

* ironing out a few typos etc

* fixing last things

* fixed md5sum - lets see if it changes

* removing file accidentally deleted

* tidy indents

* added bwamem2 alternative

* fixed entry for both tests

* changed name second test workflow entry

* fixed workflow entry names

* fixed md5sum for file generated with bwamem2

* added syntax new DSL2

* added new config location in test command line

* added new config location in test command line

* use of prefix instead of suffix because modules have been changed in this way

* explicit alias to bwa mem1 to avoid confusion

* removed param that should be an ext optional argument in fgbio groupreadsbyumi

* missing colon in config

* missing colon in module config too

* order list alphabetically

Co-authored-by: Maxime U. Garcia <maxime.garcia@scilifelab.se>

* remove params from body

Co-authored-by: Maxime U. Garcia <maxime.garcia@scilifelab.se>

* improving readability of input structure

Co-authored-by: Mahesh Binzer-Panchal <mahesh.binzer-panchal@nbis.se>

* reverting to mandatory input

* fixed tests and workflow take values

* remove param

Co-authored-by: Maxime U. Garcia <maxime.garcia@scilifelab.se>

* simplify tests params

Co-authored-by: Maxime U. Garcia <maxime.garcia@scilifelab.se>

* formatting inputs for readability

* factoring in changes to bwamem2_mem and bwa_mem sort/view inputs

* updating test md5sum for grouped file following code update in bwamem

Co-authored-by: Maxime U. Garcia <maxime.garcia@scilifelab.se>
Co-authored-by: Maxime U. Garcia <max.u.garcia@gmail.com>
Co-authored-by: Mahesh Binzer-Panchal <mahesh.binzer-panchal@nbis.se>
2021-12-09 11:16:40 +01:00
Maxime U. Garcia
31d4099f38
Add subworkflows for ensemblvep and snpeff (#1124)
* greatly simplify syntax

* feat: add subworkflows to annotate (+ bgzip/tabix index) with ensemblvep and snpeff

* feat: get versions from all tools

* add commented infor for new annotation modules
2021-12-07 17:12:35 +01:00
Harshil Patel
20d8250d9f
Update all modules to new NF DSL2 syntax (#1099)
* Add comment line for consistency

* Remove all functions.nf

* Remove include functions.nf and publishDir options

* Replace options.args3 with task.ext.args3 - 3 modules

* Replace options.args3 with task.ext.args3 - 17 modules

* Replace {task.cpus} with task.cpus

* Replace off on off off off off off off off on off on off on off off off on off off off on on off off off on on off off off off off off off on off off off off on off on on off off off on on on on off off off on off on on off on on off off on on on off on on off on off off off off on off off off on off off on off on off off off on on off on off on off off on off off off on off off off on off off off off on off off off on on on off on on off off on off on on on off on on off on on on off off off off off on on off off on off off off off off on off off on on off on on off on off off off on off off off off on on off on off off on off off on off on off off off off off off off off on on off on off off off.args with

* Add def args = task.ext.args line to all modules in script section

* Replace options.args with args and args_list

* Initialise args2 and args3 properly

* Replace container syntax

* Revert container changes for cellranger/mkref

* Replace getProcessName in all modules

* Replace getSoftwareName in all modules

* Unify modules using VERSION variable

* Replae options.suffix with task.ext.suffix

* Remove NF version restriction for CI

* Bump NF version in README

* Replace task.process.tokenize logic with task.process

* Minor tweaks to unify syntax in tests main.nf

* Add a separate nextflow.config for each module

* Transfer remaining module options to nextflow.config

* Remove addParams from tests main.nf

* Remove TODO statements

* Use -c to import module specific config

* Bump NF version to 21.10.3

* Fix tests for artic/minion

* Fix broken publishDir syntax

* Standardise and fix obvious failing module tests

* Remove kronatools to krona

* Comment out tags in subworkflow test.yml

* Fix failing module tests

* Add consistent indentation to nextflow.config

* Comment out subworklow definitions

* Fix kallistobustools/ref

* Fix rmarkdownnotebook

* Fix jupyternotebook

* Quote task.process

* Add plink2/vcf to pytest_modules.yml

* Remove NF_CORE_MODULES_TEST from pytest CI

* Fix more tests

* Move bacteroides_fragilis to prokaryotes folder

* Fix cooler merge tests

* Fix kallistobustools/count tests

* Fix kallistobustools/ref tests

* Update test_10x_1_fastq_gz file for kallistobustools/count tests

* Fix bcftools/query tests

* Fix delly/call tests

* Fix cooler/zoomify tests

* Fix csvtk/split tests

* Fix gatk4/intervallisttools tests

* Fix gatk4/variantfiltration

* Fix pydamage/filter tests

* Fix test data for unicycler

* Fix gstama/collapse module

* Fix leehom tests

* Fix metaphlan3 tests

* Fix pairtools/select tests

* Update nextflow.config

* Update nextflow.config

* feat: update syntax

* Fix arriba tests

* Fix more failing tests

* Update test syntax

* Remove comments from tests nextflow.config

* Apply suggestions from code review

* Fix kallistobustools/count module

* Update dumpsoftwareversions module

* Update custom/dumpsoftwareversions

* Add args2 to untar module

* Update leftover modules

* Remove last remaining addParams

Co-authored-by: JoseEspinosa <kadomu@gmail.com>
Co-authored-by: Gregor Sturm <mail@gregor-sturm.de>
Co-authored-by: MaxUlysse <max.u.garcia@gmail.com>
2021-11-26 07:58:40 +00:00
FriederikeHanssen
19035c99d1
Add thread option to samtools modules (#1069)
* add thread option to samtools modules

* fix tests

* fix tests

* fix tests

* Fix naming to fix tests

* cpus-1 to account for mainthread

* remove thread in ampliconclip, docu doesn't report this param

* add -1 to all other applicable samtools modules

* Update samtools version

* Update checksums

* retrigger GHA after update

* Update modules/samtools/fastq/main.nf

Co-authored-by: Patrick Hüther <patrick.huether@gmail.com>

* Update modules/samtools/fixmate/main.nf

Co-authored-by: Patrick Hüther <patrick.huether@gmail.com>

* Update modules/samtools/flagstat/main.nf

Co-authored-by: Patrick Hüther <patrick.huether@gmail.com>

* Update modules/samtools/index/main.nf

Co-authored-by: Patrick Hüther <patrick.huether@gmail.com>

* Update modules/samtools/merge/main.nf

Co-authored-by: Patrick Hüther <patrick.huether@gmail.com>

* Update modules/samtools/stats/main.nf

Co-authored-by: Patrick Hüther <patrick.huether@gmail.com>

* Update modules/samtools/view/main.nf

Co-authored-by: Patrick Hüther <patrick.huether@gmail.com>

* Fix md5sum fixmate

* Fix md5sums

* sth funny with the fixmate checksums

* more md5sums updates

Co-authored-by: Patrick Hüther <patrick.huether@gmail.com>
2021-11-18 22:24:47 +01:00
GCJMackenzie
5b975cc20d
Add gatk somatic paired calling subworkflow (#1067)
* initial commit to setup branch

* workflow finished

* Update nextflow.config

* tumour to tumor, getpileup passed as nomral and tumor

* paired_somatic renamed to tumor_normal_somatic

* Apply suggestions from code review

Co-authored-by: Maxime U. Garcia <maxime.garcia@scilifelab.se>

* Update subworkflows/nf-core/gatk_tumor_normal_somatic_variant_calling/main.nf

Co-authored-by: Maxime U. Garcia <maxime.garcia@scilifelab.se>

* updated index names in meta.yml

* changed index file names in main script and test

* Apply suggestions from code review

Co-authored-by: Maxime U. Garcia <maxime.garcia@scilifelab.se>

* Apply suggestions from code review

* fixed bug from changes

* Apply suggestions from code review

* tests should now work after the yml update

* Update pytest_modules.yml

Co-authored-by: GCJMackenzie <gavin.mackenzie@nibsc.org>
Co-authored-by: Maxime U. Garcia <maxime.garcia@scilifelab.se>
Co-authored-by: Maxime U. Garcia <max.u.garcia@gmail.com>
2021-11-17 11:34:07 +01:00
GCJMackenzie
071b1d50a8
Add gatk somatic tumour calling subworkflow (#1064)
* initial commit to set up new branch

* save changes to checkout

* workflow working, still needs test.yml and meta.yml, also fix versions file

* subworkflow finished

* Update pytest_subworkflows.yml

* Update pytest_subworkflows.yml

* Update pytest_subworkflows.yml

* fix config subworkflow name

* Update main.nf

* Update pytest_subworkflows.yml

* fixed md5sum issue likely caused by gatk version update

* tumour changed to tumor

* old dir deleted

* Comments added to explain use of placeholders '[]'

* updated index names, input channel renamed to input

* Apply suggestions from code review

* updated to perform new subworkflow testing

Co-authored-by: GCJMackenzie <gavin.mackenzie@nibsc.org>
Co-authored-by: Maxime U. Garcia <maxime.garcia@scilifelab.se>
2021-11-17 11:07:17 +01:00
Edmund Miller
2af071ed0d
Fix subworkflows seperate from modules (#933)
* ci: Remove pytest_subworkflows

* ci(bam_sort_samtools): Depend on paths-filter instead of pytest-workflow

Co-authored-by: Harshil Patel <drpatelhh@gmail.com>
Co-authored-by: Gregor Sturm <mail@gregor-sturm.de>
Co-authored-by: Jose Espinosa-Carrasco <kadomu@gmail.com>

* ci: Revert back to one job branch

* ci(align_bowtie2): Run tests that depend on bam_sort_samtools

* ci: Fix anchor not being created yet

* ci: Update sra_fastq tags and pytest_modules

* fix(bam_sort_samtools): Update nextflow.config with params

* test(subworkflows): Update gatk_create_som_pon tags

* ci: Point to subworkflow_hacks branch of nf-core tools

Co-authored-by: Harshil Patel <drpatelhh@gmail.com>
Co-authored-by: Gregor Sturm <mail@gregor-sturm.de>
Co-authored-by: Jose Espinosa-Carrasco <kadomu@gmail.com>
2021-11-15 18:18:46 +00:00
Maxime U. Garcia
3426834744
feat: fastaidx -> fai (#1073) 2021-11-15 18:03:02 +01:00
Maxime U. Garcia
3b600af50e
feat: update gatk4 from 4.2.0.0 to 4.2.3.0 (#1059)
* feat: update gatk4 from 4.2.0.0 to 4.2.3.0

* update md5checksum

* commit all files

* actually checksum was good, but I suspect something fishy with the tests
2021-11-11 08:58:59 +01:00
GCJMackenzie
e0ab4ff4ff
bugfix: panel of normals subworkflow: remove md5sum on a gzipped file (#1045)
* commiting changes to switch branch

* commit to setup remote branch

* first draft of the sompon workflow

* keep branch in line with gendb bugfixing

* Update test.yml

* tidy up main.nf

* fixed md5sum

* Update test.yml

removed md5sum from test_panel.vcf.gz.tbi

* Update test.yml

* remove md5sum from gzipped test_panel.vcf.gz.tbi

Co-authored-by: GCJMackenzie <gavin.mackenzie@nibsc.org>
2021-11-09 11:08:59 +00:00
GCJMackenzie
9573cb1bec
Add panel of normals subworkflow (#1044)
* commiting changes to switch branch

* commit to setup remote branch

* first draft of the sompon workflow

* keep branch in line with gendb bugfixing

* Update test.yml

* tidy up main.nf

* fixed md5sum

Co-authored-by: GCJMackenzie <gavin.mackenzie@nibsc.org>
2021-11-09 10:16:43 +00:00
Moritz E. Beber
c912117972
feat: add sub-workflow for SRA (#836)
* feat: add sub-workflow for SRA
* Combine prefetch and fasterq-dump into one sub-workflow
* tests: add sub-workflow to pytest config
2021-10-14 11:50:31 +02:00
Edmund Miller
c19671dca9
Subworkflow Infrastructure (#662)
* feat(subworkflows): Add align_bowtie2 subworkflow

For testing CI setup

* test(align_bowtie2): Add initial list of changes to test

* test(align_bowtie2): Add initial test

* refactor: Use tags to run subworkflows ci

For every underlying module used by workflow and allow the modules
pytest-modules definition be the source of truth.

* refactor: Use individual directories for subworkflows

* docs(align_bowtie2): Add initial meta.yml

Copied most of it from the bowtie2/align module.

* fix(align_bowtie2): Fix module include paths

* test(bam_sort_samtools): Add initial test

* ci(bam_sort_samtools): Add modules that trigger the tag

* test(bam_stats_samtools): Add initial test

* ci(bam_stats_samtools): Add keys to pick up changes

* docs(bam_samtools): Add initial meta.yml

* test(align_bowtie2): Fix path to subworkflow

* test(align_bowtie2): Update entry point

* fix(bam_sort_samtools): Update include paths

* test(bam_sort_samtools): Fix path

* style: Clean up addParams

* test(samtools_sort): Add suffix for test

* test(align_bowtie2): Add samtools_options for suffix

* test(bam_stats_samtools): Update path

* test(bam_stats_samtools): Use stats input

Otherwise it's just an example of how it's used in the bam_sort_samtools subworkflow

* ci(linting): Skip module linting of subworkflows

* ci(linting): Clean up startsWith statement

* test(bam_stats_samtools): Use single end test data for single end test

* test(bam_stats_samtools): Add expected files

* test(align_bowtie2): Add paired-end test

* test(align_bowtie2): Sort order of output

* test(align_bowtie2): Update hashes

* docs(align_bowtie2): Fix typo

* test(align_bowtie2): Update samtools output names

* test(align_bowtie2): Remove md5sums for bam/bai

* feat(subworkflows): Add nextflow.configs

These can be used for default settings in the future. They can then be
included in the conf/modules.config so that the params don't have to be
duplicated in the root nextflow.config.

* docs(subworkflows): Include modules instead of tools

* fix: Update to versions

* chore(align_bowtie2): Remove duplicate tag

* style: Format yamls

* test(subworkflows): Only check versions for modules

* chore: Update subworkflows to match rnaseq dev

* fix(subworkflows): Update paths

* fix(bam_sort_samtools): Fix sort parameters for testing

* Apply suggestions from code review

Co-authored-by: Harshil Patel <drpatelh@users.noreply.github.com>

* docs: Update TODOs with a message

* ci: Try using a matrix for strategy

* ci: Try passing an array

* Revert "ci: Try passing an array"

This reverts commit d3611fcd8332bbb9a8501e8dd299d0a623aaecaa.

Co-authored-by: Harshil Patel <drpatelh@users.noreply.github.com>
2021-10-08 16:02:42 +01:00