Commit graph

300 commits

Author SHA1 Message Date
Matthias Hörtenhuber
e745e167c1
Fix formatting in yaml files, add yamllint config (#1279)
* fix yml formatting

* allow fastq.gz and fq.gz as file input, add meta.yml and test

* fix yaml files

* Revert "allow fastq.gz and fq.gz as file input, add meta.yml and test"

This reverts commit 34002d7a7a8c7f7bb4600c3377f35c87849f71a4.

* prettier magic!

* fix comments for yamllint

* remove node version number

* fix linting errors

Co-authored-by: Harshil Patel <drpatelh@users.noreply.github.com>
2022-02-15 11:15:27 +00:00
Edmund Miller
550df3b006
ci: Remove second python setup call (#1225)
Co-authored-by: Gregor Sturm <mail@gregor-sturm.de>
2022-01-21 07:57:57 +01:00
Gregor Sturm
55ea3f2061
CI: print out pytest logs on failure (#1220) 2022-01-21 07:51:28 +01:00
Phil Ewels
4e5406c221
Add helper script to find duplicate test YAML md5s (#1167)
* Add helper script to find duplicate test YAML md5s

* Count how many modules the duplicates affect
2021-12-14 12:58:25 +01:00
James A. Fellows Yates
edbbbbf420
fix: exclude conda/singularity images from pytest workflow output (#1162) 2021-12-13 08:13:24 +00:00
Harshil Patel
20d8250d9f
Update all modules to new NF DSL2 syntax (#1099)
* Add comment line for consistency

* Remove all functions.nf

* Remove include functions.nf and publishDir options

* Replace options.args3 with task.ext.args3 - 3 modules

* Replace options.args3 with task.ext.args3 - 17 modules

* Replace {task.cpus} with task.cpus

* Replace off on off off off off off off off on off on off on off off off on off off off on on off off off on on off off off off off off off on off off off off on off on on off off off on on on on off off off on off on on off on on off off on on on off on on off on off off off off on off off off on off off on off on off off off on on off on off on off off on off off off on off off off on off off off off on off off off on on on off on on off off on off on on on off on on off on on on off off off off off on on off off on off off off off off on off off on on off on on off on off off off on off off off off on on off on off off on off off on off on off off off off off off off off on on off on off off off.args with

* Add def args = task.ext.args line to all modules in script section

* Replace options.args with args and args_list

* Initialise args2 and args3 properly

* Replace container syntax

* Revert container changes for cellranger/mkref

* Replace getProcessName in all modules

* Replace getSoftwareName in all modules

* Unify modules using VERSION variable

* Replae options.suffix with task.ext.suffix

* Remove NF version restriction for CI

* Bump NF version in README

* Replace task.process.tokenize logic with task.process

* Minor tweaks to unify syntax in tests main.nf

* Add a separate nextflow.config for each module

* Transfer remaining module options to nextflow.config

* Remove addParams from tests main.nf

* Remove TODO statements

* Use -c to import module specific config

* Bump NF version to 21.10.3

* Fix tests for artic/minion

* Fix broken publishDir syntax

* Standardise and fix obvious failing module tests

* Remove kronatools to krona

* Comment out tags in subworkflow test.yml

* Fix failing module tests

* Add consistent indentation to nextflow.config

* Comment out subworklow definitions

* Fix kallistobustools/ref

* Fix rmarkdownnotebook

* Fix jupyternotebook

* Quote task.process

* Add plink2/vcf to pytest_modules.yml

* Remove NF_CORE_MODULES_TEST from pytest CI

* Fix more tests

* Move bacteroides_fragilis to prokaryotes folder

* Fix cooler merge tests

* Fix kallistobustools/count tests

* Fix kallistobustools/ref tests

* Update test_10x_1_fastq_gz file for kallistobustools/count tests

* Fix bcftools/query tests

* Fix delly/call tests

* Fix cooler/zoomify tests

* Fix csvtk/split tests

* Fix gatk4/intervallisttools tests

* Fix gatk4/variantfiltration

* Fix pydamage/filter tests

* Fix test data for unicycler

* Fix gstama/collapse module

* Fix leehom tests

* Fix metaphlan3 tests

* Fix pairtools/select tests

* Update nextflow.config

* Update nextflow.config

* feat: update syntax

* Fix arriba tests

* Fix more failing tests

* Update test syntax

* Remove comments from tests nextflow.config

* Apply suggestions from code review

* Fix kallistobustools/count module

* Update dumpsoftwareversions module

* Update custom/dumpsoftwareversions

* Add args2 to untar module

* Update leftover modules

* Remove last remaining addParams

Co-authored-by: JoseEspinosa <kadomu@gmail.com>
Co-authored-by: Gregor Sturm <mail@gregor-sturm.de>
Co-authored-by: MaxUlysse <max.u.garcia@gmail.com>
2021-11-26 07:58:40 +00:00
Robert A. Petit III
f052dc445c
use underscores in anchors and references (#1080)
* use underscores in anchors and references

* Dummy change to trigger CI

* use dev branch

* underscore anchor
2021-11-18 18:09:09 +00:00
Edmund Miller
2af071ed0d
Fix subworkflows seperate from modules (#933)
* ci: Remove pytest_subworkflows

* ci(bam_sort_samtools): Depend on paths-filter instead of pytest-workflow

Co-authored-by: Harshil Patel <drpatelhh@gmail.com>
Co-authored-by: Gregor Sturm <mail@gregor-sturm.de>
Co-authored-by: Jose Espinosa-Carrasco <kadomu@gmail.com>

* ci: Revert back to one job branch

* ci(align_bowtie2): Run tests that depend on bam_sort_samtools

* ci: Fix anchor not being created yet

* ci: Update sra_fastq tags and pytest_modules

* fix(bam_sort_samtools): Update nextflow.config with params

* test(subworkflows): Update gatk_create_som_pon tags

* ci: Point to subworkflow_hacks branch of nf-core tools

Co-authored-by: Harshil Patel <drpatelhh@gmail.com>
Co-authored-by: Gregor Sturm <mail@gregor-sturm.de>
Co-authored-by: Jose Espinosa-Carrasco <kadomu@gmail.com>
2021-11-15 18:18:46 +00:00
Moritz E. Beber
194e598a34
Fix workflow (#817)
* tests: fix tags definition

* fix: correct typo
2021-10-12 11:58:19 +01:00
Edmund Miller
d0df4b03e3
ci: Duplicate tests for subworkflows (#815)
Causing an issue with only running the subworkflow tests and now the
modules. Just for now, we can come back to this.
2021-10-12 08:14:20 +01:00
Edmund Miller
c19671dca9
Subworkflow Infrastructure (#662)
* feat(subworkflows): Add align_bowtie2 subworkflow

For testing CI setup

* test(align_bowtie2): Add initial list of changes to test

* test(align_bowtie2): Add initial test

* refactor: Use tags to run subworkflows ci

For every underlying module used by workflow and allow the modules
pytest-modules definition be the source of truth.

* refactor: Use individual directories for subworkflows

* docs(align_bowtie2): Add initial meta.yml

Copied most of it from the bowtie2/align module.

* fix(align_bowtie2): Fix module include paths

* test(bam_sort_samtools): Add initial test

* ci(bam_sort_samtools): Add modules that trigger the tag

* test(bam_stats_samtools): Add initial test

* ci(bam_stats_samtools): Add keys to pick up changes

* docs(bam_samtools): Add initial meta.yml

* test(align_bowtie2): Fix path to subworkflow

* test(align_bowtie2): Update entry point

* fix(bam_sort_samtools): Update include paths

* test(bam_sort_samtools): Fix path

* style: Clean up addParams

* test(samtools_sort): Add suffix for test

* test(align_bowtie2): Add samtools_options for suffix

* test(bam_stats_samtools): Update path

* test(bam_stats_samtools): Use stats input

Otherwise it's just an example of how it's used in the bam_sort_samtools subworkflow

* ci(linting): Skip module linting of subworkflows

* ci(linting): Clean up startsWith statement

* test(bam_stats_samtools): Use single end test data for single end test

* test(bam_stats_samtools): Add expected files

* test(align_bowtie2): Add paired-end test

* test(align_bowtie2): Sort order of output

* test(align_bowtie2): Update hashes

* docs(align_bowtie2): Fix typo

* test(align_bowtie2): Update samtools output names

* test(align_bowtie2): Remove md5sums for bam/bai

* feat(subworkflows): Add nextflow.configs

These can be used for default settings in the future. They can then be
included in the conf/modules.config so that the params don't have to be
duplicated in the root nextflow.config.

* docs(subworkflows): Include modules instead of tools

* fix: Update to versions

* chore(align_bowtie2): Remove duplicate tag

* style: Format yamls

* test(subworkflows): Only check versions for modules

* chore: Update subworkflows to match rnaseq dev

* fix(subworkflows): Update paths

* fix(bam_sort_samtools): Fix sort parameters for testing

* Apply suggestions from code review

Co-authored-by: Harshil Patel <drpatelh@users.noreply.github.com>

* docs: Update TODOs with a message

* ci: Try using a matrix for strategy

* ci: Try passing an array

* Revert "ci: Try passing an array"

This reverts commit d3611fcd8332bbb9a8501e8dd299d0a623aaecaa.

Co-authored-by: Harshil Patel <drpatelh@users.noreply.github.com>
2021-10-08 16:02:42 +01:00
Gregor Sturm
3d0b873584
Run CI only on pull-requests and on the master-branch. (#743)
* Run CI only on pull-requests and on the master-branch.

* also for the other workflows
2021-09-27 08:03:30 +02:00
Gregor Sturm
ab67a1d41b
Update fastqc to produce multi-version versions.yml (#665)
* Update fastqc to produce multi-version versions.yml

* Update readme and pull request template

* Fix markdownlint

* remove  variable

* Change publish dir to lowercase

* Re-add getSoftwareName

* Add custom pytest-workflow test to ensure versions.yml is valid

* Add docstring

* Remove __init__.py as it is not needed

* Remove changes to README, since this part went to nf-co.re

* Add NF_CORE_TEST env var

* Fix editorconfig

* Add additional consistency checks for versions.yml

* Update multiqc module

* Fix output channel
2021-09-24 10:01:54 +01:00
Edmund Miller
669fb5caed
Homer Modules (#75)
* feat(homer): Add initial makeTagDirectory

* feat(homer): Add initial findPeaks module

* feat(homer): Update with new options

See 1d30e2c21a

* fix(homer): Correct findpeaks process name

* fix(homer): Takes a bam file instead of bed

* feat(homer): Add initial makeTagDirectory test

* fix(homer): Hardcode genome and configureHomer

I'd like to modularize configureHomer, but I need to figure out how
exactly the genomes work.

* fix(homer): bam => bed

Bam requires samtools to be present, which it's not in this docker image

* feat(homer): Add initial configureHomer script

* ci(homer): Add initial test

* test(homer): Reproducible configuration workaround

- I can't run both tests(one file and two files) at the same time because it breaks
- I can't copy the genome stuff from the configurehomer module because it's read only
- So I can't make the makeTagDirectory module depend on configureHomer

* test(homer): Add placeholder annotatepeaks

The required inputs are necessarily required for all workflows from what
I've used, but I'll need to look at the actual docs

* test(homer): Add missing B.bed

* test(homer): Rename two => groseq

Then all of the various workflows that homer provides can be e2e tested

* feat(homer): Add initial makeUCSCfile module

* test(homer): Add start to makeUCSCfile testing

* chore(homer): Add various cleanups

* test(homer): Rewrite annotatepeaks

Not passing yet

* test(homer): Rewrite configurehomer

* test(homer): Rewrite findpeaks

Still failing

* test(homer): Rewrite makeucscfile

Not passing yet

* test(homer): Rewrite maketagdirectory

All homer modules now follow the new structure. Time to make them pass.

* test(homer): Fix typo for workflow name

* fix(homer): Use correct container

* fix(homer): Accept fasta in maketagdirectory

Apparently all of the homer stuff can just take any old fasta and you
don't need to configure the genome ahead of time with configureHomer

* test(homer): makeTagDirectory passes now

* fix(homer): Update containers in makeucscfile

* test(homer): Rewrite makeucscfile

Takes input from maketagdirectory which is how the module should be used

* fix(homer): Update makeUCSCFile bedgraph path

* test(homer): Update makeucscfile expected output

* fix(homer): Update containers in findpeaks

* fix(homer): Change findpeaks args

The user is just going to have to know what they're doing for now

* test(homer): findPeaks rewrite with tagDir input

* test(homer): Update expected files for findPeaks

And bump filters

* style: Appease editorconfig

* ci: Remove old workflow

* tests(homer): Add md5sums

* test(homer): Add meta test

* style(homer): Capitalize HOMER

* docs(homer): Add maketagdirectory meta.yml

* docs(homer): Add makeucscfile meta.yml

* docs(homer): Add findpeaks meta.yml

* test(homer): Update to new test data standards

* chore: Remove stuff that got revived in the rebase

* chore: software => modules

* test(homer): Update tags

* test(homer): Update annotatepeaks

* ci: Fix uploading of artifacts

GitHub actions doesn't like the / in the tags

* test(homer): Remove annotate md5sum

This is failing and breaking new tests

* test(homer): Use bams instead of beds

* test(homer): Fix meta maketagdirectory

* test(homer): Fix input in all tests

* test(homer): Move back to bed files

Forgot samtools isn't present

* chore(homer): Add TODOs for tests

* test(homer): Add bed format arg

* test(homer): Update md5sums

* test(homer): Fix tags tsvs

* style(homer): Appease nf-core linting

* docs(homer): Be in line with what is in the main.nf file

Co-authored-by: Kevin Menden <kevin.menden@live.com>

Co-authored-by: Kevin Menden <kevin.menden@live.com>
2021-09-08 15:40:34 +00:00
Harshil Patel
e937c7950a
Rename software/ directory to modules/ to re-organise module structure (#567)
* Update README

* Rename pytest_software.yml to pytest_modules.yml

* Rename main software directory to modules

* Remove deprecated modules

* Rename tests software to modules

* Replace paths for tests in pytest_modules.yml

* Replace software with modules in Github Actions

* Replace software with modules in main.nf tests

* Rename software to modules in test.yml
2021-07-07 10:10:18 +01:00
Kevin Menden
2395734492
Adapt linting action (#556)
* bump bcftools filter

* fix bcftools

* adapt module linting
2021-07-02 14:15:32 +02:00
Edmund Miller
bdee7804ca
build: Bump version to 21.04.0 (#481)
* build: Bump version to 21.04.0

Recent stable release https://github.com/nextflow-io/nextflow/releases/tag/v21.04.0

* Update README.md

Co-authored-by: Harshil Patel <drpatelh@users.noreply.github.com>
2021-05-04 00:38:29 +01:00
Harshil Patel
f34892c1af
Test PR for nf-core modules lint (#414)
* Update functions.nf for publish_by_id

* Pass a list of publish_by_id instead of a string

* Remove NF version

* Rename code linting file

* Run separate workflow for nf-core modules lint

* Adjust nf-core update comment

* Comment out linting from pytest-workflow.yml

* Remove commented out linting code from pytest-workflow.yml

* Adjust spacing

* Adjust spacing again

* Move NF version to job level for linting

* Add CAPSULE_LOG none env when installing NF

* Remove NXF_ANSI_LOG env for linting

* Revert changes
2021-04-09 07:58:43 +01:00
Edmund Miller
a3684d9594
Lint modules ci (#389)
* ci: Add modules lint step

Moved it ahead of the nextflow install so ideally it'll fail before we
bother doing any more setup

* ci: _ => /

* Update tests/config/pytest_software.yml

Co-authored-by: Harshil Patel <drpatelh@users.noreply.github.com>
2021-04-08 09:10:52 +01:00
Edmund Miller
6e4b23a1ca
ci: Upload work directory on failure (#388)
Closes #374
2021-03-25 09:34:19 +00:00
Edmund Miller
1fc4aa260d
ci: Add 21.03.0 version to matrix (#317) 2021-03-22 22:57:59 +00:00
Gregor Sturm
bed31cd9b4
Add link to issue in pull request template (#312)
* Add link to issue in pull request template

* Update PULL_REQUEST_TEMPLATE.md
2021-03-22 10:04:20 -07:00
Maxime Garcia
e61f400167
feature: add new issue template for creating new module (#283)
* feature: add new issue template for creating new module

* Apply suggestions from code review

* Apply suggestions from code review

Co-authored-by: Kevin Menden <kevin.menden@live.com>

Co-authored-by: Kevin Menden <kevin.menden@live.com>
2021-03-19 15:39:22 +01:00
Harshil Patel
9b99b33401
Rename .github/filters.yml -> tests/config/pytest_software.yml (#271)
* Rename pytest filters.yml

* Change filters.yml name

* Update resoures for ivar trim
2021-03-15 12:51:30 +00:00
Jose Espinosa-Carrasco
572abb00b3
Remove def from module options definition in main (#270)
* Remove def from module options definition in main

* Fix bismark_deduplicate tests

* Fix bwameth_align tests

* Fixing gatk4 conda tests ("=" instead of ':' in build id)

* Same as previous commit (Fix gatk4 test)

* Fix qualimap bamqc test (no md5 check for pngs)

* Fix seqkit split2 tests. Changed to new test data

* Fix samtools tests. Some were missing initOptions include

* Removing TOOL SUBTOOL template module since now it is included on tools repo
2021-03-15 12:16:43 +00:00
Maxime Garcia
05cbbbaded
Rename gatk/createsequencedictionary -> gatk4/createsequencedictionary (#260)
* fix: update and sort filters

* fix: rename gatk/createsequencedictionary -> gatk4/createsequencedictionary
2021-03-08 14:45:46 +01:00
Kevin Menden
4566525da2
Converge test data usage (#249)
* initial data restructuing

* fixed bedtools_complement

* fixed bedtools_genomecov

* fixed bedtools_getfasta

* fixed bedtools_intersect

* fixed bedtools maskfasta

* fixed bedtools_merge

* fixed bedtools_slop

* fixed bedtools_sort

* fixed bismark_genome_preparation

* fixed blast

* fixed bowtie data

* fixed bowtie2 data

* fixed bwa data

* fixed bwamem2 data usage

* fixed cat_fastq data

* fixed cutadapt data

* fixed dsh data

* fixed fastp data

* fixed fastqc; fixed bug with wrong fastq format

* fixed gatk

* fixed data for gffread, gunzip

* fixed ivar paths

* fixed data paths for minimap2

* fixed mosdepth

* fixed multiqc, pangolin

* fixed picard data paths

* fixed data paths for qualimap, quast

* fixed salmon data paths

* fixed samtools paths

* fixed seqwish, stringtie paths

* fixed tabix, trimgalore paths

* cleaned up data

* added first description to README

* changed test data naming again; everything up to bwa fixed

* everything up to gatk4

* fixed everything up to ivar

* fixed everything up to picard

* everything up to quast

* everything fixed up to stringtie

* switched everyting to 'test' naming scheme

* fixed samtools and ivar tests

* cleaned up README a bit

* add (simulated) methylation test data

based on SARS-CoV-2 genome; simulated with Sherman --non_dir --genome sarscov2/fasta/ --paired -n 10000 -l 100 --CG 20 --CH 90

* bwameth/align: update data paths and checksums

also, build index on the go

* bwameth/index: update data paths and checksums

* methyldackel/extract: update data paths and checksums

* methyldackel/mbias: update data paths and checksums

* bismark/deduplicate: update data paths and checksums

* remove obsolete testdata

* remove empty 'dummy_file.txt'

* update data/README.md

* methyldackel: fix test

* Revert "methyldackel: fix test"

This reverts commit f175a32d144b1b0bfa0c6885da80c51e3cfe038a.

* methyldackel: fix test

for real

* move test.genome.sizes

* changed test names

* switched genomic to genome and transcriptome

* fix bedtools, blast

* fix gtf, tabix, .paf

* fix bowtie,bwa,bwameth

* fixed: bwa, bwamem, gatk, gffread, quast

* fixed bismark and blast

* fixed remaining tests

* delete bam file

Co-authored-by: phue <patrick.huether@gmail.com>
2021-03-04 10:10:57 +00:00
Maxime Garcia
ea6fde0193
Merge pull request #252 from MaxUlysse/master_tiddit
New module: tiddit
2021-02-25 09:34:33 +01:00
MaxUlysse
f62196a4a6 feat: add tiddit_sv module 2021-02-24 15:59:10 +01:00
FriederikeHanssen
48d6d88104
Update .github/filters.yml
Co-authored-by: Kevin Menden <kevin.menden@live.com>
2021-02-24 10:29:13 +01:00
FriederikeHanssen
4673c119c7 resolve merge conflicts 2021-02-24 10:03:16 +01:00
Jose Espinosa-Carrasco
8f4363ccb5
Merge pull request #247 from JoseEspinosa/bcftools-mpileup
Add bcftools mpileup module
2021-02-24 09:24:43 +01:00
Harshil Patel
5287fe5f95
Merge pull request #245 from phue/bismark_deduplicate
add bismark/deduplicate + tests
2021-02-23 21:08:24 +00:00
Harshil Patel
4703a60443
Merge pull request #220 from phue/bwameth
add bwameth modules
2021-02-23 21:01:54 +00:00
JoseEspinosa
20bcb00ea0 Fix indentation 2021-02-23 19:08:56 +01:00
JoseEspinosa
4b3c86ed9f Add tests for bcftools-mpileup 2021-02-23 19:05:57 +01:00
phue
e73bda26f5 add bismark/deduplicate + tests 2021-02-23 18:02:16 +01:00
Gregor Sturm
38349417ad
Update .github/workflows/pytest-workflow.yml
Co-authored-by: Harshil Patel <drpatelh@users.noreply.github.com>
2021-02-23 14:14:29 +01:00
Gregor Sturm
94bdc62123 Only use one thread for pytest-workflow
This avoids a race condition on the conda cache.
2021-02-23 14:06:52 +01:00
FriederikeHanssen
a0c9777d60 add seqkit to GHA 2021-02-23 12:43:30 +01:00
kevinmenden
3fee38ee2d added gatk4 revertsam 2021-02-22 15:21:39 +01:00
kevinmenden
bd585265ce added entry to filters.yml 2021-02-22 13:36:49 +01:00
Kevin Menden
0202162b17
Merge branch 'master' into gatk4-mergevcfs 2021-02-19 14:11:30 +01:00
MaxUlysse
2bf3890afb Merge remote-tracking branch 'upstream/master' into master_tiddit 2021-02-19 13:37:45 +01:00
kevinmenden
3ac5c5fad6 working test 2021-02-19 13:36:42 +01:00
kevinmenden
c0693fa3db remove whitespace 2021-02-19 13:30:40 +01:00
Kevin Menden
5c325f2942
Merge branch 'master' into master 2021-02-19 13:08:43 +01:00
kevinmenden
bd8ba42a8d added gatk4 SamToFastq 2021-02-19 12:41:29 +01:00
kevinmenden
6dcaf5ffc8 add gatk4 bedtointervallist 2021-02-19 11:26:31 +01:00
MaxUlysse
eaff4eb0ff fix: order alphabetically 2021-02-19 10:56:04 +01:00