Commit graph

190 commits

Author SHA1 Message Date
Edmund Miller
c19671dca9
Subworkflow Infrastructure (#662)
* feat(subworkflows): Add align_bowtie2 subworkflow

For testing CI setup

* test(align_bowtie2): Add initial list of changes to test

* test(align_bowtie2): Add initial test

* refactor: Use tags to run subworkflows ci

For every underlying module used by workflow and allow the modules
pytest-modules definition be the source of truth.

* refactor: Use individual directories for subworkflows

* docs(align_bowtie2): Add initial meta.yml

Copied most of it from the bowtie2/align module.

* fix(align_bowtie2): Fix module include paths

* test(bam_sort_samtools): Add initial test

* ci(bam_sort_samtools): Add modules that trigger the tag

* test(bam_stats_samtools): Add initial test

* ci(bam_stats_samtools): Add keys to pick up changes

* docs(bam_samtools): Add initial meta.yml

* test(align_bowtie2): Fix path to subworkflow

* test(align_bowtie2): Update entry point

* fix(bam_sort_samtools): Update include paths

* test(bam_sort_samtools): Fix path

* style: Clean up addParams

* test(samtools_sort): Add suffix for test

* test(align_bowtie2): Add samtools_options for suffix

* test(bam_stats_samtools): Update path

* test(bam_stats_samtools): Use stats input

Otherwise it's just an example of how it's used in the bam_sort_samtools subworkflow

* ci(linting): Skip module linting of subworkflows

* ci(linting): Clean up startsWith statement

* test(bam_stats_samtools): Use single end test data for single end test

* test(bam_stats_samtools): Add expected files

* test(align_bowtie2): Add paired-end test

* test(align_bowtie2): Sort order of output

* test(align_bowtie2): Update hashes

* docs(align_bowtie2): Fix typo

* test(align_bowtie2): Update samtools output names

* test(align_bowtie2): Remove md5sums for bam/bai

* feat(subworkflows): Add nextflow.configs

These can be used for default settings in the future. They can then be
included in the conf/modules.config so that the params don't have to be
duplicated in the root nextflow.config.

* docs(subworkflows): Include modules instead of tools

* fix: Update to versions

* chore(align_bowtie2): Remove duplicate tag

* style: Format yamls

* test(subworkflows): Only check versions for modules

* chore: Update subworkflows to match rnaseq dev

* fix(subworkflows): Update paths

* fix(bam_sort_samtools): Fix sort parameters for testing

* Apply suggestions from code review

Co-authored-by: Harshil Patel <drpatelh@users.noreply.github.com>

* docs: Update TODOs with a message

* ci: Try using a matrix for strategy

* ci: Try passing an array

* Revert "ci: Try passing an array"

This reverts commit d3611fcd8332bbb9a8501e8dd299d0a623aaecaa.

Co-authored-by: Harshil Patel <drpatelh@users.noreply.github.com>
2021-10-08 16:02:42 +01:00
Gregor Sturm
3d0b873584
Run CI only on pull-requests and on the master-branch. (#743)
* Run CI only on pull-requests and on the master-branch.

* also for the other workflows
2021-09-27 08:03:30 +02:00
Gregor Sturm
ab67a1d41b
Update fastqc to produce multi-version versions.yml (#665)
* Update fastqc to produce multi-version versions.yml

* Update readme and pull request template

* Fix markdownlint

* remove  variable

* Change publish dir to lowercase

* Re-add getSoftwareName

* Add custom pytest-workflow test to ensure versions.yml is valid

* Add docstring

* Remove __init__.py as it is not needed

* Remove changes to README, since this part went to nf-co.re

* Add NF_CORE_TEST env var

* Fix editorconfig

* Add additional consistency checks for versions.yml

* Update multiqc module

* Fix output channel
2021-09-24 10:01:54 +01:00
Edmund Miller
669fb5caed
Homer Modules (#75)
* feat(homer): Add initial makeTagDirectory

* feat(homer): Add initial findPeaks module

* feat(homer): Update with new options

See 1d30e2c21a

* fix(homer): Correct findpeaks process name

* fix(homer): Takes a bam file instead of bed

* feat(homer): Add initial makeTagDirectory test

* fix(homer): Hardcode genome and configureHomer

I'd like to modularize configureHomer, but I need to figure out how
exactly the genomes work.

* fix(homer): bam => bed

Bam requires samtools to be present, which it's not in this docker image

* feat(homer): Add initial configureHomer script

* ci(homer): Add initial test

* test(homer): Reproducible configuration workaround

- I can't run both tests(one file and two files) at the same time because it breaks
- I can't copy the genome stuff from the configurehomer module because it's read only
- So I can't make the makeTagDirectory module depend on configureHomer

* test(homer): Add placeholder annotatepeaks

The required inputs are necessarily required for all workflows from what
I've used, but I'll need to look at the actual docs

* test(homer): Add missing B.bed

* test(homer): Rename two => groseq

Then all of the various workflows that homer provides can be e2e tested

* feat(homer): Add initial makeUCSCfile module

* test(homer): Add start to makeUCSCfile testing

* chore(homer): Add various cleanups

* test(homer): Rewrite annotatepeaks

Not passing yet

* test(homer): Rewrite configurehomer

* test(homer): Rewrite findpeaks

Still failing

* test(homer): Rewrite makeucscfile

Not passing yet

* test(homer): Rewrite maketagdirectory

All homer modules now follow the new structure. Time to make them pass.

* test(homer): Fix typo for workflow name

* fix(homer): Use correct container

* fix(homer): Accept fasta in maketagdirectory

Apparently all of the homer stuff can just take any old fasta and you
don't need to configure the genome ahead of time with configureHomer

* test(homer): makeTagDirectory passes now

* fix(homer): Update containers in makeucscfile

* test(homer): Rewrite makeucscfile

Takes input from maketagdirectory which is how the module should be used

* fix(homer): Update makeUCSCFile bedgraph path

* test(homer): Update makeucscfile expected output

* fix(homer): Update containers in findpeaks

* fix(homer): Change findpeaks args

The user is just going to have to know what they're doing for now

* test(homer): findPeaks rewrite with tagDir input

* test(homer): Update expected files for findPeaks

And bump filters

* style: Appease editorconfig

* ci: Remove old workflow

* tests(homer): Add md5sums

* test(homer): Add meta test

* style(homer): Capitalize HOMER

* docs(homer): Add maketagdirectory meta.yml

* docs(homer): Add makeucscfile meta.yml

* docs(homer): Add findpeaks meta.yml

* test(homer): Update to new test data standards

* chore: Remove stuff that got revived in the rebase

* chore: software => modules

* test(homer): Update tags

* test(homer): Update annotatepeaks

* ci: Fix uploading of artifacts

GitHub actions doesn't like the / in the tags

* test(homer): Remove annotate md5sum

This is failing and breaking new tests

* test(homer): Use bams instead of beds

* test(homer): Fix meta maketagdirectory

* test(homer): Fix input in all tests

* test(homer): Move back to bed files

Forgot samtools isn't present

* chore(homer): Add TODOs for tests

* test(homer): Add bed format arg

* test(homer): Update md5sums

* test(homer): Fix tags tsvs

* style(homer): Appease nf-core linting

* docs(homer): Be in line with what is in the main.nf file

Co-authored-by: Kevin Menden <kevin.menden@live.com>

Co-authored-by: Kevin Menden <kevin.menden@live.com>
2021-09-08 15:40:34 +00:00
Harshil Patel
e937c7950a
Rename software/ directory to modules/ to re-organise module structure (#567)
* Update README

* Rename pytest_software.yml to pytest_modules.yml

* Rename main software directory to modules

* Remove deprecated modules

* Rename tests software to modules

* Replace paths for tests in pytest_modules.yml

* Replace software with modules in Github Actions

* Replace software with modules in main.nf tests

* Rename software to modules in test.yml
2021-07-07 10:10:18 +01:00
Kevin Menden
2395734492
Adapt linting action (#556)
* bump bcftools filter

* fix bcftools

* adapt module linting
2021-07-02 14:15:32 +02:00
Edmund Miller
bdee7804ca
build: Bump version to 21.04.0 (#481)
* build: Bump version to 21.04.0

Recent stable release https://github.com/nextflow-io/nextflow/releases/tag/v21.04.0

* Update README.md

Co-authored-by: Harshil Patel <drpatelh@users.noreply.github.com>
2021-05-04 00:38:29 +01:00
Harshil Patel
f34892c1af
Test PR for nf-core modules lint (#414)
* Update functions.nf for publish_by_id

* Pass a list of publish_by_id instead of a string

* Remove NF version

* Rename code linting file

* Run separate workflow for nf-core modules lint

* Adjust nf-core update comment

* Comment out linting from pytest-workflow.yml

* Remove commented out linting code from pytest-workflow.yml

* Adjust spacing

* Adjust spacing again

* Move NF version to job level for linting

* Add CAPSULE_LOG none env when installing NF

* Remove NXF_ANSI_LOG env for linting

* Revert changes
2021-04-09 07:58:43 +01:00
Edmund Miller
a3684d9594
Lint modules ci (#389)
* ci: Add modules lint step

Moved it ahead of the nextflow install so ideally it'll fail before we
bother doing any more setup

* ci: _ => /

* Update tests/config/pytest_software.yml

Co-authored-by: Harshil Patel <drpatelh@users.noreply.github.com>
2021-04-08 09:10:52 +01:00
Edmund Miller
6e4b23a1ca
ci: Upload work directory on failure (#388)
Closes #374
2021-03-25 09:34:19 +00:00
Edmund Miller
1fc4aa260d
ci: Add 21.03.0 version to matrix (#317) 2021-03-22 22:57:59 +00:00
Harshil Patel
9b99b33401
Rename .github/filters.yml -> tests/config/pytest_software.yml (#271)
* Rename pytest filters.yml

* Change filters.yml name

* Update resoures for ivar trim
2021-03-15 12:51:30 +00:00
Gregor Sturm
38349417ad
Update .github/workflows/pytest-workflow.yml
Co-authored-by: Harshil Patel <drpatelh@users.noreply.github.com>
2021-02-23 14:14:29 +01:00
Gregor Sturm
94bdc62123 Only use one thread for pytest-workflow
This avoids a race condition on the conda cache.
2021-02-23 14:06:52 +01:00
drpatelh
418bd64e80 Test module with Conda 2021-02-07 21:33:42 +00:00
Edmund Miller
05c09429a3
ci: Fix artifact path
Had to change it for singularity
2021-02-04 08:45:34 -06:00
Edmund Miller
6fe7b48b5b
Revert "Run alllll the testsgit add .github/workflows/pytest-workflow.yml"
This reverts commit ffc218e51b.
2021-02-03 20:08:11 -06:00
drpatelh
ffc218e51b
Run alllll the testsgit add .github/workflows/pytest-workflow.yml 2021-02-03 20:03:45 -06:00
drpatelh
0c88ff7458
Upgrade Singularity and change TMPDIR for pytest output 2021-02-03 20:03:43 -06:00
drpatelh
3968c18fd7 Only run Docker tests 2021-02-03 14:04:58 +00:00
drpatelh
9e44b0f4f2 Add conda clean step to action 2021-02-03 12:57:55 +00:00
drpatelh
49fecaa4f0 Switch channel order 2021-02-03 12:07:39 +00:00
drpatelh
d783e2d535 Only test Docker and Conda 2021-02-03 09:02:12 +00:00
Edmund Miller
54bb0ae7a1
ci: Name logs 2021-02-02 17:51:20 -06:00
Edmund Miller
42b8962320
ci: Add bioconda channels 2021-02-02 17:13:35 -06:00
Edmund Miller
0813e61c01
ci: Kick off all container types 2021-02-02 17:08:10 -06:00
Edmund Miller
725de25ce4
Revert "ci: Only test docker for now"
This reverts commit c63ba06777.
2021-02-02 17:03:29 -06:00
Edmund Miller
c63ba06777
ci: Only test docker for now
I can't access the artifacts.
2021-02-02 16:49:03 -06:00
Edmund Miller
bb3196ddeb
ci: Update name to use matrix 2021-02-02 16:49:03 -06:00
Edmund Miller
dcfd1089e5
ci: Add --keep-workflow-wd-on-fail 2021-02-02 16:49:03 -06:00
Edmund Miller
983b024010
ci: Use ubuntu-20.04 for singularity 2021-02-02 16:49:02 -06:00
Edmund Miller
8cd6e41df9
ci: Change singularity version 3.6.4 => 3.5.3
Per @grst having issues after 3.6
2021-02-02 16:49:02 -06:00
Edmund Miller
bf2e027fdf
ci: Remove conda to kickoff CI 2021-02-02 16:48:58 -06:00
Edmund Miller
1e9aad430d
ci: Remove ${{}} for style points
It's not necessary in if statements
2021-02-02 15:40:30 -06:00
Edmund Miller
654c6225f0
ci: Setup multiple profiles for testing
Added conda and singularity
2021-02-02 15:38:51 -06:00
Edmund Miller
35ac5c5f15
ci: Add filters for new modules
These snuck in after the 506ebe5 was written
2021-02-02 15:27:45 -06:00
Edmund Miller
341fa43ae3
ci: Add PROFILE ahead of pytest run 2021-02-02 13:07:58 -06:00
Harshil Patel
e7b1241449
Merge pull request #120 from nf-core/composite-runs
Universal CI job
2021-02-02 18:26:02 +00:00
Edmund Miller
ce2a5b32f5
refactor: Move away from composite-action 2021-02-02 12:21:36 -06:00
drpatelh
f84e5b59ca Rename bowtie_index to bowtie_build 2021-02-01 17:53:31 +00:00
Harshil Patel
0a41130276
Merge pull request #131 from KevinMenden/fastp-module
Adding fastp module
2021-02-01 13:43:35 +00:00
Harshil Patel
e851da8640
Merge pull request #130 from KevinMenden/master
Adding a QUAST module
2021-02-01 13:17:57 +00:00
kevinmenden
e9e795633d added fastp tests 2021-02-01 14:02:06 +01:00
Harshil Patel
c74d8a1711
Merge pull request #128 from JoseEspinosa/bowtie2_viralrecon
Bowtie2 viralrecon 2
2021-02-01 09:56:11 +00:00
Harshil Patel
2c962f34df
Merge pull request #110 from luslab/master
Add SEACR module containing a single CALLPEAK process
2021-02-01 09:48:15 +00:00
kevinmenden
956bf22a58 fixed bug; updated meta.yml 2021-02-01 07:44:26 +01:00
JoseEspinosa
177693de49 Adding worflow yml for bowtie2_align and bowtie2_build 2021-01-30 22:53:27 +01:00
Harshil Patel
bd59706d51
Merge pull request #118 from heuermh/dsh-bio
Added dsh-bio filter-bed as dsh_filterbed
2021-01-29 18:15:24 +00:00
Harshil Patel
0fcbc4233c
Merge pull request #126 from JoseEspinosa/samtools_viralrecon
Samtools viralrecon
2021-01-29 18:12:48 +00:00
JoseEspinosa
22dcdc692d Adding samtools_mpileup module 2021-01-29 17:58:42 +01:00