nf-core_modules/.github/workflows/pytest-workflow.yml

119 lines
3.3 KiB
YAML
Raw Normal View History

2021-01-22 19:32:54 +00:00
name: Pytest-workflow
on:
push:
branches: [master]
pull_request:
branches: [master]
2021-01-22 19:32:54 +00:00
jobs:
changes:
name: Check for changes
runs-on: ubuntu-latest
outputs:
# Expose matched filters as job 'modules' output variable
modules: ${{ steps.filter.outputs.changes }}
steps:
- uses: actions/checkout@v2
- uses: dorny/paths-filter@v2
id: filter
with:
filters: "tests/config/pytest_modules.yml"
test:
runs-on: ubuntu-20.04
Update all modules to new NF DSL2 syntax (#1099) * Add comment line for consistency * Remove all functions.nf * Remove include functions.nf and publishDir options * Replace options.args3 with task.ext.args3 - 3 modules * Replace options.args3 with task.ext.args3 - 17 modules * Replace {task.cpus} with task.cpus * Replace off on off off off off off off off on off on off on off off off on off off off on on off off off on on off off off off off off off on off off off off on off on on off off off on on on on off off off on off on on off on on off off on on on off on on off on off off off off on off off off on off off on off on off off off on on off on off on off off on off off off on off off off on off off off off on off off off on on on off on on off off on off on on on off on on off on on on off off off off off on on off off on off off off off off on off off on on off on on off on off off off on off off off off on on off on off off on off off on off on off off off off off off off off on on off on off off off.args with * Add def args = task.ext.args line to all modules in script section * Replace options.args with args and args_list * Initialise args2 and args3 properly * Replace container syntax * Revert container changes for cellranger/mkref * Replace getProcessName in all modules * Replace getSoftwareName in all modules * Unify modules using VERSION variable * Replae options.suffix with task.ext.suffix * Remove NF version restriction for CI * Bump NF version in README * Replace task.process.tokenize logic with task.process * Minor tweaks to unify syntax in tests main.nf * Add a separate nextflow.config for each module * Transfer remaining module options to nextflow.config * Remove addParams from tests main.nf * Remove TODO statements * Use -c to import module specific config * Bump NF version to 21.10.3 * Fix tests for artic/minion * Fix broken publishDir syntax * Standardise and fix obvious failing module tests * Remove kronatools to krona * Comment out tags in subworkflow test.yml * Fix failing module tests * Add consistent indentation to nextflow.config * Comment out subworklow definitions * Fix kallistobustools/ref * Fix rmarkdownnotebook * Fix jupyternotebook * Quote task.process * Add plink2/vcf to pytest_modules.yml * Remove NF_CORE_MODULES_TEST from pytest CI * Fix more tests * Move bacteroides_fragilis to prokaryotes folder * Fix cooler merge tests * Fix kallistobustools/count tests * Fix kallistobustools/ref tests * Update test_10x_1_fastq_gz file for kallistobustools/count tests * Fix bcftools/query tests * Fix delly/call tests * Fix cooler/zoomify tests * Fix csvtk/split tests * Fix gatk4/intervallisttools tests * Fix gatk4/variantfiltration * Fix pydamage/filter tests * Fix test data for unicycler * Fix gstama/collapse module * Fix leehom tests * Fix metaphlan3 tests * Fix pairtools/select tests * Update nextflow.config * Update nextflow.config * feat: update syntax * Fix arriba tests * Fix more failing tests * Update test syntax * Remove comments from tests nextflow.config * Apply suggestions from code review * Fix kallistobustools/count module * Update dumpsoftwareversions module * Update custom/dumpsoftwareversions * Add args2 to untar module * Update leftover modules * Remove last remaining addParams Co-authored-by: JoseEspinosa <kadomu@gmail.com> Co-authored-by: Gregor Sturm <mail@gregor-sturm.de> Co-authored-by: MaxUlysse <max.u.garcia@gmail.com>
2021-11-26 07:58:40 +00:00
name: ${{ matrix.tags }} ${{ matrix.profile }}
needs: changes
if: needs.changes.outputs.modules != '[]'
Subworkflow Infrastructure (#662) * feat(subworkflows): Add align_bowtie2 subworkflow For testing CI setup * test(align_bowtie2): Add initial list of changes to test * test(align_bowtie2): Add initial test * refactor: Use tags to run subworkflows ci For every underlying module used by workflow and allow the modules pytest-modules definition be the source of truth. * refactor: Use individual directories for subworkflows * docs(align_bowtie2): Add initial meta.yml Copied most of it from the bowtie2/align module. * fix(align_bowtie2): Fix module include paths * test(bam_sort_samtools): Add initial test * ci(bam_sort_samtools): Add modules that trigger the tag * test(bam_stats_samtools): Add initial test * ci(bam_stats_samtools): Add keys to pick up changes * docs(bam_samtools): Add initial meta.yml * test(align_bowtie2): Fix path to subworkflow * test(align_bowtie2): Update entry point * fix(bam_sort_samtools): Update include paths * test(bam_sort_samtools): Fix path * style: Clean up addParams * test(samtools_sort): Add suffix for test * test(align_bowtie2): Add samtools_options for suffix * test(bam_stats_samtools): Update path * test(bam_stats_samtools): Use stats input Otherwise it's just an example of how it's used in the bam_sort_samtools subworkflow * ci(linting): Skip module linting of subworkflows * ci(linting): Clean up startsWith statement * test(bam_stats_samtools): Use single end test data for single end test * test(bam_stats_samtools): Add expected files * test(align_bowtie2): Add paired-end test * test(align_bowtie2): Sort order of output * test(align_bowtie2): Update hashes * docs(align_bowtie2): Fix typo * test(align_bowtie2): Update samtools output names * test(align_bowtie2): Remove md5sums for bam/bai * feat(subworkflows): Add nextflow.configs These can be used for default settings in the future. They can then be included in the conf/modules.config so that the params don't have to be duplicated in the root nextflow.config. * docs(subworkflows): Include modules instead of tools * fix: Update to versions * chore(align_bowtie2): Remove duplicate tag * style: Format yamls * test(subworkflows): Only check versions for modules * chore: Update subworkflows to match rnaseq dev * fix(subworkflows): Update paths * fix(bam_sort_samtools): Fix sort parameters for testing * Apply suggestions from code review Co-authored-by: Harshil Patel <drpatelh@users.noreply.github.com> * docs: Update TODOs with a message * ci: Try using a matrix for strategy * ci: Try passing an array * Revert "ci: Try passing an array" This reverts commit d3611fcd8332bbb9a8501e8dd299d0a623aaecaa. Co-authored-by: Harshil Patel <drpatelh@users.noreply.github.com>
2021-10-08 15:02:42 +00:00
strategy:
fail-fast: false
matrix:
tags: ["${{ fromJson(needs.changes.outputs.modules) }}"]
profile: ["docker", "singularity", "conda"]
2021-01-22 19:32:54 +00:00
env:
NXF_ANSI_LOG: false
steps:
- uses: actions/checkout@v2
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: "3.x"
2021-01-24 17:25:13 +00:00
- uses: actions/cache@v2
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
restore-keys: |
${{ runner.os }}-pip-
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: "3.x"
- name: Install Python dependencies
run: python -m pip install --upgrade pip pytest-workflow
2021-01-24 17:35:24 +00:00
- uses: actions/cache@v2
with:
path: /usr/local/bin/nextflow
Update all modules to new NF DSL2 syntax (#1099) * Add comment line for consistency * Remove all functions.nf * Remove include functions.nf and publishDir options * Replace options.args3 with task.ext.args3 - 3 modules * Replace options.args3 with task.ext.args3 - 17 modules * Replace {task.cpus} with task.cpus * Replace off on off off off off off off off on off on off on off off off on off off off on on off off off on on off off off off off off off on off off off off on off on on off off off on on on on off off off on off on on off on on off off on on on off on on off on off off off off on off off off on off off on off on off off off on on off on off on off off on off off off on off off off on off off off off on off off off on on on off on on off off on off on on on off on on off on on on off off off off off on on off off on off off off off off on off off on on off on on off on off off off on off off off off on on off on off off on off off on off on off off off off off off off off on on off on off off off.args with * Add def args = task.ext.args line to all modules in script section * Replace options.args with args and args_list * Initialise args2 and args3 properly * Replace container syntax * Revert container changes for cellranger/mkref * Replace getProcessName in all modules * Replace getSoftwareName in all modules * Unify modules using VERSION variable * Replae options.suffix with task.ext.suffix * Remove NF version restriction for CI * Bump NF version in README * Replace task.process.tokenize logic with task.process * Minor tweaks to unify syntax in tests main.nf * Add a separate nextflow.config for each module * Transfer remaining module options to nextflow.config * Remove addParams from tests main.nf * Remove TODO statements * Use -c to import module specific config * Bump NF version to 21.10.3 * Fix tests for artic/minion * Fix broken publishDir syntax * Standardise and fix obvious failing module tests * Remove kronatools to krona * Comment out tags in subworkflow test.yml * Fix failing module tests * Add consistent indentation to nextflow.config * Comment out subworklow definitions * Fix kallistobustools/ref * Fix rmarkdownnotebook * Fix jupyternotebook * Quote task.process * Add plink2/vcf to pytest_modules.yml * Remove NF_CORE_MODULES_TEST from pytest CI * Fix more tests * Move bacteroides_fragilis to prokaryotes folder * Fix cooler merge tests * Fix kallistobustools/count tests * Fix kallistobustools/ref tests * Update test_10x_1_fastq_gz file for kallistobustools/count tests * Fix bcftools/query tests * Fix delly/call tests * Fix cooler/zoomify tests * Fix csvtk/split tests * Fix gatk4/intervallisttools tests * Fix gatk4/variantfiltration * Fix pydamage/filter tests * Fix test data for unicycler * Fix gstama/collapse module * Fix leehom tests * Fix metaphlan3 tests * Fix pairtools/select tests * Update nextflow.config * Update nextflow.config * feat: update syntax * Fix arriba tests * Fix more failing tests * Update test syntax * Remove comments from tests nextflow.config * Apply suggestions from code review * Fix kallistobustools/count module * Update dumpsoftwareversions module * Update custom/dumpsoftwareversions * Add args2 to untar module * Update leftover modules * Remove last remaining addParams Co-authored-by: JoseEspinosa <kadomu@gmail.com> Co-authored-by: Gregor Sturm <mail@gregor-sturm.de> Co-authored-by: MaxUlysse <max.u.garcia@gmail.com>
2021-11-26 07:58:40 +00:00
key: ${{ runner.os }}
2021-01-24 17:35:24 +00:00
restore-keys: |
${{ runner.os }}-nextflow-
- name: Install Nextflow
env:
CAPSULE_LOG: none
run: |
wget -qO- get.nextflow.io | bash
sudo mv nextflow /usr/local/bin/
- name: Set up Singularity
if: matrix.profile == 'singularity'
uses: eWaterCycle/setup-singularity@v5
with:
singularity-version: 3.7.1
2021-02-02 23:13:35 +00:00
- name: Setup miniconda
if: matrix.profile == 'conda'
uses: conda-incubator/setup-miniconda@v2
with:
auto-update-conda: true
2021-02-03 12:07:39 +00:00
channels: conda-forge,bioconda,defaults
python-version: ${{ matrix.python-version }}
2021-02-03 12:57:55 +00:00
- name: Conda clean
if: matrix.profile == 'conda'
run: conda clean -a
# Test the module
- name: Run pytest-workflow
# only use one thread for pytest-workflow to avoid race condition on conda cache.
Update all modules to new NF DSL2 syntax (#1099) * Add comment line for consistency * Remove all functions.nf * Remove include functions.nf and publishDir options * Replace options.args3 with task.ext.args3 - 3 modules * Replace options.args3 with task.ext.args3 - 17 modules * Replace {task.cpus} with task.cpus * Replace off on off off off off off off off on off on off on off off off on off off off on on off off off on on off off off off off off off on off off off off on off on on off off off on on on on off off off on off on on off on on off off on on on off on on off on off off off off on off off off on off off on off on off off off on on off on off on off off on off off off on off off off on off off off off on off off off on on on off on on off off on off on on on off on on off on on on off off off off off on on off off on off off off off off on off off on on off on on off on off off off on off off off off on on off on off off on off off on off on off off off off off off off off on on off on off off off.args with * Add def args = task.ext.args line to all modules in script section * Replace options.args with args and args_list * Initialise args2 and args3 properly * Replace container syntax * Revert container changes for cellranger/mkref * Replace getProcessName in all modules * Replace getSoftwareName in all modules * Unify modules using VERSION variable * Replae options.suffix with task.ext.suffix * Remove NF version restriction for CI * Bump NF version in README * Replace task.process.tokenize logic with task.process * Minor tweaks to unify syntax in tests main.nf * Add a separate nextflow.config for each module * Transfer remaining module options to nextflow.config * Remove addParams from tests main.nf * Remove TODO statements * Use -c to import module specific config * Bump NF version to 21.10.3 * Fix tests for artic/minion * Fix broken publishDir syntax * Standardise and fix obvious failing module tests * Remove kronatools to krona * Comment out tags in subworkflow test.yml * Fix failing module tests * Add consistent indentation to nextflow.config * Comment out subworklow definitions * Fix kallistobustools/ref * Fix rmarkdownnotebook * Fix jupyternotebook * Quote task.process * Add plink2/vcf to pytest_modules.yml * Remove NF_CORE_MODULES_TEST from pytest CI * Fix more tests * Move bacteroides_fragilis to prokaryotes folder * Fix cooler merge tests * Fix kallistobustools/count tests * Fix kallistobustools/ref tests * Update test_10x_1_fastq_gz file for kallistobustools/count tests * Fix bcftools/query tests * Fix delly/call tests * Fix cooler/zoomify tests * Fix csvtk/split tests * Fix gatk4/intervallisttools tests * Fix gatk4/variantfiltration * Fix pydamage/filter tests * Fix test data for unicycler * Fix gstama/collapse module * Fix leehom tests * Fix metaphlan3 tests * Fix pairtools/select tests * Update nextflow.config * Update nextflow.config * feat: update syntax * Fix arriba tests * Fix more failing tests * Update test syntax * Remove comments from tests nextflow.config * Apply suggestions from code review * Fix kallistobustools/count module * Update dumpsoftwareversions module * Update custom/dumpsoftwareversions * Add args2 to untar module * Update leftover modules * Remove last remaining addParams Co-authored-by: JoseEspinosa <kadomu@gmail.com> Co-authored-by: Gregor Sturm <mail@gregor-sturm.de> Co-authored-by: MaxUlysse <max.u.garcia@gmail.com>
2021-11-26 07:58:40 +00:00
run: TMPDIR=~ PROFILE=${{ matrix.profile }} pytest --tag ${{ matrix.tags }} --symlink --kwdof
2021-01-24 05:39:51 +00:00
2022-01-20 07:55:43 +00:00
- name: Output log on failure
if: failure()
run: |
2022-01-21 03:34:08 +00:00
sudo apt install bat
2022-01-20 07:55:43 +00:00
echo "======> log.out <======="
2022-01-21 03:34:08 +00:00
bat /home/runner/pytest_workflow_*/*/log.out
2022-01-20 07:55:43 +00:00
echo
echo
echo "======> log.err <======="
2022-01-21 03:34:08 +00:00
bat /home/runner/pytest_workflow_*/*/log.err
2022-01-20 07:55:43 +00:00
2021-01-24 05:39:51 +00:00
- name: Upload logs on failure
if: failure()
2021-01-24 05:39:51 +00:00
uses: actions/upload-artifact@v2
with:
Update all modules to new NF DSL2 syntax (#1099) * Add comment line for consistency * Remove all functions.nf * Remove include functions.nf and publishDir options * Replace options.args3 with task.ext.args3 - 3 modules * Replace options.args3 with task.ext.args3 - 17 modules * Replace {task.cpus} with task.cpus * Replace off on off off off off off off off on off on off on off off off on off off off on on off off off on on off off off off off off off on off off off off on off on on off off off on on on on off off off on off on on off on on off off on on on off on on off on off off off off on off off off on off off on off on off off off on on off on off on off off on off off off on off off off on off off off off on off off off on on on off on on off off on off on on on off on on off on on on off off off off off on on off off on off off off off off on off off on on off on on off on off off off on off off off off on on off on off off on off off on off on off off off off off off off off on on off on off off off.args with * Add def args = task.ext.args line to all modules in script section * Replace options.args with args and args_list * Initialise args2 and args3 properly * Replace container syntax * Revert container changes for cellranger/mkref * Replace getProcessName in all modules * Replace getSoftwareName in all modules * Unify modules using VERSION variable * Replae options.suffix with task.ext.suffix * Remove NF version restriction for CI * Bump NF version in README * Replace task.process.tokenize logic with task.process * Minor tweaks to unify syntax in tests main.nf * Add a separate nextflow.config for each module * Transfer remaining module options to nextflow.config * Remove addParams from tests main.nf * Remove TODO statements * Use -c to import module specific config * Bump NF version to 21.10.3 * Fix tests for artic/minion * Fix broken publishDir syntax * Standardise and fix obvious failing module tests * Remove kronatools to krona * Comment out tags in subworkflow test.yml * Fix failing module tests * Add consistent indentation to nextflow.config * Comment out subworklow definitions * Fix kallistobustools/ref * Fix rmarkdownnotebook * Fix jupyternotebook * Quote task.process * Add plink2/vcf to pytest_modules.yml * Remove NF_CORE_MODULES_TEST from pytest CI * Fix more tests * Move bacteroides_fragilis to prokaryotes folder * Fix cooler merge tests * Fix kallistobustools/count tests * Fix kallistobustools/ref tests * Update test_10x_1_fastq_gz file for kallistobustools/count tests * Fix bcftools/query tests * Fix delly/call tests * Fix cooler/zoomify tests * Fix csvtk/split tests * Fix gatk4/intervallisttools tests * Fix gatk4/variantfiltration * Fix pydamage/filter tests * Fix test data for unicycler * Fix gstama/collapse module * Fix leehom tests * Fix metaphlan3 tests * Fix pairtools/select tests * Update nextflow.config * Update nextflow.config * feat: update syntax * Fix arriba tests * Fix more failing tests * Update test syntax * Remove comments from tests nextflow.config * Apply suggestions from code review * Fix kallistobustools/count module * Update dumpsoftwareversions module * Update custom/dumpsoftwareversions * Add args2 to untar module * Update leftover modules * Remove last remaining addParams Co-authored-by: JoseEspinosa <kadomu@gmail.com> Co-authored-by: Gregor Sturm <mail@gregor-sturm.de> Co-authored-by: MaxUlysse <max.u.garcia@gmail.com>
2021-11-26 07:58:40 +00:00
name: logs-${{ matrix.profile }}
2021-01-24 05:39:51 +00:00
path: |
/home/runner/pytest_workflow_*/*/.nextflow.log
/home/runner/pytest_workflow_*/*/log.out
/home/runner/pytest_workflow_*/*/log.err
/home/runner/pytest_workflow_*/*/work
!/home/runner/pytest_workflow_*/*/work/conda
!/home/runner/pytest_workflow_*/*/work/singularity