nf-core_modules/.github/workflows/pytest-workflow.yml

113 lines
3.2 KiB
YAML
Raw Normal View History

2021-01-22 19:32:54 +00:00
name: Pytest-workflow
on:
push:
branches: [master]
pull_request:
branches: [master]
2021-01-22 19:32:54 +00:00
jobs:
changes:
name: Check for changes
runs-on: ubuntu-latest
Subworkflow Infrastructure (#662) * feat(subworkflows): Add align_bowtie2 subworkflow For testing CI setup * test(align_bowtie2): Add initial list of changes to test * test(align_bowtie2): Add initial test * refactor: Use tags to run subworkflows ci For every underlying module used by workflow and allow the modules pytest-modules definition be the source of truth. * refactor: Use individual directories for subworkflows * docs(align_bowtie2): Add initial meta.yml Copied most of it from the bowtie2/align module. * fix(align_bowtie2): Fix module include paths * test(bam_sort_samtools): Add initial test * ci(bam_sort_samtools): Add modules that trigger the tag * test(bam_stats_samtools): Add initial test * ci(bam_stats_samtools): Add keys to pick up changes * docs(bam_samtools): Add initial meta.yml * test(align_bowtie2): Fix path to subworkflow * test(align_bowtie2): Update entry point * fix(bam_sort_samtools): Update include paths * test(bam_sort_samtools): Fix path * style: Clean up addParams * test(samtools_sort): Add suffix for test * test(align_bowtie2): Add samtools_options for suffix * test(bam_stats_samtools): Update path * test(bam_stats_samtools): Use stats input Otherwise it's just an example of how it's used in the bam_sort_samtools subworkflow * ci(linting): Skip module linting of subworkflows * ci(linting): Clean up startsWith statement * test(bam_stats_samtools): Use single end test data for single end test * test(bam_stats_samtools): Add expected files * test(align_bowtie2): Add paired-end test * test(align_bowtie2): Sort order of output * test(align_bowtie2): Update hashes * docs(align_bowtie2): Fix typo * test(align_bowtie2): Update samtools output names * test(align_bowtie2): Remove md5sums for bam/bai * feat(subworkflows): Add nextflow.configs These can be used for default settings in the future. They can then be included in the conf/modules.config so that the params don't have to be duplicated in the root nextflow.config. * docs(subworkflows): Include modules instead of tools * fix: Update to versions * chore(align_bowtie2): Remove duplicate tag * style: Format yamls * test(subworkflows): Only check versions for modules * chore: Update subworkflows to match rnaseq dev * fix(subworkflows): Update paths * fix(bam_sort_samtools): Fix sort parameters for testing * Apply suggestions from code review Co-authored-by: Harshil Patel <drpatelh@users.noreply.github.com> * docs: Update TODOs with a message * ci: Try using a matrix for strategy * ci: Try passing an array * Revert "ci: Try passing an array" This reverts commit d3611fcd8332bbb9a8501e8dd299d0a623aaecaa. Co-authored-by: Harshil Patel <drpatelh@users.noreply.github.com>
2021-10-08 15:02:42 +00:00
strategy:
fail-fast: false
matrix:
filter:
["tests/config/pytest_modules.yml", "tests/config/pytest_subworkflows.yml"]
outputs:
# Expose matched filters as job 'modules' output variable
modules: ${{ steps.filter.outputs.changes }}
steps:
- uses: actions/checkout@v2
- uses: dorny/paths-filter@v2
id: filter
with:
Subworkflow Infrastructure (#662) * feat(subworkflows): Add align_bowtie2 subworkflow For testing CI setup * test(align_bowtie2): Add initial list of changes to test * test(align_bowtie2): Add initial test * refactor: Use tags to run subworkflows ci For every underlying module used by workflow and allow the modules pytest-modules definition be the source of truth. * refactor: Use individual directories for subworkflows * docs(align_bowtie2): Add initial meta.yml Copied most of it from the bowtie2/align module. * fix(align_bowtie2): Fix module include paths * test(bam_sort_samtools): Add initial test * ci(bam_sort_samtools): Add modules that trigger the tag * test(bam_stats_samtools): Add initial test * ci(bam_stats_samtools): Add keys to pick up changes * docs(bam_samtools): Add initial meta.yml * test(align_bowtie2): Fix path to subworkflow * test(align_bowtie2): Update entry point * fix(bam_sort_samtools): Update include paths * test(bam_sort_samtools): Fix path * style: Clean up addParams * test(samtools_sort): Add suffix for test * test(align_bowtie2): Add samtools_options for suffix * test(bam_stats_samtools): Update path * test(bam_stats_samtools): Use stats input Otherwise it's just an example of how it's used in the bam_sort_samtools subworkflow * ci(linting): Skip module linting of subworkflows * ci(linting): Clean up startsWith statement * test(bam_stats_samtools): Use single end test data for single end test * test(bam_stats_samtools): Add expected files * test(align_bowtie2): Add paired-end test * test(align_bowtie2): Sort order of output * test(align_bowtie2): Update hashes * docs(align_bowtie2): Fix typo * test(align_bowtie2): Update samtools output names * test(align_bowtie2): Remove md5sums for bam/bai * feat(subworkflows): Add nextflow.configs These can be used for default settings in the future. They can then be included in the conf/modules.config so that the params don't have to be duplicated in the root nextflow.config. * docs(subworkflows): Include modules instead of tools * fix: Update to versions * chore(align_bowtie2): Remove duplicate tag * style: Format yamls * test(subworkflows): Only check versions for modules * chore: Update subworkflows to match rnaseq dev * fix(subworkflows): Update paths * fix(bam_sort_samtools): Fix sort parameters for testing * Apply suggestions from code review Co-authored-by: Harshil Patel <drpatelh@users.noreply.github.com> * docs: Update TODOs with a message * ci: Try using a matrix for strategy * ci: Try passing an array * Revert "ci: Try passing an array" This reverts commit d3611fcd8332bbb9a8501e8dd299d0a623aaecaa. Co-authored-by: Harshil Patel <drpatelh@users.noreply.github.com>
2021-10-08 15:02:42 +00:00
filters: ${{ matrix.filter }}
2021-01-24 05:39:32 +00:00
test:
2021-02-02 22:26:24 +00:00
runs-on: ubuntu-20.04
2021-02-02 22:46:56 +00:00
name: ${{ matrix.tags }} ${{ matrix.profile }} ${{ matrix.nxf_version }}
needs: changes
if: needs.changes.outputs.modules != '[]'
2021-01-22 19:32:54 +00:00
strategy:
2021-02-02 23:08:10 +00:00
fail-fast: false
2021-01-22 19:32:54 +00:00
matrix:
nxf_version: ["21.04.0"]
tags: ["${{ fromJson(needs.changes.outputs.modules) }}"]
profile: ["docker", "singularity", "conda"]
2021-01-22 19:32:54 +00:00
env:
NXF_ANSI_LOG: false
steps:
- uses: actions/checkout@v2
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: "3.x"
2021-01-24 17:25:13 +00:00
- uses: actions/cache@v2
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
restore-keys: |
${{ runner.os }}-pip-
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: "3.x"
- name: Install Python dependencies
run: python -m pip install --upgrade pip pytest-workflow
2021-01-24 17:35:24 +00:00
- uses: actions/cache@v2
with:
path: /usr/local/bin/nextflow
key: ${{ runner.os }}-nextflow-${{ matrix.nxf_version }}
restore-keys: |
${{ runner.os }}-nextflow-
- name: Install Nextflow
env:
NXF_VER: ${{ matrix.nxf_version }}
CAPSULE_LOG: none
run: |
wget -qO- get.nextflow.io | bash
sudo mv nextflow /usr/local/bin/
- name: Set up Singularity
if: matrix.profile == 'singularity'
uses: eWaterCycle/setup-singularity@v5
with:
singularity-version: 3.7.1
2021-02-02 23:13:35 +00:00
- name: Setup miniconda
if: matrix.profile == 'conda'
uses: conda-incubator/setup-miniconda@v2
with:
auto-update-conda: true
2021-02-03 12:07:39 +00:00
channels: conda-forge,bioconda,defaults
python-version: ${{ matrix.python-version }}
2021-02-03 12:57:55 +00:00
- name: Conda clean
if: matrix.profile == 'conda'
run: conda clean -a
# Test the module
- name: Run pytest-workflow
# only use one thread for pytest-workflow to avoid race condition on conda cache.
run: NF_CORE_MODULES_TEST=1 TMPDIR=~ PROFILE=${{ matrix.profile }} pytest --tag ${{ matrix.tags }} --symlink --kwdof
2021-01-24 05:39:51 +00:00
- name: Upload logs on failure
if: failure()
2021-01-24 05:39:51 +00:00
uses: actions/upload-artifact@v2
with:
Homer Modules (#75) * feat(homer): Add initial makeTagDirectory * feat(homer): Add initial findPeaks module * feat(homer): Update with new options See 1d30e2c21affedc742680e8e04d60c6481d9cd11 * fix(homer): Correct findpeaks process name * fix(homer): Takes a bam file instead of bed * feat(homer): Add initial makeTagDirectory test * fix(homer): Hardcode genome and configureHomer I'd like to modularize configureHomer, but I need to figure out how exactly the genomes work. * fix(homer): bam => bed Bam requires samtools to be present, which it's not in this docker image * feat(homer): Add initial configureHomer script * ci(homer): Add initial test * test(homer): Reproducible configuration workaround - I can't run both tests(one file and two files) at the same time because it breaks - I can't copy the genome stuff from the configurehomer module because it's read only - So I can't make the makeTagDirectory module depend on configureHomer * test(homer): Add placeholder annotatepeaks The required inputs are necessarily required for all workflows from what I've used, but I'll need to look at the actual docs * test(homer): Add missing B.bed * test(homer): Rename two => groseq Then all of the various workflows that homer provides can be e2e tested * feat(homer): Add initial makeUCSCfile module * test(homer): Add start to makeUCSCfile testing * chore(homer): Add various cleanups * test(homer): Rewrite annotatepeaks Not passing yet * test(homer): Rewrite configurehomer * test(homer): Rewrite findpeaks Still failing * test(homer): Rewrite makeucscfile Not passing yet * test(homer): Rewrite maketagdirectory All homer modules now follow the new structure. Time to make them pass. * test(homer): Fix typo for workflow name * fix(homer): Use correct container * fix(homer): Accept fasta in maketagdirectory Apparently all of the homer stuff can just take any old fasta and you don't need to configure the genome ahead of time with configureHomer * test(homer): makeTagDirectory passes now * fix(homer): Update containers in makeucscfile * test(homer): Rewrite makeucscfile Takes input from maketagdirectory which is how the module should be used * fix(homer): Update makeUCSCFile bedgraph path * test(homer): Update makeucscfile expected output * fix(homer): Update containers in findpeaks * fix(homer): Change findpeaks args The user is just going to have to know what they're doing for now * test(homer): findPeaks rewrite with tagDir input * test(homer): Update expected files for findPeaks And bump filters * style: Appease editorconfig * ci: Remove old workflow * tests(homer): Add md5sums * test(homer): Add meta test * style(homer): Capitalize HOMER * docs(homer): Add maketagdirectory meta.yml * docs(homer): Add makeucscfile meta.yml * docs(homer): Add findpeaks meta.yml * test(homer): Update to new test data standards * chore: Remove stuff that got revived in the rebase * chore: software => modules * test(homer): Update tags * test(homer): Update annotatepeaks * ci: Fix uploading of artifacts GitHub actions doesn't like the / in the tags * test(homer): Remove annotate md5sum This is failing and breaking new tests * test(homer): Use bams instead of beds * test(homer): Fix meta maketagdirectory * test(homer): Fix input in all tests * test(homer): Move back to bed files Forgot samtools isn't present * chore(homer): Add TODOs for tests * test(homer): Add bed format arg * test(homer): Update md5sums * test(homer): Fix tags tsvs * style(homer): Appease nf-core linting * docs(homer): Be in line with what is in the main.nf file Co-authored-by: Kevin Menden <kevin.menden@live.com> Co-authored-by: Kevin Menden <kevin.menden@live.com>
2021-09-08 15:40:34 +00:00
name: logs-${{ matrix.profile }}-${{ matrix.nxf_version }}
2021-01-24 05:39:51 +00:00
path: |
/home/runner/pytest_workflow_*/*/.nextflow.log
/home/runner/pytest_workflow_*/*/log.out
/home/runner/pytest_workflow_*/*/log.err
/home/runner/pytest_workflow_*/*/work