Skip to content

Commit e826c0b

Browse files
pinin4fjordsclaude
andcommitted
Resolve nf-core template merge conflicts for v3.4.1
Merged dev branch into nf-core-template-merge-3.4.1, resolving conflicts by keeping pipeline-specific features while adopting template improvements. ## Conflict resolutions: ### Infrastructure improvements (adopted from template): - Update nf-test to v0.9.3 - Update minimum Nextflow version to 25.04.0 - Add new help system parameters (--help, --help_full, --show_hidden) - Update utility subworkflow git_sha values - Adopt `removeNextflowVersion` function for test snapshots - Use vars.AWS_S3_BUCKET instead of secrets for non-sensitive config ### Pipeline-specific features (retained): - Keep aligner matrix testing (star_salmon, star_rsem) in AWS full tests - Retain Sentieon authentication environment variables - Keep commit-pinned test data URL for reproducibility - Preserve pipeline metadata and real Zenodo DOI - Maintain existing CHANGELOG format ### Merged additions: - Combined test ignore patterns (both pipeline-specific and template) - Added both fastqc pattern variants for compatibility All changes follow the principle of preserving pipeline functionality while adopting template infrastructure improvements for better maintainability. Follows merge patterns from nf-core/sarek#2031 and guidance from https://nf-co.re/docs/tutorials/sync/merging_automated_prs 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
2 parents 2732654 + c3607ed commit e826c0b

File tree

622 files changed

+112837
-451
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

622 files changed

+112837
-451
lines changed

.github/actions/nf-test/action.yml

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,10 +56,20 @@ runs:
5656
channel-priority: strict
5757
conda-remove-defaults: true
5858

59+
# Set up secrets
60+
- name: Set up Nextflow secrets
61+
if: env.SENTIEON_ENCRYPTION_KEY != '' && env.SENTIEON_LICENSE_MESSAGE != ''
62+
shell: bash
63+
run: |
64+
python -m pip install cryptography
65+
nextflow secrets set SENTIEON_AUTH_DATA $(python3 .github/actions/nf-test/license_message.py encrypt --key "$SENTIEON_ENCRYPTION_KEY" --message "$SENTIEON_LICENSE_MESSAGE")
66+
5967
- name: Run nf-test
6068
shell: bash
6169
env:
6270
NFT_WORKDIR: ${{ env.NFT_WORKDIR }}
71+
SENTIEON_LICSRVR_IP: ${{ env.SENTIEON_LICSRVR_IP }}
72+
SENTIEON_AUTH_MECH: "GitHub Actions - token"
6373
run: |
6474
nf-test test \
6575
--profile=+${{ inputs.profile }} \
Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
#!/usr/bin/env python3
2+
3+
#########################################
4+
# Author: [DonFreed](https://github.com/DonFreed)
5+
# File: license_message.py
6+
# Source: https://github.com/DonFreed/docker-actions-test/blob/main/.github/scripts/license_message.py
7+
# Source+commit: https://github.com/DonFreed/docker-actions-test/blob/aa1051a9f53b3a1e801953748d062cad74dca9a9/.github/scripts/license_message.py
8+
# Download Date: 2023-07-04, commit: aa1051a
9+
# This source code is licensed under the BSD 2-Clause license
10+
#########################################
11+
12+
"""
13+
Functions for generating and sending license messages
14+
"""
15+
16+
# Modified from - https://stackoverflow.com/a/59835994
17+
18+
import argparse
19+
import base64
20+
import calendar
21+
import re
22+
import secrets
23+
import sys
24+
25+
from cryptography.hazmat.primitives.ciphers.aead import AESGCM
26+
from datetime import datetime as dt
27+
28+
MESSAGE_TIMEOUT = 60 * 60 * 24 # Messages are valid for 1 day
29+
NONCE_BYTES = 12
30+
31+
32+
class DecryptionTimeout(Exception):
33+
# Decrypting a message that is too old
34+
pass
35+
36+
37+
def generate_key():
38+
key = secrets.token_bytes(32)
39+
return key
40+
41+
42+
def handle_generate_key(args):
43+
key = generate_key()
44+
key_b64 = base64.b64encode(key)
45+
print(key_b64.decode("utf-8"), file=args.outfile)
46+
47+
48+
def encrypt_message(key, message):
49+
nonce = secrets.token_bytes(NONCE_BYTES)
50+
timestamp = calendar.timegm(dt.now().utctimetuple())
51+
data = timestamp.to_bytes(10, byteorder="big") + b"__" + message
52+
ciphertext = nonce + AESGCM(key).encrypt(nonce, data, b"")
53+
return ciphertext
54+
55+
56+
def handle_encrypt_message(args):
57+
key = base64.b64decode(args.key.encode("utf-8"))
58+
message = args.message.encode("utf-8")
59+
ciphertext = encrypt_message(key, message)
60+
ciphertext_b64 = base64.b64encode(ciphertext)
61+
print(ciphertext_b64.decode("utf-8"), file=args.outfile)
62+
63+
64+
def decrypt_message(key, ciphertext, timeout=MESSAGE_TIMEOUT):
65+
nonce, ciphertext = ciphertext[:NONCE_BYTES], ciphertext[NONCE_BYTES:]
66+
message = AESGCM(key).decrypt(nonce, ciphertext, b"")
67+
68+
msg_timestamp, message = re.split(b"__", message, maxsplit=1)
69+
msg_timestamp = int.from_bytes(msg_timestamp, byteorder="big")
70+
timestamp = calendar.timegm(dt.now().utctimetuple())
71+
if (timestamp - msg_timestamp) > timeout:
72+
raise DecryptionTimeout("The message has an expired timeout")
73+
return message.decode("utf-8")
74+
75+
76+
def handle_decrypt_message(args):
77+
key = base64.b64decode(args.key.encode("utf-8"))
78+
ciphertext = base64.b64decode(args.message.encode("utf-8"))
79+
message = decrypt_message(key, ciphertext, timeout=args.timeout)
80+
print(str(message), file=args.outfile)
81+
82+
83+
def parse_args(argv=None):
84+
parser = argparse.ArgumentParser(description=__doc__)
85+
parser.add_argument("--outfile", default=sys.stdout, type=argparse.FileType("w"), help="The output file")
86+
87+
subparsers = parser.add_subparsers(help="Available sub-commands")
88+
89+
gen_parser = subparsers.add_parser("generate_key", help="Generate a random key string")
90+
gen_parser.set_defaults(func=handle_generate_key)
91+
92+
encrypt_parser = subparsers.add_parser("encrypt", help="Encrypt a message")
93+
encrypt_parser.add_argument("--key", required=True, help="The encryption key")
94+
encrypt_parser.add_argument("--message", required=True, help="Message to encrypt")
95+
encrypt_parser.set_defaults(func=handle_encrypt_message)
96+
97+
decrypt_parser = subparsers.add_parser("decrypt", help="Decyrpt a message")
98+
decrypt_parser.add_argument("--key", required=True, help="The encryption key")
99+
decrypt_parser.add_argument("--message", required=True, help="Message to decrypt")
100+
decrypt_parser.add_argument(
101+
"--timeout",
102+
default=MESSAGE_TIMEOUT,
103+
type=int,
104+
help="A message timeout. Decryption will fail for older messages",
105+
)
106+
decrypt_parser.set_defaults(func=handle_decrypt_message)
107+
108+
return parser.parse_args(argv)
109+
110+
111+
if __name__ == "__main__":
112+
args = parse_args()
113+
args.func(args)

.github/include.yaml

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
".":
2+
- ./.github/workflows/**
3+
- ./nf-test.config
4+
- ./nextflow.config
5+
tests:
6+
- ./assets/*
7+
- ./bin/*
8+
- ./conf/*
9+
- ./main.nf
10+
- ./nextflow_schema.json

.github/workflows/awsfulltest.yml

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,9 @@ jobs:
1616
# run only if the PR is approved by at least 2 reviewers and against the master/main branch or manually triggered
1717
if: github.repository == 'nf-core/rnaseq' && github.event.review.state == 'approved' && (github.event.pull_request.base.ref == 'master' || github.event.pull_request.base.ref == 'main') || github.event_name == 'workflow_dispatch' || github.event_name == 'release'
1818
runs-on: ubuntu-latest
19+
strategy:
20+
matrix:
21+
aligner: ["star_salmon", "star_rsem"]
1922
steps:
2023
- name: Set revision variable
2124
id: revision
@@ -36,7 +39,8 @@ jobs:
3639
parameters: |
3740
{
3841
"hook_url": "${{ secrets.MEGATESTS_ALERTS_SLACK_HOOK_URL }}",
39-
"outdir": "s3://${{ vars.AWS_S3_BUCKET }}/rnaseq/results-${{ steps.revision.outputs.revision }}"
42+
"aligner": "${{ matrix.aligner }}",
43+
"outdir": "s3://${{ vars.AWS_S3_BUCKET }}/rnaseq/results-${{ steps.revision.outputs.revision }}/aligner_${{ matrix.aligner }}/"
4044
}
4145
profiles: test_full
4246

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
name: full-sized tests on cloud providers
2+
run-name: Submitting workflow to all cloud providers using full sized data
3+
on:
4+
workflow_dispatch:
5+
inputs:
6+
platform:
7+
description: "Platform to run test"
8+
required: true
9+
default: "all"
10+
type: choice
11+
options:
12+
- all
13+
- aws
14+
- azure
15+
- gcp
16+
jobs:
17+
run-full-tests-on-aws:
18+
if: ${{ github.event.inputs.platform == 'all' || github.event.inputs.platform == 'aws' || !github.event.inputs }}
19+
runs-on: ubuntu-latest
20+
strategy:
21+
matrix:
22+
aligner: ["star_salmon", "star_rsem"]
23+
steps:
24+
- uses: seqeralabs/action-tower-launch@v2
25+
with:
26+
workspace_id: ${{ secrets.TOWER_WORKSPACE_ID }}
27+
access_token: ${{ secrets.TOWER_ACCESS_TOKEN }}
28+
compute_env: ${{ secrets.TOWER_CE_AWS_CPU }}
29+
workdir: "${{ secrets.TOWER_BUCKET_AWS }}/work/rnaseq/work-${{ github.sha }}"
30+
run_name: "aws_rnaseq_full_${{ matrix.aligner }}"
31+
revision: ${{ github.sha }}
32+
profiles: test_full_aws
33+
parameters: |
34+
{
35+
"hook_url": "${{ secrets.MEGATESTS_ALERTS_SLACK_HOOK_URL }}",
36+
"aligner": "${{ matrix.aligner }}",
37+
"outdir": "${{ secrets.TOWER_BUCKET_AWS }}/rnaseq/results-${{ github.sha }}/aligner_${{ matrix.aligner }}/"
38+
}
39+
- uses: actions/upload-artifact@v3
40+
with:
41+
name: Tower debug log file
42+
path: tower_action_*.log
43+
44+
run-full-tests-on-azure:
45+
if: ${{ github.event.inputs.platform == 'all' || github.event.inputs.platform == 'azure' || !github.event.inputs }}
46+
runs-on: ubuntu-latest
47+
strategy:
48+
matrix:
49+
aligner: ["star_salmon", "star_rsem"]
50+
steps:
51+
- uses: seqeralabs/action-tower-launch@v2
52+
with:
53+
workspace_id: ${{ secrets.TOWER_WORKSPACE_ID }}
54+
access_token: ${{ secrets.TOWER_ACCESS_TOKEN }}
55+
compute_env: ${{ secrets.TOWER_CE_AZURE_CPU }}
56+
workdir: "${{ secrets.TOWER_BUCKET_AZURE }}/work/rnaseq/work-${{ github.sha }}"
57+
run_name: "azure_rnaseq_full_${{ matrix.aligner }}"
58+
revision: ${{ github.sha }}
59+
profiles: test_full_azure
60+
parameters: |
61+
{
62+
"hook_url": "${{ secrets.MEGATESTS_ALERTS_SLACK_HOOK_URL }}",
63+
"aligner": "${{ matrix.aligner }}",
64+
"outdir": "${{ secrets.TOWER_BUCKET_AZURE }}/rnaseq/results-${{ github.sha }}/aligner_${{ matrix.aligner }}/",
65+
"igenomes_base": "${{ secrets.TOWER_IGENOMES_BASE_AZURE }}"
66+
}
67+
- uses: actions/upload-artifact@v3
68+
with:
69+
name: Tower debug log file
70+
path: tower_action_*.log
71+
72+
run-full-tests-on-gcp:
73+
if: ${{ github.event.inputs.platform == 'gcp' || !github.event.inputs }}
74+
runs-on: ubuntu-latest
75+
strategy:
76+
matrix:
77+
aligner: ["star_salmon", "star_rsem"]
78+
steps:
79+
- uses: seqeralabs/action-tower-launch@v2
80+
with:
81+
workspace_id: ${{ secrets.TOWER_WORKSPACE_ID }}
82+
access_token: ${{ secrets.TOWER_ACCESS_TOKEN }}
83+
compute_env: ${{ secrets.TOWER_CE_GCP_CPU }}
84+
workdir: "${{ secrets.TOWER_BUCKET_GCP }}/work/rnaseq/work-${{ github.sha }}"
85+
run_name: "gcp_rnaseq_full_${{ matrix.aligner }}"
86+
revision: ${{ github.sha }}
87+
profiles: test_full_gcp
88+
parameters: |
89+
{
90+
"hook_url": "${{ secrets.MEGATESTS_ALERTS_SLACK_HOOK_URL }}",
91+
"aligner": "${{ matrix.aligner }}",
92+
"outdir": "${{ secrets.TOWER_BUCKET_GCP }}/rnaseq/results-${{ github.sha }}/aligner_${{ matrix.aligner }}/"
93+
}
94+
- uses: actions/upload-artifact@v3
95+
with:
96+
name: Tower debug log file
97+
path: tower_action_*.log
Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
name: small-sized tests on cloud providers
2+
run-name: Submitting workflow to all cloud providers using small sized data
3+
on:
4+
workflow_dispatch:
5+
inputs:
6+
platform:
7+
description: "Platform to run test"
8+
required: true
9+
default: "all"
10+
type: choice
11+
options:
12+
- all
13+
- aws
14+
- azure
15+
- gcp
16+
jobs:
17+
run-small-tests-on-aws:
18+
if: ${{ github.event.inputs.platform == 'all' || github.event.inputs.platform == 'aws' }}
19+
runs-on: ubuntu-latest
20+
steps:
21+
- uses: seqeralabs/action-tower-launch@v2
22+
with:
23+
workspace_id: ${{ secrets.TOWER_WORKSPACE_ID }}
24+
access_token: ${{ secrets.TOWER_ACCESS_TOKEN }}
25+
compute_env: ${{ secrets.TOWER_CE_AWS_CPU }}
26+
workdir: "${{ secrets.TOWER_BUCKET_AWS }}/work/rnaseq/work-${{ github.sha }}"
27+
run_name: "aws_rnaseq_small"
28+
revision: ${{ github.sha }}
29+
profiles: test
30+
parameters: |
31+
{
32+
"outdir": "${{ secrets.TOWER_BUCKET_AWS }}/rnaseq/results-test-${{ github.sha }}/"
33+
}
34+
- uses: actions/upload-artifact@v3
35+
with:
36+
name: Tower debug log file
37+
path: tower_action_*.log
38+
39+
run-small-tests-on-azure:
40+
if: ${{ github.event.inputs.platform == 'all' || github.event.inputs.platform == 'azure' }}
41+
runs-on: ubuntu-latest
42+
steps:
43+
- uses: seqeralabs/action-tower-launch@v2
44+
with:
45+
workspace_id: ${{ secrets.TOWER_WORKSPACE_ID }}
46+
access_token: ${{ secrets.TOWER_ACCESS_TOKEN }}
47+
compute_env: ${{ secrets.TOWER_CE_AZURE_CPU }}
48+
workdir: "${{ secrets.TOWER_BUCKET_AZURE }}/work/rnaseq/work-${{ github.sha }}"
49+
run_name: "azure_rnaseq_small"
50+
revision: ${{ github.sha }}
51+
profiles: test
52+
parameters: |
53+
{
54+
"outdir": "${{ secrets.TOWER_BUCKET_AZURE }}/rnaseq/results-test-${{ github.sha }}/"
55+
}
56+
- uses: actions/upload-artifact@v3
57+
with:
58+
name: Tower debug log file
59+
path: tower_action_*.log
60+
61+
run-small-tests-on-gcp:
62+
if: ${{ github.event.inputs.platform == 'gcp' }}
63+
runs-on: ubuntu-latest
64+
steps:
65+
- uses: seqeralabs/action-tower-launch@v2
66+
with:
67+
workspace_id: ${{ secrets.TOWER_WORKSPACE_ID }}
68+
access_token: ${{ secrets.TOWER_ACCESS_TOKEN }}
69+
compute_env: ${{ secrets.TOWER_CE_GCP_CPU }}
70+
workdir: "${{ secrets.TOWER_BUCKET_GCP }}/work/rnaseq/work-${{ github.sha }}"
71+
run_name: "gcp_rnaseq_small"
72+
revision: ${{ github.sha }}
73+
profiles: test
74+
parameters: |
75+
{
76+
"outdir": "${{ secrets.TOWER_BUCKET_GCP }}/rnaseq/results-test-${{ github.sha }}/"
77+
}
78+
- uses: actions/upload-artifact@v3
79+
with:
80+
name: Tower debug log file
81+
path: tower_action_*.log

.github/workflows/nf-test.yml

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,7 @@ jobs:
5050
env:
5151
NFT_VER: ${{ env.NFT_VER }}
5252
with:
53-
max_shards: 7
53+
max_shards: 14
5454

5555
- name: debug
5656
run: |
@@ -95,6 +95,10 @@ jobs:
9595
continue-on-error: ${{ matrix.NXF_VER == 'latest-everything' }}
9696
env:
9797
NFT_WORKDIR: ${{ env.NFT_WORKDIR }}
98+
SENTIEON_AUTH_MECH: "GitHub Actions - token"
99+
SENTIEON_ENCRYPTION_KEY: ${{ secrets.SENTIEON_ENCRYPTION_KEY }}
100+
SENTIEON_LICENSE_MESSAGE: ${{ secrets.SENTIEON_LICENSE_MESSAGE }}
101+
SENTIEON_LICSRVR_IP: ${{ secrets.SENTIEON_LICSRVR_IP }}
98102
NXF_VERSION: ${{ matrix.NXF_VER }}
99103
with:
100104
profile: ${{ matrix.profile }}

.gitignore

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,13 @@
1+
*.pyc
2+
.DS_Store
13
.nextflow*
2-
work/
4+
.nf-test.log
35
data/
6+
nf-test
7+
.nf-test*
48
results/
5-
.DS_Store
6-
testing/
9+
test.xml
710
testing*
8-
*.pyc
11+
testing/
12+
work/
913
null/

.nf-core.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ lint:
1313
nf_core_version: 3.4.1
1414
repository_type: pipeline
1515
template:
16-
author: "Harshil Patel, Phil Ewels, Rickard Hammar\xE9n"
16+
author: "Harshil Patel, Phil Ewels, Rickard Hammarén"
1717
description: RNA sequencing analysis pipeline for gene/isoform quantification and
1818
extensive quality control.
1919
force: false

0 commit comments

Comments
 (0)