Skip to content

Remote nextflow hooks #54

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
SkepticRaven merged 21 commits into from
Aug 26, 2025
Merged

Remote nextflow hooks #54

SkepticRaven merged 21 commits into from
Aug 26, 2025

Conversation

@SkepticRaven
Copy link
Contributor

@SkepticRaven SkepticRaven commented Jul 23, 2025

Adds:

  • Filtering of inputs to skip already processed data
  • rclone cloud remote hook (dropbox)
  • globus endpoint remote hook (JAX T0/T1/T2)
  • Exposes these new options to all workflow inputs
  • Some nf-tests for new remote processes

SkepticRaven added 17 commits July 8, 2025 08:58
@SkepticRaven
Changing rclone commands to properly use nextflow config vars
87a8b9c
@SkepticRaven
Adding globus-specific modules
3816540
@SkepticRaven
Adding filters for files
24a04c2 
Some syntax correction
@SkepticRaven
Starting the remote workflow logic changes.
6a85365
@SkepticRaven
Adjusting var names to read more clearly
ee58960
@SkepticRaven
Multimouse workflow used wrong property
a5da72d
@SkepticRaven
Reworking filtering structure and prepare data workflows to be more u…
5fc673d 
…niform.

Also attempting to restructure all to operate on file lists for retrieval.
@SkepticRaven
First version of failing test, but can get nf-tests running
42e522a
@SkepticRaven
Getting dropbox retrieval operating.
3e3f8d1
@SkepticRaven
Adjusting other rclone-based funcs to use adjusted variable
389ac85
@SkepticRaven
Restricting rclone to 1 transfer at a time
24a7ed5
@SkepticRaven
Adding globus remote test
510e74a
@SkepticRaven
Updating resource paths due to failing hard drive. Also making them s…
2841f69 
…impler to update in the future.
@SkepticRaven
Adding proper config to test globus transfers
a8fcd62
@SkepticRaven
Getting the globus test to actually ping globus. Currently goes into …
531049e 
…an infinite wait loop if files don't exist.
@SkepticRaven
Adding a catch for infinite task wait loop
d5582b8
@SkepticRaven
Adding an invalid file test for globus
a4e095a
@SkepticRaven SkepticRaven requested a review from bergsalex July 23, 2025 14:35
bergsalex
bergsalex approved these changes Jul 23, 2025
View reviewed changes
nextflow/tests/remote/globus_test.nf.test
Copy link
Collaborator

@bergsalex bergsalex Jul 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎉 for nextflow tests!

@SkepticRaven SkepticRaven merged commit e2ff6ef into main Aug 26, 2025
3 checks passed
@SkepticRaven SkepticRaven deleted the remote-nextflow-hooks branch August 26, 2025 19:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bergsalex bergsalex bergsalex approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@SkepticRaven @bergsalex