DEVEX-2578 dxCompiler Option for Parallel Compilation of Applets [by Fulcrum Genomics] by r-i-v-a · Pull Request #510 · dnanexus/dxCompiler

r-i-v-a · 2025-06-03T16:06:09Z

DEVEX-2579 For testing Fulcrum Genomics PR

… no longer exists on AWS; should be fixed in pricing model.

…/dxCompiler into for_testing_fulcrum_genomics_pr

Co-authored-by: Ted Brookings <ted@fulcrumgenomics.com>

) * Fix formatting * Revert rethrowing error * Fix bad type hint --------- Co-authored-by: Ted Brookings <ted@fulcrumgenomics.com>

…m_genomics_pr

--------- Co-authored-by: Ted Brookings <ted@fulcrumgenomics.com>

* Fix typo in test invocation of -executableCreationParallelism * Maybe fix applet with dockerRegistry test failing due to change in order of independent tasks --------- Co-authored-by: Ted Brookings <ted@fulcrumgenomics.com>

…order is reversed (#519) Co-authored-by: Ted Brookings <ted@fulcrumgenomics.com>

r-i-v-a · 2025-06-17T17:31:47Z

scripts/clean_build.sh

+set -euxo pipefail

-TEST=${1}
+TEST=${1:-""}


revert script

r-i-v-a · 2025-06-17T17:32:45Z

Add documentation about -executableCreationParallelism <int>

r-i-v-a · 2025-06-17T17:32:56Z

Add changelog note

…/dxCompiler into for_testing_fulcrum_genomics_pr

YuxinShi0423 · 2025-06-25T04:04:13Z

compiler/src/test/scala/dx/compiler/CompilerTest.scala

+    if (desc1.access == expected2Access && desc2.access == expected1Access) {
+      fail("Compiled applet order is reversed")
+    }
+
+    desc1.access shouldBe expected1Access
+
+    desc2.access shouldBe expected2Access
+


I dont get it why the access order could be wrong. The parallelized compilation is done on callable, and the network access is embedded in the callable object, so it would go with the callable and should not be affected by this new setting. @r-i-v-a

@YuxinShi0423 I think the test is just asserting that the new setting doesn't change the order; this was mentioned in the spec attached to the PMUX ticket.

I had introduced a bug that did reverse the order, but the resulting error message was unclear (it looked like the results were just wrong). I made the message for reversed order more clear and fixed the bug.

@TedBrookings could you point me to the commit for that fix?

d56c1cb Merge branch 'for_testing_fulcrum_genomics_pr' of github.com:dnanexus/dxCompiler into for_testing_fulcrum_genomics_pr
The changes were to Compiler.scala and ExecutableTree.scala. That said, this was fixing a bug that I had introduced earlier. I had been using Map keys instead of Vectors to track dependencies, so the order of mutually-independent executables had not been preserved correctly.

Thanks! So the access config is just the flag we used to check if applet1 and 2 are in correct order, but itself wont not be swapped.
I would still prefer another unit test since this one serves two purposes now. But I am okay with it. @r-i-v-a

YuxinShi0423

I got no more question. Thanks! LGTM

r-i-v-a added 3 commits June 3, 2025 17:40

fix typo aactions in GHA

26494a0

remove deprecated integration workflow

4a9fd5b

DEVEX-2579 For testing Fulcrum Genomics PR (#509)

66b3c3e

r-i-v-a added the integration2 Integration tests with dxcint label Jun 3, 2025

r-i-v-a mentioned this pull request Jun 3, 2025

DEVEX-2578 dxCompiler Option for Parallel Compilation of Applets #504

Closed

r-i-v-a and others added 8 commits June 4, 2025 00:42

revert task GPUSpec back to original form

5b5f9bf

temporarily ignore test that stalls waiting for an instance type that…

d4bab6f

… no longer exists on AWS; should be fixed in pricing model.

Merge branch 'for_testing_fulcrum_genomics_pr' of github.com:dnanexus…

c8888f6

…/dxCompiler into for_testing_fulcrum_genomics_pr

edit expected output with ignored test condition

d4509b8

edit instance types test expected output in 1 other place

675fc00

DEVEX-2579 For testing Fulcrum Genomics PR (#511)

826a7cc

Fix formatting --> test branch (#512)

28acb7e

Co-authored-by: Ted Brookings <ted@fulcrumgenomics.com>

Revert change to exception handling, fix type hint --> test branch (#513

9450dd6

) * Fix formatting * Revert rethrowing error * Fix bad type hint --------- Co-authored-by: Ted Brookings <ted@fulcrumgenomics.com>

r-i-v-a removed the integration2 Integration tests with dxcint label Jun 10, 2025

r-i-v-a and others added 2 commits June 10, 2025 14:20

Merge remote-tracking branch 'origin/develop' into for_testing_fulcru…

7bfaf57

…m_genomics_pr

Another unit test fix --> test branch (#514)

5fcd1d1

--------- Co-authored-by: Ted Brookings <ted@fulcrumgenomics.com>

r-i-v-a added cwl_conformance Triggers CWL conformance tools and workflows tests and removed cwl_conformance Triggers CWL conformance tools and workflows tests labels Jun 10, 2025

Fix in ConfigFactory.load re: unit test --> test branch (#515)

c308d51

--------- Co-authored-by: Ted Brookings <ted@fulcrumgenomics.com>

r-i-v-a requested a review from aogorodnikov-dna June 11, 2025 10:43

r-i-v-a and others added 2 commits June 16, 2025 17:14

For testing fulcrum genomics pr (#518)

ef2091c

* Fix typo in test invocation of -executableCreationParallelism * Maybe fix applet with dockerRegistry test failing due to change in order of independent tasks --------- Co-authored-by: Ted Brookings <ted@fulcrumgenomics.com>

Stabilize compile order, increase test verbosity to check if compile …

bfd5ca0

…order is reversed (#519) Co-authored-by: Ted Brookings <ted@fulcrumgenomics.com>

r-i-v-a requested review from YuxinShi0423 and mhrvol June 17, 2025 17:23

r-i-v-a commented Jun 17, 2025

View reviewed changes

scripts/clean_build.sh

set -euxo pipefail

TEST=${1}

TEST=${1:-""}

Copy link
Copy Markdown

Member Author

r-i-v-a Jun 17, 2025 •

edited

Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

revert script

r-i-v-a added 3 commits June 17, 2025 20:53

update documentation

0aec05f

add documentation

3929bab

changelog note

7f7d847

r-i-v-a added 3 commits June 17, 2025 20:59

Merge branch 'for_testing_fulcrum_genomics_pr' of github.com:dnanexus…

d56c1cb

…/dxCompiler into for_testing_fulcrum_genomics_pr

fix merge conflicts in documentation

5bb8dbd

revert project id in script

db296d8

r-i-v-a marked this pull request as ready for review June 19, 2025 21:54

r-i-v-a added integration2 Integration tests with dxcint cwl_conformance Triggers CWL conformance tools and workflows tests labels Jun 19, 2025

r-i-v-a changed the title ~~DEVEX-2579 For testing Fulcrum Genomics PR~~ DEVEX-2578 dxCompiler Option for Parallel Compilation of Applets [by Fulcrum Genomics] Jun 20, 2025

r-i-v-a removed cwl_conformance Triggers CWL conformance tools and workflows tests integration2 Integration tests with dxcint labels Jun 23, 2025

YuxinShi0423 reviewed Jun 25, 2025

View reviewed changes

r-i-v-a requested a review from YuxinShi0423 June 27, 2025 00:13

YuxinShi0423 approved these changes Jun 30, 2025

View reviewed changes

r-i-v-a merged commit 4881243 into develop Jun 30, 2025
18 of 20 checks passed

r-i-v-a deleted the for_testing_fulcrum_genomics_pr branch June 30, 2025 21:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DEVEX-2578 dxCompiler Option for Parallel Compilation of Applets [by Fulcrum Genomics]#510

DEVEX-2578 dxCompiler Option for Parallel Compilation of Applets [by Fulcrum Genomics]#510
r-i-v-a merged 22 commits intodevelopfrom
for_testing_fulcrum_genomics_pr

r-i-v-a commented Jun 3, 2025

Uh oh!

r-i-v-a Jun 17, 2025 •

edited

Loading

Uh oh!

r-i-v-a commented Jun 17, 2025 •

edited

Loading

Uh oh!

r-i-v-a commented Jun 17, 2025 •

edited

Loading

Uh oh!

YuxinShi0423 Jun 25, 2025 •

edited

Loading

Uh oh!

r-i-v-a Jun 27, 2025

Uh oh!

TedBrookings Jun 30, 2025

Uh oh!

YuxinShi0423 Jun 30, 2025

Uh oh!

TedBrookings Jun 30, 2025

Uh oh!

YuxinShi0423 Jun 30, 2025

Uh oh!

YuxinShi0423 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

r-i-v-a commented Jun 3, 2025

Uh oh!

r-i-v-a Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

r-i-v-a commented Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

r-i-v-a commented Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

YuxinShi0423 Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

r-i-v-a Jun 27, 2025

Choose a reason for hiding this comment

Uh oh!

TedBrookings Jun 30, 2025

Choose a reason for hiding this comment

Uh oh!

YuxinShi0423 Jun 30, 2025

Choose a reason for hiding this comment

Uh oh!

TedBrookings Jun 30, 2025

Choose a reason for hiding this comment

Uh oh!

YuxinShi0423 Jun 30, 2025

Choose a reason for hiding this comment

Uh oh!

YuxinShi0423 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

r-i-v-a Jun 17, 2025 •

edited

Loading

r-i-v-a commented Jun 17, 2025 •

edited

Loading

r-i-v-a commented Jun 17, 2025 •

edited

Loading

YuxinShi0423 Jun 25, 2025 •

edited

Loading