Dependency graph instead of tree

[antoine-morvan]

Hello,

Currently, it seems to me that the dependencies we can express in the JUBE input can only be a tree. For instance, my input could look like this :

name: testjube
outpath: testjube_bench
comment: Test JUBE

#Configuration
parameterset:
  - name: build_parameters
    parameter: 
    - {name: COMPILER,    _: "gcc:13.2.0,llvm:17.0.6"}
  - name: run_parameters
    parameter: 
    - {name: FREQ,            _: "NOMINAL,TURBO"}

#Operation
step:
  - name: prepare
    do: |
      echo "do the prepare"
  - name: build
    depend: prepare
    use: build_parameters
    do: |
      echo "do the build"
  - name: setup
    depend: build
    use: build_parameters, run_parameters
    do: |
      echo "do the build"
  - name: run
    depend: setup,build
    use: build_parameters, run_parameters
    do: |
      echo "do the build"

For the input above, the dependency of the steps would look like this :

graph TD;
    prepare-->build_gcc:13.2.0;
    build_gcc:13.2.0-->setup_gcc:13.2.0_NOMINAL;
    build_gcc:13.2.0-->setup_gcc:13.2.0_TURBO;
    setup_gcc:13.2.0_TURBO-->run_gcc:13.2.0_TURBO;
    setup_gcc:13.2.0_NOMINAL-->run_gcc:13.2.0_NOMINAL

    prepare-->build_llvm:17.0.6;
    build_llvm:17.0.6-->setup_llvm:17.0.6_NOMINAL;
    build_llvm:17.0.6-->setup_llvm:17.0.6_TURBO;
    setup_llvm:17.0.6_TURBO-->run_llvm:17.0.6_TURBO;
    setup_llvm:17.0.6_NOMINAL-->run_llvm:17.0.6_NOMINAL

Loading

However, my setup step is a bit costly in terms of compute resources, but not meaningful in terms of results (analysis): it simply produces some intermediate input from built binaries. Actually, I want to run it once only, using whatever of the build and run parameter that are specified. The dependencies I would like to express would look like this :

graph TD;
    prepare-->build_gcc:13.2.0;
    build_gcc:13.2.0-->setup_gcc:13.2.0_NOMINAL;
    setup_gcc:13.2.0_NOMINAL-->run_gcc:13.2.0_TURBO;
    setup_gcc:13.2.0_NOMINAL-->run_gcc:13.2.0_NOMINAL

    prepare-->build_llvm:17.0.6;
    build_llvm:17.0.6-->run_llvm:17.0.6_TURBO;
    build_llvm:17.0.6-->run_llvm:17.0.6_NOMINAL;
    setup_gcc:13.2.0_NOMINAL-->run_llvm:17.0.6_TURBO;
    setup_gcc:13.2.0_NOMINAL-->run_llvm:17.0.6_NOMINAL

Loading

Note that the setup is executed once for all the run steps. Because of this, some nodes have 2 parents, thus moving from tree to graph structure. I would like something like this :

  - name: setup
    depend: build
    use: build_parameters[0], run_parameters[0] # use the parameter 0 from possible values
    do: |
      echo "do the build"
  - name: run
    depend: setup,build # since there is only one instance of setup, they all depend on the same setup step; might be necessary to add support for step identification in more complex cases...
    use: build_parameters, run_parameters
    do: |
      echo "do the build"

Is there a way to specify things using JUBE ?

Best.

[thobreuer]

• Hint: since the setup step depend on the build step, the parametersets used in the build will automatically be available in the setup step without using them explicitly (see my example below).
• What you are probably looking for is a shared step with a shared do as a solution for the setup step. JUBE will still create the four directories (as in your first graph), but the shared do will only be executed once in the shared directory define in the step shared attribute for all four workpackages of the setup step. See also our adv. tutorial section about shared-operations: https://apps.fz-juelich.de/jsc/jube/jube2/docu/advanced.html#shared-operations

name: testjube
outpath: testjube_bench
comment: TEST

#Configuration
parameterset:
  - name: build_parameters
    parameter:
    - {name: COMPILER,    _: "gcc:13.2.0,llvm:17.0.6"}
  - name: run_parameters
    parameter:
    - {name: FREQ,            _: "NOMINAL,TURBO"}

#Operation
step:
  - name: prepare
    do: |
      echo "do the prepare"
  - name: build
    depend: prepare
    use: build_parameters
    do: |
      echo "do the build: $COMPILER"
  - name: setup
    depend: build
    use: run_parameters
    shared: share
    do:
      - 'echo "do the setup: $COMPILER $FREQ"'
      - {shared: true, _: 'echo "IN SHARED"'}
  - name: run
    depend: setup
    do: |
      echo "do the run: $COMPILER $FREQ"

[antoine-morvan]

Thanks for pointing this ; I'll play a bit more, but it seems to be doing what I need.

In the few tries I attempted, I used - {shared: true, _: 'echo "IN SHARED $COMPILER $FREQ"'} to check which parameters are in use. If I run with

    do:
      - 'echo "do the setup: $COMPILER $FREQ"'
      - {shared: true, _: 'echo "IN SHARED  $COMPILER $FREQ"'}

The result is IN SHARED llvm:17.0.6 TURBO. But if I use

    do:
      - {shared: true, _: 'echo "IN SHARED  $COMPILER $FREQ"'}

The result is IN SHARED gcc:13.2.0 NOMINAL

Could you explain how the instance of parameter is chosen ?

On another note, it seems that the documentation refers to Operation and Step as the same thing. I was searching for "step" and "dependency" (as shared dependency), but not "operation". It's neither defined in the JUBE tutorial, nor in the avanced one. Is there any page explaining the concepts (some structure documentation, not a tutorial) ?

Best.

[thobreuer]

• It looks as if the last parameter combination is active in the first case and the first combination in the second case (1: gcc:13.2.0 NOMINAL, 2: gcc:13.2.0 TURBO, 3: llvm:17.0.6 NOMINAL, 4: llvm:17.0.6 TURBO). If I see this correctly, then we have not documented this. Unfortunately, I can't explain it right now either. We will take a closer look at this.
• Operation and step describe different features. Operation refers to the do-tags at this point. I assume that this was described in the documentation because the do feature is called operation in the JUBE code. We will improve the documentation in this respect.
• Our "documentation" currently consists of the following parts (This structure has grown historically. However, we are considering restructuring it. Suggestions for improvement are welcome.):
  • Tutorial: https://apps.fz-juelich.de/jsc/jube/jube2/docu/tutorial.html
  • Adv tutorial: https://apps.fz-juelich.de/jsc/jube/jube2/docu/advanced.html
  • FAQ: https://apps.fz-juelich.de/jsc/jube/jube2/docu/faq.html
  • Glossar (and command line jube help): https://apps.fz-juelich.de/jsc/jube/jube2/docu/glossar.html