Enable Initializing Second Experiment From Library

[michaelmckinsey1]

Description

• Refactor programming models to be defined like scaling options are. I noticed while testing that the scaling variants were not affected from the same bug as the programming models. I.e. this is an issue with importing the programming class multiple times, which results in weird behavior with directives. This change avoids this.
• Depends on Automatically Enumerate Dryruns and MpiOnlyExperiment #904
• Issue described in Cannot initialize second experiment from library #932
• This can be worked around by running the pytests individually, but this complicates the code coverage.
  • Example of error code checking on command line tests here Require --system During Experiment Init #982
    files#diff-a3ddd7238fc6e36daeb0ef76e93fe15cc87824bd3299888075210c5ca6a474d3
  • See removed tests in .github/workflows/run.yml -> moved to pytest now with this fix. Much cleaner tests in pytests.
• Refactor all experiments
• rename models.py to programming_model.py
• Update docs/tutorial

[michaelmckinsey1]

Ready for a review, but definitely not merging because only 2 of the experiments are updated for this change

[scheibelp]

I have a question and a request

[scheibelp]

It looks like these variants are always defined on every experiment, regardless of whether it inherits the associated model types.

It should be allowed to split these variants into separate modules and imported classes (as they were originally); at the same time, customizing it all from one file is ok too, I wonder if it's possible to address the above concern though (it might not be possible to conditionally define the variants).

[michaelmckinsey1]

This is true, regardless of which models are defined, if the experiment inherits from Models(...) all programming models will show in the spec +mpi~rocm~cuda~openmp. I would like to note that this is the same behavior for the Scaling, e.g. even if the experiment does not define Throughput scaling it will show ~throughput in the spec. Not that this is correct either, but if we can find a solution for this it should be applied to both the programming models and scaling.

I don't know of a solution to the underlying bug that would enable separate classes. Perhaps something like

# mpi.py
def Mpi():
  class BaseMPI:
       variant("mpi")

# rocm.py
def Rocm():
  class BaseRocm:
       variant("rocm")

would provide the correct behavior, but the design seems maybe a bit convoluted.

It seems like the relevant piece for the behavior we desire is making sure the variant is not applied when the class is imported.

[michaelmckinsey1]

This is something we missed when we refactored ramble and spack repos

[codecov-commenter]

Codecov Report

❌ Patch coverage is 84.33735% with 13 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.53%. Comparing base (ad7d87e) to head (b4e1bb7).
⚠️ Report is 1 commits behind head on develop.

Files with missing lines	Patch %	Lines
lib/benchpark/programming_model.py	76.59%	11 Missing ⚠️
lib/benchpark/experiment.py	94.11%	1 Missing ⚠️
lib/main.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop     #931      +/-   ##
===========================================
+ Coverage    62.69%   63.53%   +0.83%     
===========================================
  Files           48       49       +1     
  Lines         3664     3762      +98     
  Branches       285      322      +37     
===========================================
+ Hits          2297     2390      +93     
- Misses        1356     1361       +5     
  Partials        11       11              

Files with missing lines	Coverage Δ
experiments/babelstream/experiment.py	40.00% <100.00%> (ø)
experiments/kripke/experiment.py	27.27% <100.00%> (ø)
experiments/stream/experiment.py	50.00% <100.00%> (ø)
experiments/test/saxpy/experiment.py	43.47% <100.00%> (-6.53%)	⬇️
lib/benchpark/accounting.py	91.91% <ø> (ø)
lib/benchpark/cmd/info.py	23.80% <ø> (ø)
lib/benchpark/test/experiment_errors.py	100.00% <100.00%> (ø)
modifiers/caliper/modifier.py	86.36% <100.00%> (ø)
lib/benchpark/experiment.py	89.25% <94.11%> (+4.40%)	⬆️
lib/main.py	55.29% <0.00%> (ø)
... and 1 more

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[scheibelp]

@michaelmckinsey1

lib/benchpark/rocm.py‎

had

        def get_spack_variants(self):
            return (
                "+rocm amdgpu_target={rocm_arch}"
                if self.spec.satisfies("+rocm")
                else "~rocm"
            )

Where did this move to? I feel like it was dropped when looking through these changes

[michaelmckinsey1]

@michaelmckinsey1

lib/benchpark/rocm.py‎

had

        def get_spack_variants(self):
            return (
                "+rocm amdgpu_target={rocm_arch}"
                if self.spec.satisfies("+rocm")
                else "~rocm"
            )

Where did this move to? I feel like it was dropped when looking through these changes

It looks like it did get dropped