Question: Specifying a custom set of metrics to helm-run.

[Erotemic]

I'm interested in running a subset of the HEIM benchmarks, but I'm not interested in all of the metrics. Currently I'm running:

helm-run --run-entries mscoco:model=huggingface/stable-diffusion-v1-4 \
    --suite my-heim-suite \
    --max-eval-instances 10 --num-threads 1

I see in helm.benchmark.run_speces.heim_run_specs there is get_mscoco_spec which is registered as a run spec function and ultimately calls into get_core_heim_metric_specs to create the metric class instances. In this example the final constructed run spec contains these metrics:

metric_specs=[MetricSpec(class_name='helm.benchmark.metrics.image_generation.lpips_metrics.LearnedPerceptualImagePatchSimilarityMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.image_generation.multi_scale_ssim_metrics.MultiScaleStructuralSimilarityIndexMeasureMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.image_generation.psnr_metrics.PeakSignalToNoiseRatioMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.image_generation.uiqi_metrics.UniversalImageQualityIndexMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.image_generation.aesthetics_metrics.AestheticsMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.image_generation.clip_score_metrics.CLIPScoreMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.image_generation.efficiency_metrics.EfficiencyMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.image_generation.fractal_dimension_metric.FractalDimensionMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.image_generation.watermark_metrics.WatermarkMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicGenerationMetric', args={'names': []}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicReferenceMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.InstancesPerSplitMetric', args={})]

But I don't think there is any way for me to modify that run entry description to impact the metrics? I think the only items I can specify there are one of the keys from the RUN_EXPANDERS (i.e.)

    'instructions'
    'prompt'
    'newline'
    'stop'
    'format_prompt'
    'follow_format_instructions'
    'add_to_stop'
    'global_prefix'
    'num_train_trials'
    'max_train_instances'
    'max_eval_instances'
    'num_outputs'
    'num_trials'
    'model'
    'model_deployment'
    'data_augmentation'
    'tokenizer'
    'num_prompt_tokens'
    'num_output_tokens'
    'chatml'
    'eval_split'
    'output_format_instructions'
    'temperature'
    'increase_temperature'
    'increase_max_tokens'
    'process_output'

Or a key that agrees with an argument from the function wrapped with run_spec_function("mscoco"), which in this case is get_mscoco_spec.

So my thought is I have to create a custom run spec, and I worked on following the information here: https://crfm-helm.readthedocs.io/en/latest/adding_new_scenarios/#custom-run-spec-function (this doc could use a bit more detail, and ideally a small tutorial).

I created a file in the current working directory:

echo '
from helm.benchmark.run_spec import RunSpec, run_spec_function
from helm.benchmark.adaptation.adapter_spec import AdapterSpec
from helm.benchmark.metrics.metric import MetricSpec
from helm.benchmark.run_specs.classic_run_specs import get_basic_metric_specs
from helm.benchmark.scenarios.scenario import ScenarioSpec
from typing import List
from helm.benchmark.run_specs.heim_run_specs import get_image_generation_adapter_spec


def get_my_core_heim_metric_specs() -> List[MetricSpec]:
    """Evaluate every image with these set of metrics."""
    return [
        MetricSpec(class_name="helm.benchmark.metrics.image_generation.clip_score_metrics.CLIPScoreMetric", args={}),
    ] + get_basic_metric_specs(names=[])


@run_spec_function("my_mscoco")
def get_my_mscoco_spec(
    for_efficiency: bool = False,
    compute_fid: bool = False,
    run_human_eval: bool = False,
    num_human_examples: int = 100,
    use_perturbed: bool = False,
    skip_photorealism: bool = False,
    skip_subject: bool = False,
) -> RunSpec:
    scenario_spec = ScenarioSpec(
        class_name="helm.benchmark.scenarios.image_generation.mscoco_scenario.MSCOCOScenario", args={}
    )

    adapter_spec: AdapterSpec
    metric_specs: List[MetricSpec]
    run_spec_name: str

    adapter_spec = get_image_generation_adapter_spec(num_outputs=4)
    metric_specs = get_my_core_heim_metric_specs()
    run_spec_name = "my_mscoco"

    return RunSpec(
        name=run_spec_name,
        scenario_spec=scenario_spec,
        adapter_spec=adapter_spec,
        metric_specs=metric_specs,
        groups=[run_spec_name],
    )
' > helm_my_run_specs.py

Using helm-run by itself did not see the new file even though it conformed to the naming scheme, but I think that is a python issue. When I explicitly set PYTHONPATH=. it picked up the new spec. And I was able to run:

PYTHONPATH=. helm-run --run-entries my_mscoco:model=huggingface/stable-diffusion-v1-4 --suite my-heim-suite --max-eval-instances 10 --num-threads 1 --log-config ./helm_debug_log_config.yaml

But this does output the results in a different folder. I'm not sure if setting the name to "mscoco" would cause issues or not.

I'm wondering if:

1. Is this the best way to compute a subset of metrics for an existing run spec?
2. Would it be worth adding something to HELM to make it easier to point at a specific file that contains the custom run specs rather than relying on a fragile naming scheme? (Also, I'm not even sure where the helm_*_run_specs pattern matching is happening, the only thing I see to discover run specs is in discover_run_spec_functions). But it might be a lot more intuitive if there was an argument or environment variable that let the user specify the module to import that contained their customized run specs.

[Erotemic]

So it turns out that the only reason I was able to run a custom run spec was because I had put the line import helm_my_run_specs inside helm/benchmark/run.py when I was debugging. Without that it looks like there isn't any mechanism to pick up the custom run spec.

Correct me if I'm wrong, but my conclusion is that the documentation saying:

HELM will search for modules with names matching these patterns for run spec functions:

• helm_*_run_specs (i.e. a root module)

is wrong. And if it used to do that it does not do so anymore. I might recommend using something like ubelt.import_module_from_path and allow the user to specify the path to the custom python module, so helm-run can pre-import the user code before it tries to resolve the run entry description.

I could separate out this feature into a helm utility so we don't need to add ubelt as a dependency (unless you just wanted to use it), and then allow the user to specify --plugins on in the command line and they can either specify one or more paths to custom python files that register new features, or names of modules if they exist in the PYTHONPATH somewhere else. That way you don't have to have the arbitrary naming convention, nor do you have to add code into the helm package itself.

If there is interest I will make a PR for this feature.

[yifanmai]

Regarding your original question, the easiest way to do change the metrics is to make a custom run spec function, as you have already done. It is fine to reuse the same original run spec name prefix (e.g. mscoco:) or to use a different one; it depends on whether you need to disambiguate between runs using the original metrics and your custom metrics.

An alternative way of doing this is to write a run expander that rewrites the RunSpec that . Unfortunately, we don't have a way to use run expanders plug-ins yet, but we should probably support this use case.

[yifanmai]

The reason that the modules in the root module don't get picked up is that you have to add the current working directory to PYTHONPATH because Python does not add the current working directory to the Python module search path running when using command line comments / Python entry points such as helm-run. See the documentation for more details.

In general, I agree that the current plugin system is not user-friendly and could be improved. We want to support plugins in plugin packages, as well as unpackaged Python modules that live in the user's current file tree. Python has some conventions and practices around plugins that are used by other frameworks like Django and Drupal, and I think we should follow those conventions instead of rolling our own.

I'm a little concerned about the approach in #3916 because it seems to be a lot of machinery to implement a non-standard Python import mechanism, which could lead to maintainability or interoperability issues down the line. I would prefer a design that is more idiomatic for Python.

Another workaround is to have the HELM entry point add the current working directory to the Python import paths, but I suspect this could causes breakages for some users.

I have this as an issue to address on my medium term roadmap. If you'd like to work on this before then , let me know if you have any proposals or ideas.

[Erotemic]

and I think we should follow those conventions instead of rolling our own.

I didn't realize that there were actual conventions for this; in the sense that pypa is recommending them - I was familiar with all of these mechanisms. Unfortunately, none of them are entirely satisfying - some less than others.

1. Naming conventions. This is by far the worst one. A package name is a holistic identifier (in the [^\W\d]\w* sense), and forcing a prefix on that name feels wrong. I get a very strong "code smell" from this. It pollutes the global namespace, it makes package reorganization more brittle, it becomes cumbersome to reference, among other reasons.

2. Package namespaces. This is similar to the previous, but addresses the key issue of identifier mangling. A proper dotted prefix is much nicer, but it is still somewhat cumbersome.

3. Package metadata. I think this solution is the most robust for developers (this is what pytest uses), but is difficult for practitioners. There are a lot of people who write python code, but can't write a pyproject.toml. The other issue is that it commits you into making a full package structure even if you just want to test out some research code, I think you want to minimize the overhead of getting custom code injected into HELM.

Another workaround is to have the HELM entry point add the current working directory to the Python import paths, but I suspect this could causes breakages for some users.

I would not recommend having HELM mangle sys.path (permanently at least), I agree it will cause issues.

I'd like to advocate for my approach a bit, as I think it is best for the core demographic that might want to use HELM. They can write whatever dirty code they'd like, and then just point to its path explicitly. IMO, this has the best UX by far, and the standard mechanisms just can't do it, and there is a reason for that: Python makes it hard to import a module from its path. This is why I wrote import_module_from_path. It does have a few edge cases, but 99.9% of the time it works and provide the --plugins <path> UX, which I think is important.

The reason for the edge cases mainly stem from the fact that PythonPathContext inspects the path of the module, temporarily adds its directory to sys.path, imports the file via its module name, and then it cleans up the sys.path reverting it to its previous state. So as long as there isn't a conflicting name that takes priority (which is also an issue with other methods) or the importing module doesn't do advanced path trickery without undoing it, then this is reasonably safe.

But I totally get that adding a custom non-standard import mechanism to HELM opens a maintainability issue. I have two solutions I can propose for this:

1. The util_import file can be massively reduced to provide minimal functionality. E.g. you don't need any of the zip import stuff. The form I provided it in is stripped straight out of ubelt with my liberator tool, so I could reduce the dependency scope by a lot. (Also because the code does come with doctests, if there was an issue with the code in the future, it would be very likely flagged by CI).

2. I could completely remove the util_import and just add a dependency on ubelt so I can use ubelt.import_module_from_path. It's my package, but I've been maintaining it for nearly 9 years now, it has 100k downloads/month, and I pride myself on its stability. The downside is that it's a dependency for 1 function, but the upside is that if there is a maintainability problem, just remove the feature.

There is a 3rd option that I think is viable. This would avoid import_module_from_path completely and force the user to specify --plugins <module_name> and then require that the module be importable. I personally like having the path option, but this one only requires importlib and the user has to manually add PYTHONPATH=. if they are importing from the current working directory (which isn't that bad) if you know you have to do it.

---

In any case, this is aligned with my needs and I would be willing to work on this in the short-term. Based on your feedback, my recommendation would be a hybrid approach. Add support for for the standard options 2 and 3 (I really dislike 1). And also add support for a --plugins CLI argument that can accept a module name (and its path if you are ok with adding ubelt as a dependency or vendoring some code, but we can omit that).

Thus if a project is in the proof of concept stage, it is quick to test out with --plugins <path> using my existing implementation import_user_plugins, but if the project matures and the developers want to make it a serious plugin they can add lines in their pyproject.toml:

[project.entry-points.helm]
myproject = "myproject.helm_plugin"

Then HELM can import these at runtime via:

from importlib.metadata import entry_points
eps = entry_points()
group = eps.select(group="helm")
for ep in group:
    # This will import mypackage.helm_plugin
    ep.load()

I think this is the best option for robustly adding a plugin, but the existing namespace package plugin style should continue to work.

---

TL;DR: The standard Python plugin conventions each have real drawbacks for HELM’s use case, especially for quick experimentation. Entry points are the most robust long-term option, but they force a full package setup. Namespace packages work but are clunky. Naming conventions are a non-starter. A hybrid approach seems best: support entry points and namespace packages for “serious” plugins, but also add a --plugins option for loading ad-hoc modules (optionally by path) to keep the UX simple for researchers.

Let me know what you think, and I'll update the PR.

[yifanmai]

Thanks for the detailed explanation. I'm leaning towards this solution:

completely remove the util_import and just add a dependency on ubelt so I can use ubelt.import_module_from_path

because it would alleviate my concerns about having the custom import code in this repository. Let me do some research and get back to you with a more concrete answer.