Bug: ValueError: tokenizer_name is None when adding a new Hugging Face model due to prod_env behavior

[ztmyjs]

I am reporting a persistent and difficult-to-debug issue where helm-run fails with a ValueError: tokenizer_name is None when adding a new, standard Hugging Face model.

After a very lengthy debugging session, we have concluded that this is not a user configuration error but rather an unexpected behavior in how HELM handles the prod_env directory, which seems to be auto-created and then overrides otherwise correct configurations.

The Problem in Detail
When a new model is correctly defined in src/helm/config/model_deployments.yaml, running helm-run still fails. The traceback points to the WindowService receiving a None value for tokenizer_name.

The logs show that even if the prod_env directory is deleted or renamed beforehand, the helm-run script appears to find or recreate it, and then enters a "local mode with base path: prod_env". This action seems to invalidate the configurations loaded from src/helm/config/.

Environment
OS: v100 machine

Python Version: 3.10

HELM Installation Method: https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E-Instruct

Steps to Reproduce
Start with a clean clone of the crfm-helm repository.

Bash

git clone https://github.com/stanford-crfm/helm.git
cd helm
Set up the environment (e.g., pip install -e .).

Ensure no prod_env directory exists.

Bash

rm -rf prod_env
Add the following complete and correct model deployment configuration to the end of src/helm/config/model_deployments.yaml:

YAML

• name: meta-llama/Llama-4-Scout-17B-16E-Instruct
  model: meta-llama/Llama-4-Scout-17B-16E-Instruct
  tokenizer_name: meta-llama/Llama-4-Scout-17B-16E-Instruct
  client_spec:
  class_name: helm.proxy.clients.huggingface_client.HuggingFaceClient
  args:
  pretrained_model_name_or_path: meta-llama/Llama-4-Scout-17B-16E-Instruct
  trust_remote_code: true
  Use any standard run spec, for example, by creating a file named my_eval.conf with the following content:

entries: [
{description: "MMLU test", priority: 1, groups: ["mmlu"]},
]
Execute the helm-run command:

Bash

helm-run --conf-paths my_eval.conf --suite my_suite --models-to-run meta-llama/Llama-4-Scout-17B-16E-Instruct --max-eval-instances 1
Expected Behavior
The helm-run command should start successfully, load the specified model, and run the evaluation without configuration errors.

Actual Behavior (The Bug)
The command fails with the ValueError: tokenizer_name is None traceback. Crucially, a prod_env directory is observed to be present during or after the run, and the log shows Running in local mode with base path: prod_env.

(Please paste the full traceback from your failed helm-run command here)

Evidence and Diagnostics (The "Smoking Gun")
To prove that the YAML configuration itself is loaded correctly by HELM's registry system, I ran a separate diagnostic script. This script successfully loads the configuration and finds the correct tokenizer_name. This proves the configuration file is correct, but the information is lost or ignored later in the helm-run process.

Output of the successful diagnostic script:

-Starting Final HELM Debug Script -

Here is the FINAL configuration HELM actually sees for your model:
{'name': 'meta-llama/Llama-4-Scout-17B-16E-Instruct', 'client_spec': ClientSpec(class_name='helm.proxy.clients.huggingface_client.HuggingFaceClient', args={'pretrained_model_name_or_path': 'meta-llama/Llama-4-Scout-17B-16E-Instruct', 'trust_remote_code': True}), 'model_name': 'meta-llama/Llama-4-Scout-17B-16E-Instruct', 'tokenizer_name': 'meta-llama/Llama-4-Scout-17B-16E-Instruct', 'window_service_spec': None, 'max_sequence_length': None, 'max_request_length': None, 'max_sequence_and_generated_tokens_length': None, 'deprecated': False}

Confirmed Workaround
The only way to make the run succeed is to manually disable the prod_env directory right before running the command:

Bash

mv prod_env prod_env_disabled
helm-run ... # This now works correctly
This confirms the issue is tied to the prod_env override logic.

Thank you for looking into this complex issue.

---

(helm) junyao@goofy-1:~/helm$ cat src/helm/config/model_deployments.yaml
model_deployments:

• name: meta-llama/Llama-4-Scout-17B-16E-Instruct
  model: meta-llama/Llama-4-Scout-17B-16E-Instruct
  tokenizer_name: meta-llama/Llama-4-Scout-17B-16E-Instruct
  client_spec:
  class_name: helm.proxy.clients.huggingface_client.HuggingFaceClient
  args:
  pretrained_model_name_or_path: meta-llama/Llama-4-Scout-17B-16E-Instruct
  trust_remote_code: true
  window_service_spec:
  class_name: helm.benchmark.window_services.local_window_service.LocalWindowService
  args:
  tokenizer_name: meta-llama/Llama-4-Scout-17B-16E-Instruct
  (helm) junyao@goofy-1:/helm$ cat src/helm/benchmark/run_specs/my_llama4_eval.conf
  entries: [
  {
  name: "mmlu:computer_security",
  description: "mmlu:subject=computer_security,method=multiple_choice_joint,model=meta-llama/Llama-4-Scout-17B-16E-Instruct",
  priority: 1,
  scenario_spec: {
  class_name: "helm.benchmark.scenarios.mmlu_scenario.MMLUScenario",
  args: {
  subject: "computer_security"
  }
  },
  adapter_spec: {
  method: "multiple_choice_joint",
  model: "meta-llama/Llama-4-Scout-17B-16E-Instruct",
  model_deployment: "meta-llama/Llama-4-Scout-17B-16E-Instruct"
  },
  metric_specs: [
  {
  class_name: "helm.benchmark.metrics.basic_metrics.BasicGenerationMetric",
  args: {
  names: ["exact_match", "quasi_exact_match", "prefix_exact_match", "quasi_prefix_exact_match"]
  }
  },
  {
  class_name: "helm.benchmark.metrics.basic_metrics.BasicReferenceMetric"
  },
  {
  class_name: "helm.benchmark.metrics.basic_metrics.InstancesPerSplitMetric"
  }
  ]
  }
  ]
  (helm) junyao@goofy-1:/helm$ cat src/helm/config/tokenizer_configs.yaml
  tokenizer_configs:
• name: meta-llama/Llama-4-Scout-17B-16E-Instruct
  tokenizer: meta-llama/Llama-4-Scout-17B-16E-Instruct
  tokenizer_spec:
  class_name: AutoTokenizer
  tokenizer_name: meta-llama/Llama-4-Scout-17B-16E-Instruct

(helm) junyao@goofy-1:~/helm$ cat src/helm/config/model_metadata.yaml
models:

• name: meta-llama/Llama-4-Scout-17B-16E-Instruct
  display_name: Llama‑4 Scout 17B
  creator_organization_name: Meta
  description: |
  Instruction‑tuned Llama‑4 Scout 17B model (16 experts), multimodal (text+image input, text output).
  access: gated
  release_date: 2024-01-30
  license: Llama 4 Community License
  tags:
  • TEXT_MODEL_TAG
  • INSTRUCTION_FOLLOWING_MODEL_TAG
    tokenizer_name: meta-llama/Llama-4-Scout-17B-16E-Instruct
    (helm) junyao@goofy-1:~/helm$

[yifanmai]

Hi @ztmyjs, thanks for opening an issue.

There are two things going on here:

1. Mode deployment override order. This recent commit 213b918 / pull request Change logic for default model deployments #3694 changed the logic for which model deployment gets chosen as the default. If you are on a more recent commit, you should put your model deployment at the end of the file, or you can put it in prod_env/model_deployments.yaml. If if you are on an older commit, you should put your model deployment at the start of the file.
2. Tokenizer names. The HELM tokenizer name does not necessarily match the Hugging Face tokenizer name. In this case, the tokenizer name is actually meta/llama-4-scout-17b-16e-instruct (see the code snippet below). There are two ways to fix this - either create a new entry in tokenizer_configs.yaml with the tokenizer name meta-llama/Llama-4-Scout-17B-16E-Instruct , or change the tokenizer_name in your model deployments entry in model_deployments.yaml to meta/llama-4-scout-17b-16e-instruct.

   helm/src/helm/config/tokenizer_configs.yaml

   Lines 428 to 434 in cc701a7

   	- name: meta/llama-4-scout-17b-16e-instruct
   	tokenizer_spec:
   	class_name: "helm.tokenizers.huggingface_tokenizer.HuggingFaceTokenizer"
   	args:
   	pretrained_model_name_or_path: meta-llama/Llama-4-Scout-17B-16E-Instruct
   	prefix_token: "<|begin_of_text|>"
   	end_of_text_token: "<|end_of_text|>"

[yifanmai]

Please also refer to the documentation on adding model deployments and tokenizers.

[ztmyjs]

Please also refer to the documentation on adding model deployments and tokenizers.

Hello Yifanmai:

Thanks so much on your kind help!

This issue is unresolvable through any configuration changes. We have now tried every possible combination, including the exact methods suggested in the previous comment.

Final Proof:

I created a "perfect" configuration, strictly following all official advice (using the internal tokenizer_name, placing configs in the correct files, etc.).

I ran the command in a single thread (--num-threads 1) to rule out any parallel processing issues.

The run still fails with the exact same ValueError: tokenizer_name is None traceback. The logs clearly show the correct tokenizer name being loaded by the main thread, but it is lost somewhere before the WindowService needs it.

Here are the following configs i tried the second solution you mentioned:

(helm) junyao@goofy-1:~/helm$ git checkout -- src/helm/config/tokenizer_configs.yaml
(helm) junyao@goofy-1:~/helm$ git checkout -- src/helm/config/model_metadata.yaml
(helm) junyao@goofy-1:~/helm$ git checkout -- src/helm/config/model_deployments.yaml

---

(helm) junyao@goofy-1:~/helm$ git log -n 1

commit 36eba5ae02df3f2f64313931da8b19f8f388b168 (HEAD -> main, origin/main, origin/HEAD)
Author: Yifan Mai <yifan@cs.stanford.edu>
Date:   Mon Jun 30 15:54:30 2025 -0700
    Add MMMLU scenario (#3699)

---

my_llama4_eval.conf:

(helm) junyao@goofy-1:~/helm$ cat src/helm/benchmark/run_specs/my_llama4_eval.conf

entries: [
  {
    name: "mmlu:computer_security",
    description: "mmlu:subject=computer_security,method=multiple_choice_joint,model=meta-llama/Llama-4-Scout-17B-16E-Instruct",
    priority: 1,
    adapter_spec: {
      method: "multiple_choice_joint",
      model: "meta-llama/Llama-4-Scout-17B-16E-Instruct",
      model_deployment: "meta-llama/Llama-4-Scout-17B-16E-Instruct",
      tokenizer: "meta-llama/Llama-4-Scout-17B-16E-Instruct"
    }
  }
]

---

model_metadata.yaml

(helm) junyao@goofy-1:~/helm$ cat prod_env/model_metadata.yaml*

models:
  - name: meta-llama/Llama-4-Scout-17B-16E-Instruct
    display_name: "Llama-4 Scout 17B Instruct"
    creator_organization_name: "Meta"
    description: "Llama-4 Scout 17B Instruct model"
    access: "gated"
    tokenizer_name: meta-llama/Llama-4-Scout-17B-16E-Instruct
    tags: ["text-generation"]

---

tokenizer_configs.yaml

(helm) junyao@goofy-1:~/helm$ cat prod_env/tokenizer_configs.yaml

tokenizer_configs:
  - name: meta-llama/Llama-4-Scout-17B-16E-Instruct
    tokenizer_spec:
      class_name: "helm.tokenizers.huggingface_tokenizer.HuggingFaceTokenizer"
      args:
        pretrained_model_name_or_path: meta-llama/Llama-4-Scout-17B-16E-Instruct

---

model_deployments.yaml

(helm) junyao@goofy-1:~/helm$ cat prod_env/model_deployments.yaml

model_deployments:
  - name: meta-llama/Llama-4-Scout-17B-16E-Instruct
    tokenizer_name: meta-llama/Llama-4-Scout-17B-16E-Instruct
    client_spec:
      class_name: "helm.clients.huggingface_client.HuggingFaceClient"
      args:
        pretrained_model_name_or_path: meta-llama/Llama-4-Scout-17B-16E-Instruct
        trust_remote_code: true

---

Error Message
(helm) junyao@goofy-1:~/helm$ helm-run --conf-paths src/helm/benchmark/run_specs/my_llama4_eval.conf --suite my_llama4_suite --models-to-run meta-llama/Llama-4-Scout-17B-16E-Instruct --max-eval-instances 100 --num-threads 1
2025-07-05T05:03:38 INFO helm_run {
2025-07-05T05:03:39 INFO Reading tokenizer configs from /home/junyao/helm/src/helm/config/tokenizer_configs.yaml...
2025-07-05T05:03:39 INFO Reading model deployments from /home/junyao/helm/src/helm/config/model_deployments.yaml...
2025-07-05T05:03:39 INFO Reading tokenizer configs from prod_env/tokenizer_configs.yaml...
2025-07-05T05:03:39 INFO Reading model deployments from prod_env/model_deployments.yaml...
2025-07-05T05:03:39 INFO Read 1 run entries from src/helm/benchmark/run_specs/my_llama4_eval.conf
2025-07-05T05:03:41 INFO 1 entries produced 1 run specs
2025-07-05T05:03:41 INFO run_specs {
2025-07-05T05:03:41 INFO RunSpec(name='mmlu:subject=computer_security,method=multiple_choice_joint,model=meta-llama_Llama-4-Scout-17B-16E-Instruct', scenario_spec=ScenarioSpec(class_name='helm.benchmark.scenarios.mmlu_scenario.MMLUScenario', args={'subject': 'computer_security'}), adapter_spec=AdapterSpec(method='multiple_choice_joint', global_prefix='', global_suffix='', instructions='The following are multiple choice questions (with answers) about computer security.\n', input_prefix='Question: ', input_suffix='\n', reference_prefix='A. ', reference_suffix='\n', chain_of_thought_prefix='', chain_of_thought_suffix='\n', output_prefix='Answer: ', output_suffix='\n', instance_prefix='\n', substitutions=[], max_train_instances=5, max_eval_instances=100, num_outputs=5, num_train_trials=1, num_trials=1, sample_train=True, model_deployment='meta-llama/Llama-4-Scout-17B-16E-Instruct', model='meta-llama/Llama-4-Scout-17B-16E-Instruct', temperature=0.0, max_tokens=1, stop_sequences=['\n'], random=None, multi_label=False, image_generation_parameters=None, reeval_parameters=None, eval_splits=None), metric_specs=[MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicGenerationMetric', args={'names': ['exact_match', 'quasi_exact_match', 'prefix_exact_match', 'quasi_prefix_exact_match']}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.BasicReferenceMetric', args={}), MetricSpec(class_name='helm.benchmark.metrics.basic_metrics.InstancesPerSplitMetric', args={})], data_augmenter_spec=DataAugmenterSpec(perturbation_specs=[], should_augment_train_instances=False, should_include_original_train=False, should_skip_unchanged_train=False, should_augment_eval_instances=False, should_include_original_eval=False, should_skip_unchanged_eval=False, seeds_per_instance=1), groups=['mmlu'], annotators=None)
2025-07-05T05:03:41 INFO } [0.0s]
2025-07-05T05:03:41 INFO Running in local mode with base path: prod_env
Looking in path: prod_env
2025-07-05T05:03:41 INFO AutoTokenizer: cache_backend_config = SqliteCacheBackendConfig(path='prod_env/cache')
2025-07-05T05:03:41 INFO AutoClient: file_storage_path = prod_env/cache
2025-07-05T05:03:41 INFO AutoClient: cache_backend_config = SqliteCacheBackendConfig(path='prod_env/cache')
2025-07-05T05:03:41 INFO AutoTokenizer: cache_backend_config = SqliteCacheBackendConfig(path='prod_env/cache')
Looking in path: prod_env
2025-07-05T05:03:41 INFO AnnotatorFactory: file_storage_path = prod_env/cache
2025-07-05T05:03:41 INFO AnnotatorFactory: cache_backend_config = SqliteCacheBackendConfig(path='prod_env/cache')
0%| | 0/1 [00:00<?, ?it/s]2025-07-05T05:03:41 INFO Running mmlu:subject=computer_security,method=multiple_choice_joint,model=meta-llama_Llama-4-Scout-17B-16E-Instruct {
2025-07-05T05:03:41 INFO scenario.get_instances {
2025-07-05T05:03:41 INFO ensure_file_downloaded {
2025-07-05T05:03:41 INFO Not downloading https://people.eecs.berkeley.edu/~hendrycks/data.tar because benchmark_output/scenarios/mmlu/data already exists
2025-07-05T05:03:41 INFO } [0.0s]
2025-07-05T05:03:41 INFO benchmark_output/scenarios/mmlu/data/auxiliary_train/computer_security_auxiliary_train.csv doesn't exist, skipping
2025-07-05T05:03:41 INFO Reading benchmark_output/scenarios/mmlu/data/dev/computer_security_dev.csv
2025-07-05T05:03:41 INFO Reading benchmark_output/scenarios/mmlu/data/val/computer_security_val.csv
2025-07-05T05:03:41 INFO Reading benchmark_output/scenarios/mmlu/data/test/computer_security_test.csv
2025-07-05T05:03:41 INFO } [0.002s]
2025-07-05T05:03:41 INFO 116 instances, 5 train instances, 100/111 eval instances
2025-07-05T05:03:41 INFO DataPreprocessor.preprocess {
2025-07-05T05:03:41 INFO } [0.0s]
2025-07-05T05:03:41 INFO MultipleChoiceJointAdapter.adapt {
2025-07-05T05:03:41 INFO 105 instances, choosing 5/5 train instances, 100 eval instances
2025-07-05T05:03:41 INFO Adapting with train_trial_index=0 {
2025-07-05T05:03:41 INFO Sampled 5 examples for trial #0.
2025-07-05T05:03:41 INFO Parallelizing computation on 100 items over 1 threads {
[DEBUG] local_context.tokenize() got request.tokenizer = meta-llama/Llama-4-Scout-17B-16E-Instruct | 0/100 [00:00<?, ?it/s]
[DEBUG] _get_tokenizer called with tokenizer_name = meta-llama/Llama-4-Scout-17B-16E-Instruct
0%| | 0/100 [00:00<?, ?it/s]
2025-07-05T05:03:41 INFO } [0.0s]
2025-07-05T05:03:41 INFO } [0.0s]
2025-07-05T05:03:41 INFO } [0.001s]
2025-07-05T05:03:41 INFO } [0.005s]
2025-07-05T05:03:41 INFO Error when running mmlu:subject=computer_security,method=multiple_choice_joint,model=meta-llama_Llama-4-Scout-17B-16E-Instruct:
Traceback (most recent call last):
File "/home/junyao/helm/src/helm/benchmark/runner.py", line 216, in run_all
self.run_one(run_spec)
File "/home/junyao/helm/src/helm/benchmark/runner.py", line 283, in run_one
request_states: List[RequestState] = adapter.adapt(instances, self.executor.execution_spec.parallelism)
File "/home/junyao/helm/src/helm/common/hierarchical_logger.py", line 127, in wrapper
return fn(*args, **kwargs)
File "/home/junyao/helm/src/helm/benchmark/adaptation/adapters/in_context_learning_adapter.py", line 63, in adapt
self._adapt_trial_index(all_train_instances, train_trial_index, eval_instances, parallelism)
File "/home/junyao/helm/src/helm/benchmark/adaptation/adapters/in_context_learning_adapter.py", line 86, in _adapt_trial_index
results: List[List[RequestState]] = parallel_map(
File "/home/junyao/helm/src/helm/common/general.py", line 239, in parallel_map
results = list(tqdm(map(process, items), total=len(items), disable=None))
File "/home/junyao/miniconda3/envs/helm/lib/python3.10/site-packages/tqdm/std.py", line 1181, in iter
for obj in iterable:
File "/home/junyao/helm/src/helm/benchmark/adaptation/adapters/in_context_learning_adapter.py", line 83, in generate_requests_for_training_trial
return self.generate_requests(eval_instance, train_trial_index, training_instances)
File "/home/junyao/helm/src/helm/benchmark/adaptation/adapters/multiple_choice_joint_adapter.py", line 57, in generate_requests
prompt = self.construct_prompt(training_instances, eval_instance, include_output=False, reference_index=None)
File "/home/junyao/helm/src/helm/benchmark/adaptation/adapters/in_context_learning_adapter.py", line 242, in construct_prompt
prompt = self._make_prompt_fit(prompt)
File "/home/junyao/helm/src/helm/benchmark/adaptation/adapters/in_context_learning_adapter.py", line 302, in _make_prompt_fit
if self.window_service.fits_within_context_window(
File "/home/junyao/helm/src/helm/benchmark/window_services/local_window_service.py", line 94, in fits_within_context_window
return self.get_num_tokens(text) + expected_completion_token_length <= self.max_request_length
File "/home/junyao/helm/src/helm/benchmark/window_services/local_window_service.py", line 87, in get_num_tokens
return len(self.encode(text).tokens)
File "/home/junyao/helm/src/helm/benchmark/window_services/local_window_service.py", line 45, in encode
response: TokenizationRequestResult = self.service.tokenize(
File "/home/junyao/helm/src/helm/benchmark/window_services/tokenizer_service.py", line 21, in tokenize
return self._context.tokenize(request)
File "/home/junyao/helm/src/helm/common/local_context.py", line 93, in tokenize
return self.tokenizer.tokenize(request)
File "/home/junyao/helm/src/helm/tokenizers/auto_tokenizer.py", line 72, in tokenize
tokenizer: Tokenizer = self._get_tokenizer(request.tokenizer)
File "/home/junyao/helm/src/helm/tokenizers/auto_tokenizer.py", line 38, in _get_tokenizer
raise ValueError("tokenizer_name is None. Please check your model_metadata.yaml and model_deployments.yaml")
ValueError: tokenizer_name is None. Please check your model_metadata.yaml and model_deployments.yaml

100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 144.96it/s]
2025-07-05T05:03:41 INFO } [2.726s]
Traceback (most recent call last):
File "/home/junyao/miniconda3/envs/helm/bin/helm-run", line 8, in
sys.exit(main())
File "/home/junyao/helm/src/helm/benchmark/run.py", line 371, in main
return helm_run(args)
File "/home/junyao/helm/src/helm/common/hierarchical_logger.py", line 127, in wrapper
return fn(*args, **kwargs)
File "/home/junyao/helm/src/helm/benchmark/run.py", line 274, in helm_run
run_benchmarking(
File "/home/junyao/helm/src/helm/benchmark/run.py", line 127, in run_benchmarking
runner.run_all(run_specs)
File "/home/junyao/helm/src/helm/benchmark/runner.py", line 225, in run_all
raise RunnerError(f"Failed runs: [{failed_runs_str}]")
helm.benchmark.runner.RunnerError: Failed runs: ["mmlu:subject=computer_security,method=multiple_choice_joint,model=meta-llama_Llama-4-Scout-17B-16E-Instruct"]
(helm) junyao@goofy-1:~/helm$ git checkout -- src/helm/config/tokenizer_configs.yaml

[ztmyjs]

@yifanmai thanks