Stop requiring CacheConfig in GenerationConfig with StaticCache

[poedator]

I have an observation that in some common use cases, configuring StaticCache during GenerationConfig initialisation is unnecessary.
it was introduced in #32830

Specifically, when the model is used with .generate, only the cache_implementation config option is relevant. The rest of the cache config is determined inside generate(), specifically in transformers.generation.utils::GenerationMixin._get_cache(). In that function, StaticCache is created based on the requested generation parameters, ignoring cache_config entirely.

Suggestion:

• do not require StaticCache config in GenerationConfig.init
• have some custom logic for ExecuTorch, that does not affect other use cases
• have default arguments for StaticCache.init() so that it would quietly get created with some default parameters (not a good idea though).
• have tests in transformers that set just cache_implementation="static" and then call .generate()

current workaround:

do not set cache_implementation in GenerationConfig constructor, but set it afterwards with:
model.generation_config.cache_implementation = "static"

system info
applies to transformers from september 2024 up to the current 4.46.3

Who can help?

@gante

[Rocketknight1]

cc @zucchini-nlp as well!

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[poedator]

up
@zucchini-nlp @gante

[gante]

Yeah, it shouldn't be required. Setting model.generation_config.cache_implementation = "static" should be enough for most cases. Having a look.

[gante]

Hi @poedator 👋

I'm trying to reproduce the issue, but I'm not sure how. Can you help me out? The script below works as expected.

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("distilgpt2")
model = AutoModelForCausalLM.from_pretrained("distilgpt2", device_map="auto")

inputs = tokenizer(["The quick brown"], return_tensors="pt").to(model.device)
gen_out = model.generate(**inputs, do_sample=False, cache_implementation="static")
print(tokenizer.batch_decode(gen_out, skip_special_tokens=True))

model.generation_config.cache_implementation = "static"
gen_out = model.generate(**inputs, do_sample=False)
print(tokenizer.batch_decode(gen_out, skip_special_tokens=True))

[poedator]

Hi, @gante !
your example works fine, but in my app I pack all generation args into GenerationConfig (because it can conveniently hold it all) and it fails when cache_implementation="static" is among the args.
The constructor requires 'batch_size' and 'max_cache_len', but I may not know them in advance. These are decided inside .generate().

Try this example, please:

from transformers import GenerationConfig
generation_config = GenerationConfig(cache_implementation="static")

>> output:
TypeError                                 Traceback (most recent call last)
Cell In[1], line 2
      1 from transformers import GenerationConfig
----> 2 generation_config = GenerationConfig(
      3     cache_implementation="static"
      4 )

File [~/conda/envs/tts39/lib/python3.9/site-packages/transformers/generation/configuration_utils.py:415](http://terranova.vla.yp-c.yandex.net:6501/lab/workspaces/tts/tree/tts/conda/envs/tts39/lib/python3.9/site-packages/transformers/generation/configuration_utils.py#line=414), in GenerationConfig.__init__(self, **kwargs)
    413 cache_config_class = NEEDS_CACHE_CONFIG[self.cache_implementation]
    414 if self.cache_config is None:
--> 415     self.cache_config = cache_config_class()
    416 elif isinstance(self.cache_config, dict):
    417     self.cache_config = cache_config_class.from_dict(self.cache_config)

TypeError: __init__() missing 2 required positional arguments: 'batch_size' and 'max_cache_len'

[gante]

Gotcha. That shouldn't happen, opening a PR to fix it.