Can't use large models with pipeline()

[sroussey]

System Info

Example:

        const p = pipeline('text-generation', 'Xenova/Phi-3-mini-4k-instruct', {
            device: 'webgpu',
            dtype: 'q4',
        });

I see this error in the console:

Uncaught (in promise) Error: Can't create a session. ERROR_CODE: 1, ERROR_MESSAGE: Deserialize tensor model.layers.5.mlp.gate_proj.MatMul.weight_Q4 failed.Failed to load external data file ""model_q4.onnx_data"", error: Module.MountedFiles is not available.

Seeing that onnx_data is the issue, I figured I needed to pass use_external_data_format along, but it does not work.

I have tried :

        const p = pipeline('text-generation', 'Xenova/Phi-3-mini-4k-instruct', {
            device: 'webgpu',
            dtype: 'q4',
            use_external_data_format: true,
        });

and

        const p = pipeline('text-generation', 'Xenova/Phi-3-mini-4k-instruct', {
            device: 'webgpu',
            dtype: 'q4',
            session_options: {use_external_data_format: true},
        });

But neither of these will load the model correctly.

Environment/Platform

• Website/web-app
• Browser extension
• Server-side (e.g., Node.js, Deno, Bun)
• Desktop app (e.g., Electron)
• Other (e.g., VSCode extension)

Description

see above

Reproduction

see above

[sroussey]

I altered the source to pass use_external_data_format through, and I see that the extra data file loads. However, it seems more difficult to use this way... like how do you use a chat template?

[xenova]

Thanks for #1180! I don't know why this wasn't added before 👀

However, it seems more difficult to use this way... like how do you use a chat template?

You can just pass in the messages object (and it will do the templating for you). See here for example code.

If you want access to the tokenizer to do things yourself, you can access pipeline.tokenizer.apply_chat_template

[sroussey]

BTW: if you run the code I did above without the fix, it loads most of the files, but not all, and so fails.

Problem number 2 is that if you then fix the transformers code, it will still fail since not all the files were downloaded correctly, but it thinks it was.

I need to open DevTools, go to Application tab, then to Cache Storage, and then delete transformers-cache by right clicking and choosing delete. Only then will the code work.

This seems brittle.

[benc-uk]

Exact same issue in #963

[tangkunyin]

@benc-uk @sroussey They had lost option parameters, not only use_external_data_format #1200

Hope it will be fixed as soon as possible. Thanks a lot @xenova

[tangkunyin]

Before the official updates, anyone can use this for temporary

"@huggingface/transformers": "git+https://github.com/tangkunyin/transformers.js.git#develop"

It works for me!

[xenova]

Thanks all for the discussion - we'll be adding this in #1212.

Unfortunately, it's not as simple as adding "use_external_data_format" to the pipeline options because a pipeline may consist of multiple modules, which may or may not require the external data format. Consider onnx-community/Janus-Pro-1B-ONNX, a model which has many modules, each with their own datatype. Of the ~5 modules, only the language model requires the external data format... but only for the fp32 and fp16 dtypes.

The proposed solution, which I am working on in #1212, will be to specify a mapping of filenames -> number of external chunks, defaulting to 0 (no external data format), and only specified when this should be overridden.

Would love to hear your thoughts!