Support proxy and user-agent in fsspec calls

[lhoestq]

Since we switched to the new HfFileSystem we no longer apply user's proxy and user-agent.

Using the HTTP_PROXY and HTTPS_PROXY environment variables works though since we use aiohttp to call the HF Hub.

This can be implemented in _prepare_single_hop_path_and_storage_options.

Though ideally the HfFileSystem could support passing at least the proxies

[zutarich]

hii @lhoestq can you assign this issue to me?

[lhoestq]

You can reply "#self-assign" to this issue to automatically get assigned to it :)
Let me know if you have any questions or if I can help

[zutarich]

#2289

[zutarich]

Actually i am quite new to figure it out how everything goes and done

You can reply "#self-assign" to this issue to automatically get assigned to it :)
Let me know if you have any questions or if I can help

when i wrote #self-assign it automatically got converted to some number is it correct or i have done it some wrong way, I am quite new to open source thus wanna try to learn and explore it

[zutarich]

#2289 #self-assign

[lhoestq]

Ah yea github tries to replace the #self-assign with an issue link. I guess you can try to copy-paste instead to see if it works

Anyway let me assign you manually

[zutarich]

thanks a lot @lhoestq ! though i have a very lil idea of the issue, i am new. as i said before, but gonna try my best shot to do it.
can you please suggest some tips or anything from your side, how basically we approach it will be really helpfull.
Will try my best!

[lhoestq]

The HfFileSystem from the huggingface_hub package can already read the HTTP_PROXY and HTTPS_PROXY environment variables. So the remaining thing missing is the user_agent that the user may include in a DownloadConfig object. The user agent can be used for regular http calls but also calls to the HfFileSystem.

• for http, the user_agent isn't passed from DownloadConfig to get_datasets_user_agent in _prepare_single_hop_path_and_storage_options in streaming_download_manager.py so we need to include it
• for HfFileSystem I think it requires a PR in https://github.com/huggingface/huggingface_hub to include it in the HfFileSystem.__init__

[ArjunJagdale]

Hi @lhoestq 👋🏼

Is anyone currently working on this? If not, I'd like to pick it up.

As I understand it:

• The user_agent from DownloadConfig isn't currently passed to get_datasets_user_agent() inside _prepare_single_hop_path_and_storage_options.
• I'll update that function to include the correct user-agent in the headers.
• For full support, we may also need a change in huggingface_hub to let HfFileSystem accept custom headers.

Please let me know if this approach sounds good or if you’d prefer it handled differently 🙌