S3: Using IRSA in Kubernetes - Images expire after one hour

[abhishekag03-cb]

I'm using S3 as the file storage strategy using IRSA as recommended in Option B here

To my surprise, the image links always expire in exactly one hour. Happens for both Agent icons and LLM Generated Images.

For the Agent icons, if I open the Agent Configuration via the Agent builder again - the URL gets refreshed and the icon starts appearing back for another hour in the Agents list.
Each icon refresh is counted as a new Agent version and now I'm having 50+ versions for each agent.

I have already tried setting both S3_URL_EXPIRY_SECONDS and S3_REFRESH_EXPIRY_MS to high values like 7 days, but the URLs still expire in an hour.

Ideally I would want the Agent icons to never expire and the LLM generated images to be there for atleast 10 days.

[danny-avila]

Some explanation on this: https://www.reddit.com/r/aws/comments/1dh1hkt/presigned_urls_are_expiring_in_1_hour_only_what/

TL;DR CloudFront needs to be implemented for the LC use case + AWS

[layrfranca-hotmart]

Hey @danny-avila ! I’m running the project on the cloud using EKS and I’m experiencing a similar issue...

I tried migrating to the S3 file strategy but had to rollback because images URLs were expiring. Even the S3 hard limit of 7 days doesn’t meet my use case. I’m using S3 CDN with IRSA (just passing the bucket name and region in the .env files) and I was hoping the backend would handle the URLs refresh.

I was expecting the LibreChat backend to detect when a URL is expired (or about to expire) and fetch a refreshed URL on demand (the refreshS3FileUrls function used in the GET files route, which gets the user’s files). Isn’t this supposed to work by now?

Also, after migrating, I noticed that documents in the "Files" collection started saving differently. Do you have any guidance or documentation on migrating existing files from the pod’s volume to S3? I’m able to upload files to S3, but my main concern is how to handle the database documents that require signed URLs.

Thanks in advance!

[abhishekag03-cb]

+1 to @layrfranca-hotmart. We already have the refresh logic implemented in the GetAgent endpoint (i.e. when the Agent is loaded in the Agent Builder settings), we should have the same thing at other places within LibreChat's backend.
Our org considers setting up IAM role as an anti-pattern and this is becoming a big issue for the users as the images generated always disappear in an hour (even though they continue to exist in S3)