How should DIFY obtain the URL address parsed by the MinerU tool when integrating Minio

[tianshiqingge]

Self Checks

• This is only for bug report, if you would like to ask a question, please head to Discussions.
• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:)
• Please do not modify this template :) and fill in all the required fields.

Dify version

1.4.0

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

I correctly configured Minio using S3 in DIFY, and the files in DIFY can be uploaded to Minio normally. The configuration is as follows:
STORAGE_TYPE=s3
S3_ENDPOINT= http://minio:9000/
S3_REGION=us-east-1
S3_BUCKET_NAME=difyai
S3_ACCESS_KEY=xxxx
S3_SECRET_KEY=xxxx
My problem is that when I use MinerU tool to convert PDF files in the workflow, the converted image address cannot be opened properly. My Files_URL= https://intense-game-marlin.ngrok-free.app/difyai If configured in this way (this address is Minio's external network address), the tool will report content errors such as: Failed to conversion tool message: PluginInvokeError: {"args":{},"error_type":"Exception","message":" The application failed to upload the URL. Status: 401, result: {"traceId": "81328028b2f3", "msgCode": "A0202", "msg": "User verification failed", "data": null, "success": false, "total": 0} "}. If Files_URL= http://api:5001/ If configured in this way, MinerU will not encounter errors when converting files, but the URL address of the parsed image cannot be opened. My MinerU is using the official API.How should I configure it?

✔️ Expected Behavior

No response

❌ Actual Behavior

No response

[dosubot[bot]]

To ensure Dify obtains accessible URLs for files processed by the MinerU tool when integrating with Minio, you need to pay close attention to the configuration of the FILES_URL and Minio authentication settings.

Key Points for Configuration

1. FILES_URL Setting
   For Docker Compose deployments, set:

   FILES_URL=http://your-ip
   

   in your .env file.
   For standard installations (not Docker Compose), use:

   FILES_URL=http://your-ip:port
   

   This approach has been confirmed to work with the MinerU plugin and ensures that generated URLs are accessible by external tools like MinerU. Do not use the Minio external address directly as the FILES_URL—use the Dify API service address instead.
   Reference

2. Minio S3 Configuration
   Your Minio settings in .env should look like:

   STORAGE_TYPE=s3
   S3_ENDPOINT=http://minio:9000/
   S3_REGION=us-east-1
   S3_BUCKET_NAME=difyai
   S3_ACCESS_KEY=xxxx
   S3_SECRET_KEY=xxxx
   S3_USE_AWS_MANAGED_IAM=false
   

   Make sure:

   • The Minio server is running and accessible from your Dify containers.
   • The bucket exists.
   • The access and secret keys are correct.
   • The endpoint URL is reachable from within the Docker network.
     Reference

3. 401 Authentication Error
   The 401 error (User verification failed) when using the external Minio address as FILES_URL suggests that MinerU is trying to access Minio directly, which requires authentication that it does not have.
   By using the Dify API address as FILES_URL, Dify acts as a proxy and handles authentication, so MinerU does not need direct access to Minio.

4. Accessing Converted Files
   When FILES_URL is set to the Dify API address, MinerU should receive URLs that are accessible via the Dify service. If you cannot open these URLs externally, ensure your Dify API service is reachable from outside your Docker network (e.g., via port mapping or reverse proxy).

5. Region Setting for Plugins
   If you use dify-plugin-daemon with Minio, set:

   PLUGIN_AWS_REGION=us-east-1
   

   to avoid initialization errors.
   Reference

Summary

• Do not set FILES_URL to the Minio external address.
• Set FILES_URL to the Dify API address (internal or external, as appropriate).
• Ensure your Dify API is accessible to MinerU and external clients if needed.
• Confirm all Minio credentials and bucket settings are correct.

If you follow these steps and still encounter issues with URL access, the problem may be related to network accessibility or reverse proxy configuration for your Dify API service. If you need more help with that, please provide details about your network setup.

To continue talking to Dosu, mention @dosu-bot.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[tianshiqingge]

@dosu-bot Docker Compose deployment, set up:
FILES_URL= http://10.0.60.165 When starting Docker, an error occurs: invalid interpolation format for services. appi. environment. FILE. URL
You may need to escape any $ with another $.
${FILES_URL: http://10.0.60.165 }

[dosubot[bot]]

The "invalid interpolation format" error in Docker Compose usually happens when the environment variable syntax is not correct, or when special characters like $ are not handled properly.

To set FILES_URL for Dify in Docker Compose, follow these steps:

1. Set FILES_URL in your .env file
   Add this line to your .env file (not directly in docker-compose.yaml):

   FILES_URL=http://10.0.60.165
   

   This is the recommended way for Dify Docker Compose deployments and should not cause interpolation errors if used in the .env file and referenced in your docker-compose.yaml as FILES_URL=${FILES_URL}.
   Reference

2. If you must set it directly in docker-compose.yaml
   If you are setting the value directly in the docker-compose.yaml under environment:, use quotes and avoid ${...} syntax unless you are referencing another variable. For example:

   environment:
     - FILES_URL=http://10.0.60.165

3. If you use ${...} syntax and default values
   If you use variable interpolation with default values (e.g., ${FILES_URL:-http://10.0.60.165}), and you see this error, escape the $ with another $:

   environment:
     - FILES_URL=$${FILES_URL:-http://10.0.60.165}

   Reference

4. Upgrade Docker and use the new command
   If you still get the error, upgrade Docker to the latest version and use the docker compose command (with a space, not a hyphen) instead of docker-compose. Older Docker Compose versions are known to have issues with interpolation syntax.
   Reference

Summary:

• Prefer setting FILES_URL in your .env file as FILES_URL=http://10.0.60.165.
• If you must use interpolation in docker-compose.yaml, escape $ as $$.
• Upgrade Docker and use docker compose if problems persist.

If you still encounter issues, please share the relevant part of your docker-compose.yaml and .env files for further troubleshooting.

To continue talking to Dosu, mention @dosu-bot.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[tianshiqingge]

@dosu-bot
According to your configuration, my dify has a problem. I can't see any created knowledge base or model. The same error is reported: Internal Server Error. I uploaded the API container log. Please help me analyze the cause of the problem

docker-api-1-20250525110137.log

[tianshiqingge]

@dosu-bot
Why is it configured as FILES_URL= http://api:5001 Or configured as FILES_URL= http://10.0.60.165:5001 ， The MinerU tool reports the same error: Failed to transform tool message: PluginInvokeError: {"args":{},"error_type":"Exception","message":"apply upload url failed. status:401 ,result:{"traceId":"bf9d6eabb0b6","msgCode":"A0202","msg":"user authenticate failed","data":null,"success":false,"total":0}"}
How can I solve this problem?

[dosubot[bot]]

The 401 authentication error ("user authenticate failed") from the MinerU tool when using FILES_URL set to http://api:5001 or http://10.0.60.165:5001 indicates that MinerU is trying to access the Dify API for file upload or retrieval, but the request is not properly authenticated.

Here’s what’s likely happening and how you can address it:

Why the 401 Error Occurs

• Authentication Required: Dify’s API endpoints for file operations require authentication. If MinerU is calling these endpoints without the correct credentials (such as an API key in the HTTP header), Dify will return a 401 error.
• FILES_URL as Dify API: Setting FILES_URL to the Dify API address is correct, but only if the client (MinerU) can authenticate with Dify’s API. If MinerU does not send the required authentication headers, access will be denied.

How to Solve

1. Check MinerU’s Authentication Support

   • Confirm whether MinerU supports sending an API key or other authentication headers when accessing FILES_URL endpoints.
   • If MinerU does not support authentication, it cannot access protected Dify API endpoints directly.

2. Configure Dify API Authentication

   • Dify supports API key authentication for its API endpoints. The API key is typically sent in the Authorization HTTP header, but this can be customized in Dify’s tool configuration.
   • Make sure you have generated an API key in Dify and that MinerU is configured to use it (if MinerU supports this).

3. Test with Public Access (Not Recommended for Production)

   • For testing, you could temporarily disable authentication on the Dify API (if possible), but this is not recommended for production environments due to security risks.

4. Proxy or Custom Integration

   • If MinerU cannot be configured to send authentication headers, you may need to implement a proxy or middleware that adds the required authentication before forwarding requests to Dify.

Key Points from Dify’s Documentation

• Dify supports different authorization methods for tools, including none and api_key. The API key is usually sent in the Authorization header, but this can be customized. If the client does not send the correct header, a 401 error will occur [source].
• FILES_URL should point to the Dify API, but the client must be able to authenticate with it [source].

What to Do Next

• Check MinerU’s documentation or settings to see if you can configure an API key or authentication header for file requests.
• If not possible, consider reaching out to MinerU’s support or community to request this feature.
• If you need to keep authentication enabled (recommended), all clients accessing FILES_URL must support Dify’s authentication method.

If you have access to the API container logs and see specific authentication errors, they may provide more details about what is missing in the request. If you need help interpreting those logs, please share the relevant error lines.

Let me know if you need more details on configuring API keys in Dify or if you have questions about MinerU’s capabilities.

To continue talking to Dosu, mention @dosu-bot.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[tianshiqingge]

@dosu-bot
Thank you very much for your answer. Now I have a new problem. My FILES_URL parameter is configured as follows: FILES_URL= http://api:5001 , when MinerU converts the image address to: http://api:5001/files/tools/5b040680 -6560-48f0-95e3-29456066e05d.jpg? timestamp=1748160691&nonce=7198bedff0c7792aad8f57a2b4bca89a&sign=eTwW9rg8jUpXiS_rS1TcW48BscQkVmViuBy-9183PJo=， This address cannot be opened. I can see in minio that the parsed image file has been uploaded successfully. Where do I have problems with configuration? How to solve it?

[dosubot[bot]]

The issue where the URL generated by MinerU (e.g., http://api:5001/files/tools/...jpg?...) cannot be opened, even though the file is present in Minio, is most likely related to how Dify signs and serves file URLs.

Here’s what you should check and how to resolve it:

---

1. FILES_URL Configuration

• For Docker Compose, set FILES_URL in your .env file to the actual address and port where your Dify API is accessible from your browser or MinerU. For example, if you access Dify at http://10.0.60.165:5001, then:

  FILES_URL=http://10.0.60.165:5001
  

  Make sure this address is reachable from the machine where you are trying to open the URL (not just from inside Docker) [source].

---

2. Port and Network Accessibility

• Ensure that port 5001 is open and accessible from your network. If you are running Dify in Docker, make sure you have mapped port 5001 to your host and there are no firewall rules blocking it.

---

3. URL Signing and Secret Key

• Dify generates signed URLs for file access using HMAC with a secret key (SECRET_KEY) and a timeout (FILES_ACCESS_TIMEOUT). If the secret key in your .env file has changed after files were uploaded, or if the URL is expired, the link will not work [source].
• Make sure the SECRET_KEY in your .env file is consistent and has not changed.
• The default timeout for file URLs is 300 seconds (5 minutes). If you try to open the URL after this period, it will be invalid. You can increase FILES_ACCESS_TIMEOUT in your .env if needed.

---

4. URL Format and Query Parameters

• The URL must include valid timestamp, nonce, and sign parameters. If any of these are missing or invalid, Dify will reject the request [source].
• If you are copying the URL, make sure there are no extra spaces or line breaks.

---

5. Internal vs. External Access

• If you use http://api:5001 as FILES_URL, this address must be resolvable and accessible from the client (your browser or MinerU). If api is only resolvable inside Docker, use your host machine’s IP address instead.

---

6. Restart Services

• After changing .env or Docker configuration, always restart your containers:

  docker-compose down
  docker-compose up -d
  

---

Troubleshooting Steps

1. Open the generated URL in a browser on the same machine where Dify is running. If it works, the problem is likely network/firewall related.
2. If you get an "Invalid request" or "Signature error", check your SECRET_KEY and FILES_ACCESS_TIMEOUT.
3. If you get a connection error, check your Docker port mapping and firewall.

---

Summary:

• Use your host’s IP and correct port for FILES_URL.
• Ensure port 5001 is open and mapped.
• Keep SECRET_KEY consistent.
• Increase FILES_ACCESS_TIMEOUT if needed.
• Restart Docker after changes.

If you still cannot access the URL, please provide the exact error message you see in your browser or client when trying to open the link.

To continue talking to Dosu, mention @dosu-bot.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}