Skip to content

Commit 8fe6fe6

Browse files
xqvvuxqvvu
committed
wip: switch to proxy mode for upload progress
1 parent 6157882 commit 8fe6fe6

File tree

139 files changed

+12578
-4338
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

139 files changed

+12578
-4338
lines changed

‎.github/workflows/fastgpt-test.yaml‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,8 +19,6 @@ jobs:
1919
ref: ${{ github.event.pull_request.head.ref }}
2020
repository: ${{ github.event.pull_request.head.repo.full_name }}
2121
- uses: pnpm/action-setup@v4
22-
with:
23-
version: 9
2422
- name: 'Install Deps'
2523
run: pnpm install
2624
- name: 'Test'

‎document/content/docs/self-host/config/object-storage.en.mdx‎

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -21,17 +21,22 @@ This guide covers environment variable configuration for object storage provider
2121
- `STORAGE_SECRET_ACCESS_KEY` Secret Access Key for the service credentials
2222
- `STORAGE_PUBLIC_BUCKET` FastGPT public resource bucket name
2323
- `STORAGE_PRIVATE_BUCKET` FastGPT private resource bucket name
24-
- `STORAGE_TRANSFER_MODE` File transfer mode. Options:
25-
- `proxy` (default): uploads and downloads are proxied by FastGPT backend; browser direct access to object storage is not required.
26-
- `presigned`: uploads and downloads use object-storage presigned URLs (legacy-compatible mode).
24+
25+
### Transfer Behavior
26+
27+
- Uploads always go through the FastGPT backend proxy.
28+
- Downloads support both `proxy` and `presigned` modes.
29+
- The default download mode is inferred from `STORAGE_EXTERNAL_ENDPOINT`:
30+
- not configured: default to `proxy`
31+
- configured: default to `presigned`
2732

2833
### Self-Hosted MinIO and AWS S3
2934

3035
> MinIO has strong AWS S3 protocol support, so MinIO and AWS S3 configurations are nearly identical — differences come from provider-specific or self-hosted requirements.
3136
> In theory, any object storage with S3 protocol support comparable to MinIO will work, such as SeaweedFS, RustFS, etc.
3237
3338
- `STORAGE_S3_ENDPOINT` Internal connection address. Can be a container ID, e.g., `http://fastgpt-minio:9000`
34-
- `STORAGE_EXTERNAL_ENDPOINT` An address accessible by both **server** and **client** to reach the bucket. Use a fixed host IP or domain name — don't use `127.0.0.1` or `localhost` (containers can't access loopback addresses). When `STORAGE_TRANSFER_MODE=presigned`, this is used to generate upload/download presigned URLs.
39+
- `STORAGE_EXTERNAL_ENDPOINT` An address accessible by both **server** and **client** to reach the bucket. Use a fixed host IP or domain name — don't use `127.0.0.1` or `localhost` (containers can't access loopback addresses). Once configured, the default download mode automatically becomes `presigned`.
3540
- `STORAGE_S3_FORCE_PATH_STYLE` [Optional] Virtual-hosted-style or path-style routing. If vendor is `minio`, this is fixed to `true`.
3641
- `STORAGE_S3_MAX_RETRIES` [Optional] Maximum request retry attempts. Default: 3
3742

@@ -46,7 +51,6 @@ STORAGE_ACCESS_KEY_ID=your_access_key
4651
STORAGE_SECRET_ACCESS_KEY=your_secret_key
4752
STORAGE_PUBLIC_BUCKET=fastgpt-public
4853
STORAGE_PRIVATE_BUCKET=fastgpt-private
49-
STORAGE_TRANSFER_MODE=proxy
5054
STORAGE_EXTERNAL_ENDPOINT=http://127.0.0.1:9000
5155
STORAGE_S3_ENDPOINT=http://127.0.0.1:9000
5256
STORAGE_S3_FORCE_PATH_STYLE=true

‎document/content/docs/self-host/config/object-storage.mdx‎

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -21,17 +21,22 @@ import FastGPTLink from '@/components/docs/linkFastGPT';
2121
- `STORAGE_SECRET_ACCESS_KEY` 服务访问密钥的 Secret Access Key
2222
- `STORAGE_PUBLIC_BUCKET` FastGPT 公开资源存储桶桶名
2323
- `STORAGE_PRIVATE_BUCKET` FastGPT 私有资源存储桶桶名
24-
- `STORAGE_TRANSFER_MODE` 文件传输模式，可选值：
25-
- `proxy`（默认）：上传和下载都通过 FastGPT 后端代理，不要求浏览器直连对象存储。
26-
- `presigned`：上传和下载都通过对象存储预签名 URL，兼容旧模式。
24+
25+
### 传输模式说明
26+
27+
- 上传固定走 FastGPT 后端代理。
28+
- 下载支持 `proxy` 和 `presigned` 两种模式。
29+
- 默认下载模式会根据 `STORAGE_EXTERNAL_ENDPOINT` 自动判断：
30+
- 未配置：默认 `proxy`
31+
- 已配置：默认 `presigned`
2732

2833
### 自部署的 MinIO 和 AWS S3
2934

3035
> MinIO 这类产品对 AWS S3 协议支持比较完整，因此使用 Minio 和AWS S3 配置几乎可以是相同的，只是因为服务商提供和自部署的区别，会有额外的配置。
3136
> 因此理论上任何对 AWS S3 协议的支持程度至少和 MinIO 相当的对象存储服务都可以使用，比如 SeaweedFS、RustFS 等。
3237
3338
- `STORAGE_S3_ENDPOINT` 内网连接地址，可以是容器 id 连接，比如 `http://fastgpt-minio:9000`
34-
- `STORAGE_EXTERNAL_ENDPOINT` 一个**服务器**和**客户端**均可访问到存储桶的地址，可以是固定的宿主机 IP 或者域名，注意不要填写成 127.0.0.1 或者 localhost 等本地回环地址(因为容器里无法使用)。当 `STORAGE_TRANSFER_MODE=presigned` 时，该配置用于签发上传/下载预签名 URL。
39+
- `STORAGE_EXTERNAL_ENDPOINT` 一个**服务器**和**客户端**均可访问到存储桶的地址，可以是固定的宿主机 IP 或者域名，注意不要填写成 127.0.0.1 或者 localhost 等本地回环地址(因为容器里无法使用)。配置后，默认下载模式会自动切换为 `presigned`。
3540
- `STORAGE_S3_FORCE_PATH_STYLE` 【可选】虚拟主机风格路由或路径路由风格，其中如果厂商填写了 `minio` 的话，该值被固定为 `true`
3641
- `STORAGE_S3_MAX_RETRIES` 【可选】请求最大尝试次数，默认为 3 次
3742

@@ -46,7 +51,6 @@ STORAGE_ACCESS_KEY_ID=your_access_key
4651
STORAGE_SECRET_ACCESS_KEY=your_secret_key
4752
STORAGE_PUBLIC_BUCKET=fastgpt-public
4853
STORAGE_PRIVATE_BUCKET=fastgpt-private
49-
STORAGE_TRANSFER_MODE=proxy
5054
STORAGE_EXTERNAL_ENDPOINT=http://127.0.0.1:9000
5155
STORAGE_S3_ENDPOINT=http://127.0.0.1:9000
5256
STORAGE_S3_FORCE_PATH_STYLE=true

‎document/data/doc-last-modified.json‎

Lines changed: 14 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -79,10 +79,10 @@
7979
"document/content/docs/introduction/guide/dashboard/workflow/question_classify.mdx": "2025-07-23T21:35:03+08:00",
8080
"document/content/docs/introduction/guide/dashboard/workflow/reply.en.mdx": "2026-02-26T22:14:30+08:00",
8181
"document/content/docs/introduction/guide/dashboard/workflow/reply.mdx": "2025-07-23T21:35:03+08:00",
82-
"document/content/docs/introduction/guide/dashboard/workflow/sandbox-v2.en.mdx": "2026-03-10T16:00:22+08:00",
83-
"document/content/docs/introduction/guide/dashboard/workflow/sandbox-v2.mdx": "2026-03-10T16:00:22+08:00",
84-
"document/content/docs/introduction/guide/dashboard/workflow/sandbox.en.mdx": "2026-03-10T16:00:22+08:00",
85-
"document/content/docs/introduction/guide/dashboard/workflow/sandbox.mdx": "2026-03-10T16:00:22+08:00",
82+
"document/content/docs/introduction/guide/dashboard/workflow/sandbox-v2.en.mdx": "2026-03-11T15:10:01+08:00",
83+
"document/content/docs/introduction/guide/dashboard/workflow/sandbox-v2.mdx": "2026-03-11T15:10:01+08:00",
84+
"document/content/docs/introduction/guide/dashboard/workflow/sandbox.en.mdx": "2026-03-11T15:10:01+08:00",
85+
"document/content/docs/introduction/guide/dashboard/workflow/sandbox.mdx": "2026-03-11T15:10:01+08:00",
8686
"document/content/docs/introduction/guide/dashboard/workflow/text_editor.en.mdx": "2026-02-26T22:14:30+08:00",
8787
"document/content/docs/introduction/guide/dashboard/workflow/text_editor.mdx": "2025-07-23T21:35:03+08:00",
8888
"document/content/docs/introduction/guide/dashboard/workflow/tfswitch.en.mdx": "2026-02-26T22:14:30+08:00",
@@ -137,8 +137,8 @@
137137
"document/content/docs/introduction/opensource/license.mdx": "2026-03-03T17:39:47+08:00",
138138
"document/content/docs/openapi/app.en.mdx": "2026-02-26T22:14:30+08:00",
139139
"document/content/docs/openapi/app.mdx": "2026-02-12T18:45:30+08:00",
140-
"document/content/docs/openapi/chat.en.mdx": "2026-03-05T15:30:51+08:00",
141-
"document/content/docs/openapi/chat.mdx": "2026-03-05T15:30:51+08:00",
140+
"document/content/docs/openapi/chat.en.mdx": "2026-03-11T15:10:01+08:00",
141+
"document/content/docs/openapi/chat.mdx": "2026-03-11T15:10:01+08:00",
142142
"document/content/docs/openapi/dataset.en.mdx": "2026-02-26T22:14:30+08:00",
143143
"document/content/docs/openapi/dataset.mdx": "2026-02-12T18:45:30+08:00",
144144
"document/content/docs/openapi/index.en.mdx": "2026-02-26T22:14:30+08:00",
@@ -159,8 +159,8 @@
159159
"document/content/docs/self-host/config/model/ppio.mdx": "2026-03-03T17:39:47+08:00",
160160
"document/content/docs/self-host/config/model/siliconCloud.en.mdx": "2026-03-03T17:39:47+08:00",
161161
"document/content/docs/self-host/config/model/siliconCloud.mdx": "2026-03-03T17:39:47+08:00",
162-
"document/content/docs/self-host/config/object-storage.en.mdx": "2026-03-03T17:39:47+08:00",
163-
"document/content/docs/self-host/config/object-storage.mdx": "2026-03-03T17:39:47+08:00",
162+
"document/content/docs/self-host/config/object-storage.en.mdx": "2026-03-03T16:25:35+08:00",
163+
"document/content/docs/self-host/config/object-storage.mdx": "2026-03-03T16:25:35+08:00",
164164
"document/content/docs/self-host/config/signoz.en.mdx": "2026-03-03T17:39:47+08:00",
165165
"document/content/docs/self-host/config/signoz.mdx": "2026-03-03T17:39:47+08:00",
166166
"document/content/docs/self-host/custom-models/bge-rerank.en.mdx": "2026-03-03T17:39:47+08:00",
@@ -235,7 +235,7 @@
235235
"document/content/docs/self-host/upgrading/4-14/4148.mdx": "2026-03-09T17:39:53+08:00",
236236
"document/content/docs/self-host/upgrading/4-14/41481.en.mdx": "2026-03-09T12:02:02+08:00",
237237
"document/content/docs/self-host/upgrading/4-14/41481.mdx": "2026-03-09T17:39:53+08:00",
238-
"document/content/docs/self-host/upgrading/4-14/4149.mdx": "2026-03-11T23:15:17+08:00",
238+
"document/content/docs/self-host/upgrading/4-14/4149.mdx": "2026-03-11T15:10:01+08:00",
239239
"document/content/docs/self-host/upgrading/outdated/40.en.mdx": "2026-03-03T17:39:47+08:00",
240240
"document/content/docs/self-host/upgrading/outdated/40.mdx": "2026-03-03T17:39:47+08:00",
241241
"document/content/docs/self-host/upgrading/outdated/41.en.mdx": "2026-03-03T17:39:47+08:00",
@@ -300,8 +300,8 @@
300300
"document/content/docs/self-host/upgrading/outdated/48.mdx": "2026-03-03T17:39:47+08:00",
301301
"document/content/docs/self-host/upgrading/outdated/481.en.mdx": "2026-03-03T17:39:47+08:00",
302302
"document/content/docs/self-host/upgrading/outdated/481.mdx": "2026-03-03T17:39:47+08:00",
303-
"document/content/docs/self-host/upgrading/outdated/4810.en.mdx": "2026-03-10T16:00:22+08:00",
304-
"document/content/docs/self-host/upgrading/outdated/4810.mdx": "2026-03-10T16:00:22+08:00",
303+
"document/content/docs/self-host/upgrading/outdated/4810.en.mdx": "2026-03-11T15:10:01+08:00",
304+
"document/content/docs/self-host/upgrading/outdated/4810.mdx": "2026-03-11T15:10:01+08:00",
305305
"document/content/docs/self-host/upgrading/outdated/4811.en.mdx": "2026-03-03T17:39:47+08:00",
306306
"document/content/docs/self-host/upgrading/outdated/4811.mdx": "2026-03-03T17:39:47+08:00",
307307
"document/content/docs/self-host/upgrading/outdated/4812.en.mdx": "2026-03-03T17:39:47+08:00",
@@ -320,8 +320,8 @@
320320
"document/content/docs/self-host/upgrading/outdated/4818.mdx": "2026-03-03T17:39:47+08:00",
321321
"document/content/docs/self-host/upgrading/outdated/4819.en.mdx": "2026-03-03T17:39:47+08:00",
322322
"document/content/docs/self-host/upgrading/outdated/4819.mdx": "2026-03-03T17:39:47+08:00",
323-
"document/content/docs/self-host/upgrading/outdated/482.en.mdx": "2026-03-10T16:00:22+08:00",
324-
"document/content/docs/self-host/upgrading/outdated/482.mdx": "2026-03-10T16:00:22+08:00",
323+
"document/content/docs/self-host/upgrading/outdated/482.en.mdx": "2026-03-11T15:10:01+08:00",
324+
"document/content/docs/self-host/upgrading/outdated/482.mdx": "2026-03-11T15:10:01+08:00",
325325
"document/content/docs/self-host/upgrading/outdated/4820.en.mdx": "2026-03-03T17:39:47+08:00",
326326
"document/content/docs/self-host/upgrading/outdated/4820.mdx": "2026-03-03T17:39:47+08:00",
327327
"document/content/docs/self-host/upgrading/outdated/4821.en.mdx": "2026-03-03T17:39:47+08:00",
@@ -408,4 +408,4 @@
408408
"document/content/docs/use-cases/external-integration/wecom.mdx": "2025-12-10T20:07:05+08:00",
409409
"document/content/docs/use-cases/index.en.mdx": "2026-02-26T22:14:30+08:00",
410410
"document/content/docs/use-cases/index.mdx": "2025-07-24T14:23:04+08:00"
411-
}
411+
}

‎packages/global/common/error/code/s3.ts‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
import { i18nT } from '../../../../web/i18n/utils';
2+
import { type ErrType } from '../errorCode';
3+
4+
/* s3: 510000 */
5+
export enum S3ErrEnum {
6+
invalidUploadFileType = 'InvalidUploadFileType',
7+
uploadFileTypeMismatch = 'UploadFileTypeMismatch',
8+
fileUploadDisabled = 'FileUploadDisabled'
9+
}
10+
11+
const s3ErrList = [
12+
{
13+
statusText: S3ErrEnum.invalidUploadFileType,
14+
message: i18nT('common:error.s3_upload_invalid_file_type')
15+
},
16+
{
17+
statusText: S3ErrEnum.uploadFileTypeMismatch,
18+
message: i18nT('common:error.s3_upload_invalid_file_type')
19+
},
20+
{
21+
statusText: S3ErrEnum.fileUploadDisabled,
22+
message: i18nT('common:error.file_upload_disabled')
23+
}
24+
];
25+
26+
export default s3ErrList.reduce((acc, cur, index) => {
27+
return {
28+
...acc,
29+
[cur.statusText]: {
30+
code: 510000 + index,
31+
statusText: cur.statusText,
32+
message: cur.message,
33+
data: null
34+
}
35+
};
36+
}, {} as ErrType<`${S3ErrEnum}`>);

‎packages/global/common/error/errorCode.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ import outLinkErr from './code/outLink';
77
import teamErr from './code/team';
88
import userErr from './code/user';
99
import commonErr from './code/common';
10+
import s3Err from './code/s3';
1011
import SystemErrEnum from './code/system';
1112
import { i18nT } from '../../../web/i18n/utils';
1213

@@ -108,5 +109,6 @@ export const ERROR_RESPONSE: Record<
108109
...userErr,
109110
...pluginErr,
110111
...commonErr,
112+
...s3Err,
111113
...SystemErrEnum
112114
};

‎packages/global/common/error/s3.ts‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
import { formatFileSize } from '../file/tools';
2+
import { S3ErrEnum } from './code/s3';
23

34
/**
45
* Parse S3 upload error and return user-friendly error message key
@@ -20,6 +21,16 @@ export function parseS3UploadError({
2021
if (typeof error === 'string' && error.includes('EntityTooLarge')) {
2122
return t('common:error:s3_upload_file_too_large', { max: maxSizeStr });
2223
}
24+
if (
25+
typeof error === 'string' &&
26+
(error.includes(S3ErrEnum.uploadFileTypeMismatch) ||
27+
error.includes(S3ErrEnum.invalidUploadFileType))
28+
) {
29+
return t('common:error:s3_upload_invalid_file_type');
30+
}
31+
if (typeof error === 'string' && error.includes(S3ErrEnum.fileUploadDisabled)) {
32+
return t('common:error.file_upload_disabled');
33+
}
2334

2435
// Handle axios error response
2536
if (error?.response?.data) {
@@ -32,6 +43,20 @@ export function parseS3UploadError({
3243
if (msg.includes('EntityTooLarge') || statusText.includes('EntityTooLarge')) {
3344
return t('common:error:s3_upload_file_too_large', { max: maxSizeStr });
3445
}
46+
if (
47+
msg.includes(S3ErrEnum.uploadFileTypeMismatch) ||
48+
statusText.includes(S3ErrEnum.uploadFileTypeMismatch) ||
49+
msg.includes(S3ErrEnum.invalidUploadFileType) ||
50+
statusText.includes(S3ErrEnum.invalidUploadFileType)
51+
) {
52+
return t('common:error:s3_upload_invalid_file_type');
53+
}
54+
if (
55+
msg.includes(S3ErrEnum.fileUploadDisabled) ||
56+
statusText.includes(S3ErrEnum.fileUploadDisabled)
57+
) {
58+
return t('common:error.file_upload_disabled');
59+
}
3560
if (
3661
msg.includes('unAuthFile') ||
3762
statusText.includes('unAuthFile') ||
@@ -46,6 +71,15 @@ export function parseS3UploadError({
4671
if (data.includes('EntityTooLarge')) {
4772
return t('common:error:s3_upload_file_too_large', { max: maxSizeStr });
4873
}
74+
if (
75+
data.includes(S3ErrEnum.uploadFileTypeMismatch) ||
76+
data.includes(S3ErrEnum.invalidUploadFileType)
77+
) {
78+
return t('common:error:s3_upload_invalid_file_type');
79+
}
80+
if (data.includes(S3ErrEnum.fileUploadDisabled)) {
81+
return t('common:error.file_upload_disabled');
82+
}
4983
if (data.includes('AccessDenied')) {
5084
return t('common:error:s3_upload_auth_failed');
5185
}

‎packages/global/common/file/tools.ts‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,3 +13,64 @@ export const formatFileSize = (bytes: number): string => {
1313
export const detectFileEncoding = (buffer: Buffer) => {
1414
return detect(buffer.slice(0, 200))?.encoding?.toLocaleLowerCase();
1515
};
16+
17+
const encodeRFC5987ValueChars = (value: string) => {
18+
return encodeURIComponent(value).replace(
19+
/['()*]/g,
20+
(char) => `%${char.charCodeAt(0).toString(16).toUpperCase()}`
21+
);
22+
};
23+
24+
const sanitizeHeaderFilename = (filename?: string) => {
25+
const normalized = `${filename || ''}`.replace(/[\r\n]/g, '').trim();
26+
if (!normalized) return 'file';
27+
28+
const replacedSeparators = normalized.replace(/[\\/]/g, '_');
29+
const dotIndex = replacedSeparators.lastIndexOf('.');
30+
const name = dotIndex > 0 ? replacedSeparators.slice(0, dotIndex) : replacedSeparators;
31+
const ext = dotIndex > 0 ? replacedSeparators.slice(dotIndex) : '';
32+
33+
const asciiName = name
34+
.replace(/[^\x20-\x7E]/g, '_')
35+
.replace(/["%;\\]/g, '_')
36+
.replace(/\s+/g, ' ')
37+
.trim();
38+
const asciiExt = ext.replace(/[^\x20-\x7E]/g, '').replace(/[^A-Za-z0-9._-]/g, '');
39+
40+
return `${asciiName || 'file'}${asciiExt}` || 'file';
41+
};
42+
43+
export const getContentDisposition = ({
44+
filename,
45+
type = 'inline'
46+
}: {
47+
filename?: string;
48+
type?: 'inline' | 'attachment';
49+
}) => {
50+
const normalizedFilename = `${filename || 'file'}`.replace(/[\r\n]/g, '').trim() || 'file';
51+
const fallbackFilename = sanitizeHeaderFilename(normalizedFilename);
52+
53+
return `${type}; filename="${fallbackFilename}"; filename*=UTF-8''${encodeRFC5987ValueChars(
54+
normalizedFilename
55+
)}`;
56+
};
57+
58+
export const parseContentDispositionFilename = (contentDisposition?: string) => {
59+
if (!contentDisposition) return '';
60+
61+
const filenameStarRegex = /filename\*=([^']*)'([^']*)'([^;\n]*)/i;
62+
const starMatches = filenameStarRegex.exec(contentDisposition);
63+
if (starMatches?.[3]) {
64+
try {
65+
return decodeURIComponent(starMatches[3]);
66+
} catch {}
67+
}
68+
69+
const filenameRegex = /filename[^;=\n]*=((['"]).*?\2|[^;\n]*)/i;
70+
const matches = filenameRegex.exec(contentDisposition);
71+
if (matches?.[1]) {
72+
return matches[1].replace(/['"]/g, '');
73+
}
74+
75+
return '';
76+
};

‎packages/global/common/type/mongo.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
import { z } from 'zod';
1+
import z from 'zod';
22

33
export const ObjectIdSchema = z.preprocess(
44
(value) => (typeof value === 'object' ? String(value) : value),

‎packages/global/core/ai/sandbox/constants.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
import type { I18nStringType } from '../../../common/i18n/type';
22
import { hashStr } from '../../../common/string/tools';
33
import type { ChatCompletionTool } from '../type';
4-
import { z } from 'zod';
4+
import z from 'zod';
55

66
// ---- 沙盒状态 ----
77
export const SandboxStatusEnum = {

0 commit comments

Comments
 (0)