fix(api): keep non-ASCII characters in uploaded file names - #3042
fix(api): keep non-ASCII characters in uploaded file names#3042ryanchou1994 wants to merge 2 commits into
Conversation
Greptile SummaryThis PR preserves Unicode upload filenames by limiting sanitization to control characters and by encoding non-ASCII S3 metadata as printable ASCII.
Confidence Score: 4/5The filename encoding collision should be fixed before merging because a valid literal ASCII filename can be changed when read from S3 metadata. The encoder leaves all printable ASCII unchanged while the decoder reserves a subset of those values as encoded words, so the new transport representation cannot round-trip every valid filename. Files Needing Attention: packages/shared/s3MetadataEncoding.ts Important Files Changed
Prompt To Fix All With AI### Issue 1
packages/shared/s3MetadataEncoding.ts:13
**Encoded-word filenames lose identity**
When a literal ASCII filename matches the encoded-word syntax, `encodeS3MetadataValue` stores it verbatim but `decodeS3MetadataValue` interprets it as encoded data, causing names such as `=?UTF-8?B?SGVsbG8=?=` to become `Hello` when S3 metadata is read and potentially persisted back to the database.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.Reviews (1): Last reviewed commit: "fix(api): keep non-ASCII characters in u..." | Re-trigger Greptile |
|
|
||
| export function encodeS3MetadataValue(value: string): string { | ||
| if (ASCII_ONLY.test(value)) { | ||
| return value; |
There was a problem hiding this comment.
Encoded-word filenames lose identity
When a literal ASCII filename matches the encoded-word syntax, encodeS3MetadataValue stores it verbatim but decodeS3MetadataValue interprets it as encoded data, causing names such as =?UTF-8?B?SGVsbG8=?= to become Hello when S3 metadata is read and potentially persisted back to the database.
Prompt To Fix With AI
This is a comment left during a code review.
Path: packages/shared/s3MetadataEncoding.ts
Line: 13
Comment:
**Encoded-word filenames lose identity**
When a literal ASCII filename matches the encoded-word syntax, `encodeS3MetadataValue` stores it verbatim but `decodeS3MetadataValue` interprets it as encoded data, causing names such as `=?UTF-8?B?SGVsbG8=?=` to become `Hello` when S3 metadata is read and potentially persisted back to the database.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.|
Good catch on the encoded-word collision — fixed in 70b0085: a literal name that already matches the |
Uploads replaced every non-ASCII character in the file name with "_", so "Prüfung Größe Öl.pdf" was stored and shown as "Pr_fung Gr__e _l.pdf" and the original name was lost (one underscore per character, not reversible). The replacement was added in 39a650f for karakeep-app#1765: the file name is also written to S3 as user-defined metadata (x-amz-meta-file-name), which travels as an HTTP header and must be ASCII, otherwise Node rejects the request with ERR_INVALID_CHAR. Sanitising the name at upload time fixed the crash but threw away the information for every backend, including the local file system and the database column that the UI displays. Move the ASCII constraint to where it actually applies: the S3 store now wraps non-ASCII metadata values as an RFC 2047 encoded-word on write and unwraps them on read, while plain ASCII values are stored verbatim so existing objects are unaffected. The upload path only replaces control characters (C0, DEL, C1) and keeps everything else as-is; the file name is stored as data and never used as a filesystem path. Fixes karakeep-app#3041
…-word A literal ASCII file name such as "=?UTF-8?B?SGVsbG8=?=" was stored verbatim but unwrapped to "Hello" when read back from S3 metadata. Encode such values too so decodeS3MetadataValue can never misinterpret a literal name.
70b0085 to
4300501
Compare
Description
Uploaded file names lost every non-ASCII character:
Prüfung Größe Öl.pdfcame back asPr_fung Gr__e _l.pdf(one_per character, so not reversible), and since asset bookmarks fall back tofileNamefor their title, that mangled name is what got displayed and searched.The replacement was introduced in 39a650f for #1765, where a non-ASCII file name crashed S3 uploads: the name is stored as S3 user-defined metadata (
x-amz-meta-file-name), which is sent as an HTTP header and must be ASCII (ERR_INVALID_CHAR). Sanitising at upload time fixed that crash but threw the information away for every backend — including the local file system store and theassets.fileNamecolumn the UI reads.This PR moves the ASCII constraint to the only place it applies:
packages/shared/s3MetadataEncoding.ts(new):encodeS3MetadataValue/decodeS3MetadataValue. Values that are already printable ASCII are stored verbatim, so existing objects and the existing S3 e2e expectations are unchanged. Anything else is wrapped as an RFC 2047 encoded-word (=?UTF-8?B?…?=), which is ASCII-only and round-trips losslessly.packages/shared/assetdb.ts: the S3 store encodesfileNamewhen writing metadata and decodes it when reading it back.packages/api/utils/fileName.ts(new) +upload.ts: the upload path now only replaces control characters (C0, DEL, C1) and keeps everything else, including non-ASCII letters. The file name is only ever stored as data (DB column, asset metadata, API response) and is never used as a filesystem path, so nothing else needs escaping there.Fixes #3041
How Has This Been Tested?
packages/api/utils/fileName.test.ts(3) andpackages/shared/s3MetadataEncoding.test.ts(4) —pnpm --filter @karakeep/api testandpnpm --filter @karakeep/shared testall green (8 and 111 tests respectively).sanitizeUploadFileNameto the old/[^\x20-\x7E]/gmakes the "preserves non-ASCII letters" test fail; makingencodeS3MetadataValuereturn its input unchanged makes the "ASCII-only encoded word" test fail. Both pass again with the fix in place.http.validateHeaderValue:報告 2026.pdfand🦞.txtare rejected withERR_INVALID_CHARwhen passed raw, and accepted once wrapped as=?UTF-8?B?…?=. (Latin-1 names such asPrüfung Größe Öl.pdfhappen to pass Node's check, but S3 user-defined metadata is still specified as US-ASCII, so they are encoded too.)pnpm --filter @karakeep/api typecheck,pnpm --filter @karakeep/shared typecheck,lintandformatall clean.packages/e2e_tests/tests/assetdb/s3-store.test.ts, needs MinIO). Its existing expectations use ASCII file names, which this change stores byte-for-byte as before.Checklist:
Please describe to which degree, if any, an LLM was used in creating this pull request.
I directed and planned this change myself: choosing the issue, deciding to move the ASCII constraint to the S3 boundary instead of sanitising at upload, and reviewing and running every test. An LLM assistant (Claude Code) was used for roughly a third of the work — mainly drafting code, tests and parts of this description. I take full responsibility for the change.