Improve attachment file type recognision#7397
Open
CrafterFB wants to merge 8 commits intooobabooga:devfrom
Open
Improve attachment file type recognision#7397CrafterFB wants to merge 8 commits intooobabooga:devfrom
CrafterFB wants to merge 8 commits intooobabooga:devfrom
Conversation
…agic numbers (file content)
…esent on windows)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Checklist:
Background
While testing vision for my install i attached a JPEG without extension (so
imageinstead ofimage.jpg).The AI stated that it didn't see an image.
New Behavior
File type recognision
The handling of attachments is now based on mime types.
It first tries to infer the type based on the file path (file name) using the python standard library mimetypes.
If it returned
None,filetypeis used to infer the type based on the magic number (file content).The branching in
chat.pynow is based on the found mime type.New supported file types
Since all image attachments are wrapped in an
Imageobject usingpillow, the bottlenecks for image type support arepillowand the detection system.The new supported types are whitelisted in the
is_mime_type_vision_supportedinimage_utils.py.The new supported types include all previously supported types,
tiffandavifas well as other lesser knows types (seeis_mime_type_vision_supportedfor the full list.All new mime types are for file types supported by pillow and are (semi) official i.e. are listed in
/usr/share/mimeor on the mime type registry.Other changes
Any place that the type of attachments was checked now uses
is_mime_type_vision_supportedto test for image attachments.The
<img />element of the attachments in the ui now includes the detected mime type.New Dependency:
filetypeThe library
filetype(pypi) was chosen because it is portable (pure python implementation).All
requirements.txtfiles have been updated with the new dependency.python-magicwas also considered but since windows doesn't ship withlibmagicand the version with binaries included isn't available for modern python versions, it was replaced withfiletype.Version without
filetypeCommit 4ceb426 has most of the new changes except the fallback if the mime type couldn't be found based on the path.
Testing
The testing was done on fedora kde 43 with the cpu-only build and Qwen3VL-2B-Instruct-Q8_0 with mmproj-Qwen3VL-2B-Instruct-Q8_0.
The AI was asked "what is in this file" and the file was attached.
I used the same easy to recognize base png image converted to various file types.
If the AI correctly identified what was in the image it was a success.
The following file types where tested:
Those include all previously supported types, the two i saw as reasonably important and a random selection of the unusual types.
It worked correctly with most types on the first try. Although:
application/octet-stream(raw binary) and thus did't work. See the next Section on why they are still included in the white list.Other Tests (all successfull)
Tested Failure cases
fallback to previous behavior, the file is attempted to be opened with utf-8 and often fails, an error is printed in the server console and the file is not attached.
fix: add type to the whitelist
fallback to previous behavior, the file is attempted to be opened with utf-8 and often fails, an error is printed in the server console and the file is not attached.
the file is attempted to be opened with pillow and fails, an error is printed in the server console and the file is not attached. The error is a different one than previously, but the result is the same.
The changes made are a strict improvement functionality wise with all failure cases falling back to behavior that is similar to the behavior before.