Skip to content

Commit 60ab81c

Browse files
authored
blob support: decode binary database values as data URLs (#1005)
* feat: Enhanced BLOB support across all database backends - Add comprehensive BLOB support for all supported databases: * PostgreSQL: BYTEA columns with data URL conversion * MySQL/MariaDB: BLOB columns with data URL conversion * MSSQL: VARBINARY, BIGVARBINARY, BINARY, IMAGE columns * SQLite: BLOB columns with data URL conversion - Create shared data URL conversion functions to eliminate code duplication - Add comprehensive tests for all database types - Update CHANGELOG.md with detailed feature description - All blob data is now consistently converted to data URLs with base64 encoding - Cross-database compatibility ensures identical blob behavior across all backends - Comprehensive testing validates functionality across PostgreSQL, MySQL, MariaDB, MSSQL, and SQLite * update changelog * fmt * fix: Address clippy linter issues - Change blob function parameters from Vec<u8> to &[u8] to avoid unnecessary copying - Use inline format args in data URL construction - Update function calls to borrow parameters correctly - Maintain backward compatibility and functionality * feat: Add smart MIME type detection for BLOB data - Implement automatic MIME type detection based on file signatures (magic bytes) - Support common file formats: PNG, JPEG, GIF, BMP, WebP, SVG, PDF, DOCX, XLSX, PPTX, JSON, XML, ZIP - Automatic fallback to 'application/octet-stream' for unknown formats - Update CHANGELOG.md with comprehensive feature description - Add comprehensive tests for MIME type detection functionality - BLOB data now automatically returns appropriate data URLs: * PNG files: 'data:image/png;base64,...' * PDF files: 'data:application/pdf;base64,...' * SVG files: 'data:image/svg+xml;base64,...' * Unknown files: 'data:application/octet-stream;base64,...' - Improves user experience by providing correct MIME types for downloads and displays - Eliminates need for manual MIME type specification in most cases * refactor: Make MIME type detection more concise - Use bytes.starts_with() for cleaner magic byte detection - Remove verbose comments for each MIME type - Maintain same functionality with cleaner, more readable code - Reduce code duplication and improve maintainability * refactor: Improve MIME type detection with byte strings - Use byte string literals (b"string") for better readability - Remove UTF-8 parsing for text-based formats, use direct byte comparisons - Maintain same functionality with cleaner, more performant code - PNG: b"\x89PNG\r\n\x1a\n" instead of hex arrays - JPEG: b"\xFF\xD8" instead of [0xFF, 0xD8] - Text formats: Direct byte matching without String::from_utf8_lossy - Update all tests to use new byte string format * refactor: Extract MIME type detection into separate module - Create new mime_detection.rs module for better code organization - Move detect_mime_type() function and tests to dedicated module - Update sql_to_json.rs to import from mime_detection module - Remove unused import from functions.rs - Maintain same functionality with improved code structure Benefits: - Better separation of concerns - Improved code organization and maintainability - Easier to extend MIME detection in the future - Cleaner module boundaries File changes: - NEW: src/webserver/database/mime_detection.rs (MIME detection + tests) - MOD: src/webserver/database/mod.rs (add mime_detection module) - MOD: src/webserver/database/sql_to_json.rs (use mime_detection module) - MOD: src/webserver/database/sqlpage_functions/functions.rs (remove unused import) * refactor: Rename mime_detection.rs to blob_to_data_url.rs - Rename module from mime_detection to blob_to_data_url for better clarity - Follow project naming conventions (similar to sql_to_json.rs) - Update all imports and references to use new module name - Maintain same functionality with improved code organization File changes: - RENAMED: src/webserver/database/mime_detection.rs → src/webserver/database/blob_to_data_url.rs - MOD: src/webserver/database/mod.rs (update module declaration) - MOD: src/webserver/database/sql_to_json.rs (update import path) - MOD: src/webserver/database/sqlpage_functions/functions.rs (update import path) * move - Add #[must_use] attribute to detect_mime_type for better usage indication - Replace empty byte check with is_empty() for clarity - Update tests for improved readability with formatted assertions - Remove unnecessary blank lines in sql_to_json.rs Benefits: - Improved code clarity and maintainability - Enhanced test readability * remove old file * refactor: Update sql_to_json.rs to use new blob_to_data_url module - Replace direct call to vec_to_data_uri_value with updated import from blob_to_data_url - Remove deprecated vec_to_data_uri and vec_to_data_uri_value functions for cleaner code - Maintain existing functionality while improving code organization * refactor: Simplify MIME type detection logic in blob_to_data_url.rs - Remove unnecessary empty byte check and streamline conditions for text-based formats - Enhance readability by consolidating checks for XML and JSON formats - Maintain existing functionality while improving code clarity * clippy * Update documentation for BLOB support and data type handling - CHANGELOG.md : details on BLOB support and automatic MIME type detection - Add examples in extensions-to-sql.md illustrating data type conversions and JSON object structure - Update SQL examples in migrations to reflect new BLOB handling capabilities
1 parent 15c0d24 commit 60ab81c

File tree

9 files changed

+306
-23
lines changed

9 files changed

+306
-23
lines changed

CHANGELOG.md

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,15 @@
1111
- Since modals have their own url inside the page, you can now link to a modal from another page, and if you refresh a page while the modal is open, the modal will stay open.
1212
- modals now have an `open` parameter to open the modal automatically when the page is loaded.
1313
- New [download](https://sql-page.com/component.sql?component=download) component to let the user download files. The files may be stored as BLOBs in the database, local files on the server, or may be fetched from a different server.
14+
- **Enhanced BLOB Support**. You can now return binary data (BLOBs) directly to sqlpage, and it will automatically convert them to data URLs. This allows you to use database BLOBs directly wherever a link is expected, including in the new download component.
15+
- supports columns of type `BYTEA` (PostgreSQL), `BLOB` (MySQL, SQLite), `VARBINARY` and `IMAGE` (mssql)
16+
- Automatic detection of common file types based on magic bytes
17+
- This means you can use a BLOB wherever an image url is expected. For instance:
18+
```sql
19+
select 'list' as component;
20+
select username as title, avatar_blob as image_url
21+
from users;
22+
```
1423

1524
## v0.36.1
1625
- Fix regression introduced in v0.36.0: PostgreSQL money values showed as 0.0

docker-compose.yml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,11 @@
44
# DATABASE_URL='postgres://root:Password123!@localhost/sqlpage'
55
# DATABASE_URL='mssql://root:Password123!@localhost/sqlpage'
66
# DATABASE_URL='mysql://root:Password123!@localhost/sqlpage'
7+
8+
# Run for instance:
9+
# docker compose up postgres
10+
# and in another terminal:
11+
# DATABASE_URL='db_url' cargo test
712
services:
813
web:
914
build: { context: "." }

examples/official-site/extensions-to-sql.md

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -206,3 +206,52 @@ SET post_id = COALESCE($post_id, 0);
206206
-- Prepared statement (SQLite syntax)
207207
SELECT COALESCE(CAST(?1 AS TEXT), 0)
208208
```
209+
210+
# Data types
211+
212+
Each database has its own rich set of data types.
213+
The data modal in SQLPage itself is simpler, mainly composed of text strings and json objects.
214+
215+
### From the user to SQLPage
216+
217+
Form fields and URL parameters may contain arrays. These are converted to JSON strings before processing.
218+
219+
For instance, Loading `users.sql?user[]=Tim&user[]=Tom` will result in a single variable `$user` with the textual value `["Tim", "Tom"]`.
220+
221+
### From SQLPage to the database
222+
223+
SQLPage sends only text strings (`VARCHAR`) and `NULL`s to the database, since these are the only possible variable and function return values.
224+
225+
### From the database to SQLPage
226+
227+
Each row of data returned by a SQL query is converted to a JSON object before being passed to components.
228+
229+
- Each column becomes a key in the json object. If a row has two columns of the same name, they become an array in the json object.
230+
- Each value is converted to the closest JSON value
231+
- all number types map to json numbers, booleans to booleans, and `NULL` to `null`,
232+
- all text types map to json strings
233+
- date and time types map to json strings containing ISO datetime values
234+
- binary values (BLOBs) map to json strings containing [data URLs](https://developer.mozilla.org/en-US/docs/Web/URI/Reference/Schemes/data)
235+
236+
#### Example
237+
238+
The following PostgreSQL query:
239+
240+
```sql
241+
select
242+
1 as one,
243+
'x' as my_array, 'y' as my_array,
244+
now() as today,
245+
'<svg></svg>'::bytea as my_image;
246+
```
247+
248+
will result in the following JSON object being passed to components for rendering
249+
250+
```json
251+
{
252+
"one" : 1,
253+
"my_array" : ["x","y"],
254+
"today":"2025-08-30T06:40:13.894918+00:00",
255+
"my_image":"data:image/svg+xml;base64,PHN2Zz48L3N2Zz4="
256+
}
257+
```

examples/official-site/extensions-to-sql.sql

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,10 @@
11
select 'http_header' as component,
22
'public, max-age=300, stale-while-revalidate=3600, stale-if-error=86400' as "Cache-Control";
33

4-
select 'dynamic' as component, properties FROM example WHERE component = 'shell' LIMIT 1;
4+
select 'dynamic' as component, json_patch(json_extract(properties, '$[0]'), json_object(
5+
'title', 'SQLPage - Extensions to SQL'
6+
)) as properties
7+
FROM example WHERE component = 'shell' LIMIT 1;
58

69
-- Article by Matthew Larkin
710
select 'text' as component,

examples/official-site/sqlpage/migrations/65_download.sql

Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -104,7 +104,25 @@ select
104104
'
105105
## Serve an image stored as a BLOB in the database
106106
107-
In PostgreSQL, you can use the [encode(bytes, format)](https://www.postgresql.org/docs/current/functions-binarystring.html#FUNCTION-ENCODE) function to encode the file content as Base64.
107+
### Automatically detect the mime type
108+
109+
If you have a table with a column `content` that contains a BLOB
110+
(depending on the database, the type may be named `BYTEA`, `BLOB`, `VARBINARY`, or `IMAGE`),
111+
you can just return its contents directly, and SQLPage will automatically detect the mime type,
112+
and convert it to a data URL.
113+
114+
```sql
115+
select
116+
''download'' as component,
117+
content as data_url
118+
from document
119+
where id = $doc_id;
120+
```
121+
122+
### Customize the mime type
123+
124+
In PostgreSQL, you can use the [encode(bytes, format)](https://www.postgresql.org/docs/current/functions-binarystring.html#FUNCTION-ENCODE) function to encode the file content as Base64,
125+
and manually create your own data URL.
108126
109127
```sql
110128
select
Lines changed: 189 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,189 @@
1+
/// Detects MIME type based on file signatures (magic bytes).
2+
/// Returns the most appropriate MIME type for common file formats.
3+
#[must_use]
4+
pub fn detect_mime_type(bytes: &[u8]) -> &'static str {
5+
// PNG: 89 50 4E 47 0D 0A 1A 0A
6+
if bytes.starts_with(b"\x89PNG\r\n\x1a\n") {
7+
return "image/png";
8+
}
9+
// JPEG: FF D8
10+
if bytes.starts_with(b"\xFF\xD8") {
11+
return "image/jpeg";
12+
}
13+
// GIF87a/89a: GIF87a or GIF89a
14+
if bytes.starts_with(b"GIF87a") || bytes.starts_with(b"GIF89a") {
15+
return "image/gif";
16+
}
17+
// BMP: 42 4D
18+
if bytes.starts_with(b"BM") {
19+
return "image/bmp";
20+
}
21+
// WebP: RIFF....WEBP
22+
if bytes.starts_with(b"RIFF") && bytes.len() >= 12 && &bytes[8..12] == b"WEBP" {
23+
return "image/webp";
24+
}
25+
// PDF: %PDF
26+
if bytes.starts_with(b"%PDF") {
27+
return "application/pdf";
28+
}
29+
// ZIP: 50 4B 03 04
30+
if bytes.starts_with(b"PK\x03\x04") {
31+
// Check for Office document types in ZIP central directory
32+
if bytes.len() >= 50 {
33+
let central_dir = &bytes[30..bytes.len().min(50)];
34+
if central_dir.windows(6).any(|w| w == b"word/") {
35+
return "application/vnd.openxmlformats-officedocument.wordprocessingml.document";
36+
}
37+
if central_dir.windows(3).any(|w| w == b"xl/") {
38+
return "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet";
39+
}
40+
if central_dir.windows(4).any(|w| w == b"ppt/") {
41+
return "application/vnd.openxmlformats-officedocument.presentationml.presentation";
42+
}
43+
}
44+
return "application/zip";
45+
}
46+
47+
if bytes.starts_with(b"<?xml") {
48+
return "application/xml";
49+
}
50+
if bytes.starts_with(b"<svg") || bytes.starts_with(b"<!DOCTYPE svg") {
51+
return "image/svg+xml";
52+
}
53+
if bytes.starts_with(b"{") || bytes.starts_with(b"[") {
54+
return "application/json";
55+
}
56+
57+
"application/octet-stream"
58+
}
59+
60+
/// Converts binary data to a data URL string.
61+
/// This function is used by both SQL type conversion and file reading functions.
62+
/// Automatically detects common file types based on magic bytes.
63+
#[must_use]
64+
pub fn vec_to_data_uri(bytes: &[u8]) -> String {
65+
let mime_type = detect_mime_type(bytes);
66+
vec_to_data_uri_with_mime(bytes, mime_type)
67+
}
68+
69+
/// Converts binary data to a data URL string with a specific MIME type.
70+
/// This function is used by both SQL type conversion and file reading functions.
71+
#[must_use]
72+
pub fn vec_to_data_uri_with_mime(bytes: &[u8], mime_type: &str) -> String {
73+
let mut data_url = format!("data:{mime_type};base64,");
74+
base64::Engine::encode_string(
75+
&base64::engine::general_purpose::STANDARD,
76+
bytes,
77+
&mut data_url,
78+
);
79+
data_url
80+
}
81+
82+
/// Converts binary data to a data URL JSON value.
83+
/// This is a convenience function for SQL type conversion.
84+
#[must_use]
85+
pub fn vec_to_data_uri_value(bytes: &[u8]) -> serde_json::Value {
86+
serde_json::Value::String(vec_to_data_uri(bytes))
87+
}
88+
89+
#[cfg(test)]
90+
mod tests {
91+
use super::*;
92+
93+
#[test]
94+
fn test_detect_mime_type() {
95+
// Test empty data
96+
assert_eq!(detect_mime_type(&[]), "application/octet-stream");
97+
98+
// Test PNG
99+
assert_eq!(detect_mime_type(b"\x89PNG\r\n\x1a\n"), "image/png");
100+
101+
// Test JPEG
102+
assert_eq!(detect_mime_type(b"\xFF\xD8\xFF\xE0"), "image/jpeg");
103+
104+
// Test GIF87a
105+
assert_eq!(detect_mime_type(b"GIF87a"), "image/gif");
106+
107+
// Test GIF89a
108+
assert_eq!(detect_mime_type(b"GIF89a"), "image/gif");
109+
110+
// Test BMP
111+
assert_eq!(detect_mime_type(b"BM\x00\x00"), "image/bmp");
112+
113+
// Test PDF
114+
assert_eq!(detect_mime_type(b"%PDF-"), "application/pdf");
115+
116+
// Test SVG
117+
assert_eq!(
118+
detect_mime_type(b"<svg xmlns=\"http://www.w3.org/2000/svg\">"),
119+
"image/svg+xml"
120+
);
121+
122+
// Test XML (non-SVG)
123+
assert_eq!(
124+
detect_mime_type(b"<?xml version=\"1.0\"?><root><data>test</data></root>"),
125+
"application/xml"
126+
);
127+
128+
// Test JSON
129+
assert_eq!(
130+
detect_mime_type(b"{\"key\": \"value\"}"),
131+
"application/json"
132+
);
133+
134+
// Test ZIP
135+
assert_eq!(detect_mime_type(b"PK\x03\x04"), "application/zip");
136+
137+
// Test unknown data
138+
assert_eq!(
139+
detect_mime_type(&[0x00, 0x01, 0x02, 0x03]),
140+
"application/octet-stream"
141+
);
142+
}
143+
144+
#[test]
145+
fn test_vec_to_data_uri() {
146+
// Test with empty bytes
147+
let result = vec_to_data_uri(&[]);
148+
assert_eq!(result, "data:application/octet-stream;base64,");
149+
150+
// Test with simple text
151+
let result = vec_to_data_uri(b"Hello World");
152+
assert_eq!(
153+
result,
154+
"data:application/octet-stream;base64,SGVsbG8gV29ybGQ="
155+
);
156+
157+
// Test with binary data
158+
let binary_data = [0, 1, 2, 255, 254, 253];
159+
let result = vec_to_data_uri(&binary_data);
160+
assert_eq!(result, "data:application/octet-stream;base64,AAEC//79");
161+
}
162+
163+
#[test]
164+
fn test_vec_to_data_uri_with_mime() {
165+
// Test with custom MIME type
166+
let result = vec_to_data_uri_with_mime(b"Hello", "text/plain");
167+
assert_eq!(result, "data:text/plain;base64,SGVsbG8=");
168+
169+
// Test with image MIME type
170+
let result = vec_to_data_uri_with_mime(&[255, 216, 255], "image/jpeg");
171+
assert_eq!(result, "data:image/jpeg;base64,/9j/");
172+
173+
// Test with empty bytes and custom MIME
174+
let result = vec_to_data_uri_with_mime(&[], "application/json");
175+
assert_eq!(result, "data:application/json;base64,");
176+
}
177+
178+
#[test]
179+
fn test_vec_to_data_uri_value() {
180+
// Test that it returns a JSON string value
181+
let result = vec_to_data_uri_value(b"test");
182+
match result {
183+
serde_json::Value::String(s) => {
184+
assert_eq!(s, "data:application/octet-stream;base64,dGVzdA==");
185+
}
186+
_ => panic!("Expected String value"),
187+
}
188+
}
189+
}

src/webserver/database/mod.rs

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
pub mod blob_to_data_url;
12
mod connect;
23
mod csv_import;
34
pub mod execute_queries;

0 commit comments

Comments
 (0)