Skip to content

Commit 38844f1

Browse files
committed
Highlight no hidden tables - key architectural difference
- Legacy attach@store and filepath@store use hidden ~external_* tables - New object type stores all metadata inline in JSON column - Benefits: simpler schema, self-contained records, easier debugging - No reference counting complexity
1 parent 6cd9b9c commit 38844f1

File tree

1 file changed

+18
-1
lines changed

1 file changed

+18
-1
lines changed

docs/src/design/tables/file-type-spec.md

Lines changed: 18 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1003,11 +1003,28 @@ azure = ["adlfs"]
10031003
| Store config | Per-attribute | Per-attribute | Per-pipeline |
10041004
| Path control | DataJoint | User-managed | DataJoint |
10051005
| DB column | binary(16) UUID | binary(16) UUID | JSON |
1006+
| Hidden tables | Yes (external) | Yes (external) | **No** |
10061007
| Backend | File/S3 only | File/S3 only | fsspec (any) |
10071008
| Partitioning | Hash-based | User path | Configurable |
1008-
| Metadata | External table | External table | Inline JSON |
1009+
| Metadata storage | External table | External table | Inline JSON |
10091010
| Deduplication | By content | By path | None |
10101011

1012+
### No Hidden Tables
1013+
1014+
A key architectural difference: the `object` type does **not** use hidden external tables.
1015+
1016+
The legacy `attach@store` and `filepath@store` types store a UUID in the table column and maintain a separate hidden `~external_*` table containing:
1017+
- File paths/keys
1018+
- Checksums
1019+
- Size information
1020+
- Reference counts
1021+
1022+
The `object` type eliminates this complexity by storing all metadata **inline** in the JSON column. This provides:
1023+
- **Simpler schema** - no hidden tables to manage or migrate
1024+
- **Self-contained records** - all information in one place
1025+
- **Easier debugging** - metadata visible directly in queries
1026+
- **No reference counting** - each record owns its object exclusively
1027+
10111028
### Legacy Type Deprecation
10121029

10131030
The existing `attach@store` and `filepath@store` types will be:

0 commit comments

Comments
 (0)