Commit 1a6c167
committed
feat: Add separate filepath_default for filepath references
Filepath storage is NOT part of the Object-Augmented Schema - it only
provides references to externally-managed files. Allow separate default
configuration for filepath references vs integrated storage.
Configuration:
- stores.default - for integrated storage (<blob>, <object>, <npy>, <attach>)
- stores.filepath_default - for filepath references (<filepath>)
This allows:
- Integrated storage on S3 or fast filesystem
- Filepath references to acquisition files on NAS or different location
Example:
{
"stores": {
"default": "main",
"filepath_default": "raw_data",
"main": {
"protocol": "s3",
"bucket": "processed-data",
"location": "lab-project"
},
"raw_data": {
"protocol": "file",
"location": "/mnt/nas/acquisition"
}
}
}
Usage:
- data : <blob> # Uses stores.default (main)
- arrays : <object> # Uses stores.default (main)
- raw : <filepath> # Uses stores.filepath_default (raw_data)
- raw : <filepath@acq> # Explicitly names store (overrides default)
Changes:
- settings.py: Add use_filepath_default parameter to get_store_spec()
- builtin_codecs.py: FilepathCodec uses use_filepath_default=True
- test_settings.py: Add 3 tests for filepath_default behavior
- settings.py: Update template to include filepath_default example
Architectural rationale:
- Hash/schema storage: integrated into OAS, DataJoint manages lifecycle
- Filepath storage: references only, users manage lifecycle
- Different defaults reflect this fundamental distinction1 parent 79f60b2 commit 1a6c167
File tree
3 files changed
+97
-7
lines changed- src/datajoint
- tests/unit
3 files changed
+97
-7
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
815 | 815 | | |
816 | 816 | | |
817 | 817 | | |
818 | | - | |
| 818 | + | |
| 819 | + | |
819 | 820 | | |
820 | 821 | | |
821 | 822 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
335 | 335 | | |
336 | 336 | | |
337 | 337 | | |
338 | | - | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
339 | 341 | | |
340 | 342 | | |
341 | 343 | | |
342 | 344 | | |
343 | 345 | | |
344 | 346 | | |
345 | | - | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
346 | 352 | | |
347 | 353 | | |
348 | 354 | | |
| |||
356 | 362 | | |
357 | 363 | | |
358 | 364 | | |
359 | | - | |
360 | | - | |
361 | | - | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
| 368 | + | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
362 | 379 | | |
363 | | - | |
| 380 | + | |
| 381 | + | |
364 | 382 | | |
365 | 383 | | |
366 | 384 | | |
| |||
778 | 796 | | |
779 | 797 | | |
780 | 798 | | |
| 799 | + | |
781 | 800 | | |
782 | 801 | | |
783 | 802 | | |
784 | 803 | | |
785 | 804 | | |
786 | 805 | | |
787 | 806 | | |
| 807 | + | |
| 808 | + | |
| 809 | + | |
| 810 | + | |
788 | 811 | | |
789 | 812 | | |
790 | 813 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
375 | 375 | | |
376 | 376 | | |
377 | 377 | | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
| 386 | + | |
| 387 | + | |
| 388 | + | |
| 389 | + | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
| 398 | + | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
| 403 | + | |
| 404 | + | |
| 405 | + | |
| 406 | + | |
| 407 | + | |
| 408 | + | |
| 409 | + | |
| 410 | + | |
| 411 | + | |
| 412 | + | |
| 413 | + | |
| 414 | + | |
| 415 | + | |
| 416 | + | |
| 417 | + | |
| 418 | + | |
| 419 | + | |
| 420 | + | |
| 421 | + | |
| 422 | + | |
| 423 | + | |
| 424 | + | |
| 425 | + | |
| 426 | + | |
| 427 | + | |
| 428 | + | |
| 429 | + | |
| 430 | + | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
| 440 | + | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
378 | 444 | | |
379 | 445 | | |
380 | 446 | | |
| |||
0 commit comments