feat: Vendor BIDS loader and lazy loading for complete consumption pipeline

[The-Obstacle-Is-The-Way]

Summary

This repo currently provides a production-only pipeline (BIDS → HuggingFace Hub).

We are missing consumption features that would make this package a complete solution for neuroimaging researchers.

Installation Note

This package is installed as a git dependency (not on PyPI):

uv add git+https://github.com/The-Obstacle-Is-The-Way/neuroimaging-go-brrrr.git
# or
pip install git+https://github.com/The-Obstacle-Is-The-Way/neuroimaging-go-brrrr.git

Current State

Feature	Status
Production (BIDS → Hub upload)	✅ Complete
Hub consumption (load_dataset("hugging-science/..."))	✅ Works (via upstream datasets)
Local BIDS consumption (load_dataset('bids', data_dir='/path'))	❌ Missing
Memory-efficient lazy loading for NIfTI	❌ Missing

Pending Upstream PRs (Unlikely to Merge)

These features exist as open PRs on huggingface/datasets but are unlikely to be prioritized:

• #7886 - BIDS dataset loader
• #7887 - Lazy loading for NIfTI (memory fix)

Proposal

Vendor the code from PRs #7886 and #7887 into this repo so that:

1. Installing this package provides a complete neuroimaging toolkit
2. Users don't have to wait for upstream merges (which may never happen)
3. This repo becomes the canonical extension for neuroimaging on HuggingFace

Implementation Plan

1. Add src/bids_hub/loader/bids.py - BIDS loader from PR #7886
2. Add lazy loading wrapper from PR #7887
3. Expose via bids_hub.load_bids() or similar API
4. Update ARCHITECTURE.md to document consumption pipeline

References

• Gap analysis: docs/to_do/consumption-pipeline-gap.md

[The-Obstacle-Is-The-Way]

Update: Clarification on BIDS Loader Priority

After deeper analysis, here's our updated understanding:

BIDS Loader Follows Established Patterns

The BIDS loader follows the same pattern as existing huggingface/datasets packaged modules:

src/datasets/packaged_modules/
├── imagefolder/    ← load_dataset('imagefolder', data_dir=...)
├── audiofolder/    ← load_dataset('audiofolder', data_dir=...)
├── videofolder/    ← load_dataset('videofolder', data_dir=...)
├── niftifolder/    ← load_dataset('niftifolder', data_dir=...)
├── bids/           ← load_dataset('bids', data_dir=...)  # PR #7886

All of these are local folder → Dataset API convenience loaders. None preserve folder structure on Hub (Hub uploads are flattened to Parquet).

Priority Assessment

Loader	User Base	Priority
imagefolder	Millions (CV)	High
audiofolder	Hundreds of thousands	Medium
niftifolder	Thousands (neuroimaging)	Low
bids	Hundreds (BIDS-specific)	Very Low

BIDS loader is valid but niche. It's useful for researchers who download BIDS from OpenNeuro and want HF Dataset API locally - but that's a small audience.

Recommendation

• Lazy loading (PR #7887): Still valuable for Hub consumption - keep in scope
• BIDS loader (PR #7886): Valid feature, but low priority for vendoring. We can vendor it if demand arises, but it's not blocking any workflows since Hub consumption already works.

The upstream PR can stay open - if HuggingFace maintainers want it, they'll merge it. We don't need to vendor it here unless there's specific demand.