You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This page covers the bring-your-own-data API: computing embeddings from imagery you already have, instead of provider-fetched imagery.
4
-
5
-
Related pages: [API: Embedding](api_embedding.md), [API: Specs and Data Structures](api_specs.md).
3
+
You already have imagery — patches from your own dataset, exported GeoTIFFs, a
4
+
training cube on disk. This page covers the bring-your-own-data API: computing
5
+
embeddings from that imagery directly, with no provider fetch and no provider
6
+
auth.
7
+
8
+
Related pages: [API: Embedding](api_embedding.md), [API: Specs and Data
9
+
Structures](api_specs.md), [Spatial ROI Handling](spatial_roi.md).
10
+
11
+
!!! abstract "The one idea"
12
+
You **register** each piece of imagery once as a `UserData` — the pixels
13
+
plus everything that describes them: which collection they came from, one
14
+
band name per channel, and where/when they were acquired. From then on,
15
+
embedding takes only a model name. The declaration, not the array shape,
16
+
is the contract: a model whose bands your declaration covers gets its
17
+
channels sliced out automatically; a model it cannot satisfy is **refused
18
+
with the exact reason**, never served silently wrong data.
6
19
7
20
---
8
21
9
-
## Concept
10
-
11
-
The flow has two steps:
12
-
13
-
1.**Register** each piece of imagery as a `UserData` — the pixels plus everything that describes them: which collection they came from, one band name per channel, and where/when they were acquired.
14
-
2.**Embed** by naming a model: `get_embedding_from_data("galileo", data)`. Nothing else is needed — the declaration already carries the full context.
22
+
## The two-step flow
15
23
16
-
Every on-the-fly model declares an input sensor (a collection plus an ordered band list). Your declaration is matched against it:
17
-
18
-
-**Superset data is accepted**: if your declaration covers all bands the model needs, the needed channels are sliced out and reordered automatically. One 12-band Sentinel-2 L2A cube can serve models that need 3, 6, or 10 of those bands.
19
-
-**Insufficient data is refused**: a collection mismatch (e.g. S2 data offered to a MODIS model) or a missing band raises `ModelError` naming exactly what is missing. Precomputed models (`tessera`, `gse`, `copernicus`) are always refused — they have no imagery input.
24
+
```mermaid
25
+
flowchart LR
26
+
REG["1. Register\nUserData(pixels, collection,\nbands, spatial, temporal)"] --> MATCH["2. Match\ndeclaration vs the model's\ninput sensor"]
27
+
MATCH -->|covers all bands| SLICE["slice + reorder\nchannels to model order"] --> EMB["Embedding"]
28
+
MATCH -->|"missing band /\nwrong collection"| REFUSE["ModelError\nnaming what is missing"]
29
+
```
20
30
21
-
Values must be **raw provider units** for the declared collection (e.g. Sentinel-2 L2A surface-reflectance DN in `0..10000`), exactly what a provider fetch would return. Per-model normalization stays inside each embedder, so you never need to know a model's normalization. Data that looks already normalized (max ≤ 1.5 on an S2 declaration) triggers a warning.
31
+
Every on-the-fly model declares an input sensor (a collection plus an ordered
32
+
band list), and your declaration is matched against it per request.
33
+
34
+
**Superset data is accepted.** If your declaration covers all bands the model
35
+
needs, the needed channels are sliced out and reordered automatically. One
36
+
12-band Sentinel-2 L2A cube serves models that need 3, 6, or 10 of those bands
37
+
— `galileo` takes its 10, an RGB model takes `B4/B3/B2`, all from the same
38
+
declaration. Band aliases resolve the same way provider fetches resolve them
39
+
(`"RED"` → `"B4"`, `"NIR_NARROW"` → `"B8A"`, …).
40
+
41
+
**Insufficient data is refused.** A collection mismatch (e.g. S2 data offered
42
+
to a MODIS model) or a missing band raises `ModelError` naming exactly what is
43
+
missing. Precomputed models (`tessera`, `gse`, `copernicus`) are always
44
+
refused — they have no imagery input. Call
45
+
[`list_models_for_data`](#list_models_for_data) to see the verdict for every
46
+
catalog model up front.
47
+
48
+
!!! warning "Values must be raw provider units"
49
+
Pass exactly what a provider fetch would return — for Sentinel-2 L2A that
50
+
is surface-reflectance DN in `0..10000`, **not** reflectance rescaled to
51
+
`0..1`. Per-model normalization stays inside each embedder, so you never
52
+
need to know a model's normalization; but data that looks already
53
+
normalized (max ≤ 1.5 on an S2 declaration) triggers a `UserWarning`,
54
+
because it would otherwise produce silently wrong embeddings.
22
55
23
56
---
24
57
@@ -28,30 +61,48 @@ Values must be **raw provider units** for the declared collection (e.g. Sentinel
28
61
from rs_embed import UserData
29
62
30
63
UserData(
31
-
data: np.ndarray, # [C,H,W] or [T,C,H,W], raw provider values
32
-
collection: str, # e.g. "COPERNICUS/S2_SR_HARMONIZED" or alias "s2"
33
-
spatial: SpatialSpec |None=None, # where the imagery is (PointBuffer / BBox)
64
+
data: np.ndarray, # [C,H,W] or [T,C,H,W], raw provider values
65
+
collection: str, # e.g. "COPERNICUS/S2_SR_HARMONIZED" or alias "s2"
66
+
spatial: SpatialSpec |None=None, # where the imagery is (PointBuffer / BBox)
34
67
bands: tuple[str, ...] |None=None, # one band name per channel; None = canonical order
35
68
temporal: TemporalSpec |None=None, # when the imagery was acquired
36
-
scale_m: int|None=None, # optional nominal pixel size, provenance only
69
+
scale_m: int|None=None, # optional nominal pixel size, provenance only
37
70
)
38
71
```
39
72
40
-
-**`spatial` is optional but supply it whenever you have it** — models whose forward pass conditions on geometry (lat/lon or GSD encodings: `clay`, `prithvi`) refuse declarations without it, because coordinates are never fabricated. All other models accept ungeoreferenced data; they just lose location provenance in the metadata. `list_models_for_data` on a spatial-less declaration reports which models refuse for this reason.
41
-
-**`temporal` travels with the data** — it is the acquisition time of *this* imagery, so it lives here rather than on the API call. Models that condition on time read it; omitting it falls back to the package default window.
42
-
-**`bands` may be omitted only for the canonical case**: an S2 L2A declaration with exactly 12 channels defaults to the canonical order `B1, B2, B3, B4, B5, B6, B7, B8, B8A, B9, B11, B12`. Any other channel count, order, or collection must name its bands — band identity is never guessed from channel count.
43
-
44
-
Collection aliases: `"s2"` / `"sentinel-2"` / `"s2-l2a"` → `COPERNICUS/S2_SR_HARMONIZED`; `"s1"` / `"sentinel-1"` → `COPERNICUS/S1_GRD`. Full collection ids pass through unchanged. Band aliases resolve the same way provider fetches resolve them (`"RED"` → `"B4"`, `"NIR_NARROW"` → `"B8A"`, …).
45
-
46
-
Multi-frame `[T,C,H,W]` arrays are only meaningful for time-series models (galileo, prithvi, olmoearth, anysat, agrifm); single-frame models reject them.
73
+
**`collection` names the sensor product — and thereby the units.** Short
74
+
aliases resolve to full ids: `"s2"` / `"sentinel-2"` / `"s2-l2a"` →
A complete example, starting from a file on disk. Say you have a Sentinel-2 L2A patch saved as a GeoTIFF — 12 bands in the canonical order `B1..B8, B8A, B9, B11, B12`, raw surface-reflectance DN (`0..10000`, i.e. exactly as downloaded, not rescaled to `0..1`):
103
+
A complete example, starting from a file on disk. Say you have a Sentinel-2
104
+
L2A patch saved as a GeoTIFF — 12 bands in the canonical order, raw
105
+
surface-reflectance DN:
55
106
56
107
```python
57
108
import rasterio # example only; not an rs-embed dependency
@@ -69,31 +120,32 @@ with rasterio.open("maize_field_2022.tif") as src:
69
120
70
121
# 2. Register the imagery: the pixels plus everything that describes them.
# 3. Embed — just name the model. Band selection is automatic: galileo slices
82
-
# out its 10 bands, an RGB model would slice B4/B3/B2, all from this one
83
-
# declaration.
129
+
# 3. Embed — just name the model.
84
130
emb = get_embedding_from_data("galileo", data)
85
131
86
132
print(emb.data.shape) # pooled feature vector, shape [D]
87
133
print(emb.meta["user_input"]) # which bands/channels were actually used
88
134
```
89
135
90
-
If your data is already a numpy array (e.g. one sample from a training dataset), skip step 1 — anything `[C,H,W]` in raw provider units works as `data=`.
136
+
If your data is already a numpy array (e.g. one sample from a training
137
+
dataset), skip step 1 — anything `[C,H,W]` in raw provider units works as
138
+
`data=`.
91
139
92
-
Returns one `Embedding`; `meta["user_input"]` records the declaration and the channel selection actually fed to the model (`declared_bands`, `bands_used`, `channel_indices`).
140
+
Returns one `Embedding`. `meta["user_input"]` records the declaration and the
141
+
channel selection actually fed to the model (`declared_bands`, `bands_used`,
142
+
`channel_indices`), and `meta["input_prep"]` records how the array was sized
143
+
(see [Input size handling](#input-size-handling)).
93
144
94
145
### get_embeddings_batch_from_data
95
146
96
-
Continuing the example above — a whole directory of patches into one feature matrix:
147
+
Continuing the example above — a whole directory of patches into one feature
X = np.stack([e.data for e in embs]) # [N, D] — ready for sklearn, clustering, ...
121
173
```
122
174
123
-
Each item is matched independently and carries its own `spatial` / `temporal`, so one batch can mix locations, dates, and even band orders. Items sharing a temporal are dispatched together (models with true batching benefit); results always come back in input order.
175
+
Each item is matched independently and carries its own `spatial` /
176
+
`temporal`, so one batch can mix locations, dates, and even band orders.
177
+
Items sharing a temporal are dispatched together (models with true batching
178
+
benefit); results always come back in input order.
124
179
125
-
`batch_size` caps how many items reach one model forward batch — set a small value to fit a small GPU. Models keep their own per-device internal default (e.g. clay 32 on CUDA / 4 on CPU) as a further cap, so `batch_size` lowers but does not raise a model's forward batch; to raise it, use the model's `RS_EMBED_<MODEL>_BATCH_SIZE` environment variable.
180
+
!!! tip "Fitting your GPU with `batch_size`"
181
+
`batch_size` caps how many items reach one model forward batch — set a
182
+
small value for a small GPU. Models keep their own per-device internal
183
+
default (e.g. `clay` 32 on CUDA / 4 on CPU) as a further cap, so
184
+
`batch_size` lowers but does not raise a model's forward batch; to raise
185
+
it, use the model's `RS_EMBED_<MODEL>_BATCH_SIZE` environment variable.
Runs the same matching against every catalog model without loading weights. Each entry has `model`, `compatible`, `bands_used`, and `reason` (why the model is incompatible).
196
+
Runs the same matching against every catalog model without loading weights.
197
+
Each entry has `model`, `compatible`, `bands_used`, and `reason` (why the
198
+
model is incompatible).
137
199
138
200
---
139
201
140
202
## Input size handling
141
203
142
-
User data follows the package-wide `input_prep` policy, defaulting to **`"tile"`** — the same fairness semantics as the provider-fetch path:
143
-
144
-
-**Larger than the model's input size**: the array is cut into model-native tiles at its own resolution, each tile embedded, and the outputs stitched. Every model sees the full detail regardless of its input size — a 256×256 patch reaches clay as one 256-px pass and galileo as a 4×4 grid of 64-px tiles, instead of galileo silently losing 15/16 of the pixels to a resize. Pass `input_prep="resize"` to opt into one-step downsampling instead (faster, lossy; `meta["input_prep"]` records which path ran).
145
-
-**At or below the model's input size**: nothing to tile — the array goes straight to the embedder, which resizes up if needed (prithvi can pad instead via `RS_EMBED_PRITHVI_PREP=pad`). Very small arrays (e.g. 16×16 into a 224 model) run fine but carry only the information your pixels have — expect weak embeddings below roughly half the model's input size.
146
-
-**Non-square**: tiling handles rectangles cleanly (edge tiles are padded, outputs cropped back). Under `"resize"`, a plain resize distorts the aspect ratio — another reason to keep the tile default for non-square patches.
147
-
-**Flexible-size models**: `olmoearth` (FlexiViT) accepts any patch-divisible input size, so your data is consumed **natively by default** — a 512×512 patch runs as one seamless pass instead of a 2×2 tile mosaic, with no configuration needed (the side length is snapped up to a patch multiple). A native pass beyond 512 px emits a warning: attention cost grows quadratically with token count, so very large patches can be slow or exhaust GPU memory — tile them via `input_prep=InputPrepSpec(mode="tile", tile_size=...)` or downsample via `"resize"`. Passing an explicit `image_size` in model kwargs restores fixed-size behavior (larger inputs tile at it).
148
-
149
-
Rule of thumb: any patch at the model's native scale (10 m for the S2 models) is handled faithfully under the tile default; patches at or below the model's input size are also the cheapest (single forward pass).
204
+
User data follows the package-wide `input_prep` policy, defaulting to
205
+
**`"tile"`** — the same fairness semantics as the provider-fetch path (see
206
+
[Spatial ROI Handling](spatial_roi.md) and the *Input size* column in
207
+
[Models Overview](models.md)).
208
+
209
+
**Larger than the model's input size — tiled, not squashed.** The array is cut
210
+
into model-native tiles at its own resolution, each tile embedded, and the
211
+
outputs stitched. Every model sees the full detail regardless of its input
212
+
size: a 256×256 patch reaches `clay` as one 256-px pass and `galileo` as a
213
+
4×4 grid of 64-px tiles, instead of `galileo` silently losing 15/16 of the
214
+
pixels to a resize. Pass `input_prep="resize"` for one-step downsampling
215
+
instead (faster, lossy); `meta["input_prep"]` records which path ran.
216
+
217
+
**At or below the model's input size — straight through.** There is nothing to
218
+
tile: the array goes to the embedder, which resizes up if needed (`prithvi`
219
+
can pad instead via `RS_EMBED_PRITHVI_PREP=pad`). Very small arrays run fine
220
+
but carry only the information your pixels have — expect weak embeddings below
221
+
roughly half the model's input size.
222
+
223
+
**Non-square — tiling handles it cleanly.** Edge tiles are padded and the
224
+
outputs cropped back. Under `"resize"`, a plain resize distorts the aspect
225
+
ratio — another reason to keep the tile default for non-square patches.
226
+
227
+
!!! note "Flexible-size models run your data natively"
228
+
`olmoearth` (FlexiViT) accepts any patch-divisible input size, so your
229
+
data is consumed **natively by default** — a 512×512 patch runs as one
230
+
seamless pass instead of a 2×2 tile mosaic, with no configuration needed
231
+
(the side length is snapped up to a patch multiple). Passing an explicit
232
+
`image_size` in model kwargs restores fixed-size behavior (larger inputs
233
+
tile at it).
234
+
235
+
!!! warning "Very large native passes"
236
+
A flexible-size native pass beyond **512 px** emits a warning: attention
237
+
cost grows quadratically with token count, so very large patches can be
238
+
slow or exhaust GPU memory. Tile them via
239
+
`input_prep=InputPrepSpec(mode="tile", tile_size=...)` or downsample via
240
+
`"resize"`.
150
241
151
242
---
152
243
153
244
## Refusal semantics
154
245
155
246
| Situation | Result |
156
-
|---|---|
247
+
|--------- |------ |
157
248
| Declaration covers all model bands | Accepted; channels sliced/reordered |
158
249
| Missing band(s) |`ModelError` listing the missing band names |
159
250
| Collection mismatch |`ModelError` (raw units differ across collections) |
160
251
| Precomputed model |`ModelError` (no imagery input) |
161
252
| Channel count ≠ declared bands |`SpecError` from `UserData.validate()`|
162
253
|`bands=None` outside the canonical case |`SpecError` (declare bands explicitly) |
163
-
| Missing `spatial` on a georef-conditioned model (clay, prithvi) |`ModelError` (coordinates are never fabricated) |
254
+
| Missing `spatial` on a georef-conditioned model (`clay`, `prithvi`) |`ModelError` (coordinates are never fabricated) |
0 commit comments