Determine the technical requirements for the embedding.

[MathewNWSH]

Input?

• Should we generate chips (tensors) of size (1, C, 224, 224) and then pass them to the model one by one, or should we stack them into a batch and pass the batches? If yes, should a batch include multitemporal data for one location, or can it contain data gathered from many places and times?
• or it's just the matter of how many tensors can be passed to used gpu?

in below example, IBM is passing 1 tensor of batch size equal to one per iteration.

# Load input data
examples = [
    '38D_378R_2_3.tif',
    '282D_485L_3_3.tif',
    '433D_629L_3_1.tif',
    '637U_59R_1_3.tif',
    '609U_541L_3_0.tif',
]

# Select example between 0 and 4
file = examples[0]

# Define modalities
modalities = ['S2L2A', 'S1RTC', 'DEM', 'LULC', 'NDVI']
data = {m: rxr.open_rasterio(f'../examples/{m}/{file}') for m in modalities}
# Tensor with shape [B, C, 224, 224]
data = {
    k: torch.Tensor(v.values, device='cpu').unsqueeze(0)
    for k, v in data.items()
}
# Run any-to-any generation (this can take a while without a GPU, consider reducing timesteps for faster inference)
outputs = {}
for m in modalities:
    print(f'Processing {m}')
    out_modalities = modalities[:]
    out_modalities.remove(m)
    
    # Init model
    model = FULL_MODEL_REGISTRY.build(
        'terramind_v1_base_generate',
        modalities=[m],
        output_modalities=out_modalities,
        pretrained=True,
        standardize=True,
    )
    _ = model.to(device)
    
    input = data[m].clone().to(device)
    with torch.no_grad():
      generated = model(input, verbose=True, timesteps=10)
    outputs[m] = generated

[MathewNWSH]

The other thing is that if chips can overlap each-other?

[MathewNWSH]

another thing is:
should we only generate e,beddings for s2l2a or for all of these guys:
['S2L2A', 'S1RTC', 'DEM', 'LULC', 'NDVI']

[mkluczek]

Hi @MathewNWSH,
The embeddings for TerraMind comes in 16x16 patches so you don't need prepare images into chips like 224, you need just give an image which you can split by 16x16. And then you get as output: [batch_size, number_of_patches, embedding_size].

from terratorch import BACKBONE_REGISTRY

print([model_name for model_name in BACKBONE_REGISTRY if "terramind" in model_name])

You will get:
['terratorch_terramind_v1_base', 'terratorch_terramind_v1_base_tim', 'terratorch_terramind_v01_base', 'terratorch_terramind_v1_large', 'terratorch_terramind_v1_large_tim']

So if you want work with all modalities you should use models with 'tim' which means Thinking in Modalities (please look at the paper)

[mkluczek]

backbone = BACKBONE_REGISTRY.build(
    "terratorch_terramind_v1_large",
    modalities=["S2L1C"],  
    pretrained=True
)
backbone.eval()

path = "/home/eouser/PROCESSING/S2B_20240804_10m_all_bands.tif"
with rasterio.open(path) as src:
    img = src.read() 

tile_size = 2048
img_tile = img[:, :tile_size, :tile_size]

if img_tile.max() > 1: 
    img_tile /= img_tile.max()

img_tensor = torch.from_numpy(img_tile).unsqueeze(0)  

image_dict = {"S2L1C": img_tensor}

with torch.no_grad():
    outputs = backbone(image_dict)  
    final_embedding = outputs[-1]