Use `load_annotation_index=False`

[ilan-gold]

Description of feature

See https://anndata.readthedocs.io/en/stable/generated/anndata.experimental.read_lazy.html#anndata.experimental.read_lazy but I strongly doubt you guys need to materialize the string index into memory. Should save some io

[dschaub95]

Just noticed that the note here might be wrong: https://anndata.readthedocs.io/en/latest/tutorials/notebooks/read_lazy.html. Currently having access to the var is the only way for the LLM to get information about gene names. If I understand correctly that would no longer be possible or would you insert the var_names at run time? We could also offer access to var_names directly in the view_raw_data tool, would need to test if the LLM understands this option.

[ilan-gold]

Ah ok @dschaub95 then you do need to materialize a string index into memory. In that case, I would probably recommend:

adata.var.index = adata.var.true_index

Basically the "real" index is in true_index.

I was indeed in my head referring only to the obs index. But you're definitely right. We don't insert at run time.

[dschaub95]

Ok seems like this leads also to some problems when performing this operation: dataset2d[columns].to_memory(). Does load_annotation_index=False prevent loading of indices for any dataframe in the anndata or just obs and var? Right now I am considering to leave it as it is if it leads to further complications in the code, not sure if that's worth the small gain in efficiency.

[ilan-gold]

Ok seems like this leads also to some problems when performing this operation: dataset2d[columns].to_memory()

Can you post a stack trace? In theory it would prevent nested dataframes as well i.e., in obsm. I could see this just being a bug actually looking at the __getitem__ code.

[dschaub95]

Code:

adata = ad.experimental.read_lazy(path, load_annotation_index=False)
extract_data_from_dataset2d(adata.obs, ["n_genes"])

Stack trace:

ValueError                                Traceback (most recent call last)
Cell In[10], line 2
      1 adata = ad.experimental.read_lazy(path, load_annotation_index=False)
----> 2 extract_data_from_dataset2d(adata.obs, ["n_genes"])

File ~/projects/biocontext_ai/anndata-mcp/src/anndata_mcp/tools/utils.py:172, in extract_data_from_dataset2d(dataset2d, columns, row_slice, index, return_shape)
    170     data = dataset2d.iloc[row_slice][columns].to_memory()
    171 else:
--> 172     data = dataset2d[columns].to_memory()
    173 if return_shape:
    174     return truncate_string(data.to_csv(index=index)), str(data.shape)

File ~/projects/biocontext_ai/anndata-mcp/.venv/lib/python3.13/site-packages/anndata/_core/xarray.py:205, in Dataset2D.__getitem__(self, key)
    201 if isinstance(ret, XDataset):
    202     # If we get an xarray Dataset, we return a Dataset2D
    203     as_2d = Dataset2D(ret)
--> 205     as_2d.true_index_dim = self.true_index_dim
    206     as_2d.is_backed = self.is_backed
    207     return as_2d

File ~/projects/biocontext_ai/anndata-mcp/.venv/lib/python3.13/site-packages/anndata/_core/xarray.py:129, in Dataset2D.true_index_dim(self, val)
    127 if val not in self.ds.data_vars:
    128     msg = f"Unknown variable `{val}`."
--> 129     raise ValueError(msg)
    130 self.ds.attrs["indexing_key"] = val

ValueError: Unknown variable `index`.

[ilan-gold]

Ok, yup fixing that now hopefully