Metadata for foundation models (FM)

[jonas-hurst]

🚀 Feature Request

Add a way to catalog EO foundation models

🔉 Motivation

Recent advances within the realm of self-supervised learning have lead to the development of foundation models for EO. The output of such foundation models are task-agnostic embeddings (i.e. a vector or matrix or tensor of numbers), which are an abstract representation of the model input.

An entry in the Task Enum is needed to properly describe what foundation models are doing and properly catalog them using stac:mlm: Their task is to compute embeddings, which are task-agnostic.

The entry could be generate-embeddings or abstract-embeddings. This term would be generic and apply not only to foundation models, but to any model in general that generates an abstract representation of the input data.

📡 Alternatives

The entry could simply be called foundation-model, but that is not a "task" and it would be less generic than the ideas mentioned above.

📎 Additional context

[fmigneault]

@rbavery @devisperessutti @simonevaccari (or anyone else, feel free to reference to others that could be interested)
I would like to get your thoughts about this.

IMO, if a model has no specific mlm:tasks, it could explicitly indicate mlm:tasks: [], since the JSON array definition does not require a minimum amount of tasks, and it would highlight that lack of explicit task. It seems somewhat misleading to have mlm:tasks: ["generate-embeddings"] if the intent of that model is for users to define derived models, rather than actually meant to be used as an "embedding" or "feature" generator. Maybe abstract-embeddings is slightly better due to "abstract", but I still think it is missing something to convey the entire intent.

I can also see a point regarding mlm:tasks: [] not being very "verbose" about the intent by itself. Therefore, a separate field might be considered to reflect that intent very explicitly, or we could simply indicate in the extension's definition that indicating mlm:tasks: [] represents this use case (but could there be other concepts using a similar empty array in the future, which would cause confusion?).

[fmigneault]

Would it make sense to call these abstract tasks encoding / decoding? Is it clear enough?

I could also see other tasks added such as tokenization, text-summarization, question-answering, etc. for various types of ML tasks.

I agree that a specific name would better guide users than the empty array, but I would like to use common literature names for these tasks, and generate-embeddings/abstract-embedding don't feel "quite right".

[rbavery]

I agree that just "embedding" as a descriptor isn't enough, because embeddings can serve different purposes, RAG, similiarity search, features for downstream fine tuning. If they are intended for downstream supervised fine-tuning (SFT) they may not completely task agnostic (many encoders for vision generate embeddings that are not suitable for dense feature prediction).

I also agree we need to do more here than specifying an empty array.

Maybe we could break out some categories of embeddings that indicate their intent. I'd love to avoid introducing a new field (there are a lot) and I think tasks indicate intent currently so if we could stick with that that would be ideal.

Some ideas:

sft-embedding - generates embeddings that are intended for downstream fine-tuning to address other tasks. other tasks can be listed in the mlm:tasks array.

rag-embedding - generates embeddings intended for search (similarity search or image question answering for example). Users can list the specific search related tasks if the embeddings are intended for a particular search task like similarity-search

what do y'all think?

On using the term "foundational model" I think the term is too overloaded and ill-defined. Would prefer to use task descriptors that refer more specifically to the models capabilities, whether they are more general/agnostic to the domain and task or more specific.

[fmigneault]

These are good points. Both embedding and foudantional model are by themselves too broad, since there are many incompatible uses of embeddings/foundational models per domains/applications. Using only those does indicate "some task", but it does not help if that task equates to "anything". That defeats the purpose of describing the task in the first place.

Maybe better understanding what is the intended behavior or goal of the embedding application for the targeted foundation model would help better define it.

[jonas-hurst]

I had the chance to speak to a colleague (@romeokienzler) last week and we agree that it makes sense to distinguish the two (sft, rag). As the mlm:tasks prop is a list, one can simply add both if the generated embeddings are suitable for both.

[fmigneault]

I think the descriptions themselves of the provided ideas are better examples of valid "tasks" (those should be added):

• supervised-fine-tuning
• retrieval-augmented-generation
• similarity-search
• question-answering
• etc.

The embedding mention in the SFT/RAG cases feels more like an implementation approach, or a means to an end, than the task itself.

Another issue that I can foresee with embedding is that there could be endless possibilities (word-embedding, sentence-embedding, document-embedding, etc.). I would like to avoid having to create spec revisions for every new approach/method out there or in the future. Ideally, the tasks should be slightly "higher level" to encompass all of these methods regardless of the implementation details or encoding approaches. It is more about "what is their purpose" than "how they achieve it".

I could see tokenization as a valid task in the same sense as encoder/decoder if that was the task itself that the model addresses, but not if it is only one of its data preparation utilities/steps/layers (just like rag-embedding feels like a encoding/preparation step for RAG, the actual task).

[jonas-hurst]

To fully catalog FMs, adding embedding or embeddings to the list of recommended dimension names here would be helpful

[fmigneault]

Yes, we can add this.

[fmigneault]

embedding is now included (#84)