Incorporate Feast into Llama Stack

[franciscojavierarceo]

🚀 Describe the new functionality needed

The Llama stack should leverage Feast to enable the Model Lifecycle.

Feast already plays an important role in the AI/ML Lifecyle in Kubeflow and the Llama Stack would benefit from using it as well (image below for reference).

💡 Why is this needed? What if we don't build it?

Why is this needed

Feast, the open source Feature Store provides important primitives for production AI/ML including support for Vector Databases. Feast is a widely used feature store and is an add on component to Kubeflow.

Feast plays a critical role in (1) serving data for production AI Applications, (2) enabling model developers to more easily build training datasets, and (3) building APIs for distributed scale (i.e., Kubernetes). Moreover, Feast already supports APIs for executing critical operations on data, including but not limited to:

1. Ingestion
2. Transformation
3. Indexing
4. Retrieval/Serving
5. Dataset Generation/Preparation
6. Data Governance

Feast supports many different offline data warehouses and online databases, which allows for flexibility for software engineers while giving Data Scientists/Machine Learning Engineers/Researchers (i.e., experts in building models) a consistent SDK for leveraging data in production systems.

Adding Feast as a component to the Llama Stack would allow flexibility in choosing data providers and leverage Feast's existing significant contributions in serving the needs of engineers and model builders.

*Documentation on "What is Feast?".

What if we don't build it?

I think inevitably many of the Feast patterns will be implemented in Llama Stack that exist in Feast (likely with slightly different implementations). In the limit, this ends up re-inventing the wheel or implementing a better wheel.

If folks want to implement a better wheel, the Feats community would be eager to incorporate the learnings into the project so the broader AI/ML community could also benefit.

Other thoughts

There are pros and cons.

Pros:

1. Complexity is abstracted away in Feast
2. Benefit from existing expertise and implementation
3. Have support for known challenges that occur during the model building lifecycle
4. Benefit from existing data integrations

Cons:

1. Complexity is not trivially visible in the Llama stack repository
2. Llama stack maintainers may not like every choice made in Feast and will have to work with Feast to make changes (I am a maintainer for Feast and can commit to being very collaborative here)

I'm sure there are more cons but I wanted to list some.

[jwm4]

I looked in to trying to find a way to incorporate Feast into InstructLab RAG but I never managed to understand Feast well enough to come up with a vision that makes sense. I think we face the same challenge here: figuring out what aspects of Feast would align with the things Llama Stack is trying to do.

For example, I start with the support for Vector Databases article and I see that Feast includes a thin wrapper around several well-known vector databases, e.g., Elasticsearch. That seems fine but unremarkable and by itself not enough to justify bringing in a big, heavy dependency like Feast unless the wrapper were extraordinarily powerful and brings in a lot of the other benefits you list above, e.g., more sophisticated and powerful Dataset Generation/Preparation and Data Governance. So I look more at the support for Vector Databases article linked, and I see the following code in that article:

from batch_score_documents import run_model, TOKENIZER, MODEL
from transformers import AutoTokenizer, AutoModel

question = "the most populous city in the U.S. state of Texas?"

tokenizer = AutoTokenizer.from_pretrained(TOKENIZER)
model = AutoModel.from_pretrained(MODEL)
query_embedding = run_model(question, tokenizer, model)
query = query_embedding.detach().cpu().numpy().tolist()[0]

# Use Feast to match the end-users query to database vectors
from feast import FeatureStore
store = FeatureStore(repo_path=".")
features = store.retrieve_online_documents(
    feature="city_embeddings:Embeddings",
    query=query,
    top_k=5
).to_dict()

This is really doing the same thing that you would do if you were using the thin wrapper around a vector database that you already get in Llama Stack or that appear in numerous other LLM frameworks (e.g., LangChain) except that those generally include instantiating and running the embedding model under the hood which makes them a little easier to use. Regardless, the construct in Feast seems fine to me, but I am not seeing any differentiated value from Feast in this tiny example, of course.

So what I think we need next is a more detailed vision of what constructs in Feast would be used and how they would be used to get a sense of why they provide some value here. I think it is clear that the Feast vector db wrappers alone using just the code in the support for Vector Databases article doesn't add significant value beyond what's already in Llama Stack; you would need to also make more use of other aspects of Feast before you started to really see value. So what aspects would we use? How would we use them? What value would they provide?

I am not sure if RAG is the right place to start for exploring Feast + Llama Stack integration. The synthetic data generation -> model training pipeline feels a little closer to Feast's original purpose. However, I think the same questions arise there about how to use Feast effectively.

[franciscojavierarceo]

I agree with all of your points, @jwm4. Thank you for reviewing too!

As I'm reviewing the existing implementations of LlamaStack, I'm not sure it makes sense to abstract over Feast's abstractions in its current state. That said, I still need to review more and I've been spending a lot more time on a proposed Kubeflow SDK. Importantly, we are discussing LlamaStack within that SDK.

From a Kubeflow perspective, Feast will be the storage/memory layer analogue and so it may just be that those things are abstracted separately. That is a little unfortunate since Feast has many existing integrations and large scale support (streaming, Spark, etc.) but I still have to review things further to see how this will fit together. I think much of that will emerge from the Kubeflow SDK work and our considerations there for LlamaStack.

[franciscojavierarceo]

I'll say that Feast is a good structure for ML Engineers and Data Scientists but AI Engineers may not see the same benefits initially (though I do believe eventually they will).

[jwm4]

Thanks! I think it will be challenging to find the right way to get the pieces to fit together to make them complement each other, but it sounds like there could be a substantial amount of value here too. Issues like governance, scale and streaming of data will almost certainly be vital to the long-term success of Lllama Stack so if there is a way for Feast to enhance those attributes or similar attributes that could turn out to be a big opportunity.

[franciscojavierarceo]

That would be my goal. I think we'd have to work with the LlamaStack folks to figure this out and that's really the spirit of this ticket (i.e., to kick off a conversation). So let me know if you want to collaborate on this. I think the benefit of Kubeflow and Red Hat's support in Feast already helps scale to the MLOps needs pretty easily. Expanding that to the AI Engineer is new territory for Feast.