Make running the entire application locally possible #6009
Description
It would be really nice for us to be able to easily run the entire application locally (maybe using llm calls to the actual apis or mocked responses?). We are at a point now that the application is significantly complex enough that having something more reliable that we can easily use locally to do most of our end to end testing would reduce bugs, pr back and forth, and improve developer speed.
In the past we attempted to use kind to accomplish this with trino but it ended up being a bit too slow to continuously use.
I would suggest we look into using k3d. We attempted to use this previously. That mostly failed because of my own setup on my personal machine which uses a rootless docker which k3d doesn't seem to like. k3d now supports rootless podman so this can now be done with k3d. The reason doing all of this with k3d might be nice is much of what we write is actually doing k8s automation. If k3d can be made very quick it could make our test infra much closer to the production infra