Change the app Dockerfile to use faiss-base as base image

[rchitale7]

Description

The publish_remote_api_image CI was failing, with this change: #98. The reason is that both core and app folders were modified. When publishing the api image, the GitHub CI used an older version of the core image, that did not have the latest modifications. This caused the CI to fail, because the code in the app folder is using the latest modifications to core. I fixed the app Dockerfile to follow the example from the e2e image: https://github.com/opensearch-project/remote-vector-index-builder/blob/main/e2e/api/Dockerfile, so that we're just copying the app and core directories onto the faiss-base image. This ensures any modifications to core will always be picked up by the publish_remote_api_image CI.

I also updated some of the docs to reflect this change, along with the change from #98.

Issues Resolved

Resolves #101

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[navneet1v]

@rchitale7 I thought there is already separate CIs that looks at the what folder changes build the images. May be it can happen that core, app and base image cis are not running in order. I think there will be a way where we can add dependencies of CIs on one another so that they can run in order. Won’t that fix the problem?

[rchitale7]

@rchitale7 I thought there is already separate CIs that looks at the what folder changes build the images. May be it can happen that core, app and base image cis are not running in order. I think there will be a way where we can add dependencies of CIs on one another so that they can run in order. Won’t that fix the problem?

That does already exist here: https://github.com/opensearch-project/remote-vector-index-builder/blob/main/.github/workflows/publish_remote_core_image.yml#L67. So I'm not sure why this CI failed: https://github.com/opensearch-project/remote-vector-index-builder/actions/runs/16426837535/attempts/1 - maybe the GH runner erroneously used a cached image of core. I thought it made more sense to just copy the core and app folders instead of using core as a base image, since then we would never have this issue.

[navneet1v]

@rchitale7 I thought there is already separate CIs that looks at the what folder changes build the images. May be it can happen that core, app and base image cis are not running in order. I think there will be a way where we can add dependencies of CIs on one another so that they can run in order. Won’t that fix the problem?

That does already exist here: https://github.com/opensearch-project/remote-vector-index-builder/blob/main/.github/workflows/publish_remote_core_image.yml#L67. So I'm not sure why this CI failed: https://github.com/opensearch-project/remote-vector-index-builder/actions/runs/16426837535/attempts/1 - maybe the GH runner erroneously used a cached image of core. I thought it made more sense to just copy the core and app folders instead of using core as a base image, since then we would never have this issue.

The problem with copying core only is we would now need to ensure that we install all the dependencies of core also + we setup the core. So basically whatever we were doing in core we need to do again. I think this could be a problem with github runners not running in order. If we can as API github runner to wait for core to complete and then ask core API runner to wait for base image CI to complete this problem will not happen.

[rchitale7]

@navneet1v thinking through this more - what i can do is add an option for docker --pull here: https://github.com/opensearch-project/remote-vector-index-builder/blob/main/.github/workflows/publish_remote_api_image.yml#L36. That should solve the issue we have with using the cached core image. But, we still have a problem where changes in app and core trigger both the standalone API CI, and the Core CI->API CI workflow path. The standalone API CI may sometimes fail whenever there's a change in app and core, because there's no way to guarantee ordering there. So I think the solution there is to move these conditions: https://github.com/opensearch-project/remote-vector-index-builder/blob/main/.github/workflows/publish_remote_api_image.yml#L8 to the publish_remote_core_image CI, and remove the direct API CI trigger: https://github.com/opensearch-project/remote-vector-index-builder/blob/main/.github/workflows/publish_remote_api_image.yml#L4. Then, the API CI will never run as a standalone CI, and will only run as part of the Core CI->API CI workflow path. Does that make sense?