Implement clearer tagging system

[Gregory-Pereira]

My goal here is to create a more clear tagging system by separating the QA and Prod workflows.

QA workflow: Triggered by commits pushed to main. The workflow will lookup the number of the last pr that merged and tag the image with both pr-<pr_number> and as latest.

Prod workflow: Triggered by a release being "released". When it gets released, it will lookup the release title and tag as release-<release-title> (ex: something like release-v1.0, and we will have to be consistent with how we name releases). The release would be created off of main rather than creating a separate release branch like we have in the past, but this shouldn't be an issue because it will still get its own tag.

For both, this will allow renovate to pick up the different tag names as code changes so argo can understand its different and generate a PR to upgrade the container images automatically.

Thoughts @nerdalert @vishnoianil ?

[vishnoianil]

My goal here is to create a more clear tagging system by separating the QA and Prod workflows.

QA workflow: Triggered by commits pushed to main. The workflow will lookup the number of the last pr that merged and tag the image with both pr-<pr_number> and as latest.

I am wondering if we can just tag it with latest and detect the newly published image using the image sha? Current QA deployment manifest uses "main" tag of the images, and it simply requires pod restart for upgrade because it pulls the latest image tagged with "main" due to the "Always" image pull policy. So i think it would be great if we don't have to update the image tags in the manifest files itself.

Prod workflow: Triggered by a release being "released". When it gets released, it will lookup the release title and tag as release-<release-title> (ex: something like release-v1.0, and we will have to be consistent with how we name releases). The release would be created off of main rather than creating a separate release branch like we have in the past, but this shouldn't be an issue because it will still get its own tag.

The reason we cut the release-1.0 branch is because that's the branch that's hosted on ui.instructlab.ai. In case any bug is discovered or any minor enhancement is required on that, we can push those changes to release-1.0 branch and manage the life cycle of the branch. If we cut release from main branch, that means all the new experimental features will be part of that release and that branch might not be backward compatible as well. Some of these experimental feature might not be ready for the prod as well. We roughly follow the same release process as kubernetes when it comes to cutting branches and minor version upgrades.

For both, this will allow renovate to pick up the different tag names as code changes so argo can understand its different and generate a PR to upgrade the container images automatically.

So once the job detects that a new image is published, is it going to update the manifest files present in the deploy/k8s to trigger the argocd pipeline?

Thoughts @nerdalert @vishnoianil ?

[Gregory-Pereira]

We discussed most of this just going to add information for provinence

I am wondering if we can just tag it with latest and detect the newly published image using the image SHA? Current QA deployment manifest uses "main" tag of the images, and it simply requires pod restart for upgrade because it pulls the latest image tagged with "main" due to the "Always" image pull policy. So i think it would be great if we don't have to update the image tags in the manifest files itself.

We can, but I think its better to use a tag. That ways its easier to debug and tell what the deployed image is as compared to the previous or next version without having to lookup and compare SHAs. QA workflows will use the PR tag that merged, ex: :pr-334, and prod will use the tag of the release, ex: v1.1. However, we will continue to update latest as well and could always move to SHAs.

The reason we cut the release-1.0 branch is because that's the branch that's hosted on ui.instructlab.ai. In case any bug is discovered or any minor enhancement is required on that, we can push those changes to release-1.0 branch and manage the life cycle of the branch. If we cut release from main branch, that means all the new experimental features will be part of that release and that branch might not be backward compatible as well. Some of these experimental feature might not be ready for the prod as well. We roughly follow the same release process as kubernetes when it comes to cutting branches and minor version upgrades.

If that is the case we can keep the release branch method. The prod workflow now will be automatically triggered by the release so the workflow is the same. We cut the release branch, ex: release-v1.1. and cherry pick or pull in whatever commits we want. We then make a release and create the tag from that branch, with the tag in this example being v1.1. The prod workflow will grab the tag_name from the release data, and will reconstruct the image tags as release-<tag_name>. After this it will create a commit to the main branch to update the images.

once the job detects that a new image is published, is it going to update the manifest files present in the deploy/k8s to trigger the argocd pipeline?

Yes, it will update the overlay kustomization files which will patch the base deployments.

[Gregory-Pereira]

I believe this is ready for merge so I can begin testing @vishnoianil and @nerdalert