Reading manifest latest in quay.io/containers/podman: unknown: Tag latest was deleted or has expired

[cevich]

On August 23rd it was discovered that the credentials for several robot service accounts with write-access to the
container-images could have leaked. Upon discovery, the credentials were invalidated. The earliest possible
leak opportunity was around March 10th, 2022.

While the investigation is ongoing, initial inspection of the images seems to indicate it is unlikely any credentials
had actually been discovered and/or used to manipulate images. Nevertheless, out of an abundance of caution,
all possibly-affected images will be disabled.

quay.io/containers/podman : tags v3 - v4
quay.io/containers/buildah : tags v1.23.1 - v1.31.0
quay.io/containers/skopeo : tags v1.5.2 - v1.13.1
quay.io/podman/stable : tags v1.6 - v4.6.0
quay.io/podman/hello:latest SHA256 afda668e706a (<= Aug 2, 2023)
quay.io/buildah/stable : tags v1.23.3 - 1.31.0
quay.io/skopeo/stable : tags v1.3.0 - 1.13.1

We realize this issue has the potential to impact not only direct but also indirect use, such as base-images.
The safety and integrity of these images have and must take priority. At this time, all images have been disabled.
We will restore originals and/or rebuild fresh copies based on further safety analysis.

We expect analysis to be complete and/or known-safe images restored, before Sept. 8th. However please keep
in mind the research is ongoing, and the situation remains somewhat fluid. When the examination work is complete,
or if any manipulation is discovered, we will issue further updates.

Thank you in advance for your patience and understanding.

[cevich]

In case it's not obvious. The quay.io/podman/hello:latest image was manually rebuilt and replaced on the 25th. The manifest-list afda668e706a along with any produced on/before Aug 2nd are assumed suspect - don't use them.

Note: The lack of new images after about Aug. 3rd is due to unrelated breakage in build-automation. It was while attempting to fix it that I noticed the possible leak. This is why there were no images tagged from about Aug 3rd - present.

---

The "latest" buildah and podman manifest-lists are nearly finished building. I will have them pushed up in the next hour or so. Followed by skopeo.

[AbhishekSheth7]

Is there any temporary solution so that customers can continue running pipelines?

[cevich]

Yes, please see the next post...

[cevich]

The following freshly (manually) built images have been pushed:

• Manifest-list SHA256 34f12c7b72ec

  • quay.io/buildah/stable:latest
  • quay.io/buildah/stable:v1
  • quay.io/buildah/stable:v1.31
  • quay.io/buildah/stable:v1.31.0

• Manifest-list SHA256 e48f5af3e9ec

  • quay.io/containers/buildah:v1.31
  • quay.io/containers/buildah:v1
  • quay.io/containers/buildah:v1.31.0
  • quay.io/containers/buildah:latest

---

• Manifest-list SHA256 6a070625cff8

  • quay.io/podman/stable:latest
  • quay.io/podman/stable:v4
  • quay.io/podman/stable:v4.6
  • quay.io/podman/stable:v4.6.1

• Manifest-list SHA256 dabb1648acaf

  • quay.io/containers/podman:v4
  • quay.io/containers/podman:v4.6
  • quay.io/containers/podman:v4.6.1
  • quay.io/containers/podman:latest

---

• Manifest-list SHA256 5590c7d14c35

  • quay.io/skopeo/stable:latest
  • quay.io/skopeo/stable:v1
  • quay.io/skopeo/stable:v1.13
  • quay.io/skopeo/stable:v1.13.2

• Manifest-list SHA256 00286f273f2d

  • quay.io/containers/skopeo:v1
  • quay.io/containers/skopeo:v1.13
  • quay.io/containers/skopeo:v1.13.2
  • quay.io/containers/skopeo:latest

[prajwalv]

do you have an ETA for skopeo images?

[HubertBos]

Thank you

[cevich]

sorry, they went up into quay, I just haven't had a chance to post the SHAs, doing that now...

[Vbhatia20]

Thankyou for the update, eagerly waiting :)

[cevich]

Skopeo images uploaded about an hour ago. Sorry I was slow in updating the sha/tag list, meetings got in the way 😞

[iakat]

In case it might be helpful to anyone, I have had luck building skopeo images thru GitHub actions here: https://github.com/katlol/containers

[cevich]

FWIW: Skopeo images built fresh and pushed up about an hour ago. I just now updated the tag/sha256 comment above (darn meetings).

[the-sun-will-rise-tomorrow]

Would it be possible to obtain a copy of the suspect images for analysis and to assess impact?

[cevich]

I don't see a technical reason why not, other than it's nearly 100gig of data. There's also a ~2mb spreadsheet with all the original SHA256 sums and tag-push timestamps (from quay), if that would be helpful. I can't make any promises at this point, but I'll add the request to my TODO list.

[cevich]

Update: I believe I've found a way to validate (with a high degree of certainty) if any given image was likely tampered with or not. Though I want to be as certain of my logic as possible, so have asked the team to vet it based on their intimate knowledge of how images "work". If it works out, and with a bit of scripting, answers should come fairly quickly.

[cevich]

Minor update: My vetting method has been as "blessed" as best as it can be. I use those words because few things in this are going to be 100% certain and fool-proof. We're working with "degrees of probability" here. Though the team feels it will be quite high, it can't be perfect. There's always the risk that a very sophisticated attacker, given sufficient technology and knowledge resources, could fool us all.

That all said, I intend on publishing the validation method eventually - for now, actually vetting the images is my priority.

Oh, and in case it wasn't obvious: I have not and don't plan on touching the 'upstream' and 'testing' image flavors. They contain the warning label right on the box.

---

Somewhat related, I think maybe somebody mentioned it earlier, but it's worth restating: If you need near-term guarantees for these images, please consider building them yourself. All the Containerfiles are public in the podman/buildah/skopeo repos under the contrib/<project>image/ directory.

[cevich]

Image restoration Update: The team has restored a slew of older podman/buildah/skopeo images we're confident haven't been messed with. All of them had push timestamps recorded by quay, prior to the earliest date of any possible credential leaks.

[jeremy-duckworth]

Hi @cevich. First off, thank you for disclosing this. And second, does the project employ cryptographic signatures (e.g., through Sigstore or alternative) that could be used for forensic integrity validation (assuming the stolen credentials did not include those that allow misuse of the sigining infrastruture, as well)?

[cevich]

You're welcome, and unfortunately they did not, though in hind-sight it would have been a smart idea. When the replacement build automation goes up, I intend to address this and other traceability concerns - like specific details/metadata from the build system that produced it. All the suspect images do have a main-branch SHA which is very fortunate, but more details would have been helpful.

[cevich]

Update: All podman/buildah/skopeo images with final tag-push dates prior to the vulnerability window have been restored on quay. I won't list all the tags, since there's quite a few. In addition, the quay.io/podman/stable:v3.4.4 image (which IS within the window) was restored after manual checking with the now "blessed" vetting method.

There won't likely be any more images restored for a few (business) days, as I'm working on scripting up the validation checks. There's also additional scrutiny being performed by other Red Hat teams, in case anybody worries "it's just cevich looking at this" - it's not.

Note: Monday Sept. 4th is a US holiday.

[TomSweeneyRedHat]

Currently, the following tags have been restored:

Skopeo: v1 -> v1.6.0, V1.13 and v1.13.2, also latest
Buildah: v1 -> v1.23.3, v1.31.0 and v1.31, also latest
Podman: v1.6->v3.4.2, v4, v4.6, v4.6.1, also latest

As Chris mentioned, the other versions were made during the window of vulnerability. We are working with others to see if we can determine the safeness of the ones that were created then that we have backed up, and if we can't determine the safeness, trying to rebuild them "by hand".

If there is a particular version(s) that you are in desperate need of, please feel free to leave a reply here, and we'll try to focus on those.

[slonopotamus]

If there is a particular version(s) that you are in desperate need of, please feel free to leave a reply here, and we'll try to focus on those.

Buildah v1.28. I created a separate issue for it, but it was closed as a duplicate of current thread.

[paleozogt]

Buildah v1.26.0 (issue).

[pfarikrispy]

podman/stable:v4.3.1 would be very helpful 🙏

[cevich]

Ugg, that's why I was afraid to ask. If you few need specific images, you betcha there is a HOARD of other people who also need their pet images 😢

Unfortunately running my validation check manually is really tedious and not very human-friendly. I would really hate to screw it up. Going through even 3 or 5 of them manually would take a lot of time. As annoying as it is, I really think we need to wait for me to finish a automation-friendly validation tool.

I understand the current situation is a PITA, but I can promise if I restore a compromised image, it will be a MUCH larger PITA.

[cevich]

Update: Over the last several days, I've finished scripting all basic/fundimental validation checks for an automation-friendly validation tool. I've written integration tests for it, against some of the known-good images we've restored. Now it may come as a shock: The tests are all passing for me locally, though CI seems unhappy at the moment (gosh, that's never happened before!) 😀

Anyway, I'm ignoring integration-CI for now, and will be pressing on with adding the final more detailed checks. Specifically the one that verifies the quay images against metadata from our CI system. I'm hoping to get finished with that tomorrow (Thursday), but it's fairly complex. In any case, I'm intending to start using the script and restoring validated images on Friday. Though if there are hiccups it could take a few more days. But we're much closer than we were last week.

I think I mentioned it before, but there are other RH teams doing validation via a completely different methods. Most of those efforts are centered around determining IF the credentials were accessed and/or used. Though even that seems a worthy question of trying to answer.

[cevich]

Update: I've finished the first working-ish implementation of a validation script. Actually the tool works fine, everything checks out perfectly on a known-good image. The problem is my data set of quay.io push timestamps are all browser-local and the 🥵 daylight-savings time is screwing up the (test) 👍 result 😠

Anyway, it's late on a Friday and my brain is completely fried. I'm sure there's a simple solution and I can start validating images first thing on Monday.

[cevich]

Daylight savings conversion of timestamp data is complete. Unfortunately I still don't have word on if it's okay to share this data. I don't see a reason not to, but it's not my call and it's still a lower for me priority ATM.

Final cleanup/polishing of the validation script is also done. It's far from perfect/pretty/fabulous but I believe it will get the job done. I'm planning to start running it against a copy of the backup images first thing tomorrow (Tuesday). Since it was requested above, I'll start working on the latest image tags first and proceed backwards through time from there.

Note: Just to keep everyone's expectations clear, I'll likely not restore images to quay immediatly. Instead I want to hold off a little until I've got a handle on the overall results, and a go-ahead from several teams. Hopefully the reasons for my cautiousness are obvious.

[cevich]

My validation checks are done (whew!). The news is good: I found zero anomalies, dependencies, or in-congruent timestamps to w/in less than 2-minutes of successful build-task completion. In many cases the margin was less than 30-seconds. Basically it would be incredibly difficult and resource-intensive to skirt these timings.

So at this point, just based on what I found, it doesn't look likely any of the possibly-leaked credentials were successfully used. If they leaked at all.

As I mentioned yesterday, I'm holding off on restoring any more images until I get a thumbs up from RH security teams doing their own checks. I'm also waiting on them for an "okay" to publish my validation data source. Though it's nothing magic, just a list of FQINs, SHAs, and push timestamps.

[cevich]

Final update: RH security has concluded their additional scrutiny, orthogonal to my image validation. Given both results together, there is a high level of confidence no credentials actually leaked nor were utilized. As I've eluded to all along, there are no 100% perfect guarantees, so please don't neglect any of your own due-diligence.

That all said, I've been given the green-light to restore the original quay images and provide a copy of the validation data I used. Unless someone strongly objects, I'm not going to overwrite any of the manually rebuilt images. I'll restore images in reverse version order, since it seems folks depend on those the most.

[cevich]

As requested, here's a copy of the data I used to validate the images mostly automatically. The script I used and some tooling is in this PR (undecided what I'm going to do with it, some have asked me to develop it further in it's own project). A few notes about the data:

• I put this together by manually copy/pasting details from several other spreadsheets. I was careful, but it's possible I made mistakes. Please let me know if you encounter any and I'll try and fix.
• I've converted the original quay.io push dates for each FQIN into a more useful format, but the original source was a literal copy-paste from the tag-history pages. Unfortunately these only represent the last time that tag was pushed. There simply isn't data for each/every push, I asked.
• The original (most recent) image digests were also copy-pasted from quay prior to pulling all the images a few weeks ago. These digests may differ from the current quay.io contents. I believe in all cases this is due to the images being manually re-built (rather than restored) out of expediency.
• Here's a key to notes I added:
  • Not affected: This means the latest push timestamp is prior to the earliest possible credential-leak date. It was also added to some images known to have been built manually (mostly very early versions). In one case, the log-entry on quay had rotated, so is no longer available.
  • Cleanly validated: This means my validation script passed all checks w/o any fuss. The image-push on quay was positively associated with an automated build on the main branch with a delta of +/- 1 minute (2 minutes total). Though in many/most cases the actual delta was 30-ish seconds if I remember correctly.
  • Cleanly validated <Commit ID>: This means the image is missing a org.opencontainers.image.revision label - due to a build script change. However semi-automated searching was able to pin it to a main-branch automated build linked to the mentioned commit.
  • Manually Rebuilt: Just what it says. A "clean-room" (as much as can be achieved) manual build from upstream sources.

quay_container_image_details.csv.txt