ImportImageFailed: Failed to import 2 image(s)

[frenzymadness]

Hello.

I have a problem building my python3 image in Koji, see these tasks for example:

• https://koji.fedoraproject.org/koji/taskinfo?taskID=80269475
• https://koji.fedoraproject.org/koji/taskinfo?taskID=80271860

The builds itself are fine but the orchestrator is not:

ImportImageFailed: Failed to import 2 image(s)
Traceback (most recent call last):
  File "/usr/bin/atomic-reactor", line 33, in <module>
    sys.exit(load_entry_point('atomic-reactor==1.6.47', 'console_scripts', 'atomic-reactor')())
  File "/usr/lib/python3.9/site-packages/atomic_reactor/cli/main.py", line 318, in run
    cli.run()
  File "/usr/lib/python3.9/site-packages/atomic_reactor/cli/main.py", line 300, in run
    args.func(args)
  File "/usr/lib/python3.9/site-packages/atomic_reactor/cli/main.py", line 98, in cli_inside_build
    build_inside(input_method=args.input, input_args=args.input_arg,
  File "/usr/lib/python3.9/site-packages/atomic_reactor/inner.py", line 615, in build_inside
    build_result = dbw.build_docker_image()
  File "/usr/lib/python3.9/site-packages/atomic_reactor/inner.py", line 568, in build_docker_image
    raise ex
  File "/usr/lib/python3.9/site-packages/atomic_reactor/inner.py", line 560, in build_docker_image
    exit_runner.run(keep_going=True)
  File "/usr/lib/python3.9/site-packages/atomic_reactor/plugin.py", line 332, in run
    raise PluginFailedException(failed_msgs[0])
atomic_reactor.plugin.PluginFailedException: plugin 'import_image' raised an exception: ImportImageFailed: Failed to import 2 image(s)

Reported on fedora-infra as well two weeks ago: https://pagure.io/fedora-infrastructure/issue/10437

[MartinBasti]

2021-12-21 09:11:59,259 platform:- - atomic_reactor.plugin - DEBUG - running plugin 'import_image'
2021-12-21 09:11:59,259 platform:- - atomic_reactor.plugin - INFO - running plugin instance with args: '{'imagestream': 'registry.fedoraproject.org-python3'}'
2021-12-21 09:11:59,260 - osbs.core - INFO - Using service account's auth token
2021-12-21 09:11:59,288 - osbs.http - DEBUG - cleaning up
2021-12-21 09:11:59,289 - osbs.core - DEBUG - [404] b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"imagestreams.image.openshift.io \\"registry.fedoraproject.org-python3\\" not found","reason":"NotFound","details":{"name":"registry.fedoraproject.org-python3","group":"image.openshift.io","kind":"imagestreams"},"code":404}\n'
2021-12-21 09:11:59,289 platform:- - atomic_reactor.plugins.import_image - INFO - Creating ImageStream registry.fedoraproject.org-python3 for registry.fedoraproject.org/f35/python3
2021-12-21 09:11:59,316 - osbs.http - DEBUG - cleaning up
2021-12-21 09:11:59,344 - osbs.http - DEBUG - cleaning up
2021-12-21 09:11:59,345 - osbs.core - DEBUG - imagestream: {'kind': 'ImageStream', 'apiVersion': 'image.openshift.io/v1', 'metadata': {'name': 'registry.fedoraproject.org-python3', 'namespace': 'osbs-fedora', 'selfLink': '/apis/image.openshift.io/v1/namespaces/osbs-fedora/imagestreams/registry.fedoraproject.org-python3', 'uid': '0d599515-623e-11ec-808a-525400de6908', 'resourceVersion': '75745980', 'generation': 1, 'creationTimestamp': '2021-12-21T09:11:59Z', 'annotations': {'openshift.io/image.insecureRepository': 'true', 'osbs/source_repo': 'registry.fedoraproject.org/f35/python3'}}, 'spec': {'lookupPolicy': {'local': False}}, 'status': {'dockerImageRepository': 'docker-registry.default.svc:5000/osbs-fedora/registry.fedoraproject.org-python3'}}
2021-12-21 09:11:59,371 - osbs.http - DEBUG - cleaning up
2021-12-21 09:11:59,372 - osbs.core - DEBUG - tags before import: []
2021-12-21 09:11:59,444 - osbs.http - DEBUG - cleaning up
2021-12-21 09:11:59,445 - osbs.core - ERROR - Error importing image latest: {'status': {'metadata': {}, 'status': 'Failure', 'message': 'dockerimage.image.openshift.io "registry.fedoraproject.org/f35/python3:latest" not found', 'reason': 'NotFound', 'details': {'name': 'registry.fedoraproject.org/f35/python3:latest', 'group': 'image.openshift.io', 'kind': 'dockerimage'}, 'code': 404}, 'tag': 'latest'}
2021-12-21 09:11:59,445 - osbs.core - ERROR - Error importing image 0: {'status': {'metadata': {}, 'status': 'Failure', 'message': 'dockerimage.image.openshift.io "registry.fedoraproject.org/f35/python3:0" not found', 'reason': 'NotFound', 'details': {'name': 'registry.fedoraproject.org/f35/python3:0', 'group': 'image.openshift.io', 'kind': 'dockerimage'}, 'code': 404}, 'tag': '0'}
2021-12-21 09:11:59,445 - osbs.core - ERROR - Failed to import 2 image(s)

@rcerven PTAL

[rcerven]

I didn't find logs in those 2 mentioned tasks, but I've found them in : Firefox flatpak: https://koji.fedoraproject.org/koji/taskinfo?taskID=80953059

First of all, you are using 2 years old atomic-reactor and osbs-client :(

From logs I can see that images are pushed to : candidate-registry.fedoraproject.org
but imagestream is using : registry.fedoraproject.org

because you have have different source_registry and registries in reactor-config-map
(and for imagestream creation is used source_registry)

what is relation between those registries?

since other builds are passing my guess is that you have some kind of synchronization between those 2 registries

so the import is failing because image wasn't synced yet to 'registry.fedoraproject.org' registry from 'candidate-registry.fedoraproject.org' registry

[frenzymadness]

Thanks a lot for your investigation. I'm not an administrator of the infrastructure so I cannot update the packages and my knowledge of the pipeline is limited to a user perspective. I'm just trying to find someone who can fix it because I'm not able to build container images for weeks now.

To answer your question. candidate-registry is for testing. AFAIK every new container image should go there first, then an update is created in Bodhi and if the update gets enough karma points or spend enough time in testing, the container image is then moved from candidate-registry to the default registry where users can pull it.

[rcerven]

so from that firefox flatpak build, I can see that stable and latest tags are actually there: https://registry.fedoraproject.org/v2/firefox/tags/list

but openshift still reports 404, so openshift is having issue with checking that registry
you can also check created imagestream and check in status if there is more error info than just 404

you can verify if imagestream tag import is working with oc image-import command, which is exactly what is osbs doing via API

[nirik]

It indeed is erroring:

sh-5.0# oc import-image registry.fedoraproject.org-firefox
error: tag latest failed: dockerimage.image.openshift.io "registry.fedoraproject.org/firefox:latest" not found
imagestream.image.openshift.io/registry.fedoraproject.org-firefox imported with errors

Name:                   registry.fedoraproject.org-firefox
Namespace:              osbs-fedora
Created:                5 days ago
Labels:                 <none>
Annotations:            openshift.io/image.dockerRepositoryCheck=2022-01-06T09:05:08Z
Docker Pull Spec:       docker-registry.default.svc:5000/osbs-fedora/registry.fedoraproject.org-firefox
Image Lookup:           local=false
Unique Images:          0
Tags:                   2

latest
  tagged from registry.fedoraproject.org/firefox:latest
    will use insecure HTTPS or HTTP connections

  ! error: Import failed (NotFound): dockerimage.image.openshift.io "registry.fedoraproject.org/firefox:latest" not found
      5 days ago

stable
  tagged from registry.fedoraproject.org/firefox:stable
    will use insecure HTTPS or HTTP connections

  ! error: Import failed (NotFound): dockerimage.image.openshift.io "registry.fedoraproject.org/firefox:stable" not found
      5 days ago

I've asked the folks who know more about this setup to look too, they should hopefully see this tomorrow eu time.

[cverna]

what is relation between those registries?

As @frenzymadness mentioned, OSBS only pushes the images to our candidate-registry, then the image maintainer creates an Bodhi update which once pushed to "stable" copies the container image from the candidate-registry.fp.o to registry.fp.o.

I can't remember why we use registry.fp.o in the imagestream, I guess this is because it is where all "stable" images eventually ends up. But maybe it would be better to actually have OSBS look only at the candidate-registry ?

[rcerven]

well osbs uses source_registry for the imagestream and also as default registry if you omit registry in FROM pullspec

imagestreams are created for autorebuilds, if you aren't using that feature it doesn't really matter

[cverna]

Yeah we don't use the autorebuilds, since the fedora base image is not built by OSBS (we don't have all the arch available for that)

[cverna]

So it looks like we had to disable the import_image plugin in the worker_customize.json file in addition to the orchestrator_customize.json. https://pagure.io/fedora-infra/ansible/pull-request/927#request_diff

It looks like the orchestrator_customize was ignored, not sure why tho

[rcerven]

that seems like bug, that it uses worker_customize

I remember that support customize configs was dropped at some point

but you can always just remove it from orchestrator_inner:6.json

but anyway that is just workaround for your infra issue, where ocp can't connect to that registry