#10437 Container builds are failing
Closed: Fixed 6 days ago by cverna. Opened a month ago by lbalhar.

Hello.

I don't know why but my last two attempts to build python3 container on top of Fedora 35 failed.

https://koji.fedoraproject.org/koji/taskinfo?taskID=80269475
https://koji.fedoraproject.org/koji/taskinfo?taskID=80271860

From the log it seems that the builds themselves finished successfully but the orchestrator has some problem:

ImportImageFailed: Failed to import 2 image(s)
Traceback (most recent call last):
  File "/usr/bin/atomic-reactor", line 33, in <module>
    sys.exit(load_entry_point('atomic-reactor==1.6.47', 'console_scripts', 'atomic-reactor')())
  File "/usr/lib/python3.9/site-packages/atomic_reactor/cli/main.py", line 318, in run
    cli.run()
  File "/usr/lib/python3.9/site-packages/atomic_reactor/cli/main.py", line 300, in run
    args.func(args)
  File "/usr/lib/python3.9/site-packages/atomic_reactor/cli/main.py", line 98, in cli_inside_build
    build_inside(input_method=args.input, input_args=args.input_arg,
  File "/usr/lib/python3.9/site-packages/atomic_reactor/inner.py", line 615, in build_inside
    build_result = dbw.build_docker_image()
  File "/usr/lib/python3.9/site-packages/atomic_reactor/inner.py", line 568, in build_docker_image
    raise ex
  File "/usr/lib/python3.9/site-packages/atomic_reactor/inner.py", line 560, in build_docker_image
    exit_runner.run(keep_going=True)
  File "/usr/lib/python3.9/site-packages/atomic_reactor/plugin.py", line 332, in run
    raise PluginFailedException(failed_msgs[0])
atomic_reactor.plugin.PluginFailedException: plugin 'import_image' raised an exception: ImportImageFailed: Failed to import 2 image(s)

Metadata Update from @zlopez:
- Issue close_status updated to: Upstream
- Issue status updated to: Closed (was: Open)

22 days ago

Could we please keep this issue open until it's really fixed? Other might have the same problem and open a new issue.
I can try to report it but with limited knowledge of the build system, I'm not sure, I know and will be able to provide all the necessary details. https://github.com/containerbuildsystem/atomic-reactor/issues/1742

Metadata Update from @lbalhar:
- Issue status updated to: Open (was: Closed)

22 days ago

Metadata Update from @mohanboddu:
- Issue priority set to: Waiting on Assignee (was: Needs Review)
- Issue tagged with: medium-gain, medium-trouble, ops

14 days ago

So the atomic-reactor and osbs-client are two years old in the infrastructure: https://github.com/containerbuildsystem/atomic-reactor/issues/1742#issuecomment-1009279161

Could you please check my comment there and correct me if I'm wrong?

Yes, thats correct, they are old.

However, that doesn't explain why things stopped working. I'll investigate.

Hi, I've added additional info in https://github.com/containerbuildsystem/atomic-reactor/issues/1742

so you can verify what is up with ocp vs registry connection

so, let me correct a few things:

the atomic-reator and osbs-client are in the base builder container, they are installed from fedora 33 (I know, it's EOL). So, to update those we would need to update them in fedora and update the image.

I can't see why it's getting a 404 there. I did a build and oc rsh'ed in and skopeo was happy to see the image. ;(

CC: @dkirwan @mobrien if you all could look more into this that would be lovely...

oc rsh to build container isn't the same as oc image-import

you have to verify that openshift is really able to access that registry via that command

as one thing is what you can access from build container, another thing is what openshift itself can access

It started to fail differently today (https://koji.fedoraproject.org/koji/taskinfo?taskID=81356127):

2022-01-17 12:11:03,215 platform:- - atomic_reactor.plugin - DEBUG - Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/atomic_reactor/plugin.py", line 217, in get_available_plugins
    plugin_class = self.plugin_classes[plugin_name]
KeyError: 'check_user_settings'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/atomic_reactor/inner.py", line 480, in build_docker_image
    prebuild_runner = PreBuildPluginsRunner(self.builder.tasker, self,
  File "/usr/lib/python3.10/site-packages/atomic_reactor/plugin.py", line 439, in __init__
    super(PreBuildPluginsRunner, self).__init__(dt, workflow, 'PreBuildPlugin', plugins_conf,
  File "/usr/lib/python3.10/site-packages/atomic_reactor/plugin.py", line 356, in __init__
    super(BuildPluginsRunner, self).__init__(plugin_class_name, plugins_conf, *args, **kwargs)
  File "/usr/lib/python3.10/site-packages/atomic_reactor/plugin.py", line 136, in __init__
    self.available_plugins = self.get_available_plugins()
  File "/usr/lib/python3.10/site-packages/atomic_reactor/plugin.py", line 225, in get_available_plugins
    raise exc
atomic_reactor.plugin.PluginFailedException: no such plugin: 'check_user_settings', did you set the correct plugin type?

Maybe related to the builder restart?

https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/M3ZEUB32XX4SQ355BQ64DU5JYHCRX47R/

there is now installed newer osbs-client (osbs-client-1.11.0) and still 2 years old atomic-reactor (atomic_reactor-1.6.47)

The new failure today was due to a mistake on my part, I was testing building an f35 based image and didn't tag it correctly so it got taken as the new buildroot image. I have reverted so we can try to solve the initial problem first

@lbalhar could you give it a try before we close the ticket ?

Yeah, redis built fine.

I've tried a scratch build in the morning and it worked. I think this can be closed now. Thanks for all the effort.

Great, thanks @darknao for the help on investigating the problem.

Metadata Update from @cverna:
- Issue close_status updated to: Fixed
- Issue status updated to: Closed (was: Open)

6 days ago

Login to comment on this ticket.

Metadata
Boards 1
ops Status: Backlog