Migrate RabbitMQ queues in ci-aio #1620

MoteHue · 2025-04-23T13:24:06Z

No description provided.

MoteHue · 2025-04-23T13:24:15Z

This relies on the following PR: #1619

.github/workflows/stackhpc-all-in-one.yml

Alex-Welsh · 2025-04-29T07:47:26Z

@MoteHue you might want to cherry-pick 3d847fa to fix the lint jobs

MoteHue · 2025-04-29T12:37:41Z

This needs to merge first: #1629

Alex-Welsh

LGTM if CI passes

MoteHue · 2025-04-30T14:30:39Z

Error in kayobe image building:

 #11 31.51 Error: 
#11 31.51  Problem: package gcc-11.5.0-5.el9_5.x86_64 from appstream requires glibc-devel >= 2.2.90-12, but none of the providers can be installed
#11 31.51   - conflicting requests
#11 31.51   - nothing provides glibc = 2.34-125.el9_5.3 needed by glibc-devel-2.34-125.el9_5.3.x86_64 from appstream
#11 31.51   - nothing provides glibc = 2.34-125.el9_5.3 needed by glibc-devel-2.34-125.el9_5.3.i686 from appstream
#11 31.51 (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

Alex-Welsh · 2025-05-01T07:33:04Z

Upgrade job fails with

  stderr: 'mv: cannot move ''/var/lib/docker/volumes/prometheus_v2/_data'' to ''/var/lib/docker/volumes/prometheus/_data'': Directory not empty'

The OVS jobs both fail with Neutron errors
The Nova API log says:

neutronclient.common.exceptions.InternalServerError: Request Failed: internal server error while processing your request.

And Neutron Server says:

2025-04-30 15:58:06.057 27 INFO neutron.api.wsgi [None req-59fde41f-414b-4ea6-96f5-1c65e8f322b8 edd1dd0003a748f1b6f7f45aa138f47b 7f50da72d1a0478aaba23725a2a3879e - - default default] 192.168.33.2,192.168.33.3 "GET /v2.0/network-ip-availabilities HTTP/1.1" status: 200  len: 1298 time: 0.0107384
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation [None req-72faf510-e07d-4ff9-8f46-1e4e426652c4 7149963603b447caa9e1173800f7f834 1d056ff1fc0046efa37c0aca7e86e114 - - default default] GET failed.: AttributeError: 'NoneType' object has no attribute 'tags'
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation Traceback (most recent call last):
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/pecan/core.py", line 682, in __call__
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     self.invoke_controller(controller, args, kwargs, state)
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/pecan/core.py", line 573, in invoke_controller
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     result = controller(*args, **kwargs)
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     setattr(e, '_RETRY_EXCEEDED', True)
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     ectxt.value = e.inner_exc
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron_lib/db/api.py", line 187, in wrapped
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     context_reference.session.rollback()
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron_lib/db/api.py", line 181, in wrapped
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     return f(*dup_args, **dup_kwargs)
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/pecan_wsgi/controllers/utils.py", line 65, in wrapped
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/pecan_wsgi/controllers/resource.py", line 135, in index
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     return self.get(*args, **kwargs)
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/pecan_wsgi/controllers/resource.py", line 144, in get
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     return {self.collection: self.plugin_lister(*lister_args,
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron_lib/db/api.py", line 223, in wrapped
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     return f_with_retry(*args, **kwargs,
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron_lib/db/api.py", line 139, in wrapped
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     setattr(e, '_RETRY_EXCEEDED', True)
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron_lib/db/api.py", line 135, in wrapped
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_db/api.py", line 154, in wrapper
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     ectxt.value = e.inner_exc
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_db/api.py", line 142, in wrapper
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     return f(*args, **kwargs)
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron_lib/db/api.py", line 187, in wrapped
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     context_reference.session.rollback()
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     self.force_reraise()
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     raise self.value
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron_lib/db/api.py", line 181, in wrapped
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     return f(*dup_args, **dup_kwargs)
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1183, in wrapper
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     return fn(*args, **kwargs)
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/db/db_base_plugin_v2.py", line 1682, in get_ports
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     items = [self._make_port_dict(c, fields, bulk=True) for c in query]
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/db/db_base_plugin_v2.py", line 1682, in <listcomp>
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     items = [self._make_port_dict(c, fields, bulk=True) for c in query]
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/db/db_base_plugin_common.py", line 257, in _make_port_dict
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     resource_extend.apply_funcs(
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_utils/timeutils.py", line 323, in wrapper
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     result = func(*args, **kwargs)
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron_lib/db/resource_extend.py", line 91, in apply_funcs
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     resolved_func(response, db_object)
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/services/tag/tag_plugin.py", line 54, in _extend_tags_dict
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation     tag_db.tag for tag_db in db_data.standard_attr.tags
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation AttributeError: 'NoneType' object has no attribute 'tags'
2025-04-30 15:58:06.094 27 ERROR neutron.pecan_wsgi.hooks.translation

L3 agent logs say:

dhcp agent logs have:

oslo_messaging.rpc.client.RemoteError: Remote error: SubnetInUse Unable to complete operation on subnet 14855e14-668e-419d-aa9c-73b7d097265b: This subnet is being modified by another concurrent operation.

and again:

ERROR neutron.agent.dhcp.agent AttributeError: 'NoneType' object has no attribute 'tags'

MoteHue · 2025-05-01T12:11:55Z

Prometheus upgrade failure seems to be a bug in KA, I've raised it here: https://bugs.launchpad.net/kolla-ansible/+bug/2109784

Alex-Welsh · 2025-05-07T11:00:22Z

Looking at the logs for the latest CI failure:
It failed 11 tempest tests, they're all test_list_servers_negative.ListServersNegativeTestJSON.*

The errors are all related to a nova api issue, which in turn resolved to this error in neutron server:

2025-05-07 09:23:09.327 36 ERROR neutron.pecan_wsgi.hooks.translation [req-f7c9f2d8-09af-469e-aafc-bdb0198d26fb req-8d00134f-203f-4ea7-ac05-4ba78e58e716 192c1fb7030a41f5a534ebc14e7e19ce 06e0e4b7f2fc47bfb2eab5e38c461c60 - - default default] GET failed.: AttributeError: 'NoneType' object has no attribute 'tags'
2025-05-07 09:23:09.327 36 ERROR neutron.pecan_wsgi.hooks.translation Traceback (most recent call last):
2025-05-07 09:23:09.327 36 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib/python3.12/site-packages/pecan/core.py", line 682, in __call__
...
<long python trace>
...
2025-05-07 09:23:09.327 36 ERROR neutron.pecan_wsgi.hooks.translation   File "/var/lib/kolla/venv/lib/python3.12/site-packages/neutron/services/tag/tag_plugin.py", line 54, in _extend_tags_dict
2025-05-07 09:23:09.327 36 ERROR neutron.pecan_wsgi.hooks.translation     tag_db.tag for tag_db in db_data.standard_attr.tags
2025-05-07 09:23:09.327 36 ERROR neutron.pecan_wsgi.hooks.translation                              ^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-05-07 09:23:09.327 36 ERROR neutron.pecan_wsgi.hooks.translation AttributeError: 'NoneType' object has no attribute 'tags'

I'm rerunning it and hoping it's some general transient error which gets fixed magically, but if not we'll have to take another look

Migrate RabbitMQ queues in ci-aio

ec95a9c

MoteHue requested a review from a team as a code owner April 23, 2025 13:24

product-auto-label bot added size: s workflows Workflow files have been modified labels Apr 23, 2025

MoteHue closed this Apr 23, 2025

MoteHue reopened this Apr 23, 2025

Alex-Welsh requested changes Apr 24, 2025

View reviewed changes

.github/workflows/stackhpc-all-in-one.yml Outdated Show resolved Hide resolved

.github/workflows/stackhpc-all-in-one.yml Outdated Show resolved Hide resolved

Alex-Welsh mentioned this pull request Apr 29, 2025

Sync stackhpc/2024.1 into master #1618

Merged

MoteHue force-pushed the epoxy-rabbitmq-queue-migration branch from c83b800 to 7674d51 Compare April 29, 2025 09:56

product-auto-label bot added size: m and removed size: s labels Apr 29, 2025

MoteHue force-pushed the epoxy-rabbitmq-queue-migration branch from 7674d51 to bb7682d Compare April 29, 2025 10:00

Bump ansible-core version in lint jobs

c0d6b00

MoteHue force-pushed the epoxy-rabbitmq-queue-migration branch from bb7682d to c0d6b00 Compare April 29, 2025 12:37

product-auto-label bot added size: s and removed size: m labels Apr 29, 2025

Run upgrade prerequisites instead

9a4b5b3

Alex-Welsh previously approved these changes Apr 29, 2025

View reviewed changes

MoteHue closed this Apr 29, 2025

MoteHue reopened this Apr 29, 2025

Alex-Welsh mentioned this pull request Apr 30, 2025

Update to Prometheus v3 #1630

Merged

Merge branch 'stackhpc/master' into epoxy-rabbitmq-queue-migration

3fdb4f2

Alex-Welsh dismissed their stale review via 066b277 April 30, 2025 14:59

Alex-Welsh force-pushed the epoxy-rabbitmq-queue-migration branch from 066b277 to 3fdb4f2 Compare April 30, 2025 15:02

Alex-Welsh mentioned this pull request May 2, 2025

RabbitMQ 4.0 #1632

Merged

Alex-Welsh force-pushed the epoxy-rabbitmq-queue-migration branch from 039aa68 to 3fdb4f2 Compare May 6, 2025 08:32

MoteHue closed this May 6, 2025

MoteHue reopened this May 6, 2025

MoteHue closed this May 6, 2025

MoteHue reopened this May 6, 2025

Alex-Welsh approved these changes May 7, 2025

View reviewed changes

Alex-Welsh merged commit b863ab5 into stackhpc/master May 7, 2025
53 of 65 checks passed

Alex-Welsh deleted the epoxy-rabbitmq-queue-migration branch May 7, 2025 11:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Migrate RabbitMQ queues in ci-aio #1620

Migrate RabbitMQ queues in ci-aio #1620

Uh oh!

MoteHue commented Apr 23, 2025

Uh oh!

MoteHue commented Apr 23, 2025

Uh oh!

Uh oh!

Uh oh!

Alex-Welsh commented Apr 29, 2025

Uh oh!

MoteHue commented Apr 29, 2025

Uh oh!

Alex-Welsh left a comment

Uh oh!

MoteHue commented Apr 30, 2025

Uh oh!

Alex-Welsh commented May 1, 2025 •

edited

Loading

Uh oh!

MoteHue commented May 1, 2025

Uh oh!

Alex-Welsh commented May 7, 2025

Uh oh!

Uh oh!

Uh oh!

Migrate RabbitMQ queues in ci-aio #1620

Migrate RabbitMQ queues in ci-aio #1620

Uh oh!

Conversation

MoteHue commented Apr 23, 2025

Uh oh!

MoteHue commented Apr 23, 2025

Uh oh!

Uh oh!

Uh oh!

Alex-Welsh commented Apr 29, 2025

Uh oh!

MoteHue commented Apr 29, 2025

Uh oh!

Alex-Welsh left a comment

Choose a reason for hiding this comment

Uh oh!

MoteHue commented Apr 30, 2025

Uh oh!

Alex-Welsh commented May 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MoteHue commented May 1, 2025

Uh oh!

Alex-Welsh commented May 7, 2025

Uh oh!

Uh oh!

Uh oh!

Alex-Welsh commented May 1, 2025 •

edited

Loading