Skip to content

Commit 5b4cb92

Browse files
melwittmelwitt
committed
Use force=True for os-brick disconnect during delete
The 'force' parameter of os-brick's disconnect_volume() method allows callers to ignore flushing errors and ensure that devices are being removed from the host. We should use force=True when we are going to delete an instance to avoid leaving leftover devices connected to the compute host which could then potentially be reused to map to volumes to an instance that should not have access to those volumes. We can use force=True even when disconnecting a volume that will not be deleted on termination because os-brick will always attempt to flush and disconnect gracefully before forcefully removing devices. Conflicts: nova/tests/unit/virt/libvirt/volume/test_lightos.py nova/virt/libvirt/volume/lightos.py NOTE(melwitt): The conflicts are because change Ic314b26695d9681d31a18adcec0794c2ff41fe71 (Lightbits LightOS driver) is not in Xena. NOTE(melwitt): The difference from the cherry picked change is because of the following additional affected volume driver in Wallaby: * nova/virt/libvirt/volume/net.py Closes-Bug: #2004555 Change-Id: I3629b84d3255a8fe9d8a7cea8c6131d7c40899e8 (cherry picked from commit db45554) (cherry picked from commit efb0198) (cherry picked from commit 8b4b991) (cherry picked from commit 4d8efa2) (cherry picked from commit b574901)
1 parent 9813eac commit 5b4cb92

39 files changed

+406
-113
lines changed

‎doc/source/admin/configuration/cross-cell-resize.rst‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -271,7 +271,7 @@ Troubleshooting
271271
Timeouts
272272
~~~~~~~~
273273

274-
Configure a :ref:`service user <user_token_timeout>` in case the user token
274+
Configure a :ref:`service user <service_user_token>` in case the user token
275275
times out, e.g. during the snapshot and download of a large server image.
276276

277277
If RPC calls are timing out with a ``MessagingTimeout`` error in the logs,

‎doc/source/admin/configuration/index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ A list of config options based on different topics can be found below:
1919
.. toctree::
2020
:maxdepth: 1
2121

22+
/admin/configuration/service-user-token
2223
/admin/configuration/api
2324
/admin/configuration/resize
2425
/admin/configuration/cross-cell-resize

‎doc/source/admin/configuration/service-user-token.rst‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
.. _service_user_token:
2+
3+
===================
4+
Service User Tokens
5+
===================
6+
7+
.. note::
8+
9+
Configuration of service user tokens is **required** for every Nova service
10+
for security reasons. See https://bugs.launchpad.net/nova/+bug/2004555 for
11+
details.
12+
13+
Configure Nova to send service user tokens alongside regular user tokens when
14+
making REST API calls to other services. The identity service (Keystone) will
15+
authenticate a request using the service user token if the regular user token
16+
has expired.
17+
18+
This is important when long-running operations such as live migration or
19+
snapshot take long enough to exceed the expiry of the user token. Without the
20+
service token, if a long-running operation exceeds the expiry of the user
21+
token, post operations such as cleanup after a live migration could fail when
22+
Nova calls other service APIs like block-storage (Cinder) or networking
23+
(Neutron).
24+
25+
The service token is also used by services to validate whether the API caller
26+
is a service. Some service APIs are restricted to service users only.
27+
28+
To set up service tokens, create a ``nova`` service user and ``service`` role
29+
in the identity service (Keystone) and assign the ``service`` role to the
30+
``nova`` service user.
31+
32+
Then, configure the :oslo.config:group:`service_user` section of the Nova
33+
configuration file, for example:
34+
35+
.. code-block:: ini
36+
37+
[service_user]
38+
send_service_user_token = true
39+
auth_url = https://104.130.216.102/identity
40+
auth_strategy = keystone
41+
auth_type = password
42+
project_domain_name = Default
43+
project_name = service
44+
user_domain_name = Default
45+
username = nova
46+
password = secretservice
47+
...
48+
49+
And configure the other identity options as necessary for the service user,
50+
much like you would configure nova to work with the image service (Glance) or
51+
networking service (Neutron).
52+
53+
.. note::
54+
55+
Please note that the role assigned to the :oslo.config:group:`service_user`
56+
needs to be in the configured
57+
:oslo.config:option:`keystone_authtoken.service_token_roles` of other
58+
services such as block-storage (Cinder), image (Glance), and networking
59+
(Neutron).

‎doc/source/admin/live-migration-usage.rst‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -320,4 +320,4 @@ To make live-migration succeed, you have several options:
320320

321321
If live migrations routinely timeout or fail during cleanup operations due
322322
to the user token timing out, consider configuring nova to use
323-
:ref:`service user tokens <user_token_timeout>`.
323+
:ref:`service user tokens <service_user_token>`.

‎doc/source/admin/migrate-instance-with-snapshot.rst‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -67,7 +67,7 @@ Create a snapshot of the instance
6767
6868
If snapshot operations routinely fail because the user token times out
6969
while uploading a large disk image, consider configuring nova to use
70-
:ref:`service user tokens <user_token_timeout>`.
70+
:ref:`service user tokens <service_user_token>`.
7171

7272
#. Use the :command:`openstack image list` command to check the status
7373
until the status is ``ACTIVE``:

‎doc/source/admin/support-compute.rst‎

Lines changed: 0 additions & 64 deletions
Original file line numberDiff line numberDiff line change
@@ -478,67 +478,3 @@ Ensure the ``compute`` endpoint in the identity service catalog is pointing
478478
at ``/v2.1`` instead of ``/v2``. The former route supports microversions,
479479
while the latter route is considered the legacy v2.0 compatibility-mode
480480
route which renders all requests as if they were made on the legacy v2.0 API.
481-
482-
483-
.. _user_token_timeout:
484-
485-
User token times out during long-running operations
486-
---------------------------------------------------
487-
488-
Problem
489-
~~~~~~~
490-
491-
Long-running operations such as live migration or snapshot can sometimes
492-
overrun the expiry of the user token. In such cases, post operations such
493-
as cleaning up after a live migration can fail when the nova-compute service
494-
needs to cleanup resources in other services, such as in the block-storage
495-
(cinder) or networking (neutron) services.
496-
497-
For example:
498-
499-
.. code-block:: console
500-
501-
2018-12-17 13:47:29.591 16987 WARNING nova.virt.libvirt.migration [req-7bc758de-b2e4-461b-a971-f79be6cd4703 313d1247d7b845da9c731eec53e50a26 2f693c782fa748c2baece8db95b4ba5b - default default] [instance: ead8ecc3-f473-4672-a67b-c44534c6042d] Live migration not completed after 2400 sec
502-
2018-12-17 13:47:30.097 16987 WARNING nova.virt.libvirt.driver [req-7bc758de-b2e4-461b-a971-f79be6cd4703 313d1247d7b845da9c731eec53e50a26 2f693c782fa748c2baece8db95b4ba5b - default default] [instance: ead8ecc3-f473-4672-a67b-c44534c6042d] Migration operation was cancelled
503-
2018-12-17 13:47:30.299 16987 ERROR nova.virt.libvirt.driver [req-7bc758de-b2e4-461b-a971-f79be6cd4703 313d1247d7b845da9c731eec53e50a26 2f693c782fa748c2baece8db95b4ba5b - default default] [instance: ead8ecc3-f473-4672-a67b-c44534c6042d] Live Migration failure: operation aborted: migration job: canceled by client: libvirtError: operation aborted: migration job: canceled by client
504-
2018-12-17 13:47:30.685 16987 INFO nova.compute.manager [req-7bc758de-b2e4-461b-a971-f79be6cd4703 313d1247d7b845da9c731eec53e50a26 2f693c782fa748c2baece8db95b4ba5b - default default] [instance: ead8ecc3-f473-4672-a67b-c44534c6042d] Swapping old allocation on 3e32d595-bd1f-4136-a7f4-c6703d2fbe18 held by migration 17bec61d-544d-47e0-a1c1-37f9d7385286 for instance
505-
2018-12-17 13:47:32.450 16987 ERROR nova.volume.cinder [req-7bc758de-b2e4-461b-a971-f79be6cd4703 313d1247d7b845da9c731eec53e50a26 2f693c782fa748c2baece8db95b4ba5b - default default] Delete attachment failed for attachment 58997d5b-24f0-4073-819e-97916fb1ee19. Error: The request you have made requires authentication. (HTTP 401) Code: 401: Unauthorized: The request you have made requires authentication. (HTTP 401)
506-
507-
Solution
508-
~~~~~~~~
509-
510-
Configure nova to use service user tokens to supplement the regular user token
511-
used to initiate the operation. The identity service (keystone) will then
512-
authenticate a request using the service user token if the user token has
513-
already expired.
514-
515-
To use, create a service user in the identity service similar as you would when
516-
creating the ``nova`` service user.
517-
518-
Then configure the :oslo.config:group:`service_user` section of the nova
519-
configuration file, for example:
520-
521-
.. code-block:: ini
522-
523-
[service_user]
524-
send_service_user_token = True
525-
auth_type = password
526-
project_domain_name = Default
527-
project_name = service
528-
user_domain_name = Default
529-
password = secretservice
530-
username = nova
531-
auth_url = https://104.130.216.102/identity
532-
...
533-
534-
And configure the other identity options as necessary for the service user,
535-
much like you would configure nova to work with the image service (glance)
536-
or networking service.
537-
538-
.. note::
539-
540-
Please note that the role of the :oslo.config:group:`service_user` you
541-
configure needs to be a superset of
542-
:oslo.config:option:`keystone_authtoken.service_token_roles` (The option
543-
:oslo.config:option:`keystone_authtoken.service_token_roles` is configured
544-
in cinder, glance and neutron).

‎doc/source/install/compute-install-obs.rst‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -92,6 +92,26 @@ Install and configure components
9292
Comment out or remove any other options in the ``[keystone_authtoken]``
9393
section.
9494

95+
* In the ``[service_user]`` section, configure :ref:`service user
96+
tokens <service_user_token>`:
97+
98+
.. path /etc/nova/nova.conf
99+
.. code-block:: ini
100+
101+
[service_user]
102+
send_service_user_token = true
103+
auth_url = https://controller/identity
104+
auth_strategy = keystone
105+
auth_type = password
106+
project_domain_name = Default
107+
project_name = service
108+
user_domain_name = Default
109+
username = nova
110+
password = NOVA_PASS
111+
112+
Replace ``NOVA_PASS`` with the password you chose for the ``nova`` user in
113+
the Identity service.
114+
95115
* In the ``[DEFAULT]`` section, configure the ``my_ip`` option:
96116

97117
.. path /etc/nova/nova.conf

‎doc/source/install/compute-install-rdo.rst‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,26 @@ Install and configure components
8484
Comment out or remove any other options in the ``[keystone_authtoken]``
8585
section.
8686

87+
* In the ``[service_user]`` section, configure :ref:`service user
88+
tokens <service_user_token>`:
89+
90+
.. path /etc/nova/nova.conf
91+
.. code-block:: ini
92+
93+
[service_user]
94+
send_service_user_token = true
95+
auth_url = https://controller/identity
96+
auth_strategy = keystone
97+
auth_type = password
98+
project_domain_name = Default
99+
project_name = service
100+
user_domain_name = Default
101+
username = nova
102+
password = NOVA_PASS
103+
104+
Replace ``NOVA_PASS`` with the password you chose for the ``nova`` user in
105+
the Identity service.
106+
87107
* In the ``[DEFAULT]`` section, configure the ``my_ip`` option:
88108

89109
.. path /etc/nova/nova.conf

‎doc/source/install/compute-install-ubuntu.rst‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,26 @@ Install and configure components
7474
Comment out or remove any other options in the
7575
``[keystone_authtoken]`` section.
7676

77+
* In the ``[service_user]`` section, configure :ref:`service user
78+
tokens <service_user_token>`:
79+
80+
.. path /etc/nova/nova.conf
81+
.. code-block:: ini
82+
83+
[service_user]
84+
send_service_user_token = true
85+
auth_url = https://controller/identity
86+
auth_strategy = keystone
87+
auth_type = password
88+
project_domain_name = Default
89+
project_name = service
90+
user_domain_name = Default
91+
username = nova
92+
password = NOVA_PASS
93+
94+
Replace ``NOVA_PASS`` with the password you chose for the ``nova`` user in
95+
the Identity service.
96+
7797
* In the ``[DEFAULT]`` section, configure the ``my_ip`` option:
7898

7999
.. path /etc/nova/nova.conf

‎doc/source/install/controller-install-obs.rst‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -260,6 +260,26 @@ Install and configure components
260260
Comment out or remove any other options in the ``[keystone_authtoken]``
261261
section.
262262

263+
* In the ``[service_user]`` section, configure :ref:`service user
264+
tokens <service_user_token>`:
265+
266+
.. path /etc/nova/nova.conf
267+
.. code-block:: ini
268+
269+
[service_user]
270+
send_service_user_token = true
271+
auth_url = https://controller/identity
272+
auth_strategy = keystone
273+
auth_type = password
274+
project_domain_name = Default
275+
project_name = service
276+
user_domain_name = Default
277+
username = nova
278+
password = NOVA_PASS
279+
280+
Replace ``NOVA_PASS`` with the password you chose for the ``nova`` user in
281+
the Identity service.
282+
263283
* In the ``[DEFAULT]`` section, configure the ``my_ip`` option to use the
264284
management interface IP address of the controller node:
265285

0 commit comments

Comments
 (0)