Merge "Use force=True for os-brick disconnect during delete" into stable/wallaby

Author: Zuul

doc/source/admin/configuration/cross-cell-resize.rst

@@ -271,7 +271,7 @@ Troubleshooting
 Timeouts
 ~~~~~~~~
 
-Configure a :ref:`service user <user_token_timeout>` in case the user token
+Configure a :ref:`service user <service_user_token>` in case the user token
 times out, e.g. during the snapshot and download of a large server image.
 
 If RPC calls are timing out with a ``MessagingTimeout`` error in the logs,

doc/source/admin/configuration/index.rst

@@ -19,6 +19,7 @@ A list of config options based on different topics can be found below:
 .. toctree::
    :maxdepth: 1
 
+   /admin/configuration/service-user-token
    /admin/configuration/api
    /admin/configuration/resize
    /admin/configuration/cross-cell-resize

doc/source/admin/configuration/service-user-token.rst

@@ -0,0 +1,59 @@
+.. _service_user_token:
+
+===================
+Service User Tokens
+===================
+
+.. note::
+
+   Configuration of service user tokens is **required** for every Nova service
+   for security reasons. See https://bugs.launchpad.net/nova/+bug/2004555 for
+   details.
+
+Configure Nova to send service user tokens alongside regular user tokens when
+making REST API calls to other services. The identity service (Keystone) will
+authenticate a request using the service user token if the regular user token
+has expired.
+
+This is important when long-running operations such as live migration or
+snapshot take long enough to exceed the expiry of the user token. Without the
+service token, if a long-running operation exceeds the expiry of the user
+token, post operations such as cleanup after a live migration could fail when
+Nova calls other service APIs like block-storage (Cinder) or networking
+(Neutron).
+
+The service token is also used by services to validate whether the API caller
+is a service. Some service APIs are restricted to service users only.
+
+To set up service tokens, create a ``nova`` service user and ``service`` role
+in the identity service (Keystone) and assign the ``service`` role to the
+``nova`` service user.
+
+Then, configure the :oslo.config:group:`service_user` section of the Nova
+configuration file, for example:
+
+.. code-block:: ini
+
+   [service_user]
+   send_service_user_token = true
+   auth_url = https://104.130.216.102/identity
+   auth_strategy = keystone
+   auth_type = password
+   project_domain_name = Default
+   project_name = service
+   user_domain_name = Default
+   username = nova
+   password = secretservice
+   ...
+
+And configure the other identity options as necessary for the service user,
+much like you would configure nova to work with the image service (Glance) or
+networking service (Neutron).
+
+.. note::
+
+   Please note that the role assigned to the :oslo.config:group:`service_user`
+   needs to be in the configured
+   :oslo.config:option:`keystone_authtoken.service_token_roles` of other
+   services such as block-storage (Cinder), image (Glance), and networking
+   (Neutron).

doc/source/admin/live-migration-usage.rst

@@ -320,4 +320,4 @@ To make live-migration succeed, you have several options:
 
 If live migrations routinely timeout or fail during cleanup operations due
 to the user token timing out, consider configuring nova to use
-:ref:`service user tokens <user_token_timeout>`.
+:ref:`service user tokens <service_user_token>`.

doc/source/admin/migrate-instance-with-snapshot.rst

@@ -67,7 +67,7 @@ Create a snapshot of the instance
 
    If snapshot operations routinely fail because the user token times out
    while uploading a large disk image, consider configuring nova to use
-   :ref:`service user tokens <user_token_timeout>`.
+   :ref:`service user tokens <service_user_token>`.
 
 #. Use the :command:`openstack image list` command to check the status
    until the status is ``ACTIVE``:

doc/source/admin/support-compute.rst

@@ -478,67 +478,3 @@ Ensure the ``compute`` endpoint in the identity service catalog is pointing
 at ``/v2.1`` instead of ``/v2``. The former route supports microversions,
 while the latter route is considered the legacy v2.0 compatibility-mode
 route which renders all requests as if they were made on the legacy v2.0 API.
-
-
-.. _user_token_timeout:
-
-User token times out during long-running operations
----------------------------------------------------
-
-Problem
-~~~~~~~
-
-Long-running operations such as live migration or snapshot can sometimes
-overrun the expiry of the user token. In such cases, post operations such
-as cleaning up after a live migration can fail when the nova-compute service
-needs to cleanup resources in other services, such as in the block-storage
-(cinder) or networking (neutron) services.
-
-For example:
-
-.. code-block:: console
-
-  2018-12-17 13:47:29.591 16987 WARNING nova.virt.libvirt.migration [req-7bc758de-b2e4-461b-a971-f79be6cd4703 313d1247d7b845da9c731eec53e50a26 2f693c782fa748c2baece8db95b4ba5b - default default] [instance: ead8ecc3-f473-4672-a67b-c44534c6042d] Live migration not completed after 2400 sec
-  2018-12-17 13:47:30.097 16987 WARNING nova.virt.libvirt.driver [req-7bc758de-b2e4-461b-a971-f79be6cd4703 313d1247d7b845da9c731eec53e50a26 2f693c782fa748c2baece8db95b4ba5b - default default] [instance: ead8ecc3-f473-4672-a67b-c44534c6042d] Migration operation was cancelled
-  2018-12-17 13:47:30.299 16987 ERROR nova.virt.libvirt.driver [req-7bc758de-b2e4-461b-a971-f79be6cd4703 313d1247d7b845da9c731eec53e50a26 2f693c782fa748c2baece8db95b4ba5b - default default] [instance: ead8ecc3-f473-4672-a67b-c44534c6042d] Live Migration failure: operation aborted: migration job: canceled by client: libvirtError: operation aborted: migration job: canceled by client
-  2018-12-17 13:47:30.685 16987 INFO nova.compute.manager [req-7bc758de-b2e4-461b-a971-f79be6cd4703 313d1247d7b845da9c731eec53e50a26 2f693c782fa748c2baece8db95b4ba5b - default default] [instance: ead8ecc3-f473-4672-a67b-c44534c6042d] Swapping old allocation on 3e32d595-bd1f-4136-a7f4-c6703d2fbe18 held by migration 17bec61d-544d-47e0-a1c1-37f9d7385286 for instance
-  2018-12-17 13:47:32.450 16987 ERROR nova.volume.cinder [req-7bc758de-b2e4-461b-a971-f79be6cd4703 313d1247d7b845da9c731eec53e50a26 2f693c782fa748c2baece8db95b4ba5b - default default] Delete attachment failed for attachment 58997d5b-24f0-4073-819e-97916fb1ee19. Error: The request you have made requires authentication. (HTTP 401) Code: 401: Unauthorized: The request you have made requires authentication. (HTTP 401)
-
-Solution
-~~~~~~~~
-
-Configure nova to use service user tokens to supplement the regular user token
-used to initiate the operation. The identity service (keystone) will then
-authenticate a request using the service user token if the user token has
-already expired.
-
-To use, create a service user in the identity service similar as you would when
-creating the ``nova`` service user.
-
-Then configure the :oslo.config:group:`service_user` section of the nova
-configuration file, for example:
-
-.. code-block:: ini
-
-  [service_user]
-  send_service_user_token = True
-  auth_type = password
-  project_domain_name = Default
-  project_name = service
-  user_domain_name = Default
-  password = secretservice
-  username = nova
-  auth_url = https://104.130.216.102/identity
-  ...
-
-And configure the other identity options as necessary for the service user,
-much like you would configure nova to work with the image service (glance)
-or networking service.
-
-.. note::
-
-  Please note that the role of the :oslo.config:group:`service_user` you
-  configure needs to be a superset of
-  :oslo.config:option:`keystone_authtoken.service_token_roles` (The option
-  :oslo.config:option:`keystone_authtoken.service_token_roles` is configured
-  in cinder, glance and neutron).

doc/source/install/compute-install-obs.rst

@@ -92,6 +92,26 @@ Install and configure components
         Comment out or remove any other options in the ``[keystone_authtoken]``
         section.
 
+   * In the ``[service_user]`` section, configure :ref:`service user
+     tokens <service_user_token>`:
+
+     .. path /etc/nova/nova.conf
+     .. code-block:: ini
+
+        [service_user]
+        send_service_user_token = true
+        auth_url = https://controller/identity
+        auth_strategy = keystone
+        auth_type = password
+        project_domain_name = Default
+        project_name = service
+        user_domain_name = Default
+        username = nova
+        password = NOVA_PASS
+
+     Replace ``NOVA_PASS`` with the password you chose for the ``nova`` user in
+     the Identity service.
+
    * In the ``[DEFAULT]`` section, configure the ``my_ip`` option:
 
      .. path /etc/nova/nova.conf

doc/source/install/compute-install-rdo.rst

@@ -84,6 +84,26 @@ Install and configure components
         Comment out or remove any other options in the ``[keystone_authtoken]``
         section.
 
+   * In the ``[service_user]`` section, configure :ref:`service user
+     tokens <service_user_token>`:
+
+     .. path /etc/nova/nova.conf
+     .. code-block:: ini
+
+        [service_user]
+        send_service_user_token = true
+        auth_url = https://controller/identity
+        auth_strategy = keystone
+        auth_type = password
+        project_domain_name = Default
+        project_name = service
+        user_domain_name = Default
+        username = nova
+        password = NOVA_PASS
+
+     Replace ``NOVA_PASS`` with the password you chose for the ``nova`` user in
+     the Identity service.
+
    * In the ``[DEFAULT]`` section, configure the ``my_ip`` option:
 
      .. path /etc/nova/nova.conf

doc/source/install/compute-install-ubuntu.rst

@@ -74,6 +74,26 @@ Install and configure components
         Comment out or remove any other options in the
         ``[keystone_authtoken]`` section.
 
+   * In the ``[service_user]`` section, configure :ref:`service user
+     tokens <service_user_token>`:
+
+     .. path /etc/nova/nova.conf
+     .. code-block:: ini
+
+        [service_user]
+        send_service_user_token = true
+        auth_url = https://controller/identity
+        auth_strategy = keystone
+        auth_type = password
+        project_domain_name = Default
+        project_name = service
+        user_domain_name = Default
+        username = nova
+        password = NOVA_PASS
+
+     Replace ``NOVA_PASS`` with the password you chose for the ``nova`` user in
+     the Identity service.
+
    * In the ``[DEFAULT]`` section, configure the ``my_ip`` option:
 
      .. path /etc/nova/nova.conf

doc/source/install/controller-install-obs.rst

@@ -260,6 +260,26 @@ Install and configure components
         Comment out or remove any other options in the ``[keystone_authtoken]``
         section.
 
+   * In the ``[service_user]`` section, configure :ref:`service user
+     tokens <service_user_token>`:
+
+     .. path /etc/nova/nova.conf
+     .. code-block:: ini
+
+        [service_user]
+        send_service_user_token = true
+        auth_url = https://controller/identity
+        auth_strategy = keystone
+        auth_type = password
+        project_domain_name = Default
+        project_name = service
+        user_domain_name = Default
+        username = nova
+        password = NOVA_PASS
+
+     Replace ``NOVA_PASS`` with the password you chose for the ``nova`` user in
+     the Identity service.
+
    * In the ``[DEFAULT]`` section, configure the ``my_ip`` option to use the
      management interface IP address of the controller node: