Asynchronous Web API

As a Magento Developer, I would like to make WebAPI calls asynchronously so that my integrations don't have to wait for my requests to get processed to complete when it's not necessary.

In the product update context, this means that I can invoke single or bulk product APIs and get an immediate response that my request was accepted, and I'm offered a separate status reporting API to check on the progress of my requests by ID. Before Message Queue interfaces are moved to CE, this functionality will only be available in Commerce and not Open Source.

Asynchronous Endpoints

The role of asynchronous endpoint is to intercept the message to the Web API and write it to the message queue together with identifying information, such as UUID of operation.

Asynchronous endpoints will be one per operation.

Example:

POST /async/V1/poducts
PUT /async/V1/poducts
DELETE /async/V1/products

It will allow to accept both one entity and an array of entities.

Configuration

Configuration of the Async Web API will define the relation between endpoint url="/async/V1/products" method="POST" and Message Queue topic topic="products.created" The endpoint will accept messages in the format defined by following rules:

1. take the schema of the message configured for the topic
2. accept array of messages identified by this schema Separate configuration file async-webapi.xml will be used to define this mapping.

It should be possible both to put array of the messages to the queue as a single message and to have separate message per every element of the array. The is an example of configuration which assign a topic for every element of the input array:

<route url="/async/V1/products" method="POST">
    <item topic="products.created" />
</route>

The configuration which will map one topic for the whiole array of messages defined in the following way:

<route url="/async/V1/products" method="POST">
    <input topic="products.created" />
</route>

Endpoint will be mapped to the topic configured in configuration.xml:

<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework:Communication/etc/communication.xsd">
    <topic name="catalog.product.create" schema="Magento\Catalog\Api\ProductRepositoryInterface::save"></topic>
</config>

Response

The response of the service is constructed in the following way:

HTTP/1.1 202 Accepted
Location: /bulk/status/12345

"operation": "PUT /async/V1/products"
"id": "12314",
"entities": [
   {
      "id": "12134",
      "status": "accepted"
   },
   {
     "id": "12135",
     "status": "in_progress"
  },
  {
    "id": "12136",
    "status": "processed"
  }    
]

All operations are executed in the context of the particular store.

Questions

• Should async bulk request be POST only, or PUT and DELETE also needed?
• Should we support different operations as a part of one API request?
• Can we have one endpoint for all requests?
• Should the request support Swagger schema?
• Should more than one entity come in the request?
• Can we use AMQP instead?

Response

interface AsyncStatus {

   // status of async operatiuon
   public getStatus();

   // statuses for entities: (uuid, status)
   public getEntityStatuses();
}

Swagger Schema

Swagger schema generator should be updated in order to reflect the asynchronous response of the async endpoint

Status of asynchronous request

Request

GET /bulk/status/12345

Response

"operation": "PUT /async/V1/products"
"id": "12314",
"entities":
[
   {
      "id": "1213",
      "status": "accepted"
   },
   {
     "id": "1213",
     "status": "in_progress"
  },
  {
    "id": "1213",
    "status": "processed"
  }    
]

Writing of the input data to the

re-use Asynchronous Operations interfaces

\Magento\AsynchronousOperations\Model\BulkManagement::scheduleBulk

Writes bulk data to the database and to the queue. Generates the UI to track the status of the Bulk operation. Interfaces should be improved if certain use cases are not possible to implement.

Questions

• how to update the status of the bulk operation
• how to track individual items of the bulk operation
• how to leverage Redis instead of the database for the bulk operations

AMQP feature parity

Same functionality should be available by writing the messages to the RabbitMQ via AMQP protocol.

Multiple messages can be sent to the queue, processor can be bootstraped to process them. What features are missing in comparisoon to Bulk Wev API?

References

http://restcookbook.com/Resources/asynchroneous-operations/

Acceptance Criteria

• User can call product bulk update web API asynchronously
• Code Generation for Async responses should be possible via typed Swagger schema
• Bulk API should be performant under load, better mechanism for the Bulk API tracking might be introouced