Importers managed through the API should have basic functionality

[carlosthe19916]

Current status

• We create an importer through /api/v1/importer/{name}
• The server automatically starts the ingestion of data configured in the importer

Questions:

• What if I want to execute the importer manually? e.g. the period configured is every month but I want to execute it right now.
• How to monitor the execution of the importer? has it started? it failed? it succeeded? How the Ui can now it is running?
• How to cancel the execution of an importer?

Proposal

In my opinion there are 2 abstractions that should be decoupled:

• Importer: contains the configuration of where and how files should be fetched. e.g. serverUrl, credentials, etc. No other info should be there.
• Task or Execution: contains information about a single execution of an importer

The idea is that a single importer can be executed multiple times. Otherwise we will need to create a new importer every time we need to import data.

Here are more details:

• Create an importer POST /importers:

{
  "source": "http://myurl.com",
}

• Create a Task (execute an importer once) POST /tasks

{
  "importerId": 1, // This is the id of the importer we want to execute
}

• In case we want to execute an importer multiple times (cron) we can create a task like POST /tasks

{
  "importerId": 1, // This is the id of the importer we want to execute
  "cron": "0 0 1 * *"

• For monitoring the status a task we can do: GET /tasks?status=scheduled|running
  This should return a list of tasks running in the system. Note that multiple tasks could be running at the same time

• For canceling a task we can do PUT /task/{taskId}

{
  "id": 1,
  "status": "cancel"
}

[ctron]

We already have that partially, I think it makes sense to check what's there.

[carlosthe19916]

yep, I know we have it partially. That is why I opened this discussion. To share some thoughts about how to cover full functionality of importers life-cycle. I didn't mean, we have to do it right now. But open the discussion

[bobmcwhirter]

Somewhat a tangent, but how do we feel about singular-vs-plural?

Should we PUT to /tasks/{id} or /task/{id}?

Should we GET from /tasks for the collection?

[carlosthe19916]

About task/{id} that is my mistake. I think it should be /tasks/{id}.
In general I think we should follow common practices as I described here #151

• GET /personas: List of personas
• POST /personas: Create a single persona. Any info required goes in the body of the POST request
• GET /personas/{id}: get a single persona
• PUT /personas/{id}: update a persona
• DELETE /personas/{id}: delete a persona

So endpoints in plural. Not a big sin to use singular but just common practices.

[ctron]

Can we please stop claiming "common practices" without any reference.

[bobmcwhirter]

While I have no strong opinion, I do like consistency, and if push comes to shove, I prefer singular, because English is a stupid language and as soon as we have a collection of index items, we devolve into indexes vs indices.

#cactus