Skip to content

Implement HTTP request/response source #839

@kerinin

Description

@kerinin

Idea is something along the lines of the code block shown below. Things to consider:

  1. Support both "read-only sources" (things which always respond 200 and are used in defining other responses)
  2. Support "read/write sources" (things which compute responses using kaskada)

It's OK if this only works with the new partitioned execution code paths (eg., using the Source / Destination traits).

# this produces `{ request_id: i64, data: <schema> }`
source = kd.sources.Http(
  schema = ...,
  # Optional response. This allows for requests that append data.
  # If not specified, then a later `run_and_respond` is necessary.
  respond_with = 200,
)

output = kd.record({
  request_id: source.request_id,
  x_plus_y = source.data.x + source.data.y,
)}

source.run_and_respond(
  # Use Kaskada to define responses.
  respond_with = output,
  # Report a warning if the output is not available
  # for a request within this timelimit.
  # this may happen if the request gets buffered in
  # a shift, etc.
  timelimit = 1s
)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions