Pattern for collating paginated data from http.send

[devonsparks]

Many HTTP API services that might provide input to Rego policies (e.g., via http.send) use paged responses, where a continuation token or reference for the "next page" is provided in the current response. This makes resource collections (generally terminating) linked lists of pages, where a client can walk over them in sequence to collate as many results as it needs.

Without support for recursion, is there a Rego idiom or pattern for collating data across pages of an HTTP resource collection without having to resort to a custom built-in?

As a concrete example, assume you have the following partial collections:

GET /basket/1/items?page=1

{"items":[{"value":"A"}, {"value":"B"}], "next":"/basket/2/page=2"}

GET /basket/1/items?page=2

{"items":[{"value":"C"}, {"value":"D"}], "next":"/basket/2/page=3"}

GET /basket/1/items?page=3

{"items":[{"value":"E"}], "next":null}

And what we want (through the use of Rego rules and http.send) is the aggregate set:

[{"value":"A"}, {"value":"B"}, {"value":"C"}, {"value":"D"}, {"value":"E"}]

Paginated collections on HTTP APIs are common enough I figured there may be a standard solution to the problem. Any insight?

[anderseknert]

Paginated collections on HTTP APIs are common

That is true, but rarely used in policy, as evaluation time is often critical. But if it isn't – is the total number of pages known? If so I suppose you could iterate over the pages in a comprehension:

collected := [item |
    n := numbers.range(1, stop)
    some item in http.send({"method": "GET", "url": sprintf("%s%d", [base, n])}).body.items
]

Another option would be to put a tiny service in front of this API that would deal with the pagination, and have OPA served a single page.

[devonsparks]

Thanks @anderseknert! Appreciate the guidance. This validates my hunch.