Cached endpoints for dumping data

[pederhan]

It has become a requirement to be able to dump all data about hosts. Iterating over all hosts, page by page, with a max page size of 1000 takes far too long with ~50000 hosts.

There are multiple reasons for this. Among them are:

• No re-use of database connections between requests
• Maximum page size of 1000 (this becomes exponentially worse because of the above)
• Somewhat complex lookups required to construct a Host object (IPs, RRs, Policies, host groups, communities)
• Django
• Python runtime

Possible solutions

Cached /hosts endpoint with a larger maximum page size

Using the drf-extensions package, we can use the PaginateByMaxMixin mixin class, to provide a way for requestors to specify that they want all hosts in a single response.

We could then use the built-in Django caching mechanism (optionally augmented by diskcache's Django support) to cache the response to /hosts?page_size=max, which then serves as the only path to get a cached response from that endpoint.

Separate endpoint(s) for dumping data

MREG deals with more than just hosts. It could be a good idea to have a common endpoint for data dumping (potentially limited to a subset of users) that provides cached dumps of hosts, networks, RRs, etc.

[oyvindkolbu]

Caching is hard, as then you need some kind of method to invalidate the cache. Scripts repeatedly fetching data should do some due diligence before doing a full dump. Like amount of objects and the newest object.

I think the latter approach with some specialized endpoints for dumping data is the way forward. The dumps are typically used by ldap exports and they don't want the full shebang of a Host object, but rather a small subset.