ETag support with pre-gzipped content stored in static variables

[JosePineiro]

willmmiles commented about ETag support form FS files:

Very cool! In WLED we've got similar logic being applied to statically linked resources, ie. pre-gzipped content stored in static variables and served via beginResponse(200, contentType, data_ptr, data_size). We do it this way so the web UI can be updated with a regular firmware OTA, without having to make a copy of the content to the filesystem. Perhaps there's room to add a send_gzip() function that invokes the same code for compiled-in static content?

I assume the function you're using is beginResponse(int code, const char *contentType, const uint8_t *content, size_t len, AwsTemplateProcessor callback).
The simplest option would be to use the void send(int code, const char *contentType, const uint8_t *content, size_t len, AwsTemplateProcessor callback = nullptr) function.
The strategy might be to check if the content has the GZIP Magic number (to make sure it is a gz content). If so, use eTag and disable AwsTemplateProcessor callback.
This way, it wouldn't be necessary to create new functions, and the improvement would work in the existing code.

[willmmiles]

Yes, that's the call we're using. The tradeoff of making the ETag generation implicit is that every user ends up paying the cost of doing the check, even though in many real scenarios the code knows the outcome (either way!) statically at compile time. I'm not sure I'm in the best position to make that call for all use cases - @mathieucarbou what do you think?

The strategy might be to check if the content has the GZIP Magic number (to make sure it is a gz content). If so, use eTag and disable AwsTemplateProcessor callback.

If we do want implicit analysis, I'd suggest going the other way around -- do the magic number check if and only if the callback is unset, ie. if (!_callback && has_magic_number(content)). I would presume any user that has actually specified a callback is expecting it to be used -- while it might be truly weird to tempate-process something that begins with the magic number of a .gz file, I don't want to decide that on a user's behalf.

[mathieucarbou]

let's defer to @me-no-dev 😁

Personally I would be favor in bending the rfc rules to do as few as possible flash disk access (answering also some comments from late PRs).

I favor being able to process a lot of concurrent requests (and fast), without too much heap usage more than anything else.

Keeping track / caching these numbers mean more heap.

That being said, if we go this route, why not opening the ability (parameter or public utility function ?) to help user get the etag from a file (fif gzip, from the magic number and if another file, like it was done before - from the size if I remember). This way, someone still has the ability to do his own magic to check incoming headers and answer appropriately if one wants to be really rfc compliant.

And for the static caching, I think it could be nice to refactor the caching logic that sits at many places now into a middleware or something more common if we want that improvement.

We could have a CacheMiddleware that is initalized with some settings:

• do we want etag support ?
• do we want etag computed faster (i.e. file size) or more reliable (involves reading gzip)
• do we want to cache them on heap (more heap usage) ? If so, what is the expiry mechanism ?
• what kind of cache params we set in the response ?
• etc

This middleware could then be added to all or some routes depending on the need.

Note: could be a middleware or some config object / utility object passes to routes because it needs some information from the resource behind that is requested so I don't know if a middleware could be used - but maybe...

@JosePineiro As i saw from your PR right now, we seem to have caching code at different places - this is kind of a mess, that you are trying to improve and refactor a lot lately and this is really appreciated because this part required some grunt work!