VIP9: Expand VCL object support

Synopsis

Currently objects are limited to global objects which have a lifetime of the entire VCL. This VIP is to have objects which can be created during the request and their scope is limited to that request. When the request is done, the objects are destructed.

Why?

The driver for this is creating a new curl/http vmod. (There are several other vmods which would benefit from this as well.) When making curl requests from VCL, you may want to have multiple outstanding (async) requests, so we need to have the ability to encapsulate these requests in isolated, independent, and request scoped objects.

Another use case is creating simple type objects, like VCL_STRING, VCL_INT, and VCL_BLOB. We can wrap these types into an object and we now have VCL variables which don't require conversion into headers and we can define them on the fly per request.

How

Here is a VCL snippet which compiles and works with the patch [0] and uses a new libvmod_types [1].

import types;

sub vcl_init
{
  //global objects, these are unchanged from 4.0
  new s = types.string("Hello!");
  new reqs = types.integer(0);
}

sub vcl_recv
{
  //new req scoped objects
  new req.var.slocal = types.string("Request scoped string");
  new req.var.s2 = types.string("request string two");
  new req.var.count = types.integer(1);
}

sub vcl_backend_fetch
{
  //new bereq scoped objects
  new bereq.var.sbe = types.string("berequest string v1");
  set bereq.http.sbe = bereq.var.sbe.value();

  bereq.var.sbe.set("berequest string v2");
  set bereq.http.sbe2 = bereq.var.sbe.value();
}

sub vcl_deliver
{
  //referencing a mix of global and req scoped objects
  set resp.http.X-s = s.value();
  set resp.http.X-s-length = s.length();

  set resp.http.X-slocal = req.var.slocal.value();
  set resp.http.X-slocal-length = req.var.slocal.length();

  req.var.count.increment(10);
  set resp.http.count = req.var.count.value();
  set resp.http.reqs = reqs.increment_get(1);
}

A theoretical curl/http example:

import http;

sub vcl_recv
{
  //http request #1
  new req.var.h1 = http.request();
  req.var.h1.set_header("foo", "bar");
  req.var.h1.set_url("POST", "http://host1/blah?ok=true");
  req.var.h1.send();

  //http request #2 (we dont read it so its async)
  new req.var.h2 = http.request();
  req.var.h2.set_url("GET", "http://host2/ping");
  req.var.h2.send();
}

sub vcl_deliver
{
  //reference and read http request #1 and block for result
  set resp.http.X-test-response-code = req.var.h1.get_response_code();
}

Regarding the patch, I left the legacy global objects alone in code and syntax. I introduced 2 new variable name scopes: req.var.* and bereq.var.*. This is completely cosmetic as these variables can still be request scoped without the (be)req.var prefix. However, the reason for adding it is to give the user some kind of indication that their variable is tied to a frontend, backend, or global scope. Otherwise I have the feeling having a bunch of un-prefixed variables throwing vcc scope errors when used incorrectly will be confusing. However, these prefixes may be too verbose, so there is going to be a trade off here.

Also, the implementation is fairly simple because I piggybacked on the vmod/vrt priv_task implementation. Request scoped objects are basically given a shimmed struct vmod_priv. I had to jump thru a few small hoops in vcc code to get the priv->priv to cast into an actual struct that the VMOD expects. This may or may not be related to VIP#1, but it would be cleaner to move objects to something more priv like than trying to pass in an explicit struct. However, for the patch, I kept the object interface the same and made use of the previously mentioned vcc/vrt shims.

The patch is enough to have the examples work and give you guys an idea of how it would work.

[0] https://github.com/rezan/varnish-cache/commit/b547bd9ad2fca9db1ef17ee73b8e9b7df9950c34 [1] https://github.com/rezan/libvmod-types