Skip to content

Mustash Varnish plugin

Jump to bottom
Mark Croxton edited this page Jul 4, 2017 · 36 revisions

The Mustash Varnish plugin can be activated on the Settings screen. Once activated, go to the cache-breaking rules screen and add a rule for each of the two hooks it exposes: stash_delete and stash_flush_cache. There is usually no need to narrow the rules by scope or bundle, so leave these blank.

The stash_delete hook is triggered whenever individual Stash variables are deleted, either manually through the Mustash interface, when variables are deleted by cache-breaking rules attached to editing events, when expired variables are pruned, or with the {exp:stash:destroy} tag. If the variable cleared is a page URL (such as a full-page cache), Mustash will send a special header to Varnish that will cause the corresponding Varnish cache item to be banned from it's cache.

The stash_flush_cache hook is triggered when the entire cache is deleted, either from the 'Clear cached variables' screen in Mustash or with the {exp:stash:flush_cache} tag. In this case Mustash will send a different header to Varnish to ban the cache for the entire domain.

The plugin requires Varnish to be installed on your server, and that some special rules are added to the Varnish VCL file (usually located at /etc/varnish/default.vcl) to intercept the two custom headers from Mustash (EE_PURGE and EE_PURGE_URL).

default.vcl

  • If you have set a custom cookie_prefix in your site config, change exp_sessionid to your_prefix_sessionid in the code below.

  • If you have renamed admin.php, change the references to it in the code below with the new filename.

  • You may want to customise the TTL and grace periods.

  • Content is never cached for logged-in users, and logged-in users will always see un-cached content.

  • You may wish to add additional paths to be excluded from caching, e.g /members, /cart etc.

  • We're stripping ALL cookies for cached content, so make sure your pages do not rely on certain cookies being set to alter their state. For example:

    • The Cookie Consent module will not work, as it requires a cookie to be set after consent.
    • Multi-language sites that use a cookie to select the currently displayed language will not work (use a unique url segment for each language instead).
    • Pages containing forms that make use of EE's built-in CSRF tokens will not work. One solution is to include them via AJAX to a URL path that you have specifically excluded.

Varnish 3.x

# Configure backend with probing
backend default {
    .host = "127.0.0.1"; # The domain or IP address of your ExpressionEngine Site
    .port = "8080"; # The port of your EE site. In production, Varnish should be on port 80, so consider changing your Apache server to port 8080 
    .max_connections = 300;
    .probe = {
        #.url = "/"; # short easy way (GET /)
        # We prefer to only do a HEAD /
        .request =
            "HEAD / HTTP/1.1"
            "Host: localhost"
            "Connection: close";

        .interval  = 5s; # check the health of each backend every 5 seconds
        .timeout   = 1s; # timing out after 1 second.
        .window    = 5;  # If 3 out of the last 5 polls succeeded the backend is considered healthy, otherwise it will be marked as sick
        .threshold = 3;
    }

    .first_byte_timeout     = 300s;   # How long to wait before we receive a first byte from our backend?
    .connect_timeout        = 5s;     # How long to wait for a backend connection?
    .between_bytes_timeout  = 2s;     # How long to wait between bytes received from our backend?
}

# Be sure to add your server IP here:
acl purge {
   "localhost";
   "127.0.0.1";
   "::1";
}

sub vcl_recv {

    # Request headers from user
    
    # Forward client IP to backend
    remove req.http.X-Forwarded-For;
    set req.http.X-Forwarded-For = client.ip;

    # Strip hash, server doesn't need it
    if (req.url ~ "\#") {
        set req.url = regsub(req.url, "\#.*$", "");
    }

    # Strip a trailing ? if it exists
    if (req.url ~ "\?$") {
        set req.url = regsub(req.url, "\?$", "");
    }

    # Some generic URL manipulation, useful for all templates that follow
    # First remove the Google Analytics added parameters, useless for our backend
    if (req.url ~ "(\?|&)(utm_source|utm_medium|utm_campaign|utm_content|gclid|cx|ie|cof|siteurl)=") {
        set req.url = regsuball(req.url, "&(utm_source|utm_medium|utm_campaign|utm_content|gclid|cx|ie|cof|siteurl)=([A-z0-9_\-\.%25]+)", "");
        set req.url = regsuball(req.url, "\?(utm_source|utm_medium|utm_campaign|utm_content|gclid|cx|ie|cof|siteurl)=([A-z0-9_\-\.%25]+)", "?");
        set req.url = regsub(req.url, "\?&", "?");
        set req.url = regsub(req.url, "\?$", "");
    }
    
    # Properly handle different encoding types
    if (req.http.Accept-Encoding) {
        if (req.url ~ "\.(jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf|ico)$") {
            # No point in compressing these
            remove req.http.Accept-Encoding;
        } elsif (req.http.Accept-Encoding ~ "gzip") {
            set req.http.Accept-Encoding = "gzip";
        } elsif (req.http.Accept-Encoding ~ "deflate") {
            set req.http.Accept-Encoding = "deflate";
        } else {
            # unknown algorithm
            remove req.http.Accept-Encoding;
        }
    }

    # Allow purging
    if (req.request == "PURGE") {
        if (!client.ip ~ purge) {
            error 405 "This IP is not allowed to send PURGE requests.";
        }
        return (lookup);
    }
  
    # Clear the cache for an entire domain
    if (req.request == "EE_PURGE") {
        if (!client.ip ~ purge) {
            error(403, "Not allowed.");
        }
        ban("req.url ~ ^/.*$ && req.http.host == "+req.http.host);
        error 200 "Purged";
    }

    # Clear any cached object that matches the exact req.url
    if (req.request == "EE_PURGE_URL") {
        if (!client.ip ~ purge) {
            error(403, "Not allowed.");
        }
        ban("req.url == "+req.url+" && req.http.host == "+req.http.host);
        error 200 "Purged";
    }
  
    # Only deal with GET and HEAD by default - don't cache dynamic content
    if (req.request != "GET" && req.request != "HEAD") {
        return (pass);
    }
  
    # Don't cache these paths
    # Don't cache Expressionengine logged-in user sessions
    if (req.url ~ "^/admin\.php" ||
        req.url ~ "ACT=" ||
        req.http.Cookie ~ "exp_sessionid")  
    {
        return (pass);
    }
        
    # Remove cookies
    unset req.http.Cookie;

    # Allow a grace period for offering "stale" data in case backend lags
    if (req.backend.healthy) {
        set req.grace = 30s;
    } else {
        set req.grace = 1h;
    }   
  
    return (lookup);
}

sub vcl_hit {
    if (req.request == "PURGE") {
            purge;
            error 200 "Purged.";
    }
}

sub vcl_miss {
    if (req.request == "PURGE") {
            purge;
            error 200 "Purged.";
    }
}

sub vcl_fetch {

    # Response headers from backend 

    # These status codes should always pass through and never cache
    if (beresp.status == 503 || beresp.status == 500) {
        set beresp.http.X-Cacheable = "NO: beresp.status";
        set beresp.http.X-Cacheable-status = beresp.status;
        return (hit_for_pass);
    }
    if (beresp.status == 404) {
        set beresp.http.magicmarker = "1";
        set beresp.http.X-Cacheable = "YES";
        set beresp.ttl = 20s;
        return (deliver);
    }

    # Marker for vcl_deliver to reset age:
    set beresp.http.magicmarker = "1";
    
    # Check if the requested page is cacheable
    if (bereq.url !~ "^/admin\.php" && 
        bereq.url !~ "ACT=" && 
        bereq.http.Cookie !~ "exp_sessionid" &&
        (bereq.request ~ "GET" || bereq.request ~ "HEAD")) {
            
            /* Remove set-cookies from response */
            unset beresp.http.set-cookie;
            set beresp.http.X-Cacheable = "YES";
        
            /* Remove expires header from response */    
            unset beresp.http.expires;
    }
    else
    {
        set beresp.http.X-Cacheable = "NO";
    }

    # Our cache TTL
    if (bereq.url ~ "\.(js|css|jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf|pdf|ico)$" && ! (bereq.url ~ "\.(php)") ) {
        /* TTL for static files */
        set beresp.ttl = 20m;
    } else {
        /* standard TTL */
        set beresp.ttl = 5m;
    }
    
    # Grace period
    set beresp.grace = 1h;

    return(deliver);
}

sub vcl_deliver {
    
    # From http://varnish-cache.org/wiki/VCLExampleLongerCaching
    if (resp.http.magicmarker) {
        /* Remove the magic marker */
        unset resp.http.magicmarker;

        /* By definition we have a fresh object */
        set resp.http.age = "0";
    }

    # Add cache hit data
    if (obj.hits > 0) {
        # if hit add hit count
        set resp.http.X-Cache = "HIT";
        set resp.http.X-Cache-Hits = obj.hits;
    }
    else 
    {
        set resp.http.X-Cache = "MISS";
    }

    # Add cache object url
    set resp.http.X-Url = req.url;
   
    return (deliver);
}

Varnish 4.x

# Required for VCL 4.0
vcl 4.0;

# Import VMod's
import std;

# Configure backend with probing
backend default {
    .host = "127.0.0.1"; # The domain or IP address of your ExpressionEngine Site
    .port = "8080"; # The port of your EE site. In production, Varnish should be on port 80, so consider changing your Apache server to port 8080 
    .max_connections = 300;
    .probe = {
        #.url = "/"; # short easy way (GET /)
        # We prefer to only do a HEAD /
        .request =
            "HEAD / HTTP/1.1"
            "Host: localhost"
            "Connection: close";

        .interval  = 5s; # check the health of each backend every 5 seconds
        .timeout   = 1s; # timing out after 1 second.
        .window    = 5;  # If 3 out of the last 5 polls succeeded the backend is considered healthy, otherwise it will be marked as sick
        .threshold = 3;
    }

    .first_byte_timeout     = 300s;   # How long to wait before we receive a first byte from our backend?
    .connect_timeout        = 5s;     # How long to wait for a backend connection?
    .between_bytes_timeout  = 2s;     # How long to wait between bytes received from our backend?
}

# Be sure to add your server IP here:
acl purge {
   "localhost";
   "127.0.0.1";
   "::1";
}

sub vcl_recv {

    # Request headers from user
    
    # Forward client IP to backend (first request)
    if (req.restarts == 0) {
        if (req.http.X-Forwarded-For) { # set or append the client.ip to X-Forwarded-For header
            set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip;
        } else {
            set req.http.X-Forwarded-For = client.ip;
        }
    }

    # Normalize the query arguments
    set req.url = std.querysort(req.url);

    # Strip hash, server doesn't need it
    if (req.url ~ "\#") {
        set req.url = regsub(req.url, "\#.*$", "");
    }

    # Strip a trailing ? if it exists
    if (req.url ~ "\?$") {
        set req.url = regsub(req.url, "\?$", "");
    }

    # Some generic URL manipulation, useful for all templates that follow
    # First remove the Google Analytics added parameters, useless for our backend
    if (req.url ~ "(\?|&)(utm_source|utm_medium|utm_campaign|utm_content|gclid|cx|ie|cof|siteurl)=") {
        set req.url = regsuball(req.url, "&(utm_source|utm_medium|utm_campaign|utm_content|gclid|cx|ie|cof|siteurl)=([A-z0-9_\-\.%25]+)", "");
        set req.url = regsuball(req.url, "\?(utm_source|utm_medium|utm_campaign|utm_content|gclid|cx|ie|cof|siteurl)=([A-z0-9_\-\.%25]+)", "?");
        set req.url = regsub(req.url, "\?&", "?");
        set req.url = regsub(req.url, "\?$", "");
    }
    
    # Normalize Accept-Encoding header
    if (req.http.Accept-Encoding) {
        if (req.url ~ "\.(jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf|ico)$") {
            # No point in compressing these
            unset req.http.Accept-Encoding;
        } elsif (req.http.Accept-Encoding ~ "gzip") {
            set req.http.Accept-Encoding = "gzip";
        } elsif (req.http.Accept-Encoding ~ "deflate") {
            set req.http.Accept-Encoding = "deflate";
        } else {
            # unknown algorithm
            unset req.http.Accept-Encoding;
        }
    }

    # Allow purging
    if (req.method == "PURGE") {
        if (!client.ip ~ purge) { # purge is the ACL defined at the beginning
            # Not from an allowed IP? Then die with an error.
            return (synth(405, "This IP is not allowed to send PURGE requests."));
        }
        # If you got this stage (and didn't error out above), purge the cached result
        return (purge);
    }
  
    # Clear the cache for an entire domain
    if (req.method == "EE_PURGE") {
        if (!client.ip ~ purge) {
            return(synth(403, "Not allowed."));
        }
        ban("req.url ~ ^/.*$ && req.http.host == "+req.http.host);
        return(synth(200, "Purged"));
    }
    
    # Clear any cached object that matches the exact req.url
    if (req.method == "EE_PURGE_URL") {
        if (!client.ip ~ purge) {
            return(synth(403, "Not allowed."));
        }
        ban("req.url == "+req.url+" && req.http.host == "+req.http.host);
        return(synth(200, "Purged"));
    }

    # Implementing websocket support (https://www.varnish-cache.org/docs/4.0/users-guide/vcl-example-websockets.html)
    if (req.http.Upgrade ~ "(?i)websocket") {
        return (pipe);
    }
  
    # Only deal with GET and HEAD by default - don't cache dynamic content
    if (req.method != "GET" && req.method != "HEAD") {
        return (pass);
    }
  
    # Don't cache these paths
    # Don't cache Expressionengine logged-in user sessions
    if (req.url ~ "^/admin\.php" ||
        req.url ~ "ACT=" ||
        req.http.Cookie ~ "exp_sessionid")  
    {
        return (pass);
    }
        
    # Remove cookies
    unset req.http.Cookie;

    # Send Surrogate-Capability headers to announce ESI support to backend
    set req.http.Surrogate-Capability = "key=ESI/1.0";

    # Set default grace
    set req.http.grace = "none";
  
    return (hash);
}

sub vcl_pipe {
  # Called upon entering pipe mode.
  # In this mode, the request is passed on to the backend, and any further data from both the client
  # and backend is passed on unaltered until either end closes the connection. Basically, Varnish will
  # degrade into a simple TCP proxy, shuffling bytes back and forth. For a connection in pipe mode,
  # no other VCL subroutine will ever get called after vcl_pipe.

  # Note that only the first request to the backend will have
  # X-Forwarded-For set.  If you use X-Forwarded-For and want to
  # have it set for all requests, make sure to have:
  # set bereq.http.connection = "close";
  # here.  It is not set by default as it might break some broken web
  # applications, like IIS with NTLM authentication.

  set bereq.http.Connection = "Close";

  # Implementing websocket support (https://www.varnish-cache.org/docs/4.0/users-guide/vcl-example-websockets.html)
  if (req.http.upgrade) {
    set bereq.http.upgrade = req.http.upgrade;
  }

  return (pipe);
}

sub vcl_pass {
  # Called upon entering pass mode. In this mode, the request is passed on to the backend, and the
  # backend's response is passed on to the client, but is not entered into the cache. Subsequent
  # requests submitted over the same client connection are handled normally.

  # return (pass);
}

sub vcl_hash {

    # The data on which the hashing will take place
    # Called after vcl_recv to create a hash value for the request. This is used as a key
    # to look up the object in Varnish.

    hash_data(req.url);

    if (req.http.host) {
        hash_data(req.http.host);
    } else {
        hash_data(server.ip);
    }

    # Hash cookies for requests that have them 
    # Note: we're removing cookies from request so this should never be run
    if (req.http.Cookie) {
        hash_data(req.http.Cookie);
    }
}

sub vcl_hit {

    # Called when a cache lookup is successful.

    if (obj.ttl >= 0s) {
        # normal hit
        return (deliver);
    }
    # We have no fresh fish. Lets look at the stale ones.
    if (std.healthy(req.backend_hint)) {
        # Backend is healthy. Limit age to 10s.
        if (obj.ttl + 10s > 0s) {
            set req.http.grace = "normal(limited)";
            return (deliver);
        } else {
            # No candidate for grace. Fetch a fresh object.
            return(fetch);
        }
    } else {
        # backend unhealthy - use full grace
        if (obj.ttl + obj.grace > 0s) {
            set req.http.grace = "full";
            return (deliver);
        } else {
            # no graced object.
            return (fetch);
        }
    }

    # fetch & deliver once we get the result
    return (fetch); # Dead code, keep as a safeguard
}

sub vcl_miss {
  # Called after a cache lookup if the requested document was not found in the cache. Its purpose
  # is to decide whether or not to attempt to retrieve the document from the backend, and which
  # backend to use.

  return (fetch);
}


sub vcl_backend_response {

    # Response headers from backend 

    # Pause ESI request and remove Surrogate-Control header
    if (beresp.http.Surrogate-Control ~ "ESI/1.0") {
        unset beresp.http.Surrogate-Control;
        set beresp.do_esi = true;
    }

    # These status codes should always pass through and never cache
    if (beresp.status == 503 || beresp.status == 500) {
        set beresp.http.X-Cacheable = "NO: beresp.status";
        set beresp.http.X-Cacheable-status = beresp.status;
        set beresp.uncacheable = true;
        return (deliver);
    }
    if (beresp.status == 404) {
        set beresp.http.magicmarker = "1";
        set beresp.http.X-Cacheable = "YES";
        set beresp.ttl = 20s;
        return (deliver);
    }

    # Sometimes, a 301 or 302 redirect formed via Apache's mod_rewrite can mess with the HTTP port that is being passed along.
    # This often happens with simple rewrite rules in a scenario where Varnish runs on :80 and Apache on :8080 on the same box.
    # A redirect can then often redirect the end-user to a URL on :8080, where it should be :80.
    # This may need finetuning on your setup.
    #
    # To prevent accidental replace, we only filter the 301/302 redirects for now.
    if (beresp.status == 301 || beresp.status == 302) {
        set beresp.http.Location = regsub(beresp.http.Location, ":[0-9]+", "");
    }

    # Marker for vcl_deliver to reset age:
    set beresp.http.magicmarker = "1";
    
    # Check if the requested page is cacheable
    if (bereq.url !~ "^/admin\.php" && 
        bereq.url !~ "ACT=" && 
        bereq.http.Cookie !~ "exp_sessionid" &&
        (bereq.method ~ "GET" || bereq.method ~ "HEAD")) {
            
            /* Remove set-cookies from response */
            unset beresp.http.set-cookie;
            set beresp.http.X-Cacheable = "YES";
        
            /* Remove expires header from response */    
            unset beresp.http.expires;
    }
    else
    {
        set beresp.http.X-Cacheable = "NO";
    }

    # Large static files are delivered directly to the end-user without
    # waiting for Varnish to fully read the file first.
    # Varnish 4 fully supports Streaming, so use streaming here to avoid locking.
    if (bereq.url ~ "^[^?]*\.(mp[34]|rar|tar|tgz|gz|wav|zip|bz2|xz|7z|avi|mov|ogm|mpe?g|mk[av]|webm)(\?.*)?$") {
        unset beresp.http.set-cookie;
        set beresp.do_stream = true;  # Check memory usage it'll grow in fetch_chunksize blocks (128k by default) if the backend doesn't send a Content-Length header, so only enable it for big objects
        set beresp.do_gzip = false;   # Don't try to compress it for storage
    }

    # Our cache TTL
    if (bereq.url ~ "\.(js|css|jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf|pdf|ico)$" && ! (bereq.url ~ "\.(php)") ) {
        /* TTL for static files */
        set beresp.ttl = 20m;
    } else {
        /* standard TTL */
        set beresp.ttl = 5m;
    }
    
    # Grace period on incoming objects
    set beresp.grace = 1h;

    return(deliver);
}


sub vcl_deliver {
    
    # From http://varnish-cache.org/wiki/VCLExampleLongerCaching
    if (resp.http.magicmarker) {
        /* Remove the magic marker */
        unset resp.http.magicmarker;

        /* By definition we have a fresh object */
        set resp.http.age = "0";
    }

    # Add cache hit data
    # Please note that obj.hits behaviour changed in 4.0, now it counts per objecthead, not per object
    # and obj.hits may not be reset in some cases where bans are in use. See bug 1492 for details.
    # So take hits with a grain of salt
    if (obj.hits > 0) {
        # if hit add hit count
        set resp.http.X-Cache = "HIT";
        set resp.http.X-Cache-Hits = obj.hits;
    }
    else 
    {
        set resp.http.X-Cache = "MISS";
    }

    # Add cache object url
    set resp.http.X-Url = req.url;

    # Copy grace header from request
    set resp.http.X-Grace = req.http.grace;

    # Security: remove some headers: PHP version, Apache version & OS
    unset resp.http.X-Powered-By;
    unset resp.http.Server;
    unset resp.http.X-Varnish;
    unset resp.http.Via;
    unset resp.http.Link;
    unset resp.http.X-Generator;
   
    return (deliver);
}

sub vcl_purge {
  # Only handle actual PURGE HTTP methods, everything else is discarded
  if (req.method != "PURGE") {
    # restart request
    set req.http.X-Purge = "Yes";
    return(restart);
  }
}

sub vcl_fini {
  # Called when VCL is discarded only after all requests have exited the VCL.
  # Typically used to clean up VMODs.

  return (ok);
}

Installing Varnish on a cPanel server

Unixy sell a Varnish plugin for cPanel, which provides a handy control panel for managing Varnish: http://www.unixy.net/varnish/

Once you have downloaded the zip from Unixy, upload it to /usr/src. Then SSH into your server and follow the installation instructions provided by Unixy.

When you're done login to WHM and find the new Varnish control panel.

On the advanced configuration page:

  • Set Cache time to Live to 300 seconds (go higher if you dare or have heavy traffic).
  • Set Memory Cache to 250M (can go up to 2G, depends on how big site is and how much memory you have available).
  • Add these URLs using the 'URL opt-out' field: admin\.php and ACT=. (Add any other URLs you want to exclude from caching, e.g. 'system' if you still have that in your webroot).

Now you will need to customize /etc/varnish/default.vcl. Unixy's default.vcl contains some additional includes that you should leave in the config, but you should be able to copy and pasted the relevant rules from the config above. Restart Varnish (from WHM control panel) after editing default.vcl.

Home

Getting started

Using Stash

Using Mustash

Template design patterns

Tag reference

Clone this wiki locally