Run a test-instance of Varnish for Magento 2

Full stack in Docker

First of all, there are numerous solutions out there that allow you to run the entire stack of Magento on Docker: Nginx, PHP-FPM, MySQL, Redis and often also including Varnish. This blog is not about such a setup.

Instead, I'm running an Ubuntu environment with a native stack: Apache with mod_php, MySQL and Redis. It runs just fine as it is. I don't need Nginx. And actually the speed is awesome. And I've been hearing people struggle with various Docker setups, while for me, the stack simply works. The benefit of running Linux.

And obviously I have Docker and magerun2 installed.

Turning on and off Varnish

I usually work with Magento 2 without Varnish. With extension development, custom project development, headless development, there is no need to test for Varnish. Either testing things with the regular Full Page Cache is good enough. Or Full Page Cache is not relevant (like with headless development).

However, once in a while, I receive a request for one of my extensions where Varnish seems to be giving issues. Almost in all of these cases, the problem deals with the specific client project after all. But investing an hour into such an issue, only to find out that it is not a problem with my extension is frustrating. Nonetheless, that level of support is requested: Does my extension give a specific issue under Varnish or not?

My need became to quickly turn on and off Varnish, whenever I wanted. Let's create a script for that.

Explaining my setup

On my Linux box, I can run and download the official Docker image of Varnish (varnish) and I currently prefer to test things with version 6 (varnish:6). However, I do not want my Magento also to live in Docker as well: I already have that running natively, remember.

I have choosen to run Varnish in Docker with a static IP (172.20.0.2). My Linux box runs a Docker daemon, which assigns it a static IP as well (172.20.0.1), so that I'm able to access the Varnish container without issues. (In the past, I have played around with port forwarding, IP masquarading, proxies, but as of yet, the simple static IP setup is still the most straight-forward for me.)

To make sure I can easily access both Magento and Varnish, I have given them names which I will keep a secret. But let's refer to them as magento.local and varnish.local. This leads to a simple /etc/hosts file like this:

172.20.0.1 magento.local
172.20.0.2 varnish.local

I'll use the same /etc/hosts for Docker as well, but that's coming up in a bit. Note that your network range might be different.

Generating the Varnish configuration

Magento 2 includes a perfect tool to generate a Varnish configuration. Just refer to the Magento devdocs for details. The form in the Store Configuration could be filled with the following details:

• Access list: magento.local; varnish.local
• Backend host: magento.local
• Backend port: 80

Next, save the configuration and file varnish.vcl should be generated.

Modifying the Varnish configuration

Next, I always open up the file. First of all, to check whether the generated stuff actually works the way I want it to work:

backend default {
    .host = "magento.local";
    .port = "80";
    ...
    .probe = {
        .url = "/health_check.php";
        ...
   }
}

acl purge {
    "172.20.0.1";
    "172.20.0.2";
}

But more importantly, by default, the file is full with references to the pub/ path, that I want gone. In my environment, the DocumentRoot of Apache is referring to the pub/ folder of Magento, so that a reference like /pub/health_check.php needs to be changed into /health_check.php. This also needs to be done for some other paths like media and static. A bit of work, but I only do this with new major upgrades of either Magento or Varnish.

Save the VCL file to somewhere like /path/to/varnish.vcl.

Running the Docker instance

Next, let's pull in the Docker image. Running the container requires only a few lines of code:

docker run \
    --name=varnish-local \
    --rm -it -d \
    -v /path/to/varnish.vcl:/etc/varnish/default.vcl \
    -v /etc/hosts:/etc/hosts \
    --ip=172.20.0.2 \
    varnish:6

The Docker instance is given a name of your liking, some flags make sure that it is easily reachable (-it -d). And there are two important volumes: My local /etc/hosts is mapped to the virtual one. And the Varnish VCL is mapped, so that Varnish picks up on it automatically.

The result should be that Varnish is running. Double-check this with docker ps. Or even cooler, you should be able to login to the container using this:

docker exec -i -t  varnish-local bash

We'll play around with this in a bit. As of yet, the Varnish instance should be reachable through a URL like http://varnish.local. The domain name is mapped to an IP using /etc/hosts. And the IP should be accessible, because we are on the same network as the Docker instance.

Magento stuff

Next, let's add in Magento to the mix. I'm assuming that Magento is reachable on IP address 127.20.0.1 already and that you can see its frontend on http://magento.local. The goal is now to visit the http://varnish.local domain instead and see the same Magento frontend.

Stop redirecting

There could be a couple of things preventing you from seeing the right frontend. First of all, it might be Magento is still configured with the old magento.local URL and any attempt to access the new varnish.local URL simply redirects you to the old URL.

To stop redirects, we can use the following Magerun command:

magerun2 config:set web/url/redirect_to_base 0
magerun2 cache:clean config

After this, we should be able to visit the varnish.local URL and see some Magento output, even though it might be having other issues with things like CORS security warnings, causing CSS and JavaScript not loading properly.

Troubleshooting Varnish its access to Magento

If you don't see any Magento output when accessing the Varnish URL, then perhaps something in the Docker setup is incorrect. Perhaps it is a networking issue of some kind. First let's login to the Docker container using docker exec -i -t varnish-local bash.

Next on the command-line, you can use a tool like varnishlog to see if traffic goes through. Fire up the command, keep it running and point your browser to the Varnish URL to see if something happens.

Also great is the varnishadm tool, which gives you its own shell:

$ varnishadm
varnish> ping
varnish>

When troubleshooting things, I often also use basic tools like ping and wget to see if things are working:

apt-get update
apt-get install ping wget
ping magento.local

Wget also allows you to fetch the Magento health_check.php script to see if Magento is still accessible. It is something that Varnish tries to do as well.

wget http://magento.local/health_check.php

This actually leads to another nice hint: You should be able to see this request in the Magento webserver access-logs.

Hopefully, you can make this a success.

Fixing asset URLs

Next, the Varnish URL might be loading the Magento HTML, but it might also be that all of the static URLs are giving CORS warnings, because they are still loaded from the original Magento domain. This can be fixed in multiple ways: You could install a CORS-module into Magento (I built one!), you could remove all URL configurations from the database so that Magento loads this dynamically (which I actually prefer in a development environment, but sometimes things are just messed up).

Or you could use Magerun to change the URLs. The settings could be changed for both the frontend and the backend, but then your backend is no longer accessible. Or you could keep the backend at the original settings and only change the Store View on the frontend (in my case, a Store View with code default):

magerun2 config:set --scope=default web/unsecure/base_static_url http://magento.local/static/
magerun2 config:set --scope=store --scope-code=default web/unsecure/base_static_url http://varnish.local/static/

Even though, static versioning should work fine, I often see myself disabling it, because that way I gain more confidence on what static files are doing:

magerun2 config:set dev/static/sign 0

And clean the cache:

magerun2 cache:clean config

Enable Full Page Cache

This should be working. However, there is one final step to make: Enabling the Full Page Cache and making sure that the Full Page Cache is not using the original Magento caching anymore but is prepared for Varnish:

magerun2 config:set system/full_page_cache/caching_application 2
magerun2 cache:enable full_page

Funnily enough, that configuration setting is not entirely needed in development. Well, depending on what you are developing. The category menu will be turned into a Edge Side Include, which offers better performance. But in development, this might be less of an issue, unless you're working on performance. Make sure to set the flag back again, once you're done playing with Varnish, because otherwise the menu will be gone in the regular Magento frontend.

One final thing I'd like to see if fill in the Varnish settings in the Magento Store Configuration. It is not needed at all, because the settings are only used for generating the VCL file. However, I like to see which values I played with.

magerun2 config:set system/full_page_cache/varnish_access_list "172.20.0.1, 172.20.0.2"
magerun2 config:set system/full_page_cache/backend_host "magento.local"
magerun2 config:set system/full_page_cache/backend_port "80"

Scripted approach

Most importantly, all of the steps I took were based on the CLI. So I actually take all of these steps (and the opposite steps) by running a simple script.

Cross-fingers that it also works for you.

And why the blog picture on painting? Because Varnish seems to a paint brand as well.