Faster Magento 2 Integration Tests

Disclaimer

I have to admit, I'm a knowledgable developer but not a TDD-expert. I actually write code most of the time without tests, only to realize afterwards that tests would have been useful and sometimes badly needed because of some bug. Maybe it is also due to the fact that I deal with a lot of legacy code.

That being said, I am currently in the flow of testing: I use TDD on numerous microservices I develop, I use TDD for refactoring and module development. And I like it. But once you get to that point of running tests constantly, tests that take ages to run will break your flow. This needs to be optimized.

Obvious points first

Let's deal with the obvious points first - assuming that you want to write your own code with Integration Tests: Don't run all of the Integration Tests of the core. Instead, copy the original phpunit.xml.dist, possibly remove the existing testsuites and add your own.

What I personally do a lot now is add a suite YireoApp (for code in app/code) and a suite YireoVendor (for code in vendor/). But because these suites might run slow, because of dozens of Yireo extensions, I either add a YireoCustom suite for a specific extension or add a specific PHPStorm configuration. When I want to be in the flow, I only run tests of my own extension.

Also, make sure to know the trick of TESTS_CLEANUP. With the setting being enabled, every test run will reinstall Magento 2 in your test database. With the setting being disabled, the setup is reused. Google for tips on this.

What is slow?

So now that we know we are only going to run our own testing suite, let's see how much time this takes. I'm assuming the following: Magento 2.3; PHP 7.2 with XDebug disabled and OPCache enabled; MySQL 5.7; SSD drive; one single test HelloWorldTest with a dummy test like $this->assertTrue(false).

In an unoptimized environment, this is rather slow. As soon as the Integration Test suite kicks in, the setup scripts take more than a minute. Once the setup is completed, I toggle the TESTS_CLEANUP constant to disabled. If I rerun the tests, with the existing database (so without rerunning the setup), it still takes more than 6-8 seconds.

This breaks my flow.

Remove unneeded dependencies

One of the reasons why things are slow is that Magento is big. Making it slimmer will definitely help. Following the excellent IntegerNet blog Make Magento 2 Small Again, we can use the composer replace trick to remove all packages that are not needed.

I do this usually for production sites. But I also do this for development. If I have documented the dependencies of my own module carefully (and I can easily track my dependencies with the Yireo ExtensionChecker), I can assume that my code and my tests will run without unneeded dependencies being installed. No GraphQL, no MSI, no third-party modules.

Still, it is needed to test things with a full Magento 2 installation. This is not something I do on-demand, but in a Continuous Integration workflow using GitLab CI: Then it is fine for the tests to take up more time.

Boost the database

Another important tip is to make the database bloody fast. With a local setup, you can work on tuning MySQL parameters (buffering, query cache). However, I'm favoring the usage of a Docker image with a MySQL 5.7 server. And with a little trick, you can run the entire (!) MySQL installation in tmpfs. It can't be faster than that!

In the past, I've been writing about this in the following blog: Boost MySQL speed of Magento 2. The Docker engine will drop the database once restarted, but this is fine for running integration tests.

Intermediate results

With the composer replace trick in place and with MySQL in tmpfs, it is time to review how fast things are now. My personal results are that I can run the installation of Magento 2 itself within 20-25 seconds (TESTS_CLEANUP=enabled). And if I rerun the tests with TESTS_CLEANUP=disabled, my sample test completes within 2 seconds.

This is great. A re-installation no longer requires a coffee break. And 2 seconds is kind of enough to keep me in the flow (and keep my tests in the green). But it still can be improved.

ReachDigitals quick integration tests

The Dutch company ReachDigital (formerly, H&O), has developed an even better approach: In the past, I hacked around in the PHP files of the Integration Testing Framework and discovered that various things could be improved there.

The guys of ReachDigital must have thought the same thing. And they released a drop-in alternative, the Quick Integration Testing Framework, that applies various tricks to boost performance. Once installed via composer, you point your tests to make use of their folder dev/tests/quick-integration and you're done.

With all results in place, my test now runs in 100ms to 200ms, which is blazing fast. When I rerun the entire setup (TESTS_CLEANUP=enabled), things are still at 20-25 seconds, so no improvement there - but that's not what ReachDigital tried to accomplish.

Toggling TESTS_CLEANUP

One thing I found annoying was that I needed to toggle the value TESTS_CLEANUP again and again by modifying the phpunit.xml file. I now fixed this by creating a bin/magento yireo_devhacks:toggle_testscleanup command (shipped with the module Yireo DevHacks which will bundle more of my personal developer tricks soon).

RAM disk for tmp folder

Yet one more thing to tune: The Magento Testing Framework bootstrap copies various files to a temporary folder dev/tests/integration/tmp/sandbox-xyz. By making this folder blazing fast, multiple things are speed up: The cache of the test application, the copying process, the restoring of a database dump and other things. Under Linux, you can copy this folder to a RAM disk or a tmpfs mount as follows:

sudo mount -t tmpfs -o size=512m tmpfs /your/magento/dev/tests/integration/tmp/
sudo chmod 777 /your/magento/dev/tests/integration/tmp/

Other notes

I'm keen to experiment things with MySQL 8 - it is not yet supported but adds additional performance. However, because my database engine is already completely in memory, I'm not sure if this will shave off much performance.

Conclusion

You should be able to run your own Integration Test in 100ms to 200ms, assuming the Magento 2 setup has completed and you are re-running things. You should be able to install the Magento 2 test application in less than half a minute.