Running Only Previously Failed PHPUnit Tests cover image

Running Only Previously Failed PHPUnit Tests

Chris Di Carlo • July 18, 2021

Update: As of version 1.2, it now automatically detects if you have Pest installed and runs the appropriate runner! Check it out here

I love TDD, or test-driven development. It does take getting used to the seemingly slower pace of development but to me that's an illusion. Sure, I can't just spike out an app in 3 hours but my more mature self now asks the more prosaic questions: is that app really working as I envisioned or designed, and more importantly, am I confident that I can actually fix bugs or improve the application going forward without breaking something? The trade-off of a little longer up-front development time more than pays off down the road in terms of less bugs and easier application functionality extension.

Don't get me wrong - I still sometimes spike out something with no thought to testing but it's usually more of a quick validation of the original idea or design. Most of the time, I end up scrapping it and beginning anew with TDD and just use the quick and dirty version as a reference.

But it's not all unicorns and lollipops - there are some things in the TDD workflow that grate on me. I can get around them but it usually involves more work that I would like and hey, I'm kind of lazy in that respect.

The pain point that got my goat recently was being able to quickly re-run any failed tests while using VSCode. I use Better PHPUnit, which makes running tests quick - I've got keyboard shortcuts for common tasks like run current method, run current file, run entire suite, etc. But I didn't have a way to quickly rerun any tests that failed.

The Scenario

Consider this scenario where some tests are passing and some are failing (I added my personal keyboard shortcuts to give an idea of the actual workflow in the editor):

Now if that is successful, I have a dilemma: what do I do now? Run all the tests in current file (CTRL + K CTRL + F), even if there might not be any other failed methods? Scroll through the terminal output to see other failed tests (which doesn't work if the terminal is reused for test runs)? Run the whole test suite again to get to the next error? The last option is the easiest but with a big test suite that can take some time.

No, what I really want to do is just rerun all of the failed tests. Your wish is my command! Err... Um... How exactly can I do that? Enter a couple of hours of web searches and reading and I stumbled on an old Gist Bash script that used TAP logging in PHPUnit to extract a list of only the failed tests and pass that along to the --filter method of PHPUnit. Great! I'll just fork it and off to the races...

Sadly, no dice - the original version was written assuming an ancient version of PHPUnit and no longer works. It was also fairly long and internally used PHP to to do some of the text extraction.

Using the idea as inspiration, I decided to see if I could come up with a solution that still ran as a Bash script but didn't rely on spawning a PHP process to do the heavy lifting.

The Solution

The solution I came up with uses a command-line tool available on Linux and OSX/macOS either natively or via homebrew (probably could be installed in Windows, as well, but I have not researched that possibility as I don't use Windows except in dire need, e.g. running TamoSoft Site Survey). The tool is XMLStarlet.

My original version of this solution actually used 2 tools: XMLStarlet and sed. But as I was writing this blog post, I found that I could do everything simply using an XSLT with some slight tweaking. Just goes to show that leaving the code for a bit gives you new insights into how it works, or could work!

Here's the breakdown of how it works:

  1. XMLStarlet transforms a PHPUnit logfile (more on that in a minute) using an XSLT to extract just the failed tests, as a regex string;
  2. Call PHPUnit with the --filter parameter to only fun the failed tests.

Sounds fairly simple, right? In truth, it is; it just took me a good couple of hours to get the whole thing working. Let's unpack each step of the process!

The Log

First things first - we need to have a way to determine which tests actually failed during the last run. PHPUnit can be configured to log output in a variety of formats but for my purposes, I chose TestDox XML so that I can process it using XML tools. To start logging using this format, add the following to your <logging> element in your phpunit.xml file (if it doesn't exist, just create it):

<testdoxXml outputFile="testdox.xml"/>

Now, whenever we run any tests PHPUnit will log everything to that file. The key is that it's persisted.

Determing Failed Tests

In the Testdox XML format, failed tests are marked with a status attribute value of 3, so we need to do something get all of those test method names. The solution I came up with uses XMLStarlet and an XLST:

xmlstarlet tr --omit-decl failed-tests.xsl testdox.xml

Keep in mind this command as-written assumes the XSLT and the log file are both in the current directory.

So what does it do? If you were to write it in English, the command would say:

Using xmlstarlet, transform (tr) but omit the XML declaration in the output (--omit-decl) using the specified XSLT (failed-tests.xsl) and the specified XML source testdox.xml.

Most of that is self-explanatory from a command-line perspective but the real meat is inside the XSLT file. Let's explore that next.

Transforming the Log

Here is the XSLT used to process the PHPUnit log:

<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet
    version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

    <xsl:output method="xml" indent="yes"/>

    <!-- Match the root element of the document -->
    <xsl:template match="/">
        <xsl:apply-templates />
    </xsl:template>

    <!--
        Match any <test> elements under the <tests> parent element
        that have failed, i.e. status attribute has a value of 3
    -->
    <xsl:template match="tests">
        <xsl:apply-templates select="./test[@status = '3']" />
    </xsl:template>

    <!-- Transform each <test> element into a pipe-delimited string -->
    <xsl:template match="test">
        <xsl:value-of select="@methodName" />
        <xsl:if test="position() != last()">
            <xsl:text>|</xsl:text>
        </xsl:if>
    </xsl:template>

</xsl:stylesheet>

A little explanation is probably in order if you are not familiar with XSLT - I hadn't used one in a couple of years and had to refresh my memory.

The Root Element

<xsl:template match="/">
    <xsl:apply-templates />
</xsl:template>

This just matches the root node of the XML document and tells the processor to apply any templates.

The <tests> Element

<xsl:template match="tests">
    <xsl:apply-templates select="./test[@status = '3']" />
</xsl:template>

This matches any <test> subnode under <tests> that has an attribute name status with a value of 3. Basically, filter out everything except the failed tests.

The Magic

<xsl:template match="test">
    <xsl:value-of select="@methodName" />
    <xsl:if test="position() != last()">
        <xsl:text>,</xsl:text>
    </xsl:if>
</xsl:template>

This is where the magic happens: for each test that matched the previous template, grab the value of the methodName attribute, and if it's not the last one, also add a pipe (|). Basically, we're creating a pipe-delimited string, which just so happens to be the format for a regex that matches any of the values between the pipes.

Run PHPUnit

With our pipe-delimited string, we can now pass it into PHPUnit's --filter paramater like so:

phpunit --filter "'failed_test_one|failed_test_two'"

Making It Useable

We now have the pieces to the puzzle: the PHPUnit log file and the XSLT file to transform it into a format that's useable in the --filter parameter. You could run that one-liner directly in the terminal and it works but what happens if the PHPUnit log file doesn't exist?

To make the solution easier to use, I wrapped everything inside a Bash script. This allows some basic error checking like making sure the log file exists, and if not, run the whole suite to get a baseline. It also checks if there are any failed tests in the log file and just exits.

Here's the first version:

#!/bin/bash

logfile=./testdox.xml

if test -f "$logfile"; then
    echo "Logfile found. Filter for failed..."
    failed_tests="$(xmlstarlet tr --omit-decl failed-tests.xsl testdox.xml)"

    if [ "$failed_tests" = "" ]; then
        echo "No failed tests!  Great job!"
    else
        ./vendor/bin/phpunit --filter "'$failed_tests'"
    fi

else
   echo "Logfile not found. Running the full test suite..."
   ./phpunit
fi

exit 0

Now you could put this shell script somewhere in your path and it would work, as long as you also copied the failed-tests.xsl into the directory of your PHP project. Kind of a pain; with that in mind, I wrapped the shell script inside a basic Composer package that I could require as a --dev dependency. This allows me to make the XSLT path relative to the PHP project, i.e. ./vendor/chrisdicarlo/phpunit-failed-runner/failed-tests.xsl. With this small change:

failed_tests="$(xmlstarlet tr --omit-decl ./vendor/chrisdicarlo/phpunit-failed-runner/failed-tests.xsl testdox.xml)"

I can now get the functionality into my project and not have to worry about copying a Gist or file into my project.

For the sake of completeness, here's how you would include it in your project:

composer require --dev chrisdicarlo/phpunit-failed-runner

Integration With VSCode

The last step is to make it quick to use the package from within VSCode; for this, I use Tasks. If it doesn't already exist, create a tasks.json file in your project under the .vscode directory, with the following content:

{
    "version": "2.0.0",
    "tasks": [
        {
            "label": "Run failed tests",
            "type": "shell",
            "command": "./vendor/bin/phpunit-failed-runner",
            "group": "test",
            "presentation": {
                "echo": true,
                "reveal": "always",
                "focus": false,
                "panel": "dedicated",
                "showReuseMessage": true,
                "clear": false
            }
        }
    ]
}

Now I can run the task (I use a keyboard shortcut CTRL + ALT + SHIFT + / to bring up the Run Task palette:

image

My New Workflow

My new workflow when I'm dealing with failed tests is now typically:

Each run will keep filtering down the number of failed tests, until eventually all the failed tests pass.

Since each subsequent run of the tasks filters the already failed tests, the last step is to rerun the full suite, just to make sure I haven't broken anything else.

Rinse and repeat.

Happy coding! Cheers!