Symflower test-runner

This feature performs package-level test impact analysis on your Go repositories to identify tests that are affected by the source code changes specified in your query:

symflower test-runner --commit-from=${commit revision} --${test command to be executed}

The command takes the following arguments:

• Commit revision: Specifies the commit from which the changes are taken into account. If the filter is not provided, it defaults to HEAD.
• Test command to be executed: Currently, only the go test command is supported for test execution, with the flags that command offers.

An example of a valid command:

symflower test-runner --commit-from HEAD~ -- go test -v -count=1

Use this feature to limit test execution to relevant tests (i.e. tests of whole packages that need to be executed for a given set of code changes). Using symflower test-runner helps optimize test execution time and resource usage.

Evidence: Reducing execution times with symflower test-runner

Benchmarks for symflower test-runner using highly rated Go projects sourced from GitHub showed an average 29% reduction in test execution times:

Repository	With test runner	Without test runner	Time saved	Time saved (%)
ebiten	1m26s	2m33s	1m7s	43.61%
tailscale	10m55s	17m39s	6m44s	38.14%
go-ethereum	13m7s	20m45s	7m37s	36.76%
gonum	12m20s	19m26s	7m5s	36.51%
ugo	22m5s	24m57s	2m52s	11.49%
doit	33m7s	36m58s	3m51s	10.44%

How does symflower test-runner work?

1. Execute git diff to retrieve the list of all files modified up to the commit revision specified in the command (default: HEAD).
2. Analyze each package in the project to check for (direct or indirect) package dependencies that contain a change discovered by the git diff command.
3. Collect package names for execution.

Note that if a go.mod file was modified, the command runs all tests.

info

In case the testing code contains actions based on the contents of a file (e.g. a text file) and the content of that file changes, the analysis does not detect what tests need to be run.

Tutorial: symflower test-runner

• Example: A project that handles operations related to geometric shapes.

Project structure:

shapes
├── area
│   ├── area.go
│   └── area_test.go
├── go.mod
├── go.sum
└── perimeter
    ├── perimeter.go
    └── perimeter_test.go

Function #1: area.go contains a function that calculates the area of a circle:

package area

import (
    "errors"
    "math"
)

func CircleArea(radius float64) (area float64, err error) {
    if radius <= 0 {
        return 0, errors.New("radius must be a positive value")
    }

    return math.Pi * radius * radius, nil
}

area_test.go contains the tests for the CircleArea function:

package area

import (
    "errors"
    "testing"

    "github.com/stretchr/testify/assert"
)

func TestCircleArea(t *testing.T) {
    type testCase struct {
        Name string

        Radius float64

        ExpectedArea float64
        ExpectedErr  error
    }

    validate := func(t *testing.T, tc *testCase) {
        t.Run(tc.Name, func(t *testing.T) {
            actualArea, actualErr := CircleArea(tc.Radius)

            assert.InDelta(t, actualArea, tc.ExpectedArea, 0.1)
            assert.Equal(t, tc.ExpectedErr, actualErr)
        })
    }

    validate(t, &testCase{
        Name: "Negative radius",

        Radius: -1,

        ExpectedErr: errors.New("radius must be a positive value"),
    })
    validate(t, &testCase{
        Name: "Positive radius",

        Radius: 5.0,

        ExpectedArea: 78.5,
    })
}

Function #2: perimeter.go contains a function that calculates the perimeter of a circle:

package perimeter

import (
    "errors"
    "math"
)

func CirclePerimeter(radius float64) (perimeter float64, err error) {
    if radius <= 0 {
        return 0, errors.New("radius must be a positive value")
    }

    return 2 * math.Pi * radius, nil
}

perimeter_test.go contains the tests for the above CirclePerimeter function:

package perimeter

import (
    "errors"
    "testing"

    "github.com/stretchr/testify/assert"
)

func TestCirclePerimeter(t *testing.T) {
    type testCase struct {
        Name string

        Radius float64

        ExpectedPerimeter float64
        ExpectedErr       error
    }

    validate := func(t *testing.T, tc *testCase) {
        t.Run(tc.Name, func(t *testing.T) {
            actualPerimeter, actualErr := CirclePerimeter(tc.Radius)

            assert.InDelta(t, actualPerimeter, tc.ExpectedPerimeter, 0.1)
            assert.Equal(t, tc.ExpectedErr, actualErr)
        })
    }

    validate(t, &testCase{
        Name: "Negative radius",

        Radius: -1,

        ExpectedErr: errors.New("radius must be a positive value"),
    })
    validate(t, &testCase{
        Name: "Positive radius",

        Radius: 5.0,

        ExpectedPerimeter: 31.4,
    })
}

We need to initialize a Git repository within this project and commit all the above files:

git init
git add .
git commit -m "Init"

The next step is to modify one of the files in the example. Let's add a function to calculate the area of a square in area.go:

func SquareArea(side float64) (area float64, err error) {
    if side <= 0 {
        return 0, errors.New("side must be a positive value")
    }

    return side * side, nil
}

Corresponding test cases are added to area_test.go:

func TestSquareArea(t *testing.T) {
    type testCase struct {
        Name string

        Side float64

        ExpectedArea float64
        ExpectedErr  error
    }

    validate := func(t *testing.T, tc *testCase) {
        t.Run(tc.Name, func(t *testing.T) {
            actualArea, actualErr := SquareArea(tc.Side)

            assert.InDelta(t, actualArea, tc.ExpectedArea, 0.1)
            assert.Equal(t, tc.ExpectedErr, actualErr)
        })
    }

    validate(t, &testCase{
        Name: "Negative side",

        Side: -1,

        ExpectedErr: errors.New("side must be a positive value"),
    })
    validate(t, &testCase{
        Name: "Positive side",

        Side: 5.0,

        ExpectedArea: 25.0,
    })
}

git status shows us what files were modified:

On branch master
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   area/area.go
        modified:   area/area_test.go

no changes added to commit (use "git add" and/or "git commit -a")

Let's run symflower test-runner to execute relevant tests:

symflower test-runner –- go test -v

The command's output shows us that only the tests affected by our changes were executed:

Detected changes:
 - "/home/rui/dev/go/shapes/area/area.go"
 - "/home/rui/dev/go/shapes/area/area_test.go"

Affected by change:
 - "shapes/area"
 - "shapes/area [shapes/area.test]"
 - "shapes/area.test"

Executing "go test -v shapes/area"
=== RUN   TestCircleArea
=== RUN   TestCircleArea/Negative_radius
=== RUN   TestCircleArea/Positive_radius
--- PASS: TestCircleArea (0.00s)
    --- PASS: TestCircleArea/Negative_radius (0.00s)
    --- PASS: TestCircleArea/Positive_radius (0.00s)
=== RUN   TestSquareArea
=== RUN   TestSquareArea/Negative_side
=== RUN   TestSquareArea/Positive_side
--- PASS: TestSquareArea (0.00s)
    --- PASS: TestSquareArea/Negative_side (0.00s)
    --- PASS: TestSquareArea/Positive_side (0.00s)
PASS
ok      shapes/area     0.003s