Skip to main content

Symflower test-runner

This feature performs package-level test impact analysis on your Go repositories to identify tests that are affected by the source code changes specified in your query:

symflower test-runner --commit-from=${commit revision} --${test command to be executed}

The command takes two arguments:

  • Commit revision: Specifies the commit from which the changes are taken into account. If the filter is not provided, it defaults to HEAD.
  • Test command to be executed: Currently, only the go test command is supported for test execution, with the flags that command offers.

An example of a valid command:

symflower test-runner --commit-from HEAD~ -- go test -v -count=1

Use this feature to limit test execution to relevant tests (i.e. tests of whole packages that need to be executed for a given set of code changes). Using symflower test-runner helps optimize test execution time and resource usage.

Evidence: Reducing execution times with symflower test-runner

Benchmarks for symflower test-runner using highly rated Go projects sourced from GitHub showed an average 29% reduction in test execution times:

RepositoryWith test runnerWithout test runnerTime savedTime saved (%)
ebiten1m26s2m33s1m7s43.61%
tailscale10m55s17m39s6m44s38.14%
go-ethereum13m7s20m45s7m37s36.76%
gonum12m20s19m26s7m5s36.51%
ugo22m5s24m57s2m52s11.49%
doit33m7s36m58s3m51s10.44%

How does symflower test-runner work?

  1. Execute git diff to retrieve the list of all files modified up to the commit revision specified in the command (default: HEAD).
  2. Analyze each package in the project to check for (direct or indirect) package dependencies that contain a change discovered by the git diff command.
  3. Collect package names for execution.

Note that if a go.mod file was modified, the command runs all tests.

info

In case the testing code contains actions based on the contents of a file (e.g. a text file) and the content of that file changes, the analysis does not detect what tests need to be run.

Tutorial: symflower test-runner

  • Example: A project that handles operations related to geometric shapes.

Project structure:

shapes
├── area
│ ├── area.go
│ └── area_test.go
├── go.mod
├── go.sum
└── perimeter
├── perimeter.go
└── perimeter_test.go

Function #1: area.go contains a function that calculates the area of a circle:

package area

import (
"errors"
"math"
)

func CircleArea(radius float64) (area float64, err error) {
if radius <= 0 {
return 0, errors.New("radius must be a positive value")
}

return math.Pi * radius * radius, nil
}

area_test.go contains the tests for the CircleArea function:

package area

import (
"errors"
"testing"

"github.com/stretchr/testify/assert"
)

func TestCircleArea(t *testing.T) {
type testCase struct {
Name string

Radius float64

ExpectedArea float64
ExpectedErr error
}

validate := func(t *testing.T, tc *testCase) {
t.Run(tc.Name, func(t *testing.T) {
actualArea, actualErr := CircleArea(tc.Radius)

assert.InDelta(t, actualArea, tc.ExpectedArea, 0.1)
assert.Equal(t, tc.ExpectedErr, actualErr)
})
}

validate(t, &testCase{
Name: "Negative radius",

Radius: -1,

ExpectedErr: errors.New("radius must be a positive value"),
})
validate(t, &testCase{
Name: "Positive radius",

Radius: 5.0,

ExpectedArea: 78.5,
})
}

Function #2: perimeter.go contains a function that calculates the perimeter of a circle:

package perimeter

import (
"errors"
"math"
)

func CirclePerimeter(radius float64) (perimeter float64, err error) {
if radius <= 0 {
return 0, errors.New("radius must be a positive value")
}

return 2 * math.Pi * radius, nil
}

perimeter_test.go contains the tests for the above CirclePerimeter function:

package perimeter

import (
"errors"
"testing"

"github.com/stretchr/testify/assert"
)

func TestCirclePerimeter(t *testing.T) {
type testCase struct {
Name string

Radius float64

ExpectedPerimeter float64
ExpectedErr error
}

validate := func(t *testing.T, tc *testCase) {
t.Run(tc.Name, func(t *testing.T) {
actualPerimeter, actualErr := CirclePerimeter(tc.Radius)

assert.InDelta(t, actualPerimeter, tc.ExpectedPerimeter, 0.1)
assert.Equal(t, tc.ExpectedErr, actualErr)
})
}

validate(t, &testCase{
Name: "Negative radius",

Radius: -1,

ExpectedErr: errors.New("radius must be a positive value"),
})
validate(t, &testCase{
Name: "Positive radius",

Radius: 5.0,

ExpectedPerimeter: 31.4,
})
}

We need to initialize a Git repository within this project and commit all the above files:

git init
git add .
git commit -m "Init"

The next step is to modify one of the files in the example. Let's add a function to calculate the area of a square in area.go:

func SquareArea(side float64) (area float64, err error) {
if side <= 0 {
return 0, errors.New("side must be a positive value")
}

return side * side, nil
}

Corresponding test cases are added to area_test.go:

func TestSquareArea(t *testing.T) {
type testCase struct {
Name string

Side float64

ExpectedArea float64
ExpectedErr error
}

validate := func(t *testing.T, tc *testCase) {
t.Run(tc.Name, func(t *testing.T) {
actualArea, actualErr := SquareArea(tc.Side)

assert.InDelta(t, actualArea, tc.ExpectedArea, 0.1)
assert.Equal(t, tc.ExpectedErr, actualErr)
})
}

validate(t, &testCase{
Name: "Negative side",

Side: -1,

ExpectedErr: errors.New("side must be a positive value"),
})
validate(t, &testCase{
Name: "Positive side",

Side: 5.0,

ExpectedArea: 25.0,
})
}

git status shows us what files were modified:

On branch master
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
modified: area/area.go
modified: area/area_test.go

no changes added to commit (use "git add" and/or "git commit -a")

Let's run symflower test-runner to execute relevant tests:

symflower test-runner –- go test -v

The command's output shows us that only the tests affected by our changes were executed:

Detected changes:
- "/home/rui/dev/go/shapes/area/area.go"
- "/home/rui/dev/go/shapes/area/area_test.go"

Affected by change:
- "shapes/area"
- "shapes/area [shapes/area.test]"
- "shapes/area.test"

Executing "go test -v shapes/area"
=== RUN TestCircleArea
=== RUN TestCircleArea/Negative_radius
=== RUN TestCircleArea/Positive_radius
--- PASS: TestCircleArea (0.00s)
--- PASS: TestCircleArea/Negative_radius (0.00s)
--- PASS: TestCircleArea/Positive_radius (0.00s)
=== RUN TestSquareArea
=== RUN TestSquareArea/Negative_side
=== RUN TestSquareArea/Positive_side
--- PASS: TestSquareArea (0.00s)
--- PASS: TestSquareArea/Negative_side (0.00s)
--- PASS: TestSquareArea/Positive_side (0.00s)
PASS
ok shapes/area 0.003s