Symflower test-runner
This feature performs package-level test impact analysis on your Go repositories to identify tests that are affected by the source code changes specified in your query:
symflower test-runner --commit-from=${commit revision} --${test command to be executed}
The command takes two arguments:
- Commit revision: Specifies the commit from which the changes are taken into account. If the filter is not provided, it defaults to
HEAD
. - Test command to be executed: Currently, only the
go test
command is supported for test execution, with the flags that command offers.
An example of a valid command:
symflower test-runner --commit-from HEAD~ -- go test -v -count=1
Use this feature to limit test execution to relevant tests (i.e. tests of whole packages that need to be executed for a given set of code changes). Using symflower test-runner
helps optimize test execution time and resource usage.
Evidence: Reducing execution times with symflower test-runner
Benchmarks for symflower test-runner
using highly rated Go projects sourced from GitHub showed an average 29% reduction in test execution times:
Repository | With test runner | Without test runner | Time saved | Time saved (%) |
---|---|---|---|---|
ebiten | 1m26s | 2m33s | 1m7s | 43.61% |
tailscale | 10m55s | 17m39s | 6m44s | 38.14% |
go-ethereum | 13m7s | 20m45s | 7m37s | 36.76% |
gonum | 12m20s | 19m26s | 7m5s | 36.51% |
ugo | 22m5s | 24m57s | 2m52s | 11.49% |
doit | 33m7s | 36m58s | 3m51s | 10.44% |
How does symflower test-runner
work?
- Execute
git diff
to retrieve the list of all files modified up to the commit revision specified in the command (default:HEAD
). - Analyze each package in the project to check for (direct or indirect) package dependencies that contain a change discovered by the
git diff
command. - Collect package names for execution.
Note that if a go.mod
file was modified, the command runs all tests.
In case the testing code contains actions based on the contents of a file (e.g. a text file) and the content of that file changes, the analysis does not detect what tests need to be run.
Tutorial: symflower test-runner
- Example: A project that handles operations related to geometric shapes.
Project structure:
shapes
├── area
│ ├── area.go
│ └── area_test.go
├── go.mod
├── go.sum
└── perimeter
├── perimeter.go
└── perimeter_test.go
Function #1: area.go
contains a function that calculates the area of a circle:
package area
import (
"errors"
"math"
)
func CircleArea(radius float64) (area float64, err error) {
if radius <= 0 {
return 0, errors.New("radius must be a positive value")
}
return math.Pi * radius * radius, nil
}
area_test.go
contains the tests for the CircleArea
function:
package area
import (
"errors"
"testing"
"github.com/stretchr/testify/assert"
)
func TestCircleArea(t *testing.T) {
type testCase struct {
Name string
Radius float64
ExpectedArea float64
ExpectedErr error
}
validate := func(t *testing.T, tc *testCase) {
t.Run(tc.Name, func(t *testing.T) {
actualArea, actualErr := CircleArea(tc.Radius)
assert.InDelta(t, actualArea, tc.ExpectedArea, 0.1)
assert.Equal(t, tc.ExpectedErr, actualErr)
})
}
validate(t, &testCase{
Name: "Negative radius",
Radius: -1,
ExpectedErr: errors.New("radius must be a positive value"),
})
validate(t, &testCase{
Name: "Positive radius",
Radius: 5.0,
ExpectedArea: 78.5,
})
}
Function #2: perimeter.go
contains a function that calculates the perimeter of a circle:
package perimeter
import (
"errors"
"math"
)
func CirclePerimeter(radius float64) (perimeter float64, err error) {
if radius <= 0 {
return 0, errors.New("radius must be a positive value")
}
return 2 * math.Pi * radius, nil
}
perimeter_test.go
contains the tests for the above CirclePerimeter
function:
package perimeter
import (
"errors"
"testing"
"github.com/stretchr/testify/assert"
)
func TestCirclePerimeter(t *testing.T) {
type testCase struct {
Name string
Radius float64
ExpectedPerimeter float64
ExpectedErr error
}
validate := func(t *testing.T, tc *testCase) {
t.Run(tc.Name, func(t *testing.T) {
actualPerimeter, actualErr := CirclePerimeter(tc.Radius)
assert.InDelta(t, actualPerimeter, tc.ExpectedPerimeter, 0.1)
assert.Equal(t, tc.ExpectedErr, actualErr)
})
}
validate(t, &testCase{
Name: "Negative radius",
Radius: -1,
ExpectedErr: errors.New("radius must be a positive value"),
})
validate(t, &testCase{
Name: "Positive radius",
Radius: 5.0,
ExpectedPerimeter: 31.4,
})
}
We need to initialize a Git repository within this project and commit all the above files:
git init
git add .
git commit -m "Init"
The next step is to modify one of the files in the example. Let's add a function to calculate the area of a square in area.go
:
func SquareArea(side float64) (area float64, err error) {
if side <= 0 {
return 0, errors.New("side must be a positive value")
}
return side * side, nil
}
Corresponding test cases are added to area_test.go
:
func TestSquareArea(t *testing.T) {
type testCase struct {
Name string
Side float64
ExpectedArea float64
ExpectedErr error
}
validate := func(t *testing.T, tc *testCase) {
t.Run(tc.Name, func(t *testing.T) {
actualArea, actualErr := SquareArea(tc.Side)
assert.InDelta(t, actualArea, tc.ExpectedArea, 0.1)
assert.Equal(t, tc.ExpectedErr, actualErr)
})
}
validate(t, &testCase{
Name: "Negative side",
Side: -1,
ExpectedErr: errors.New("side must be a positive value"),
})
validate(t, &testCase{
Name: "Positive side",
Side: 5.0,
ExpectedArea: 25.0,
})
}
git status
shows us what files were modified:
On branch master
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
modified: area/area.go
modified: area/area_test.go
no changes added to commit (use "git add" and/or "git commit -a")
Let's run symflower test-runner
to execute relevant tests:
symflower test-runner –- go test -v
The command's output shows us that only the tests affected by our changes were executed:
Detected changes:
- "/home/rui/dev/go/shapes/area/area.go"
- "/home/rui/dev/go/shapes/area/area_test.go"
Affected by change:
- "shapes/area"
- "shapes/area [shapes/area.test]"
- "shapes/area.test"
Executing "go test -v shapes/area"
=== RUN TestCircleArea
=== RUN TestCircleArea/Negative_radius
=== RUN TestCircleArea/Positive_radius
--- PASS: TestCircleArea (0.00s)
--- PASS: TestCircleArea/Negative_radius (0.00s)
--- PASS: TestCircleArea/Positive_radius (0.00s)
=== RUN TestSquareArea
=== RUN TestSquareArea/Negative_side
=== RUN TestSquareArea/Positive_side
--- PASS: TestSquareArea (0.00s)
--- PASS: TestSquareArea/Negative_side (0.00s)
--- PASS: TestSquareArea/Positive_side (0.00s)
PASS
ok shapes/area 0.003s