Symflower fix
symflower fix
is a static analysis tool to repair code generated by LLMs. Use it on local files to post-process the output of LLMs:
symflower fix --language=$language --workspace=$workspace
The automatic repair logic in symflower fix
provides the following functionality:
- Remove unused imports: Unused imports are removed as they are compile errors in Go.
- Repair packages: Resolve errors from LLMs generating the wrong package names (e.g.
light_test
instead of justlight
). - Add missing imports: Add the necessary imports (e.g.
fmt.Println
) to make the code compile. - Declare undeclared variables: Variables that appear in the code but are undeclared are declared with the first assigned value.
Use symflower fix
to improve the efficiency of generating code with LLMs. symflower fix
may be applied as part of a Retrieval-Augmented Generation (RAG) workflow to improve LLM performance.
Background: Common compile errors in LLM-generated code
The latest results from the DevQualityEval benchmark show that most LLMs demonstrate limited ability to generate compiling code. A large percentage of the underlying issues fall into one of the following categories with the potential to be fixed with simple static analysis:
- Missing or incorrect package statements
- Missing or unnecessary imports
- Undeclared variables
- Assigning negative values to
uint
- Wrong object creation for nested class
The DevQualityEval benchmark uses symflower fix
whenever a model generates code that doesn't compile. symflower fix
is also used in the write-test
and transpile
tasks in case the generated code fails the included tests.
Benchmarking results show that static code analysis with symflower fix
can positively impact LLM performance in generating compilable code (tested in Go):
- +22.9% in scores across 45 applicable models
- +26.2% in response files compiled (avg. 17 files, 150 tasks total)
Results suggest that using symflower fix
, a weaker and cheaper model (with static analysis) can outperform a larger model (without static analysis) at generating compilable code:
Model | Score | # Compilable files | Beats (without static code repair) |
---|---|---|---|
mistral-tiny | +71% | +109% | mistral-small, mistral-medium |
DBRX | +19% | +26% | GPT-4o-mini, GPT-4o |
Gemma 2 27B | +16% | +13% | GPT-4o-mini, GPT-4o, LLama 3.1 405B |
GPT-4o-mini | +14% | +11% | GPT-4-turbo, Claude Sonnet 3.5 |
GPT-4-turbo | +8% | +8% | Claude Sonnet 3.5 + static code repair |
Tutorial: symflower fix
- Example: A function that calculates the area of a circle
- Command:
symflower fix --language=golang
Remove unused imports
In this example, strings
and fmt
are unused imports:
package area
import (
"errors"
"math"
"strings"
"fmt"
)
func CircleArea(radius float64) (area float64, err error) {
if radius <= 0 {
return 0, errors.New("radius must be positive")
}
return math.Pi * radius * radius, nil
}
After running symflower fix
, the strings
and fmt
imports are removed:
package area
import (
"errors"
"math"
)
func CircleArea(radius float64) (area float64, err error) {
if radius <= 0 {
return 0, errors.New("radius must be positive")
}
return math.Pi * radius * radius, nil
}
Repair packages
Source file:
package area
import (
"errors"
"math"
)
func CircleArea(radius float64) (area float64, err error) {
if radius <= 0 {
return 0, errors.New("radius must be positive")
}
return math.Pi * radius * radius, nil
}
The test file with the wrong package:
package area_test
import (
"area"
"errors"
"testing"
"github.com/stretchr/testify/assert"
)
func TestCircleArea(t *testing.T) {
type testCase struct {
Name string
Radius float64
ExpectedArea float64
ExpectedErr error
}
validate := func(t *testing.T, tc *testCase) {
t.Run(tc.Name, func(t *testing.T) {
actualArea, actualErr := area.CircleArea(tc.Radius)
assert.Equal(t, tc.ExpectedArea, actualArea)
assert.Equal(t, tc.ExpectedErr, actualErr)
})
}
validate(t, &testCase{
Name: "Negative radius",
Radius: -1,
ExpectedErr: errors.New("radius must be positive"),
})
}
After running symflower fix
, the package is fixed:
package area
import (
"errors"
"testing"
"github.com/stretchr/testify/assert"
)
func TestCircleArea(t *testing.T) {
type testCase struct {
Name string
Radius float64
ExpectedArea float64
ExpectedErr error
}
validate := func(t *testing.T, tc *testCase) {
t.Run(tc.Name, func(t *testing.T) {
actualArea, actualErr := CircleArea(tc.Radius)
assert.Equal(t, tc.ExpectedArea, actualArea)
assert.Equal(t, tc.ExpectedErr, actualErr)
})
}
validate(t, &testCase{
Name: "Negative radius",
Radius: -1,
ExpectedErr: errors.New("radius must be positive"),
})
}
Add missing imports
This time, we only import math
, missing the errors
import:
package area
import (
"math"
)
func CircleArea(radius float64) (area float64, err error) {
if radius <= 0 {
return 0, errors.New("radius must be positive")
}
return math.Pi * radius * radius, nil
}
After running symflower fix
, the errors
import is added:
package area
import (
"errors"
"math"
)
func CircleArea(radius float64) (area float64, err error) {
if radius <= 0 {
return 0, errors.New("radius must be positive")
}
return math.Pi * radius * radius, nil
}
Declare undeclared variables
In this example, we're storing a circle's area in a result
variable, but the variable is not declared:
package area
import (
"errors"
"math"
)
func CircleArea(radius float64) (area float64, err error) {
if radius <= 0 {
return 0, errors.New("radius must be positive")
}
result = math.Pi * radius * radius
return result, nil
}
After running symflower fix
, the result
variable is declared:
package area
import (
"errors"
"math"
)
func CircleArea(radius float64) (area float64, err error) {
if radius <= 0 {
return 0, errors.New("radius must be positive")
}
result := math.Pi * radius * radius
return result, nil
}