Symflower fix

This page provides an overview of symflower fix, a static analysis tool to repair code generated by LLMs.

Usage

symflower [OPTIONS] fix [fix-OPTIONS] [filters...]

The command takes the following arguments:

• workspace options
  • --language: Analyze with this language. By default all languages are used.
  • --workspace: The root directory of the project to analyze. (default: .)

Details

Some examples of the functionality provided by the automatic repair logic in symflower fix:

• Remove unused imports: Unused imports are removed as they are compile errors in Go.
• Repair packages: Resolve errors from LLMs generating the wrong package names (e.g. light_test instead of just light).
• Add missing imports: Add the necessary imports (e.g. fmt.Println) to make the code compile.
• Declare undeclared variables: Variables that appear in the code but are undeclared are declared with the first assigned value.

Use symflower fix to improve the efficiency of generating code with LLMs. symflower fix may be applied as part of a RAG workflow to improve LLM performance.

Background: Common compile errors in LLM-generated code

Results from the DevQualityEval benchmark show that most LLMs demonstrate limited ability to generate compiling code. A large percentage of the underlying issues fall into one of the following categories with the potential to be fixed with simple static analysis:

• Missing or incorrect package statements
• Missing or unnecessary imports
• Undeclared variables
• Assigning negative values to uint
• Wrong object creation for nested class

The DevQualityEval benchmark uses symflower fix whenever a model generates code that doesn't compile. symflower fix is also used in the write-test and transpile tasks in case the generated code fails the included tests.

Benchmarking results (see previous results from DevQualityEval v0.6 and DevQualityEval v1.0, and check out the latest deep dive for up-to-date results) show that static code analysis with symflower fix can positively impact LLM performance in generating compilable code (tested in Go).

Results suggest that using static code repair and static analysis, a weaker and cheaper model can outperform a larger model (without static analysis or static code repair) at generating compilable code.

Tutorial

The following tutorials provide examples of using some of the features in symflower fix. We'll be taking the example of a function that calculates the area of a circle. The following command is used:

symflower fix --language=golang

Removing unused imports

In this example, strings and fmt are unused imports:

package area

import (
    "errors"
    "math"
    "strings"
    "fmt"
)

func CircleArea(radius float64) (area float64, err error) {
    if radius <= 0 {
        return 0, errors.New("radius must be positive")
    }

    return math.Pi * radius * radius, nil
}

After running symflower fix, the strings and fmt imports are removed:

package area

import (
    "errors"
    "math"
)

func CircleArea(radius float64) (area float64, err error) {
    if radius <= 0 {
        return 0, errors.New("radius must be positive")
    }

    return math.Pi * radius * radius, nil
}

Repairing packages

Source file:

package area

import (
    "errors"
    "math"
)

func CircleArea(radius float64) (area float64, err error) {
    if radius <= 0 {
        return 0, errors.New("radius must be positive")
    }

    return math.Pi * radius * radius, nil
}

The test file with the wrong package:

package area_test

import (
    "area"
    "errors"
    "testing"

    "github.com/stretchr/testify/assert"
)

func TestCircleArea(t *testing.T) {
    type testCase struct {
        Name string

        Radius float64

        ExpectedArea float64
        ExpectedErr  error
    }

    validate := func(t *testing.T, tc *testCase) {
        t.Run(tc.Name, func(t *testing.T) {
            actualArea, actualErr := area.CircleArea(tc.Radius)

            assert.Equal(t, tc.ExpectedArea, actualArea)
            assert.Equal(t, tc.ExpectedErr, actualErr)
        })
    }

    validate(t, &testCase{
        Name: "Negative radius",

        Radius: -1,

        ExpectedErr: errors.New("radius must be positive"),
    })
}

After running symflower fix, the package is fixed:

package area

import (
    "errors"
    "testing"

    "github.com/stretchr/testify/assert"
)

func TestCircleArea(t *testing.T) {
    type testCase struct {
        Name string

        Radius float64

        ExpectedArea float64
        ExpectedErr  error
    }

    validate := func(t *testing.T, tc *testCase) {
        t.Run(tc.Name, func(t *testing.T) {
            actualArea, actualErr := CircleArea(tc.Radius)

            assert.Equal(t, tc.ExpectedArea, actualArea)
            assert.Equal(t, tc.ExpectedErr, actualErr)
        })
    }

    validate(t, &testCase{
        Name: "Negative radius",

        Radius: -1,

        ExpectedErr: errors.New("radius must be positive"),
    })
}

Adding missing imports

This time, we only import math, missing the errors import:

package area

import (
    "math"
)

func CircleArea(radius float64) (area float64, err error) {
    if radius <= 0 {
        return 0, errors.New("radius must be positive")
    }

    return math.Pi * radius * radius, nil
}

After running symflower fix, the errors import is added:

package area

import (
    "errors"
    "math"
)

func CircleArea(radius float64) (area float64, err error) {
    if radius <= 0 {
        return 0, errors.New("radius must be positive")
    }

    return math.Pi * radius * radius, nil
}

Declaring undeclared variables

In this example, we're storing a circle's area in a result variable, but the variable is not declared:

package area

import (
    "errors"
    "math"
)

func CircleArea(radius float64) (area float64, err error) {
    if radius <= 0 {
        return 0, errors.New("radius must be positive")
    }

    result = math.Pi * radius * radius

    return result, nil
}

After running symflower fix, the result variable is declared:

package area

import (
    "errors"
    "math"
)

func CircleArea(radius float64) (area float64, err error) {
    if radius <= 0 {
        return 0, errors.New("radius must be positive")
    }

    result := math.Pi * radius * radius

    return result, nil
}