Symflower fix

symflower fix is a static analysis tool to repair code generated by LLMs. Use it on local files to post-process the output of LLMs:

symflower fix --language=$language --workspace=$workspace

Some examples of the functionality provided by the automatic repair logic in symflower fix:

• Remove unused imports: Unused imports are removed as they are compile errors in Go.
• Repair packages: Resolve errors from LLMs generating the wrong package names (e.g. light_test instead of just light).
• Add missing imports: Add the necessary imports (e.g. fmt.Println) to make the code compile.
• Declare undeclared variables: Variables that appear in the code but are undeclared are declared with the first assigned value.

Use symflower fix to improve the efficiency of generating code with LLMs. symflower fix may be applied as part of a Retrieval-Augmented Generation (RAG) workflow to improve LLM performance.

Background: Common compile errors in LLM-generated code

The latest results from the DevQualityEval benchmark show that most LLMs demonstrate limited ability to generate compiling code. A large percentage of the underlying issues fall into one of the following categories with the potential to be fixed with simple static analysis:

• Missing or incorrect package statements
• Missing or unnecessary imports
• Undeclared variables
• Assigning negative values to uint
• Wrong object creation for nested class

The DevQualityEval benchmark uses symflower fix whenever a model generates code that doesn't compile. symflower fix is also used in the write-test and transpile tasks in case the generated code fails the included tests.

Benchmarking results show that static code analysis with symflower fix can positively impact LLM performance in generating compilable code (tested in Go):

• +22.9% in scores across 45 applicable models
• +26.2% in response files compiled (avg. 17 files, 150 tasks total)

Results suggest that using symflower fix, a weaker and cheaper model (with static analysis) can outperform a larger model (without static analysis) at generating compilable code:

Model	Score	# Compilable files	Beats (without static code repair)
mistral-tiny	+71%	+109%	mistral-small, mistral-medium
DBRX	+19%	+26%	GPT-4o-mini, GPT-4o
Gemma 2 27B	+16%	+13%	GPT-4o-mini, GPT-4o, LLama 3.1 405B
GPT-4o-mini	+14%	+11%	GPT-4-turbo, Claude Sonnet 3.5
GPT-4-turbo	+8%	+8%	Claude Sonnet 3.5 + static code repair

Tutorial: symflower fix

The following tutorials provide examples of using some of the features in symflower fix.

• Example: A function that calculates the area of a circle
• Command: symflower fix --language=golang

Removing unused imports with symflower fix

In this example, strings and fmt are unused imports:

package area

import (
    "errors"
    "math"
    "strings"
    "fmt"
)

func CircleArea(radius float64) (area float64, err error) {
    if radius <= 0 {
        return 0, errors.New("radius must be positive")
    }

    return math.Pi * radius * radius, nil
}

After running symflower fix, the strings and fmt imports are removed:

package area

import (
    "errors"
    "math"
)

func CircleArea(radius float64) (area float64, err error) {
    if radius <= 0 {
        return 0, errors.New("radius must be positive")
    }

    return math.Pi * radius * radius, nil
}

Repairing packages with symflower fix

Source file:

package area

import (
    "errors"
    "math"
)

func CircleArea(radius float64) (area float64, err error) {
    if radius <= 0 {
        return 0, errors.New("radius must be positive")
    }

    return math.Pi * radius * radius, nil
}

The test file with the wrong package:

package area_test

import (
    "area"
    "errors"
    "testing"

    "github.com/stretchr/testify/assert"
)

func TestCircleArea(t *testing.T) {
    type testCase struct {
        Name string

        Radius float64

        ExpectedArea float64
        ExpectedErr  error
    }

    validate := func(t *testing.T, tc *testCase) {
        t.Run(tc.Name, func(t *testing.T) {
            actualArea, actualErr := area.CircleArea(tc.Radius)

            assert.Equal(t, tc.ExpectedArea, actualArea)
            assert.Equal(t, tc.ExpectedErr, actualErr)
        })
    }

    validate(t, &testCase{
        Name: "Negative radius",

        Radius: -1,

        ExpectedErr: errors.New("radius must be positive"),
    })
}

After running symflower fix, the package is fixed:

package area

import (
    "errors"
    "testing"

    "github.com/stretchr/testify/assert"
)

func TestCircleArea(t *testing.T) {
    type testCase struct {
        Name string

        Radius float64

        ExpectedArea float64
        ExpectedErr  error
    }

    validate := func(t *testing.T, tc *testCase) {
        t.Run(tc.Name, func(t *testing.T) {
            actualArea, actualErr := CircleArea(tc.Radius)

            assert.Equal(t, tc.ExpectedArea, actualArea)
            assert.Equal(t, tc.ExpectedErr, actualErr)
        })
    }

    validate(t, &testCase{
        Name: "Negative radius",

        Radius: -1,

        ExpectedErr: errors.New("radius must be positive"),
    })
}

Adding missing imports with symflower fix

This time, we only import math, missing the errors import:

package area

import (
    "math"
)

func CircleArea(radius float64) (area float64, err error) {
    if radius <= 0 {
        return 0, errors.New("radius must be positive")
    }

    return math.Pi * radius * radius, nil
}

After running symflower fix, the errors import is added:

package area

import (
    "errors"
    "math"
)

func CircleArea(radius float64) (area float64, err error) {
    if radius <= 0 {
        return 0, errors.New("radius must be positive")
    }

    return math.Pi * radius * radius, nil
}

Declaring undeclared variables with symflower fix

In this example, we're storing a circle's area in a result variable, but the variable is not declared:

package area

import (
    "errors"
    "math"
)

func CircleArea(radius float64) (area float64, err error) {
    if radius <= 0 {
        return 0, errors.New("radius must be positive")
    }

    result = math.Pi * radius * radius

    return result, nil
}

After running symflower fix, the result variable is declared:

package area

import (
    "errors"
    "math"
)

func CircleArea(radius float64) (area float64, err error) {
    if radius <= 0 {
        return 0, errors.New("radius must be positive")
    }

    result := math.Pi * radius * radius

    return result, nil
}