Test Generation via LLMs (experimental)

info

Test Generation via LLMs is an experimental feature currently available in VS Code (from Symflower version 39199)

This experimental feature provides access to LLM-generated test suites in Visual Studio Code. Rather than Symflower's symbolic execution engine, this feature runs your query through the LLM specified in the configuration.

Functionality:

• Generate test suites for file via LLM
• Generate test suites for function via LLM

To use this feature, open up the command palette and search for Symflower: Generate Test Suites for File via LLM or Symflower: Generate Test Suites for Function via LLM.

User queries to the LLM are automatically appended with the following prompt:

Given the following Go code file "plain.go" with package "plain", provide a test file for this code. The tests should produce 100 percent code coverage and must compile. The response must contain only the test code and nothing else.

The LLM's output (generated tests) are added to a virtual file. Copy and paste the desired code to your test file.

Enable Test Generation via LLMs

Enable and configure LLM: Test Generation in Symflower's extension settings in Visual Studio Code. You'll need to enter your chosen LLM provider's:

• URL: An OpenAPI-compatible endpoint. Example: https://api.openai.com/v1 for GPT. The default setting is OpenRouter: https://openrouter.ai/api/v1.
• API token: The token for the model provider's API.
• Model: The model to be used for LLM queries. Example: mistralai/codestral-mamba.

caution

Data privacy

Symflower's Test Generation via LLM feature forwards your code to the configured LLM's provider. The request is sent to the LLM and the response is written into a virtual file. Symflower does not use or store your data. Read the data policy of your selected LLM provider to understand how your data is used.

Tutorial: Test Generation via LLMs

For this example, we have used Claude 3.5 Sonnet to generate tests. We configured Symflower's extension settings as follows:

• Symflower > Llm:Model: anthropic/claude-3.5-sonnet
• Symflower > Llm > Provider:Token: your API token

We started with the following source code:

package com.symflower.area;

public class Circle {
    public static double CircleArea(double radius) {
        if (radius <= 0) {
            throw new IllegalArgumentException("radius must be positive");
        }

        return Math.PI * radius * radius;
    }
}

For this example, we then ran Symflower: Generate Test Suites for Function via LLM in Visual Studio Code. This resulted in the following output:

package com.symflower.area;

import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.function.Executable;
import static org.junit.jupiter.api.Assertions.*;

class CircleTest {

    @Test
    void testCircleAreaPositiveRadius() {
        double radius = 5.0;
        double expectedArea = Math.PI * radius * radius;
        double actualArea = Circle.CircleArea(radius);
        assertEquals(expectedArea, actualArea, 0.0001, "Area calculation for positive radius should be correct");
    }

    @Test
    void testCircleAreaZeroRadius() {
        double radius = 0.0;
        Executable executable = () -> Circle.CircleArea(radius);
        IllegalArgumentException exception = assertThrows(IllegalArgumentException.class, executable, "Should throw IllegalArgumentException for zero radius");
        assertEquals("radius must be positive", exception.getMessage(), "Exception message should match");
    }

    @Test
    void testCircleAreaNegativeRadius() {
        double radius = -5.0;
        Executable executable = () -> Circle.CircleArea(radius);
        IllegalArgumentException exception = assertThrows(IllegalArgumentException.class, executable, "Should throw IllegalArgumentException for negative radius");
        assertEquals("radius must be positive", exception.getMessage(), "Exception message should match");
    }
}