Skip to main content

Generating test suites

Generate high-coverage test suites for your code to dramatically reduce the time and effort costs of testing your applications. Generated test suites include both the boilerplate and the actual input values that help provide high coverage over all possible execution paths in your code.

Symflower offers two methods for generating test suites:

Generate test suite via symbolic execution​

Find edge cases, reproduce bugs faster, and decide which tests you want to add to your test suite. Symflower's symbolic execution engine explores all possible execution paths and calculates test values to determine all relevant test scenarios.

Usage​

Open the file you want to analyze and choose any of the following options to trigger the generation of a test suite for a given function:

Context menu​

Right-click into a function and select "Symflower: Generate Test Suite for Function" from the context menu:

Trigger the context menu.

Keyboard shortcut​

Define a keyboard shortcut to trigger Symflower's test suite generation. Position your cursor within a function and hit your specified keybinding:

Define a keyboard shortcut.

Command palette​

Use the command palette.

Details​

Symflower currently supports the JUnit 4 and JUnit 5 testing frameworks. Have Symflower automatically detect the framework you are using, or select the framework manually in Symflower's configuration options.

Tutorial​

Step 1: Tutorial example​

Let's see an example of a simple implementation of a class that stores and categorizes triangles. Add the following code to a new file called Triangle.java:

public class Triangle {
  private int sideA;
  private int sideB;
  private int sideC;

  public Triangle(int sideA, int sideB, int sideC) {
      this.sideA = sideA;
      this.sideB = sideB;
      this.sideC = sideC;
  }
  private int sideA;
  private int sideB;
  private int sideC;

  public Triangle(int sideA, int sideB, int sideC) {
      this.sideA = sideA;
      this.sideB = sideB;
      this.sideC = sideC;
  }

public TriangleType getType() {
// Each side needs to be greater than 0
  if (sideA <= 0 || sideB <= 0 || sideC <= 0) {
      return TriangleType.invalid;
  }

// Two sides need to be bigger than the third
if(this.sideA + this.sideB < sideC || this.sideB + this.sideC < this.sideA || this.sideA + this.sideC < this.sideB){
return TriangleType.invalid;
}
public TriangleType getType() {
// Each side needs to be greater than 0
  if (sideA <= 0 || sideB <= 0 || sideC <= 0) {
      return TriangleType.invalid;
  }

// Two sides need to be bigger than the third
if(this.sideA + this.sideB < sideC || this.sideB + this.sideC < this.sideA || this.sideA + this.sideC < this.sideB){
return TriangleType.invalid;
}

  if (this.sideA == this.sideB && this.sideB == this.sideC) {
  if (this.sideA == this.sideB && this.sideB == this.sideC) {
      return TriangleType.equilateral;
  }
  }

  if (this.sideA == this.sideB || this.sideB == this.sideC || this.sideA == this.sideC) {
  if (this.sideA == this.sideB || this.sideB == this.sideC || this.sideA == this.sideC) {
      return TriangleType.isosceles;
  }
  }

  return TriangleType.scalene;
}
  return TriangleType.scalene;
}
}

enum TriangleType {
  equilateral,
  isosceles,
  scalene,
  invalid
}

Step 2: Generate test suite​

Use any of the usage options above to generate a test suite for this file via symbolic execution. A new test file TriangleSymflowerTest.java will be created with the following contents:

import org.junit.*;
import static org.junit.Assert.*;

public class TriangleSymflowerTest {
    @Test
    public void getType1() {
        Triangle t = new Triangle(0, 0, 0);
        TriangleType expected = TriangleType.invalid;
        TriangleType actual = t.getType();

        assertEquals(expected, actual);
    }

    @Test
    public void getType2() {
        Triangle t = new Triangle(1, 0, 0);
        TriangleType expected = TriangleType.invalid;
        TriangleType actual = t.getType();

        assertEquals(expected, actual);
    }

    @Test
    public void getType3() {
        Triangle t = new Triangle(1, 1, 0);
        TriangleType expected = TriangleType.invalid;
        TriangleType actual = t.getType();

        assertEquals(expected, actual);
    }

    @Test
    public void getType4() {
        Triangle t = new Triangle(1, 1, 1);
        TriangleType expected = TriangleType.equilateral;
        TriangleType actual = t.getType();

        assertEquals(expected, actual);
    }

    @Test
    public void getType5() {
        Triangle t = new Triangle(1, 2147483646, 3);
        TriangleType expected = TriangleType.invalid;
        TriangleType actual = t.getType();

        assertEquals(expected, actual);
    }

    @Test
    public void getType6() {
        Triangle t = new Triangle(1, 3, 1);
        TriangleType expected = TriangleType.invalid;
        TriangleType actual = t.getType();

        assertEquals(expected, actual);
    }

    @Test
    public void getType7() {
        Triangle t = new Triangle(128, 1, 1);
        TriangleType expected = TriangleType.invalid;
        TriangleType actual = t.getType();

        assertEquals(expected, actual);
    }

    @Test
    public void getType8() {
        Triangle t = new Triangle(1342177280, 469764097, 872415232);
        TriangleType expected = TriangleType.invalid;
        TriangleType actual = t.getType();

        assertEquals(expected, actual);
    }

    @Test
    public void getType9() {
        Triangle t = new Triangle(2, 1, 1);
        TriangleType expected = TriangleType.isosceles;
        TriangleType actual = t.getType();

        assertEquals(expected, actual);
    }

    @Test
    public void getType10() {
        Triangle t = new Triangle(2147483647, 1, 1);
        TriangleType expected = TriangleType.invalid;
        TriangleType actual = t.getType();

        assertEquals(expected, actual);
    }

    @Test
    public void getType11() {
        Triangle t = new Triangle(32768, 1, 32768);
        TriangleType expected = TriangleType.isosceles;
        TriangleType actual = t.getType();

        assertEquals(expected, actual);
    }

    @Test
    public void getType12() {
        Triangle t = new Triangle(342289410, 194581503, 536870914);
        TriangleType expected = TriangleType.invalid;
        TriangleType actual = t.getType();

        assertEquals(expected, actual);
    }

    @Test
    public void getType13() {
        Triangle t = new Triangle(524287, 524287, 1);
        TriangleType expected = TriangleType.isosceles;
        TriangleType actual = t.getType();

        assertEquals(expected, actual);
    }

    @Test
    public void getType14() {
        Triangle t = new Triangle(8191, 8190, 1);
        TriangleType expected = TriangleType.scalene;
        TriangleType actual = t.getType();

        assertEquals(expected, actual);
    }
}

Step 3: Review and add tests​

This test suite explores all possible paths in your code and provides full coverage of all test scenarios. Choose more descriptive names for your tests and add selected tests or the entire test suite to your test file.

Generate test suite via LLMs​

Access LLM-generated test suites in your IDE. Run your query through the LLM specified in the configuration and get the test suite as a result.

Usage​

Configure LLM​

Open Symflower's configuration options. After enabling LLM test generation, specify the LLM you want to use.

You'll need to enter your chosen LLM provider's:

  • URL: An OpenAPI-compatible endpoint. Example: https://api.openai.com/v1 for GPT. The default setting is OpenRouter: https://openrouter.ai/api/v1.
  • API token: The token for the model provider's API.
  • Model: The model to be used for LLM queries. Example: mistralai/codestral-mamba.

The LLM's output (generated tests) are added to a virtual file. Copy and paste the desired code to your test file.

caution

Data privacy

Symflower's Test Generation via LLM feature forwards your code to the configured LLM's provider. The request is sent to the LLM and the response is written into a virtual file. Symflower does not use or store your data. Read the data policy of your selected LLM provider to understand how your data is used.

Configuring test generation via LLMs.

Generate test suite via LLM​

Once configuration is done, open the file you want to analyze and choose any of the following options to trigger the generation of a test suite via LLM for a selected function or file:

1. Code lens​

Enable, then click the "generate tests (LLM)" code lens above any function to generate a test suite via LLM for that function:

Use code lens.

2. Keyboard shortcut​

Define a keyboard shortcut to trigger test suite generation via LLMs for the selected function or file. Position your cursor within a function and hit your specified keybinding:

Define a keyboard shortcut.

3. Command palette​

Position your cursor into a function. Open the command palette of your editor and start typing "Symflower". Select the entry "Symflower: Generate Test Suites for Function via LLM" (or use "Symflower: Generate Test Suites for File via LLM" if you want to generate a test suite for the entire file):

Use the command palette.

Tutorial​

Step 1: Configure LLM test generation​

For this example, we used Claude 3.5 Sonnet to generate tests. We configured Symflower's extension settings as follows:

  • Symflower > LLM:Model: anthropic/claude-3.5-sonnet
  • Symflower > LLM > Provider:Token: your API token

Step 2: Tutorial example​

In this tutorial, we will use the following Java code to demonstrate how to generate test suites via LLMs:

package com.symflower.area;

public class Circle {
    public static double CircleArea(double radius) {
        if (radius <= 0) {
            throw new IllegalArgumentException("radius must be positive");
        }

        return Math.PI * radius * radius;
    }
}

Copy this code into a file named Circle.java and save it in a directory of your choice.

Step 3: Generate test suite via LLM​

Use any of the available usage options to generate a test suite via LLM for the Circle class.

For this example, we ran Symflower: Generate Test Suites for Function via LLM from the command palette in Visual Studio Code. This resulted in the following output:

package com.symflower.area;

import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.function.Executable;
import static org.junit.jupiter.api.Assertions.*;

class CircleTest {

    @Test
    void testCircleAreaPositiveRadius() {
        double radius = 5.0;
        double expectedArea = Math.PI * radius * radius;
        double actualArea = Circle.CircleArea(radius);
        assertEquals(expectedArea, actualArea, 0.0001, "Area calculation for positive radius should be correct");
    }

    @Test
    void testCircleAreaZeroRadius() {
        double radius = 0.0;
        Executable executable = () -> Circle.CircleArea(radius);
        IllegalArgumentException exception = assertThrows(IllegalArgumentException.class, executable, "Should throw IllegalArgumentException for zero radius");
        assertEquals("radius must be positive", exception.getMessage(), "Exception message should match");
    }

    @Test
    void testCircleAreaNegativeRadius() {
        double radius = -5.0;
        Executable executable = () -> Circle.CircleArea(radius);
        IllegalArgumentException exception = assertThrows(IllegalArgumentException.class, executable, "Should throw IllegalArgumentException for negative radius");
        assertEquals("radius must be positive", exception.getMessage(), "Exception message should match");
    }
}

Step 4: Review and add tests​

Review the generated test cases to check if the test case names are descriptive enough and if the test values are appropriate. Make any necessary adjustments to the test cases. Copy and paste the desired test code from the generated virtual file to your test file.

Using generated tests​

Learn about managing tests with Symflower.

After generating tests, continue by reviewing the generated test cases, adding selected ones to your test file, and running the tests. To save test execution time, use test impact analysis with Symflower test-runner.