Skip to content

Latest commit

 

History

History
278 lines (191 loc) · 14.9 KB

README-UnitGen.md

File metadata and controls

278 lines (191 loc) · 14.9 KB
Raw

keploy logo

⚡️ Generate unit tests with LLMs, that actually works ⚡️

🌟 The must-have tool for developers in the AI-Gen era 🌟

Keploy Twitter Help us reach 4k stars! Keploy CNCF Landscape

Slack LinkedIn Twitter

Keploy-gen uses LLMs to understand code semantics and generates meaningful unit tests. It's inspired by the Automated Unit Test Improvement using LLM at Meta.

Objectives

  • Automate unit test generation (UTG): Quickly generate comprehensive unit tests and reduce the redundant manual effort.

  • Improve edge cases: Extend and improve the scope of tests to cover more complex scenarios that are often missed manually.

  • Boost test coverage: As codebase grows, ensuring exhaustive coverage should become feasible.

Core Components

Phase Activities Tools/Technologies
Code Analysis Analyze the code structure and dependencies. Static analysis tools, LLMs
Prompt Engineering Generation of targeted prompts to guide the LLM in producing relevant tests. LLMs, Custom scripts
Iterative Test Refinement Cyclic process of refining tests by running them, analyzing coverage, and incorporating feedback. Testing frameworks (e.g., JUnit, pytest)

Process Overview

Referred from Meta's research, TestGen-LLM top level architecture.

Test refinement process of unit test generator

Prerequisites

AI model Setup - Set the environment variable API_KEY.

export API_KEY=xxxx

API_KEY can be from either of one these:

  • OpenAI's GPT-4o directly [preferred].

  • Alternative LLMs via litellm.

  • Azure OpenAI

Installation

Install Keploy locally by running the following command:

➡ Linux/Mac

 curl --silent -O -L https://keploy.io/install.sh && source install.sh

➡ Windows

  • Download and move the keploy.exe file to C:\Windows\System32

NodeJS ➡ Running with Node.js/TypeScript applications

  • Ensure you've set the API key, as mentioned in pre-requisites above:

    export API_KEY=xxxx

  • Ensure Cobertura formatted coverage reports, edit jest.config.js or package.json: 

    // package.json
"jest": {
      "coverageReporters": ["text", "cobertura"],
    }

    or

      // jest.config.js
  module.exports = {
    coverageReporters: ["text", "cobertura"],
  };

Generating Unit Tests

  • Run the following command in the root of your application.

    • For Single Test File: If you prefer to test a smaller section of your application or to control costs, consider generating tests for a single source and its corresponding test file:

       keploy gen --sourceFilePath="<path to source file>" --testFilePath="<path to test file for above source file>" --testCommand="npm test" --coverageReportPath="<path to coverage.xml>"

    • For Entire Application use the following command to generate tests across:

      ⚠️ Warning: Executing this command will generate unit tests for all files in the application. Depending on the size of the codebase, this process may take between 20 minutes to an hour and will incur costs related to LLM usage.

      keploy gen --testCommand="npm test" --testDir="test" --coverageReportPath="<path to coverage.xml>"

    🎉 You should see improved test cases and code-coverage. ✅ Enjoy coding with enhanced unit test coverage! 🫰

Go → Running with Golang applications

  • Ensure you've set the API key, as mentioned in pre-requisites above:

    export API_KEY=xxxx

  • To ensure Cobertura formatted coverage reports, add:

     go install github.com/axw/gocov/[email protected]
 go install github.com/AlekSi/[email protected]

Generating Unit Tests

  • Run the following command in the root of your application.

    • For Single Test File: If you prefer to test a smaller section of your application or to control costs, consider generating tests for a single source and its corresponding test file:

      keploy gen --sourceFilePath="<path to source file>" --testFilePath="<path to test file for above source file>" --testCommand="go test -v ./... -coverprofile=coverage.out && gocov convert coverage.out | gocov-xml > coverage.xml" --coverageReportPath="<path to coverage.xml>"

    • For Entire Application use the following command to generate tests across:

      ⚠️ Warning: Executing this command will generate unit tests for all files in the application. Depending on the size of the codebase, this process may take between 20 minutes to an hour and will incur costs related to LLM usage.

      keploy gen --testDir="." --testCommand="go test -v ./... -coverprofile=coverage.out && gocov convert coverage.out | gocov-xml > coverage.xml" --coverageReportPath="<path to coverage.xml>"

      🎉 You should see improved test cases and code-coverage. ✅ Enjoy coding with enhanced unit test coverage! 🫰

→ Setup for Other Languages

  • Ensure you've set the API key, as mentioned in pre-requisites above:

    export API_KEY=xxxx

  • Ensure that your unit test report format is Cobertura(it's very common).

  • Generate tests using keploy-gen:

    keploy gen --sourceFilePath="<path to source code file>" --testFilePath="<path to existing unit test file>" --testCommand="<cmd to execute unit tests>" --coverageReportPath="<path to cobertura-coverage.xml>"

Configuration

Configure Keploy using command-line flags:

  --sourceFilePath ""
  --testFilePath ""
  --coverageReportPath "coverage.xml"
  --testCommand ""
  --coverageFormat "cobertura"
  --expectedCoverage 100
  --maxIterations 5
  --testDir ""
  --llmBaseUrl "https://api.openai.com/v1"
  --model "gpt-4o"
  --llmApiVersion "
  • sourceFilePath: Path to the source file for which tests are to be generated.
  • testFilePath: Path where the generated tests will be saved.
  • coverageReportPath: Path to generate the coverage report.
  • testCommand (required): Command to execute tests and generate the coverage report.
  • coverageFormat: Type of the coverage report (default "cobertura").
  • expectedCoverage: Desired coverage percentage (default 100%).
  • maxIterations: Maximum number of iterations for refining tests (default 5).
  • testDir: Directory where tests will be written.
  • llmBaseUrl: Base url of the llm.
  • model: Specifies the AI model to use (default "gpt-4o").
  • llmApiVersion: API version of the llm if any (default "")

Frequently Asked Questions

  1. What is Keploy's Unit Test Generator (UTG)?

    • Keploy's UTG automates the creation of unit tests based on code semantics, enhancing test coverage and reliability.

  2. Does Keploy send your private data to any cloud server for test generation?

    • No, Keploy does not send any user code to remote systems, except when using the unit test generation feature. When using the UT gen feature, only the source code and the unit test code will be sent to the Large Language Model (LLM) you are using. By default, Keploy uses - litellm to support vast number of LLM backends. Yes, if your organization has its own LLM(a private one), you can use it with Keploy. This ensures that data is not sent to any external systems.

  3. How does Keploy contribute to improving unit test coverage?

    • By providing a zero code platform for automated testing, Keploy empowers developers to scale up their unit test coverage without extensive coding knowledge. This integration enhances testing reports, ultimately boosting confidence in the product's quality.

  4. Is Keploy cost-effective for automated unit testing?

    • Yes, Keploy optimizes costs by automating repetitive testing tasks and improving overall test efficiency.

  5. How does Keploy generate coverage reports?

    • Keploy generates detailed Cobertura format reports, offering insights into test effectiveness and code quality.

  6. Can Keploy handle large codebases efficiently?

    • Yes, Keploy is designed to handle large codebases efficiently, though processing time may vary based on project size and complexity.

🙋🏻‍♀️ Questions? 🙋🏻‍♂️

Reach out to us. We're here to answer!

Slack LinkedIn YouTube Twitter

📝 Sample QuickStarts

🌐 Language Support

Go NodeJS

Other language may be supported, we've not tested them yet. If your coverage reports are of Cobertura format then you should be able to use keploy-gen in any language.

Dev Support

Keploy-gen is not just a project but an attempt to make developers life easier testing applications. It aims to simplify the creation and maintenance of tests, ensuring high coverage, and adapts to the complexity of modern software development.

Prompt Generation

Referred from Meta's research, the four primary prompts used in the deployment for the December 2023 Instagram and Facebook app test-a-thons

Prompt Name Template
extend_test Here is a Kotlin unit test class: {existing_test_class}. Write an extended version of the test class that includes additional tests to cover some extra corner cases.
extend_coverage Here is a Kotlin unit test class and the class that it tests: {existing_test_class} {class_under_test}. Write an extended version of the test class that includes additional unit tests that will increase the test coverage of the class under test.
corner_cases Here is a Kotlin unit test class and the class that it tests: {existing_test_class} {class_under_test}. Write an extended version of the test class that includes additional unit tests that will cover corner cases missed by the original and will increase the test coverage of the class under test.
statement_to_complete Here is a Kotlin class under test {class_under_test} This class under test can be tested with this Kotlin unit test class {existing_test_class}. Here is an extended version of the unit test class that includes additional unit test cases that will cover methods, edge cases, corner cases, and other features of the class under test that were missed by the original unit test class:

Limitation: This project currently doesn't generate quality fresh tests if there are no existing tests to learn from.

Enjoy coding with enhanced unit test coverage! 🫰