Worker

Coverage-Driven Unit Test Generation with Cover-Agent

A worker that generates unit tests for a source file, runs them, and only keeps tests that compile, pass, and measurably increase code coverage — iterating until it hits a target.

What This Builds

This recipe builds an automated test-writing worker. You point it at a source file and an existing (possibly minimal) test file, and it generates new test cases, runs the suite, and keeps only the tests that compile, pass reliably, and increase code coverage. It iterates up to a budget you set, raising coverage toward a target.

It is built on Qodo’s Cover-Agent (formerly CodiumAI Cover-Agent), the first open-source implementation of Meta’s “TestGen-LLM” / “Assured LLMSE” approach. The key idea is the validation loop: generated tests are filtered, not trusted. In the freeCodeCamp walkthrough, this took a sample calculator.py from 50% to 100% coverage.

Product Shape

This is a worker that produces a verified artifact (new passing tests), not a chat assistant. The discipline that makes it trustworthy is the filter: any generated test that does not build, does not pass, or does not add coverage is discarded — guaranteeing no regression in the existing suite.

The Stack

  • Cover-Agent — the test-generation engine with the Test Runner, Coverage Parser, Prompt Builder, and AI Caller components.
  • A coverage tool such as pytest-cov producing a Cobertura XML report (the format Cover-Agent reads).
  • OpenAI for Startups credits (or any supported model) — Cover-Agent calls an LLM, configured via --openai-model, e.g. gpt-4o.
  • GitHub Actions — runs the worker on a schedule or on PRs to top up coverage automatically.
  • GitHub repository — the code under test and where regenerated tests land.

Step-by-Step Outline

  1. Install Cover-Agent (pip install git+https://github.com/qodo-ai/qodo-cover.git) and set your LLM API key.
  2. Create a skeleton test file with the imports and at least one passing test.
  3. Generate a baseline coverage report (pytest --cov=. --cov-report=xml).
  4. Run cover-agent with --source-file-path, --test-file-path, --code-coverage-report-path, --test-command, --desired-coverage, and --max-iterations.
  5. The agent loops: build a prompt from the code, call the LLM, run the new tests, parse coverage, and discard tests that fail or add nothing.
  6. Wire it into GitHub Actions to run against changed files in a PR so coverage trends up over time.

Why This Shape Works

Generic LLMs happily emit plausible-looking tests that do not compile or do not add value. The filter loop — compile, pass, coverage-increase — is what turns “an LLM that writes tests” into “a worker whose output you can merge.” Capping --max-iterations keeps token cost bounded.

Source