Developer tools
Verified by Toolspoke

Flaky test finder

By Toolspoke

Reads CircleCI's flaky-test report each Monday, ranks the tests that fail most and keeps one GitHub tracking issue up to date.

Needs a Toolspoke account. You choose which project it installs into.

An illustration with made-up examples. It is not a recording.
CircleCI + GitHub
Required integrations
Runs on a schedule
Mondays at 9am by default
Say what you did
What its last step does
Some actions can delete
2 of 43 actions in its tools
On this page
  1. The short version
  2. What it will never do
  3. What Flaky test finder does
  4. How it works
  5. What a weekly comment looks like
  6. Who uses Flaky test finder
  7. What it connects to
  8. When it runs
  9. How to set it up
  10. Make Flaky test finder fit your team
  11. Compared with other options
  12. Questions
  13. Related agents

The short version

What it is
A ready-made Toolspoke agent that finds your flakiest CircleCI tests and keeps one GitHub issue listing them.
What it reads
CircleCI's flaky-test report and each workflow's success rate for the last seven days, for the projects you chose.
What it finds
The tests that flake most often, which are new this week, and which have gone.
What you get
One comment a week on a single tracking issue, ranked, so the team can pick the next test to fix.
How often
Every Monday once you switch it on.
What it never does
Rerun, cancel or approve a workflow, or skip a test.
What you need
A Toolspoke account, a CircleCI personal API token, and a GitHub repository for the issue.

What Flaky test finder will never do

Flaky test finder is told never to rerun, cancel or approve a workflow, never to skip or quarantine a test, and never to open more than one issue. It finds flaky tests and writes them down.

Flaky test finder will never

  • Rerun, cancel or approve a workflow, or trigger a pipeline.
  • Open more than one issue a run, or a second tracking issue when one is open.
  • Edit, close or relabel an issue, or assign one to anybody.
  • Push a commit, open a branch or pull request, or skip or quarantine a test.
  • Call a test flaky that CircleCI's report does not list.

Why it will not rerun a build

A rerun that passes makes a flaky test look fixed. An approval can let a deploy through, and a cancel stops work part-way. None of that is a report's job.

Why it will not skip a test

Skipping or quarantining a test hides whatever it was catching. Which test to fix, and how, is the team's call.

What a careless bot might doMarked applies coupon after login as skipped to unblock the build.
What this agent writes instead1. applies coupon after login in checkout.spec.ts, job e2e. Flaked 23 times, last on 26 October. New this week.

Why it keeps to one issue

A new issue every week becomes a pile nobody closes. One issue, with a comment a week, keeps the history in one place.

These are written into Flaky test finder's instructions. They guide the agent, but they are not a permission. What it is allowed to touch is set by the access you grant.

Flaky test finder reads CircleCI and keeps the list. Which tests to fix, and every build that runs, stays with your team.

What Flaky test finder does

Flaky test finder is an AI agent that reads CircleCI's flaky-test report every Monday, ranks the tests by how often they flaked, and keeps one GitHub issue up to date with the list. Your team picks which to fix.

A flaky test costs more than its own failures. Every red build it causes gets rerun until it passes, and after a while people rerun real failures too.

CircleCI already spots them: a test that passed and failed on the same commit. The report sits in CircleCI's insights, and nobody reads it until a flaky test blocks a release.

Flaky test finder brings the report to where your team tracks work. Each Monday it ranks the flaky tests across your projects, marks which are new and which have gone, and adds the list to one tracking issue. It never reruns a build to make the problem disappear.

In practice, every Monday you have:

  • Every flaky test across your projects, ranked by how often it flaked
  • Which are new this week, which are still flaky and which have gone
  • Workflows that carry flaky tests and fell below 90% success
  • All of it on one GitHub issue, not a new one each week

How it works

Flaky test finder works in four steps each week: it reads the flaky tests, reads the workflows, compares with last week and writes to the tracking issue. All of it comes from the agent's written instructions, which you can read and edit after you install it.

  1. Step 1: Read the flaky tests

    CircleCI's report for each project you chose.

  2. Step 2: Read the workflows

    Success rate and run count for the last seven days.

  3. Step 3: Compare with last week

    From the latest comment on the tracking issue.

  4. Step 4: Update the issue

    One ranked comment, or a new issue the first time.

Drawn from this agent's own tools and instructions. It is not a recording of a run.

It uses CircleCI's definition

A test is flaky when CircleCI saw it both pass and fail on the same commit. The agent lists only what get_flaky_tests reports, and never calls a test flaky on a hunch.

It keeps one issue

The first run opens an issue titled Flaky tests with your label. Every run after that adds one comment to it, so the history of each test is in one place.

It leaves the issue to you

It never edits the issue's title or body, closes it or assigns it. Whoever keeps the issue can write in it freely.

It never makes a test pass

A rerun that passes hides a flaky test. The agent never reruns, cancels or approves a workflow. The limits are listed under what it will never do.

What a weekly comment looks like

A run of Flaky test finder adds one comment to the tracking issue. It leads with tests new this week, then the ones still flaky, then those gone from the report.

GitHub issue #318, Flaky tests

Flaky test finderAPP9:00 AM

9 flaky tests across 2 projects, 2 new this week. Below 90% success: gh/acme/web workflow build-and-test, 84%.

New this week (2)

  • 1. applies coupon after login in checkout.spec.ts, gh/acme/web, job e2e. Flaked 23 times, last on 26 October.
  • 4. test_large_csv in test_export.py, gh/acme/api, job unit. Flaked 4 times, last on 25 October.

Still flaky (7)

  • 2. test_retry_backoff in test_webhooks.py, gh/acme/api, job unit. Flaked 14 times, up from 11.
  • 3. shows results while typing in search.spec.ts, gh/acme/web, job e2e. Flaked 9 times, down from 12.

Gone from the report (3)

login redirects to dashboard, test_rate_limit_reset, renders empty state. Not seen flaking this week.

One comment. The company, projects and tests are made up.
New this week

Flaky tests that were not on last week's list, with their rank, file, project, job, times flaked and when they last did.

Still flaky

Tests that were on last week's list and still are, with the change in how often they flaked.

Gone from the report

Tests that were flaky last week and are no longer in CircleCI's report. Fixed, or just quiet: the team can tell which.

When the report is clear

If no project reports a flaky test, it comments that the report is clear, or creates nothing if there is no tracking issue yet.

If a project stores no test results, CircleCI cannot find flaky tests in it, and the comment says the project needs store_test_results rather than calling it clear.

Who uses Flaky test finder

Flaky test finder suits an engineering team that builds on CircleCI and has started rerunning red builds without reading them. The examples use made-up names.

A tech lead planning the week

Nadia Petrova picks the top new flaky test each Monday and gives it to whoever is on maintenance duty.

In their issueNeeds you1. applies coupon after login, checkout.spec.ts. Flaked 23 times.

A developer tired of reruns

Kofi Boateng reruns the API build twice a day. The issue shows him it is always the same webhook test.

In their issueWorth knowing2. test_retry_backoff, test_webhooks.py. Flaked 14 times, up from 11.

A platform engineer watching CI health

Hiro Yamada watches the first line for workflows below 90% success.

In their issueWorth knowingBelow 90% success: build-and-test, 84%.

An engineering manager tracking progress

Clara Jensen reads the gone from the report group to see the fixes land.

In their issueEverything elseGone from the report (3).

What Flaky test finder connects to

Flaky test finder connects to two tools: it reads CircleCI and writes to one GitHub issue. You connect each one yourself, and Toolspoke stores the connection encrypted, so the agent never sees the credential.

  • CircleCI

    Required

    Read failed pipelines, workflows, jobs, test results and artifacts, and trigger, rerun or cancel CircleCI workflows.

    Authentication: Personal API token

    View integration for CircleCI

  • GitHub

    Required

    Manage repositories, issues, pull requests, and Actions.

    Authentication: Personal access token

    View integration for GitHub

What Flaky test finder does and cannot do in each connected tool
ToolWhat Flaky test finder doesWhat it cannot do
CircleCIReads the flaky-test report and workflow metrics for each project (get_flaky_tests, get_workflow_metrics).Delete anything: the connector has no delete action. It can trigger a pipeline, rerun or cancel a workflow and approve a job, and the agent is told never to.
GitHubFinds the tracking issue, reads its comments, and opens it or comments on it (list_issues, list_issue_comments, create_issue, comment_on_issue).Delete an issue or a comment: the connector has no delete action. It can edit issues, push commits, open pull requests and run workflows, and the agent is told only to open one issue and comment on it.

A CircleCI personal token has no scopes: it acts as you, with every permission you have. So that the agent never reruns or cancels a build is an instruction to the agent, not a lock. If you want a lock, a project can switch individual actions off. Turn off the CircleCI write and destructive actions and no agent in the project can call them. The access control guide explains how.

The tracking issue lives in the repository you chose. If that repository is public, test names and file paths are public too, so choose a private one if they should not be.

When Flaky test finder runs

Flaky test finder runs every Monday at 9 AM, but only after you switch its schedule on. It arrives switched off, so nothing runs and nothing is charged until you decide.

  • Each run reads CircleCI's flaky-test report for the projects you chose.
  • It reads workflow success for the last seven days.
  • It adds one comment to the tracking issue, or opens it the first time.
  • If the report is clear, it says so in one comment.

Each run uses model credit from your workspace balance, so it makes one run a week. The Toolspoke tool calls themselves are free.

Your copy is an ordinary agent in your project. You can pause its schedule there, run it once on demand, change the day and edit its instructions. The agent's page lists recent runs and their cost.

How to set up Flaky test finder

To start finding flaky tests, add Flaky test finder to a Toolspoke project, connect CircleCI and GitHub, choose your projects and switch on the schedule. It takes a few minutes, and nothing runs until the last step.

  1. Step 1: Create a Toolspoke account or sign in

    Choose Use this agent. If you are new, you will be asked to sign up first, and you come straight back to this agent afterwards.

  2. Step 2: Add it to a project

    Pick the project it should live in. You get your own editable copy, and you can see the two tools it brings before anything connects.

  3. Step 3: Connect CircleCI

    Create a personal API token in CircleCI's user settings and paste it. Project tokens do not work with CircleCI's API.

  4. Step 4: Choose your projects

    List the CircleCI projects by slug, such as gh/acme/api. Each needs to store test results for flaky tests to show up.

  5. Step 5: Connect GitHub and make the label

    Sign in to GitHub, choose the repository for the tracking issue, and create the label it will use, flaky-tests by default.

  6. Step 6: Switch on the schedule

    The schedule arrives switched off. Run it once, read the issue it opens, then switch it on.

Make Flaky test finder fit your team

Flaky test finder's instructions are plain English and you can edit them after you install it. Installing copies the agent, so your edits stay with your copy. Open the agent in Toolspoke and tell the builder what to change, and it rewrites the instructions.

Paste any sentence below into the builder chat, or say the same thing in your own words.

A shorter list
Keep the top 10 instead of the top 20.
A stricter health bar
Name workflows below 95% success instead of 90%.
Only one project
Only read gh/acme/web, and leave the others out.
Mention the owners
Mention @acme/web-team in the comment when a test from gh/acme/web is new this week.
Run every two weeks
Run every second Monday at 9 AM.

This changes when the agent wakes up, so the builder edits its schedule, not the instructions.

None of these edits needs more access or touches the Never rules. They change what Flaky test finder is told, not what the connections allow.

How it compares to other options

Flaky test finder beats the alternatives when CircleCI already knows which tests are flaky and nobody reads it, and loses to them when you want flaky tests retried or quarantined for you.

Flaky test finder compared with CircleCI Insights, Automatic retries, A spreadsheet someone keeps
Compared onFlaky test finderCircleCI InsightsAutomatic retriesA spreadsheet someone keeps
Finds flaky testsFrom CircleCI's reportYesNo, hides themBy hand
Ranks them across projectsYesPer projectNoIf kept up
Shows what is new and what has goneEvery weekNoNoSometimes
Lives where the team tracks workA GitHub issueIn CircleCINoA separate file
Keeps builds greenNo, it reportsNoYes, by hiding failuresNo
CostA run a weekIncludedExtra CI minutesSomeone's time

Automatic retries are the better choice when you need a green build today. Flaky test finder is for the week after, when you want the list of tests to fix rather than another rerun.

CircleCI Insights holds the same report, and the agent reads it. The difference is that it ranks across projects, says what changed since last week, and puts it on an issue your team already watches.

Frequently asked questions

Can Flaky test finder rerun or cancel my builds?

The CircleCI connection can, because a personal token has no scopes. The agent is told never to. To make it a lock, switch the CircleCI write and destructive actions off in your project.

How does it decide a test is flaky?

It uses CircleCI's own report: a test that passed and failed on the same commit. It never adds a test the report does not list.

Why is a project missing from the list?

CircleCI can only find flaky tests in a project that stores test results with store_test_results. The comment says which projects report nothing for that reason.

Will it open a new issue every week?

No. It opens one tracking issue the first time and adds a comment each week after that. If more than one is open with the label, it uses the oldest and says so.

Does it work with GitHub Actions?

Not as shipped, because it reads CircleCI's flaky-test report. You can ask the agent builder for another version. It needs its own instructions.

Is Flaky test finder free?

It has no price of its own, and every Toolspoke plan includes agents. Tool calls cost nothing, but each run draws on your usage balance. Pricing has the details.

Fix the tests you keep rerunning

Connect CircleCI and GitHub, choose your projects and switch on the schedule. Your first ranked list lands on Monday.