Flaky test finder
By Toolspoke
Reads CircleCI's flaky-test report each Monday, ranks the tests that fail most and keeps one GitHub tracking issue up to date.
Needs a Toolspoke account. You choose which project it installs into.
- CircleCI + GitHub
- Required integrations
- Runs on a schedule
- Mondays at 9am by default
- Say what you did
- What its last step does
- Some actions can delete
- 2 of 43 actions in its tools
On this page
The short version
- What it is
- A ready-made Toolspoke agent that finds your flakiest CircleCI tests and keeps one GitHub issue listing them.
- What it reads
- CircleCI's flaky-test report and each workflow's success rate for the last seven days, for the projects you chose.
- What it finds
- The tests that flake most often, which are new this week, and which have gone.
- What you get
- One comment a week on a single tracking issue, ranked, so the team can pick the next test to fix.
- How often
- Every Monday once you switch it on.
- What it never does
- Rerun, cancel or approve a workflow, or skip a test.
- What you need
- A Toolspoke account, a CircleCI personal API token, and a GitHub repository for the issue.
What Flaky test finder will never do
Flaky test finder is told never to rerun, cancel or approve a workflow, never to skip or quarantine a test, and never to open more than one issue. It finds flaky tests and writes them down.
Flaky test finder will never
- Rerun, cancel or approve a workflow, or trigger a pipeline.
- Open more than one issue a run, or a second tracking issue when one is open.
- Edit, close or relabel an issue, or assign one to anybody.
- Push a commit, open a branch or pull request, or skip or quarantine a test.
- Call a test flaky that CircleCI's report does not list.
Why it will not rerun a build
A rerun that passes makes a flaky test look fixed. An approval can let a deploy through, and a cancel stops work part-way. None of that is a report's job.
Why it will not skip a test
Skipping or quarantining a test hides whatever it was catching. Which test to fix, and how, is the team's call.
Why it keeps to one issue
A new issue every week becomes a pile nobody closes. One issue, with a comment a week, keeps the history in one place.
These are written into Flaky test finder's instructions. They guide the agent, but they are not a permission. What it is allowed to touch is set by the access you grant.
Flaky test finder reads CircleCI and keeps the list. Which tests to fix, and every build that runs, stays with your team.
What Flaky test finder does
Flaky test finder is an AI agent that reads CircleCI's flaky-test report every Monday, ranks the tests by how often they flaked, and keeps one GitHub issue up to date with the list. Your team picks which to fix.
A flaky test costs more than its own failures. Every red build it causes gets rerun until it passes, and after a while people rerun real failures too.
CircleCI already spots them: a test that passed and failed on the same commit. The report sits in CircleCI's insights, and nobody reads it until a flaky test blocks a release.
Flaky test finder brings the report to where your team tracks work. Each Monday it ranks the flaky tests across your projects, marks which are new and which have gone, and adds the list to one tracking issue. It never reruns a build to make the problem disappear.
In practice, every Monday you have:
- Every flaky test across your projects, ranked by how often it flaked
- Which are new this week, which are still flaky and which have gone
- Workflows that carry flaky tests and fell below 90% success
- All of it on one GitHub issue, not a new one each week
How it works
Flaky test finder works in four steps each week: it reads the flaky tests, reads the workflows, compares with last week and writes to the tracking issue. All of it comes from the agent's written instructions, which you can read and edit after you install it.
Step 1: Read the flaky tests
CircleCI's report for each project you chose.
Step 2: Read the workflows
Success rate and run count for the last seven days.
Step 3: Compare with last week
From the latest comment on the tracking issue.
Step 4: Update the issue
One ranked comment, or a new issue the first time.
It uses CircleCI's definition
A test is flaky when CircleCI saw it both pass and fail on the same commit. The agent lists only what get_flaky_tests reports, and never calls a test flaky on a hunch.
It keeps one issue
The first run opens an issue titled Flaky tests with your label. Every run after that adds one comment to it, so the history of each test is in one place.
It leaves the issue to you
It never edits the issue's title or body, closes it or assigns it. Whoever keeps the issue can write in it freely.
It never makes a test pass
A rerun that passes hides a flaky test. The agent never reruns, cancels or approves a workflow. The limits are listed under what it will never do.
What a weekly comment looks like
A run of Flaky test finder adds one comment to the tracking issue. It leads with tests new this week, then the ones still flaky, then those gone from the report.
GitHub issue #318, Flaky tests
9 flaky tests across 2 projects, 2 new this week. Below 90% success: gh/acme/web workflow build-and-test, 84%.
New this week (2)
- 1. applies coupon after login in
checkout.spec.ts, gh/acme/web, jobe2e. Flaked 23 times, last on 26 October. - 4. test_large_csv in
test_export.py, gh/acme/api, jobunit. Flaked 4 times, last on 25 October.
Still flaky (7)
- 2. test_retry_backoff in
test_webhooks.py, gh/acme/api, jobunit. Flaked 14 times, up from 11. - 3. shows results while typing in
search.spec.ts, gh/acme/web, jobe2e. Flaked 9 times, down from 12.
Gone from the report (3)
login redirects to dashboard, test_rate_limit_reset, renders empty state. Not seen flaking this week.
- New this week
Flaky tests that were not on last week's list, with their rank, file, project, job, times flaked and when they last did.
- Still flaky
Tests that were on last week's list and still are, with the change in how often they flaked.
- Gone from the report
Tests that were flaky last week and are no longer in CircleCI's report. Fixed, or just quiet: the team can tell which.
When the report is clear
If no project reports a flaky test, it comments that the report is clear, or creates nothing if there is no tracking issue yet.
If a project stores no test results, CircleCI cannot find flaky tests in it, and the comment says the project needs store_test_results rather than calling it clear.
Who uses Flaky test finder
Flaky test finder suits an engineering team that builds on CircleCI and has started rerunning red builds without reading them. The examples use made-up names.
A tech lead planning the week
Nadia Petrova picks the top new flaky test each Monday and gives it to whoever is on maintenance duty.
A developer tired of reruns
Kofi Boateng reruns the API build twice a day. The issue shows him it is always the same webhook test.
A platform engineer watching CI health
Hiro Yamada watches the first line for workflows below 90% success.
An engineering manager tracking progress
Clara Jensen reads the gone from the report group to see the fixes land.
What Flaky test finder connects to
Flaky test finder connects to two tools: it reads CircleCI and writes to one GitHub issue. You connect each one yourself, and Toolspoke stores the connection encrypted, so the agent never sees the credential.
CircleCI
RequiredRead failed pipelines, workflows, jobs, test results and artifacts, and trigger, rerun or cancel CircleCI workflows.
Authentication: Personal API token
GitHub
RequiredManage repositories, issues, pull requests, and Actions.
Authentication: Personal access token
| Tool | What Flaky test finder does | What it cannot do |
|---|---|---|
| CircleCI | Reads the flaky-test report and workflow metrics for each project (get_flaky_tests, get_workflow_metrics). | Delete anything: the connector has no delete action. It can trigger a pipeline, rerun or cancel a workflow and approve a job, and the agent is told never to. |
| GitHub | Finds the tracking issue, reads its comments, and opens it or comments on it (list_issues, list_issue_comments, create_issue, comment_on_issue). | Delete an issue or a comment: the connector has no delete action. It can edit issues, push commits, open pull requests and run workflows, and the agent is told only to open one issue and comment on it. |
A CircleCI personal token has no scopes: it acts as you, with every permission you have. So that the agent never reruns or cancels a build is an instruction to the agent, not a lock. If you want a lock, a project can switch individual actions off. Turn off the CircleCI write and destructive actions and no agent in the project can call them. The access control guide explains how.
The tracking issue lives in the repository you chose. If that repository is public, test names and file paths are public too, so choose a private one if they should not be.
When Flaky test finder runs
Flaky test finder runs every Monday at 9 AM, but only after you switch its schedule on. It arrives switched off, so nothing runs and nothing is charged until you decide.
- Each run reads CircleCI's flaky-test report for the projects you chose.
- It reads workflow success for the last seven days.
- It adds one comment to the tracking issue, or opens it the first time.
- If the report is clear, it says so in one comment.
Each run uses model credit from your workspace balance, so it makes one run a week. The Toolspoke tool calls themselves are free.
Your copy is an ordinary agent in your project. You can pause its schedule there, run it once on demand, change the day and edit its instructions. The agent's page lists recent runs and their cost.
How to set up Flaky test finder
To start finding flaky tests, add Flaky test finder to a Toolspoke project, connect CircleCI and GitHub, choose your projects and switch on the schedule. It takes a few minutes, and nothing runs until the last step.
Step 1: Create a Toolspoke account or sign in
Choose Use this agent. If you are new, you will be asked to sign up first, and you come straight back to this agent afterwards.
Step 2: Add it to a project
Pick the project it should live in. You get your own editable copy, and you can see the two tools it brings before anything connects.
Step 3: Connect CircleCI
Create a personal API token in CircleCI's user settings and paste it. Project tokens do not work with CircleCI's API.
Step 4: Choose your projects
List the CircleCI projects by slug, such as gh/acme/api. Each needs to store test results for flaky tests to show up.
Step 5: Connect GitHub and make the label
Sign in to GitHub, choose the repository for the tracking issue, and create the label it will use, flaky-tests by default.
Step 6: Switch on the schedule
The schedule arrives switched off. Run it once, read the issue it opens, then switch it on.
Make Flaky test finder fit your team
Flaky test finder's instructions are plain English and you can edit them after you install it. Installing copies the agent, so your edits stay with your copy. Open the agent in Toolspoke and tell the builder what to change, and it rewrites the instructions.
Paste any sentence below into the builder chat, or say the same thing in your own words.
Keep the top 10 instead of the top 20.
Name workflows below 95% success instead of 90%.
Only read gh/acme/web, and leave the others out.
Mention @acme/web-team in the comment when a test from gh/acme/web is new this week.
Run every second Monday at 9 AM.
This changes when the agent wakes up, so the builder edits its schedule, not the instructions.
None of these edits needs more access or touches the Never rules. They change what Flaky test finder is told, not what the connections allow.
How it compares to other options
Flaky test finder beats the alternatives when CircleCI already knows which tests are flaky and nobody reads it, and loses to them when you want flaky tests retried or quarantined for you.
| Compared on | Flaky test finder | CircleCI Insights | Automatic retries | A spreadsheet someone keeps |
|---|---|---|---|---|
| Finds flaky tests | From CircleCI's report | Yes | No, hides them | By hand |
| Ranks them across projects | Yes | Per project | No | If kept up |
| Shows what is new and what has gone | Every week | No | No | Sometimes |
| Lives where the team tracks work | A GitHub issue | In CircleCI | No | A separate file |
| Keeps builds green | No, it reports | No | Yes, by hiding failures | No |
| Cost | A run a week | Included | Extra CI minutes | Someone's time |
Automatic retries are the better choice when you need a green build today. Flaky test finder is for the week after, when you want the list of tests to fix rather than another rerun.
CircleCI Insights holds the same report, and the agent reads it. The difference is that it ranks across projects, says what changed since last week, and puts it on an issue your team already watches.
Frequently asked questions
Can Flaky test finder rerun or cancel my builds?
The CircleCI connection can, because a personal token has no scopes. The agent is told never to. To make it a lock, switch the CircleCI write and destructive actions off in your project.
How does it decide a test is flaky?
It uses CircleCI's own report: a test that passed and failed on the same commit. It never adds a test the report does not list.
Why is a project missing from the list?
CircleCI can only find flaky tests in a project that stores test results with store_test_results. The comment says which projects report nothing for that reason.
Will it open a new issue every week?
No. It opens one tracking issue the first time and adds a comment each week after that. If more than one is open with the label, it uses the oldest and says so.
Does it work with GitHub Actions?
Not as shipped, because it reads CircleCI's flaky-test report. You can ask the agent builder for another version. It needs its own instructions.
Is Flaky test finder free?
It has no price of its own, and every Toolspoke plan includes agents. Tool calls cost nothing, but each run draws on your usage balance. Pricing has the details.
Fix the tests you keep rerunning
Connect CircleCI and GitHub, choose your projects and switch on the schedule. Your first ranked list lands on Monday.