Tiny Refactor a Day: small AI Experiment

Every workday, before any of us has finished our morning coffee, an AI agent opens a merge request against some of our repositories. It picks a small piece of code that looks like it could be a little clearer - a duplicated block, a method that does two things - and proposes a tidy-up.

This post is about what happens when you put the Boy Scout Rule on a cron schedule, and what we've learned from a few weeks of letting an AI agent propose small refactorings every workday. So far, 40 of them have landed in production.

Background - why we can even try this

Letting an AI agent draft refactorings against a production codebase in a daily schedule sounds reckless until you look at what sits between the agent's branch and main branch. The experiment is built on top of safeguards we put in place long before LLMs were writing pull requests:

• ~93% test coverage across the codebase, catching subtle behavioral changes before they ship.
• Checkstyle enforcing formatting, basic style rules and guarding low code complexity rules.
• ArchUnit tests pinning architectural boundaries.
• SonarQube as a quality gate on every merge request.
• Since 2025, an automated AI review can also block a merge request.
• Then 100% of code is reviewed by a human before it lands in main branch, no exceptions. The agent's MRs go through the same review as a junior engineer's, a senior engineer's, or anyone else's. The difference is the size - these reviews take 2 minutes.

The first naive scheduled prompt

We used Devin Schedules for this, though Claude Code and a cron job would do the same.

The first version of the schedule was an amateur first stab. The whole prompt was this:

Create a small improvement according to DRY or Clean Code for given service under given ticket. If service and ticket are not provided, stop and explain failure.

Procedure:
1. Checkout the given service repository.
2. Create a small merge request under the given ticket with a focus on DRY principle or Clean Code. Find the best candidate for the merge request. The change should be small enough to be easy to review and merge without testing.
3. Notify me on Slack about the merge request.

The prompt was too simplistic.

The current scheduled prompt

The current iteration is less a prompt and more a small playbook. The shape itself matters - instead of a three-step procedure, it has parameters, constants, and a procedure that walks the agent through everything from picking a candidate to dealing with review comments:

# DRY / Clean Code Improvement Playbook

Create a small, focused code quality improvement following the DRY principle or Clean Code practices.

## Parameters
- Linear team (required) — the Linear team to create the issue in (e.g. "OPEX: OneOps")
- Linear labels (optional) — labels to apply to the Linear issue (e.g. inbound, tech debt)
- Repository (optional) — defaults to the repository configured in the scheduled job's context
- Assignee (optional) — defaults to the author of the scheduled job (for MR assignment, Linear assignment, and Slack notification)

If Linear team is not provided, stop and explain failure.

## Constants
- MR label: "continuous-improvement" — applied to every MR this playbook creates, used by the backlog guard

## Procedure
1. Checkout the repository.
2. Read AGENTS.md and understand project conventions.
3. Backlog guard: list open MRs in this repository with the "continuous-improvement" label. If 3 or more are still open/unmerged, skip the run and notify the assignee on Slack with the list instead of creating a new one.
4. Pick a candidate that is:
   - Small enough to review and merge without testing
   - Focused on DRY or Clean Code (extracting duplication, simplifying boilerplate, consolidating repeated patterns)
   - Not a duplicate: Review the last 10 MRs with the "continuous-improvement" label (open and merged) and try to pick a different area of the codebase, a different topic, and a different type of change.
5. Create a Linear issue for the specified team with the specified labels. Set state to "In Progress". Write a short description summarizing what will be refactored and why; once the MR exists, append its link to the description.
6. Create a feature branch prefixed with the new Linear issue code (e.g. OOT-4552/extract-exception-handling).
7. Implement the improvement and commit following the repository's commit message conventions.
8. Create a small draft MR. Add [no-canary] and the Linear ticket ID to the title. Apply the "continuous-improvement" label. Assign and set reviewer to the assignee. Link the MR to the Linear issue.
9. Wait for CI. If CI fails, attempt to fix it once. If it still fails, leave the MR as-is and flag it as "needs attention" in the Slack notification.
10. After CI passes, check the MR for review comments (from the Sonarqube or AI agent). Address each comment with a follow-up commit. If a comment is not actionable or you disagree, reply explaining why. Re-wait for CI after pushing fixes.
11. link the Devin session to the created linear ticket

The backlog guard caps the work-in-progress. If there are already three open continuous-improvement MRs sitting in review, the agent doesn't open a fourth - it sends a Slack ping with the list instead.

The diversity rule - if left to itself, AI will happily refactor the same file three days in a row, finding ever-smaller things to extract. Asking it to review the last ten continuous-improvement MRs and deliberately pick a different area, topic and type of change keeps the proposals spread across the codebase. It's not perfect - the agent still has favorites - but it's a noticeable improvement.

It's still a moving target. We are thinking about adding more human intent to the prompts, e.g. concrete god classes which should be split and so on. We rewrite this playbook every couple of weeks as we notice new failure modes.

What it actually does

After dozens of merge requests, the proposals fall into a handful of recognizable shapes. Roughly in order of how often they show up:

• Extracting duplicated error handling - near-identical try/catch blocks pulled into a shared handler. The most common by far.
• Extracting shared logic - repeated iteration patterns lifted into one place.
• Consolidating boilerplate - enum converters, repeated lookups, validation mapping folded into a single utility.
• Standardizing dependency injection - field injection migrated to constructor injection / @RequiredArgsConstructor.
• Replacing a custom utility with the standard library - our favourite: a hand-rolled Filters/Predicate pair that predated Java 8, deleted in favour of the Stream API.
• Modernizing idioms - @RequestMapping to @GetMapping, Collectors.toList() to Stream.toList().
• Removing dead code - unreachable catch blocks, branches that can't be hit.

There's a pattern worth naming here. The first four types are structural - they move code around without changing what it does, and they're safe almost by construction. The last three touch behaviour, even when they look purely cosmetic - and that's where the danger lives.

Example: a real merge request

Here's one that landed last week. It is, in every sense, an unspectacular MR.

The story is short. Four implementations of our partner order service - for four different partners - each contained an identical guard when cancelling an order that hadn't been sent yet. All four built the same CancelOrderResponse with result=true, the order ID and the same message string. AI spotted the duplication, extracted it into a single static factory method on the response type, and updated the four call sites.

Five files changed, +14 / −20 lines, no behavioural change. The agent opened the MR, the CI suite ran, a reviewer on the team read the diff, agreed it was a real improvement, and merged it.

The actual diff: four call sites collapse to CancelOrderResponse.notSentYet(orderId), with the shared construction moved into one static factory on the response type.

It's small enough to review in under a minute - and a human still made the call, deciding the consolidation was correct and the message preserved before merging. That's the point.

Not every MR looks like this. Some are weaker, some are wrong. But this is the median proposal per repository on a good day.

The merge ratio

So far the schedule has opened roughly 50 merge requests, and we've merged about 40 of them.

Why the rejected ~20%? The ten or so MRs we've closed are where most of the learning has come from. Some were closed because the "duplication" wasn't really duplication - two call sites that looked identical answered to different concepts. Some were closed because the refactor made the code technically DRY-er but harder to read. Every closed MR is a hint at something the next version of the prompt needs to handle.

What we're not measuring (yet). A merged MR isn't proof that the change made the codebase better. It's proof that one reviewer, on one morning, decided it didn't make things worse. We don't have a good metric for the long-term effect on readability, defect rate or onboarding time, but we suspect that when a codebase is readable to humans, it also makes LLMs more effective (with fewer hallucinations).

Where the idea comes from

The schedule isn't an original idea. It's a recombination of two older intuitions about how engineers should treat code, and one piece of tooling that has been quietly running the same playbook for dependencies for years.

The Boy Scout Rule. Robert C. Martin's one-liner - leave the code a little cleaner than you found it - is the oldest of the three. It says nothing about what to clean: a rename, a deleted dead branch, a shorter method. The bet is that small, opportunistic improvements compound, and that the marginal cost of one tidy-up, paid while the code is fresh in your head, is much smaller than the cost of letting the rot accumulate.

Tidy First? Kent Beck's more recent take is the same bet under a stricter discipline: treat the structural tidy-ups as their own commits, separate from behavioral changes, and decide first whether to do them. The valuable contribution isn't "always tidy" - it's the explicit decision that tidies are a different kind of change. They're cheap to review, cheap to revert, and they don't belong tangled up with the feature work that ships next to them.

Renovate. And there's already an industry-standard tool that runs essentially this loop, just in a narrower domain. Renovate watches for upstream dependency releases, opens a small MR for each one, waits for CI, and pings a human to merge. We've used it for years. It works. It has the same overall shape - bot proposes, CI verifies, human approves, MR merges - and the same overall risk profile too.

Why we still review every MR

An 80% merge rate is high enough that someone reading this might reasonably ask: do you actually still need to review these? Couldn't you just merge the green ones and spot-check?

No. And we have a production incident that makes the case better than any argument could.

One of the schedule's MRs replaced collect(Collectors.toList()) with the shorter Stream.toList() across a helper class - a textbook "modernize the idiom" change. It looked completely safe. CI was green. It merged. About an hour after it deployed, one of the screens our warehouse workers use during goods receiving started returning HTTP 500 on a routine action, across every warehouse. It stayed broken until we shipped a fix - a few hundred failed requests later.

The catch: Stream.toList() returns an immutable list, where the old Collectors.toList() returned a mutable one. Downstream code that modified the returned list now threw an exception instead of quietly mutating it. No test covered that exact path, so nothing went red - and a reviewer wouldn't necessarily catch it either, unless they knew to ask the one question that mattered: is anything modifying the list that comes back?

Here's the part that actually taught us something. This wasn't the first time the agent proposed this exact refactor. The first time, we explicitly asked it to verify that none of the returned lists were being modified by their callers. It checked, reported back that they were safe, and the change went in without incident. The second time, the diff looked the same, we'd seen it before, and we trusted it without asking for that check. Same transformation, same agent - the only variable that changed was whether a human decided to verify the risky part.

That's the whole thesis of this post in one bug. The safeguard isn't green CI or the agent's own confidence. It's a human deciding what's worth verifying. The agent is very good at the mechanical edit and genuinely bad at knowing which mechanical edits are dangerous.

Sometimes the failure is less dramatic - just bad rather than broken. A method extracted in a way that obscures what was already clear. A constant pulled into a shared file where it doesn't belong. A "duplication" eliminated by creating a coupling between two parts of the codebase that had no business knowing about each other. The tests pass, but a human reads the diff and says "no, this is worse," and closes it.

Even the people who build these agents review everything. Cognition - the company behind Devin and Windsurf - told us in a workshop session that their own engineers spend most of their day reviewing AI-generated PRs. That's the team with the deepest possible context on what the agent can and can't do, and they still don't let it merge unsupervised. If they don't, neither should we.

Maintenance, not magic

Most of the excitement about coding agents right now is about generation - scaffolding a new app, shipping a feature from a prompt, going zero to demo overnight. This experiment is the unglamorous opposite: an agent pointed at a mature codebase with one job, make what's already here a little better. Maintenance, not creation. It's a smaller promise, but it's one the technology can actually keep today - the work is bounded, reversible, and easy to verify, which is exactly what makes it safe to automate.

And the role it leaves for the human is the interesting part. The agent does the typing; the human does the deciding - which cleanups are worth making, and which "safe" change is quietly the dangerous one. As more of the mechanical work gets automated, that judgement is the part that doesn't, and the daily review is where it lives.

Isn't this just creating more noise in our already-busy lives then? It is, but it takes a minute in the morning, so some of us are actually finding it a fun thing to do.

---

- Maroš Vranec & Filip Velemínský