Technology14 min read

Testing AI-Generated Code: Lint the Pattern, Test the Decision

By · Published by Everything Blog

In short

### Testing AI-Generated Code: Lint the Pattern, Test the Decision Testing AI-generated code often involves writing tests that cover the patterns and patterns' implementations. This ensures that every file adheres to the pattern, and tests focus on the files that make decisions, rather than the repetitive implementation checks.

Key points

  • ### Testing AI-Generated Code: Lint the Pattern, Test the Decision Testing AI-generated…: ### Testing AI-Generated Code: Lint the Pattern, Test the Decision Testing AI-generated code often involves writing tests that cover the patterns and patterns' implementations.
  • This ensures that every file adheres to the pattern, and tests focus on the files that ma…: This ensures that every file adheres to the pattern, and tests focus on the files that make decisions, rather than the repetitive implementation checks.

Sources checked by our analysis pipeline: 1

Coding agents love to write tests. Ask one for a new handler, a form or a combobox, and it comes back with a test file next to it, unasked. In a large codebase where agents write most of the code from documented patterns, that sounds like the testing problem solves itself.

It does not, because those tests are not streamlined. Every agent-written test brings its own mock setup, copied from whatever test file it saw last and changed a little. Many of them assert things a linter should catch: that the session was checked, that the error was mapped, that a translation key is the expected string. And most of them test files that do nothing but follow the pattern, so they restate the implementation and never find a bug.

As a freelance team lead software engineer, I helped my team streamline this. The goal was never a fully tested codebase. The goal was coverage that follows the structure of the code: linters hold every file to its pattern, one shared harness per kind of file makes tests cheap, and a test sits on top only where a file goes beyond its pattern. This post is about unit and integration tests. End-to-end tests are a topic of their own and not covered here.

Here is what that changes for a single test file:

The streamlined test is a fraction of the size, and every line it has left asserts a decision the file makes. The rest did not disappear. It moved to places where it covers every file instead of one.

Most Files Follow a Pattern

An agentic codebase runs on patterns. Every kind of file has a pattern document that says what it looks like, and both humans and agents follow it. A write handler, for example, has the same shape in every file:

├─ await requireSession()

│

└─ try {

├─ const client = createServiceClient(Projects)

└─ await client.archiveProject({ id })

} catch (error) {

return toFailure(error)

}

return toSuccess("project.archived")

I call that shape the skeleton. Most handlers are nothing but the skeleton: check the session, call one backend method, map the error, return a result. The same is true for forms, comboboxes, table columns and endpoints: each kind has its skeleton, and most files of that kind are nothing more.

A test for a pure skeleton file restates the implementation line by line. It breaks when the code changes, and it never catches a bug, because there is no decision in the file that could be wrong. But the skeleton itself can break, and with agents it breaks in a particular way. An agent rarely forgets the whole pattern. It drops one line of it: the session check in a handler that “is only called from a signed-in page”, or a backend call that slips out of the try

during a refactor. A test only notices that in the one file it tests.

So instead of one test per file, every kind of file went through the same three steps:

- Lint the skeletonA script enforces the invariants the pattern document states.Covers: every file of the kind, including the ones agents have not written yet

- Build one shared harnessFake what the kind of file reaches outside its own code, once.Covers: the setup every test would otherwise copy, so a test file holds only its scenario

- Write the rule, then test the riskiestThe pattern document says which files earn a test and what it asserts.Covers: the files that decide something the skeleton does not

Write handlers went first, and they are the running example. Forms, comboboxes and the other kinds followed the same way, and they come at the end.

Step 1: Lint the Skeleton

The first step covers every file of a kind, including the ones no agent has written yet, without a single test. A small script parses every exported handler with the TypeScript compiler API and checks the invariants the pattern document states:

- Session firstThe first statement is await requireSession(). An exported handler is a public endpoint, whoever calls it.

- Service call in a tryEvery createServiceClient() sits inside a try that has a catch, so a backend failure reaches it.

- The catch maps the errorEvery catch returns the result of toFailure(), or rethrows into an outer try.

- Input through the helperThe submitted input only travels back through toSuccess() or toFailure(), which strip secret fields.

None of these rules is clever. Each one is a sentence from the pattern document turned into a check. The pattern document is what the agents already follow when they write a file, and what our review agents already match a change against. The lint makes the same patterns hold for every file, instead of hoping that every agent read them and every reviewer remembers them.

// Command Line

npm run lint:handlers

projects/archive-project.ts:7

`archiveProject` [session first] does not start with `await requireSession()`

invoices/list-invoices.ts:12

`listInvoices` [call in try] calls `createServiceClient(...)` outside a `try` block

For an agent, a lint like this is the best kind of feedback there is. The message names the file, the line and the rule, so the agent fixes it in the same session without a human pointing at it. A comment in a code review arrives a day later. A lint error arrives before the agent says it is done.

Getting a lint like this into a codebase that agents have been writing for months took four steps, and the same four repeated for every kind of file:

- Survey the candidate rulesAn agent runs each rule over every file and reports the hits. The hits decide which rules make it.keepSession first, 3 hitskeepService call in a try, 1 hitkeepNo stand-in data next to an error, 1 hitdropNever wrap a mapped error again, 0 hits, the type checker already enforces it

- Fix every violation firstIn their own commit, before the lint lands, so the lint starts green.archive-project.tslist-invoices.tsget-owner.ts

- Land the lint, with almost no exemptionsA real violation gets fixed, never exempted. The single exemption carries its reason next to it.exempt: 1, with reason

- Delete the tests the lint made redundantAssertions the lint now covers for every file go, and so do test files with nothing else to say.expect(requireSession).toHaveBeenCalled()expect(result.status).toBe("ERROR")

The survey is the step I would not skip. An agent ran every candidate rule over every file before a line of the lint existed, and the hits decided which rules made it: one rule was dropped because the type checker already made its violation impossible, and the others found real bugs, like a read handler that returned an empty list next to an error, so the UI could not tell “failed” from “empty”. The last step is where the streamlining starts: every test assertion the lint now covers for all files goes away.

Step 2: Build One Shared Harness

The second step makes the remaining tests cheap and uniform. A handler reaches outside its own code all the time: the backend client, the session, translations, redirects, cache invalidation, cookies. If every test file fakes those on its own, every test file starts with forty lines of mocks, and every one of them fakes the backend slightly differently. Agents are particularly good at copying forty lines of mocks, which makes the problem worse with every file they write.

So there is one harness per kind of file. The test runner picks it by file name: every *-handler.test.ts

gets a setup file that fakes everything a handler touches, and the test file holds only its scenario.

The interesting part is the backend. The easy fake is a mock function per client method that returns whatever the test wants. Our harness is stricter: it keeps the real generated client and replaces only the network with an in-memory transport. Every request is serialized and deserialized again, exactly like on its way to the real backend.

The fake backend has three rules, and each one exists because a looser fake would let a bug through:

- A method the test did not serve answers “unimplemented”. The handler under test sees a real backend error, not

undefined

. - A fake handler that crashes fails the test. Otherwise a bug in the fake would pass for a backend failure, and the test would assert the wrong thing with a straight face.

- Requests are decoded with the same limits as the production server. Not with the client library’s defaults.

That last one was not on anyone’s list. When we moved the existing tests onto the real transport, a deeply nested request that production accepts every day was rejected by the fake. The client library and the server disagreed about how deep a message may nest. The old mocks had never serialized anything, so they had hidden that mismatch for as long as the tests existed.

Step 3: Write the Rule, Then Test the Riskiest

Only then came the tests themselves. Before writing any, the pattern document got a section that says which files of this kind earn a test and what that test asserts. This section is what keeps agents from going back to one test per file. Without it, “add tests for the handlers” produces a test for every skeleton handler, all of them green, none of them useful.

- checks the session, calls one method, returns its result

- create vs. update in one handler

- redirects after success

- a read that only guards the write

- a branch over a kind, a scope or a state

- a write built from a read, and a failed read that must stop it

- a partial update: which fields reach the write

- more than one write: their order, a partial failure

- a security decision, like deciding a value on the server

- an error it remaps or swallows, a payload it reshapes

- the requests sent, and none after a failure

- the returned result

- side effects: redirects, invalidated caches, cookies

- that the session was checked

- a translation key for its own sake

A handler earns a test when it decides something the skeleton does not: a branch over a kind or a state, a write built from a read, more than one write, a security decision, or an error it remaps. Everything else is skeleton, and the lint already covers it.

The rule also says what a test asserts: what leaves the handler. That is the requests it sent (and the ones it must not send after a failure), the result it returns, and its side effects. It does not assert that the session was checked, because the lint does that. Here is a test for a handler that reads a project and writes its members back:

beforeEach(() => {

fake.serve(Projects, {

getProject: () => ({ id: "p1", members: ["ada"] }),

updateProject: () => ({}),

});

});

it("adds the new members in front of the existing ones", async () => {

const result = await addMembers("p1", ["grace", "linus"]);

expect(fake.calls(Projects, "updateProject")).toMatchObject([

{ project: { id: "p1", members: ["grace", "linus", "ada"] } },

]);

expect(result).toMatchObject({ status: "SUCCESS" });

});

it("writes nothing when the project cannot be read", async () => {

fake.serve(Projects, {

getProject: () => {

throw new ServiceError("unavailable");

},

});

const result = await addMembers("p1", ["grace"]);

expect(result).toMatchObject({ status: "ERROR" });

expect(fake.calls(Projects, "updateProject")).toEqual([]);

expect(effects.invalidated).toEqual([]);

});

The beforeEach

serves the happy path for every method the handler calls. Each test overrides only the one method it is about, runs the handler, and checks what went out. There is no mock setup in the file at all, because the harness owns it. That is also why agents write these tests well: the file is all scenario, and the rule tells them which scenarios matter.

These tests paid for themselves right away. Two of them turned up permission bugs. One handler sent a grant with an empty scope when the role did not allow the requested one. Another took a value from the form that the server should have decided on its own. Neither the lint nor the type checker could have seen either of them, because both were wrong decisions, not wrong shapes.

The Same Steps for Forms and Comboboxes

Handlers are the obvious candidates, because they talk to a backend. But the method works for every kind of file with a pattern, and in a frontend codebase most of them are UI.

A form has a skeleton as strict as a handler’s: it is built on the shared form component, it validates with a schema, it submits through a handler, and every field has a label and a slot for its error. The lint checks exactly that. Its harness renders the form with a fake handler and the real schema. A form earns a test only when it decides something: a field that appears depending on another one, or input that is reshaped before it is submitted.

A combobox that loads its options is the same story. It is built on the shared async combobox and gets its options from a read handler, never by fetching on its own. Its harness renders it on the same fake backend the handlers use, so a combobox test serves the read handler’s backend method like any handler test would:

it("only offers the members of the selected team", async () => {

fake.serve(Members, {

listMembers: ({ teamId }) => ({

members: teamId === "t1" ? [{ id: "ada", name: "Ada" }] : [],

}),

});

const view = renderWithHarness(<TeamMemberCombobox teamId="t1" />);

await view.open();

expect(view.options()).toEqual(["Ada"]);

expect(fake.calls(Members, "listMembers")).toMatchObject([

{ teamId: "t1" },

]);

});

A combobox that only lists what a read handler returns gets no test at all. This one filters by another field, so it earns one. The same reasoning works for table columns, endpoints and every other kind with a pattern document:

Not every kind needs all three steps. Table columns need no harness, because their lint already checks the one thing that tends to break. Endpoints are one-line calls into a few shared implementations, so the shared implementations got the tests and the thin files did not. And plain utility functions have no skeleton at all: they are tested directly, as they always were.

Streamlined, Not Fully Tested

After the first round per kind, tests stopped being a campaign. Whoever touches a file that matches its kind’s rule adds its test, and since the rule lives in the pattern document, agents pick it up the same way they pick up the skeleton. That turns the three steps into a loop the agents run on their own:

- The pattern documentinstructionThe skeleton, its variants, and which files earn a test.

- The agent writes or changes a fileIt follows the skeleton, and adds a test when the file decides something.

- The lint runsfeedbackFile, line and rule in the message, so the agent fixes it in the same session.

- The tests run on the shared harnessfeedbackA wrong decision fails a behavior test before anyone reviews the change.

- A human reviews lastIs this the right change, was it asked for? The pattern matching is already done.

The pattern document is the agent’s instruction. The lint and the tests are the agent’s feedback. A human reviews last and asks the questions an agent answers confidently and badly: is this the right change, and was it asked for? The pattern matching is already done.

Agents will write tests whether you ask them to or not. The question is whether those tests line up with how the codebase is built. A lint covers every file of a kind in one go, a shared harness turns each test into a few lines of scenario, and a written rule puts a test on top only where a file leaves its pattern. The result is not a fully tested codebase, and it does not need to be. It is a codebase where every file is held to its pattern, and every test that exists has a reason to.

Original source: www.robinwieruch.de

Technology