20,118 Tests and 3 Red Team Agents: Why AI-Built Software Doesn't Have to Be Fragile

JobIntel had 20,118 automated tests when release 3.17.2 shipped on October 16, 2026. It launched in March with 4,869. That release carries 1.68 lines of test code for every line of production code.

Every engineer who hears that JobIntel is built by one person and a team of AI agents asks the same question first. Yeah, but does it actually work?

Fair question. The default assumption about AI-generated code is that it is brittle: it compiles, it passes the tests it wrote for itself, and then it breaks on the first input nobody anticipated. As a default, that assumption is not wrong. Most AI-generated code ships without the discipline that makes software reliable. But the model is not the problem. The process is.

Here is the process, with the numbers from release 3.17.2.

The suite, layer by layer

Layer	Tool	At release 3.17.2
Backend unit and integration tests	pytest	12,642
Frontend component tests	Jest	5,364, in 483 files
End-to-end tests in a real browser, at three screen sizes	Playwright	2,112, in 310 spec files
Automated tests in total		20,118
Behavior scenarios in plain Given/When/Then language	pytest-bdd	681, in 79 feature files
Property-based invariants	Hypothesis	hundreds of generated inputs per rule
API fuzzing against the OpenAPI contract	Schemathesis	every night
Performance and stress	k6, Locust	p95 latency checks on every deploy, stress runs every night

The counts are test definitions in the source at the release tag: pytest functions, Jest and Playwright test blocks. The behavior scenarios are counted on their own.

Each layer catches something the others cannot.

The behavior scenarios are the requirements, written so a non-engineer can read them and a machine can run them. "Registration fails with duplicate email" is a scenario. So is "Survey responses have no user identifier." When a requirement lives in a wiki, it drifts. When it runs on every build, it either holds or the build goes red.

The property-based tests do not check examples. They state rules and let the framework hunt for inputs that break them. A match score always lands between 0 and 100. Adding a matched skill never lowers it. Normalizing a company name twice gives the same answer as normalizing it once. Hypothesis generates hundreds of cases for each rule, and the cases it finds are the ones I would never have written by hand.

The browser tests run at three screen widths, 375, 768 and 1280 pixels, because a layout that works on my monitor proves nothing about a phone.

The fuzzer reads the API contract and sends the requests nobody writes on purpose: boundary values, malformed bodies, wrong types. Any response that breaks the contract is a finding.

A test nobody has seen fail is not a test

Test count is the easiest number in software to inflate, and an AI agent will inflate it if nothing stops it.

Consider a check that asserts a search response is a list. It passes when the search returns the right jobs. It also passes when the search returns nothing at all, because an empty list is still a list. Coverage goes up, the suite stays green, and the test asserts nothing a user would ever notice.

So two rules sit on top of the suite.

The failing test comes first, and the git history proves it. For features and bug fixes, the test that fails has to be committed before the code that makes it pass. The sprint gate reads the commit order. If the implementation landed first, the sprint fails. Styling, accessibility and copy-only changes are the exception.

Every new guard has to be seen red. When a behavior scenario is added or changed, the agent breaks the production code it protects, on purpose, and records whether the scenario caught it. A gate nobody has seen fail is a report, not a gate.

Simple truth is, a test count tells you how much was written. Only a test you have watched fail tells you it works.

Four tiers, sized by what they cost

Not every check can run on every change. Run everything every time and you get a merge gate so slow that people learn to work around it. So the checks are tiered by what they cost.

1. Before every push, about 7 minutes. A local preflight, plus the same end-to-end suite the pull request gate will run.
2. Before every merge, about 15 minutes, blocking. The sprint completion protocol: the tests, security scanning with bandit and semgrep, and the test-first check.
3. Every night, for hours. The expensive work: API fuzzing, k6 and Locust stress runs, container scanning with Trivy, an accessibility audit, and the Red Team. Every failure files its own Jira ticket, and my morning starts with that queue.
4. At every phase boundary, about 30 minutes. A rollup audit that reads the nightly record instead of re-running it.

Above all four sits a rule no gate replaces. Changes move through three environments, test, preview and production, and a production deploy needs my explicit approval, every time.

The Red Team, and what it costs

On day 10 of the original 24-day build, I added three adversarial agents: a Security Sentinel, a Regulatory Auditor and a Data Integrity Adversary. They are read-only. Their job is to attack what the other agents built: injection, authentication and authorization bypass, exposed secrets, server-side request forgery, data that can drift into an invalid state, and compliance gaps.

At launch they ran on every pull request. They don't anymore, and the reason matters more than the original design.

Adversarial review by AI agents is not free. A full Red Team pass today runs six reviews, covering security, infrastructure, regulatory, requirements compliance, data integrity and performance, plus a seventh job that weighs the findings. It costs about $13.54 in API spend. In April, running the review on every change was most of an AI bill on course for $59,500 a year, with essentially zero paying customers.

So the Red Team now runs once a night against the day's work, and skips the night when nothing changed. A run outside that schedule needs a stated reason.

Adversarial review has a price, which makes its cadence a budget decision. Per pull request made sense while everything was new. Nightly makes sense for a system that ships roughly one release a week.

The symmetry is still the point. AI builds the code, and AI attacks it. If you trust a model enough to write your software, you should trust it enough to try to break it. Just decide on purpose how often you can afford to.

What the tests do not tell you

AI-generated code is not inherently fragile. It is not inherently robust either. It reflects the process around it.

Twenty thousand tests did not arrive with the right assertions. Someone had to decide which invariants matter, which boundaries to probe, which journeys a real user takes, and which green checks were telling the truth. The Red Team did not choose its own threat model or its own budget. The test-first gate did not impose itself. Each of those was a decision, made by someone who spent 25 years in enterprise programs watching what happens when a team skips the review because the dashboard is green.

In March I wrote that building JobIntel with AI worked because of the enterprise discipline around it. Seven months and four times the tests later, that has not changed. The number was never the point. The habits behind it were.

The tools do not determine the outcome. The discipline does.

---

The Trust, Match and Ready scores JobIntel puts on every job are built and tested this way. See every job. Know if you fit, at jobintel.com.