Why the designs stopped looking like designs
A four-week investigation into a system that passed every test and produced plans nobody would plant.
- Type
- Investigation · Recovery
- Period
- September 2026
- Status
- Recovery complete and frozen at a known commit; five approved reference plans captured.
The test suite was green. The engine was deterministic. Every guard passed. And the plans it produced were flat, monotonous, and frequently a single species spread over half the bed. Nothing had regressed in any way the tests could see, because the tests were measuring the wrong thing.
| 4 | independent root causes in one symptom | |
| 3 | separate gates that declined every plan | each individually correct on every rule |
| 0 | of 152 plants could satisfy one goal | tests green throughout |
The challenge
Design quality is not a boolean, and it does not fail loudly. There is no exception, no red test, no error in a log. There is a plan that is technically valid and that a gardener would look at and quietly decline.
Worse, the engine had grown seven modules that judge a plan and comparatively few that design one. It had become very good at telling me the plan was bad.
What I found
- A column read as the wrong type
- Pollinator value is stored 0–5. One filter compared it to the string “high”. Zero of 152 plants could ever satisfy the pollinator goal, and the test suite was green the entire time.
- Guesses scored as facts
- Where a plant had no recorded design role, one was invented — and then scored as a known poor fit. Invented data was being penalised as though it were measured data. This was the root cause of the one-species plans.
- One checkbox capping every plant
- A single option capped the mature height of every plant in every bed at three feet. It explained every flat plan in the sample.
- A structural backbone chosen on the wrong criterion
- A groundcover kept winning the structural role because spread-fit outweighed role-fit in the score, and one catalog phrase made it eligible.
- Recurrence was four bugs, not one
- The simplest thing a designer does — repeat a plant down a border — was blocked by an adjacency rule, a dead target field, weak repetition scoring, and a reuse cap, all at once.
- Part sun blocked every perennial
- Eight of nine flowering perennials were ineligible on a part-sun bed for missing tolerance data — and they outscored the plant that won.
My approach
I stopped trusting the test suite as evidence of quality and started generating real plans against the real catalog, looking at them, and asking a different question of each one: not is this rule correct but how often does this rule fire, and on what.
That reframing found the pattern that runs through the whole recovery. Three separate times, a gate was individually correct on every rule it contained and declined every plan I gave it, including the known-good control. A rule being right and a product working are different questions, and only one of them has a test.
Key decisions
- Keep a known-good control
- One synthetic fixture, unchanged for months, run through every gate alongside the real beds. If the control declines, the gate is wrong, whatever the rules say.
- Measure incidence, bluntly
- Alongside every quality rule, a count: how many of the last N plans did this decline? A rule that declines 100% is not strict, it is broken.
- Measure placed geometry, not metadata
- A composition contract that reads species lists can be satisfied by a plan that looks nothing like a design. Dominance has to be measured on coordinates — on plants, not species.
- Freeze and capture
- When the recovery reached plans I would accept, I froze it at a named commit and captured five complete records — intake, pattern spec, inventory provenance, assignments, quantities and every coordinate. References, not thresholds. Nothing asserts against them.
The lesson
Every one of those four root causes was a small, locally reasonable decision. None was a bad line of code. The failure was that nothing in the system was measuring the thing the product was actually for, so six correct components composed into an output no one would buy.
The written standard now sits at the top of the reference folder: tests passing is not success. The engine is held to whether a person looks at the planting and accepts it.