Why do my tests pass but the feature does not work?

A unit test proves that a function, when called, does what you expected. It does not prove that anything calls it. If your test imports the domain layer and calls the function directly, it will stay green even when no path exists from the user interface to that function. Detect it by listing every public method in the domain layer and searching the interface layer for each name. Any method with zero callers is either dead or unreachable.

How do I find unreachable features in a codebase?

Run a caller sweep. Extract every public mutating method name from your domain or simulation layer, then grep the user interface layer for each name. Methods with no match are candidates. On a real project this returned 46 public mutators, 21 with no caller anywhere, and 5 that were meant to be user actions. This finds a class of bug that no assertion was looking for.

Does AI generated code fail differently from human code?

In the author's experience across two shipped projects, the difference is not quality but reachability. An assistant is effective at producing a function that satisfies a description, and has no view of whether a path exists from a user action to that function, because that path lives in a different file and often a different conversation. The result is code that runs, returns, and satisfies every assertion, while the user gets nothing.

Why is a bare except dangerous in Python?

A bare except catches every exception, including NameError and AttributeError from code that never worked, plus SystemExit and KeyboardInterrupt, which makes clean shutdown unreliable. In one documented case a bare except swallowed a NameError on every call for four months with no crash, no log and no warning. The rule of thumb that fixed it: swallow reads, never writes. A failed read can fall back to a default. A failed write must be loud, because the user believes it happened.

Why does my code work on my machine but crash on other Python versions?

Annotation evaluation changed. PEP 649, final in Python 3.14, defers evaluation of annotations. A module that uses a name in an annotation without importing it will run without error on 3.14 and raise on import on earlier versions. If you develop on a newer runtime than you support, your own environment hides the bug from you. Run the test suite on the oldest version you claim to support.

Why do two parts of my program report different values for the same thing?

Because the quantity is computed in two places and nothing compares the results. Each calculation is internally correct and each has a passing test, so the disagreement has no owner. In one documented case a projection screen quoted eleven days for an operation that finished in one, and both numbers came from the same codebase. If the same quantity is computed in two places, that is not duplication, it is a future disagreement with a date on it. The fix is one function that both callers use, plus a test that fails the build when a second copy appears.

Should validation live in the user interface or in the domain layer?

In the domain layer. A limit enforced only in the interface stops existing the moment a second entry point to that action appears: another screen, a loaded save file, an edited config file, or an API call. The interface should display the rule, not be the rule. Verify it with a test that bypasses the interface entirely, constructing the illegal state directly against the domain layer and asserting that it refuses. Related trap: validation that runs after the value has already been read is not validation.

How can a developer working alone catch bugs that a code reviewer would find?

Replace the reviewer with something mechanical, because these failures are invisible from the inside and a solo developer only asks the questions they already know to ask. Automated sweeps do not get tired, do not share your assumptions and do not skip the file they read last week. Run them as one pre-release script that prints a report rather than blocking the build, and reserve hard build failures for a narrow set of ratchets such as bare excepts, dead imports and duplicated single-source helpers. Real testers remain the strongest complement: three of them found six stability issues and seventeen non-responding features in one evening on a release that was considered finished.

The Silent Failure Catalogue: Code That Passes Every Test and Does Nothing

By Marcin Firmuga·2026-08-21·Living document

A crash is honest. It interrupts you, it prints a stack trace, it tells you where to look. You fix it because it insists.

This catalogue is about the other kind. Code that runs. Code that returns. Code that satisfies every assertion you thought to write, and hands the person using your software absolutely nothing.

I have shipped this class of bug at least twenty-four times across two projects: a Windows system monitor that is on the Microsoft Store, and a strategy game in development. Both are built solo and in public, so the record of these mistakes is not a memory. It is a commit history I cannot edit.

What follows is the taxonomy I wish someone had handed me fourteen months ago. Eight classes. Every case is real and dated. Every sweep is one I actually run.

The one sentence this whole page is built on. A test proves that a function, when called, does what you expected. It does not prove that anything calls it. Almost every entry below is a variation on that gap.

On this page

The eight classes at a glance
SF-01 · The function nobody calls
SF-02 · The key that never existed
SF-03 · The import that never worked
SF-04 · The success message that is a lie
SF-05 · The silent catch
SF-06 · The environment that hides the bug
SF-07 · Two sources of one truth
SF-08 · The rule that only exists in the interface
Running all of this as one pre-release check
Why one person misses every one of these
Where AI assistance actually changes this
What the app does about it

The eight classes at a glance

If you only read one thing on this page, read the middle column. Every entry below is a different answer to the same question: what does your test suite think it is proving, and what is it actually proving?

INDEXSymptom to class, in one step

SF-01

Feature finished, user never gets it. Your test calls the function. Nothing else does. Caller sweep

SF-02

Permanent placeholder or empty state. Your fixture has the key you believe in. Production does not. Key diff

SF-03

Subsystem that has never run once. The import fails and something eats the failure. Undeclared import scan

SF-04

Confirmation of work that did not happen. You asserted on a return; the product is a side effect. Success-without-write grep

SF-05

A path raising on every call, for months. The catch turned a failure into a plausible fallback. Bare except census

SF-06

Correct on your machine, broken on theirs. Your runner is your dev box. Version matrix and write probe

SF-07

Two numbers that should be one number. Both are computed, neither is compared. Duplicate computation sweep

SF-08

A limit that stops holding the moment a second entry point exists. Domain-level enforcement test

The catalogue

SF-01The function nobody calls

Symptom

A feature is finished, tested and documented. The user never receives it. There is no error anywhere, because the code that would produce the result is never invoked.

Why tests pass

The test imports the domain layer and calls the function directly. The function works. It has always worked. The missing piece is the path from a user action to that call, and no assertion in a unit test can see a path.

Detection

List every public mutating method in the domain layer, then search the interface layer for each name. Zero callers means dead or unreachable, and you need to know which.

grep -rhoE "public [A-Za-z<>]+ (Try[A-Z]\w+|Set[A-Z]\w+|Add[A-Z]\w+)" Simulation/ \ | awk '{print $NF}' | sort -u \ | while read m; do n=$(grep -rl "\b$m\b" UI/ | wc -l) [ "$n" -eq 0 ] && echo "UNREACHABLE $m" done

Real case

2026-08-15 · Scaling Laws · commit 16a8774 A research node printed exactly what it granted: a curated corpus, a mixture-of-experts architecture. The player paid research points, cash and four months of in-game calendar. They received neither, in any campaign, ever. 526 tests passed on it, because every test called TryAcquireDataSource and TryAdoptArchitecture directly and nothing in the interface layer ever did. The sweep above returned 46 public mutators, 21 with no caller anywhere, and 5 that were meant to be player actions.

More cases

2026-06-19. Thermal baselines using Welford's algorithm and voltage anomaly detection ran for months and were fully tested. The chat assistant never imported either engine. It was answering questions while sitting on a brain it could not reach.
2026-07-10. A method called set_turbo() existed on the always-on optimizer daemon and had no caller in the codebase. The entire coupling it was written for was dead from the day it was written.
2026-07-03. Sixteen assistant intents matched user phrasings at full confidence and had no handler behind them. Confidence 1.00, then silence.

Fix

Write one test per mechanism that walks the whole path: the action is triggered the way the product triggers it, then you read the state the product reads. The five replacement tests here are explicitly forbidden from calling the granting function. They finish the research node the way the day loop does, then assert on what the company owns afterwards.

SF-02The key that never existed

Symptom

A screen renders a placeholder, a default, or an empty state, permanently. The real data is present in storage the whole time.

Why tests pass

The test builds its input by hand. A hand-written fixture contains the keys you believe exist, which is the exact belief under test. Meanwhile the lookup falls through to a default instead of raising, so nothing anywhere reports a problem.

Detection

Compare the keys your code reads against the columns your storage actually produces. Do it mechanically, not by reading.

python - <<'PY' import re, sqlite3, pathlib db = sqlite3.connect("data/logs/app.db") cols = {r[1] for r in db.execute("PRAGMA table_info(events)")} used = set() for f in pathlib.Path("ui").rglob("*.py"): used |= set(re.findall(r'evt\[[\'"](\w+)[\'"]\]', f.read_text(encoding="utf-8"))) print("read but never stored:", sorted(used - cols)) PY

Real case

2026-07-09 · PC Workman Every row in the events log rendered as "Unknown event", for months. The database held the type, the metric, the value, the baseline and a written human-readable description. The interface read evt["message"]. That key did not exist in any layer of the codebase, ever. The real answer was one dictionary key away and the most informative label the product ever showed a user was the word "Unknown".

Variant

2026-07-05. A learning engine reported that it was working and taught nothing. It learned exactly one metric, cpu_temp, and the query feeding it filtered WHERE cpu_temp > 0. Reading CPU temperature on Windows needs a sensor service most people do not run, so on those machines the column was always zero and the filter excluded every row. Worth noting honestly: a correctness rule I had added the week before, never learn from an estimated temperature, is what zeroed the column. A good decision created a silent outage.

Fix

Test against row shapes pulled from the real store, never against fixtures you typed. Where a missing key means the feature is broken, make the lookup raise instead of defaulting.

SF-03The import that never worked

Symptom

An entire subsystem has never run once, on any machine, since the day it was written. Everything downstream degrades to a default that looks plausible.

Why tests pass

Either the subsystem is mocked in tests, or the import failure is caught and swallowed. Both leave you with a green suite and a feature that has never executed in production.

Detection

Diff every module you import against the dependencies you actually declare and bundle.

python - <<'PY' import ast, pathlib, sys declared = set(open("requirements.txt").read().split()) std = set(sys.stdlib_module_names) for f in pathlib.Path(".").rglob("*.py"): for n in ast.walk(ast.parse(f.read_text(encoding="utf-8"))): if isinstance(n, ast.Import): for a in n.names: root = a.name.split(".")[0] if root not in std and root not in declared: print(f"{f}:{n.lineno} undeclared import: {root}") PY

Real case

2026-07-16 · PC Workman The assistant's hardware scanner had never worked, on any machine, since it was written. It ran import wmi, a package that was never installed and never bundled. The scan failed on every machine and the failure was caught by a bare except: pass. The first successful hardware identity write in the project's history happened the day I rewired it to the scanner that already worked.

Variant

2026-08-10. I published three times, in a changelog and in two articles, that psutil.sensors_temperatures() returns an empty dict on Windows. It does not. On Windows the attribute does not exist at all, so every call raised AttributeError into a try block. I only checked because I was writing an article about repeating claims from memory. The correction is on that page and this line exists so the wrong version does not outlive it.

Fix

A test that asserts the dead import is absent from the source. Three lines, and it makes the class unshippable. For optional platform APIs, check hasattr explicitly and record which branch you took, so an absent sensor is visibly different from a normal reading.

SF-04The success message that is a lie

Symptom

The product confirms an action it did not perform. Users do not report this, because they cannot tell the difference between "it saved" and "it said it saved". They assume they did something wrong and stop using the feature.

Why tests pass

The test asserts on the return value or on the message, and the message is a string sitting next to the call rather than a consequence of it. Where the real product is a side effect, a return value is decoration.

Detection

Find every success string in the interface and check that a write actually happens in the same function.

Real case

2026-07-18 · PC Workman The fan curve editor had an Apply button. It flashed "applied successfully" and persisted nothing. Every restart discarded the user's curve in silence, and it had been doing that for two releases. Nobody reported it. The fix took ten minutes. Finding it took months.

More cases

2026-08-08. The diagnostic console was meant to hide after launch. Under Windows Terminal, GetConsoleWindow() returns a hidden proxy window, so ShowWindow(SW_HIDE) succeeds, returns success, and changes nothing on screen. A tester reported it and I could not reproduce it, because my machine used a different terminal host.
2026-08-08. The Store build's "create desktop shortcut" feature wrote a shortcut pointing at an Application Id absent from the published manifest. Clicking it opened nothing. Nobody reports a shortcut like that. They stop using it.

Fix

Never assert on a return where a side effect is the product. Read the file back. Read the window class back. If your assertion and your user are looking at two different things, only one of them is testing your software.

SF-05The silent catch

Symptom

A code path has been raising on every single call for months. No crash, no log, no warning, no degraded behaviour that anyone can point at. The product simply does slightly less than it claims.

Why tests pass

The catch converts a hard failure into a fallback, and the fallback is often reasonable. Tests assert the reasonable outcome. A bare except also swallows NameError and AttributeError from code that never worked, plus SystemExit and KeyboardInterrupt, which makes clean shutdown unreliable on top of everything else.

Detection

grep -rn "except:" --include=*.py . | sed 's/^/BARE EXCEPT /' grep -rn -A2 "except Exception" --include=*.py . \ | grep -B1 -E "^\s*(pass|continue)\s*$" \ | sed 's/^/SWALLOWED /'

Real case

2026-06-24 · PC Workman A routing engine contained except: followed by pass. Two lines above it, the code read a variable belonging to a different function. NameError, on every call, for four months. The entire fallback path was dead from day one. I found it because some answers felt slightly worse than they should, which is not a debugging method.

More cases

2026-07-16. A refactor split a 6,533 line module into seven and dropped a module-level singleton. The caller wrapped its import in a broad except, so a feature flag quietly became False and the product's main AI layer switched itself off. Everything ran. Tests were green. A guard test written the same day, for a blind spot I had not predicted, is the only reason I know.
2026-07-15. A repository-wide count found 53 bare except: clauses.

Fix

Ban bare except in CI, then adopt one rule of thumb: swallow reads, never writes. A failed read can fall back to a default, because a default is a reasonable answer to "I do not know". A failed write must be loud, because the user believes it happened.

SF-06The environment that hides the bug

Symptom

The code is correct on your machine and broken on the machines you ship to. This is the class that turns "works on my machine" from a joke into a warning.

Why tests pass

Your test runner is your development environment. If the runtime, the filesystem permissions or the host application differ in production, the suite is measuring a machine no user has.

Detection

Run the suite on the oldest runtime you claim to support, and probe the environment instead of naming it.

for v in 3.9 3.10 3.11 3.12 3.13 3.14; do echo "=== python $v ===" py -$v -m unittest discover tests -q 2>&1 | tail -3 done

Real case

2026-06-17 · PC Workman A module used Optional[str] in annotations and never imported Optional. On my machine it worked perfectly, because I develop on Python 3.14, where PEP 649 defers annotation evaluation and the missing import never fires. On 3.9 through 3.13, the versions I officially support, it raised on import and the whole module silently turned itself off. My own runtime was hiding the bug from me.

More cases

2026-07-04. Microsoft approved the app into C:\Program Files\WindowsApps, which is read only. The app wrote its database, preferences and learning baselines next to its own executable, so every write failed silently on every Store install. Certification does not check this, and users do not report it, because nothing appears to go wrong. The app is simply permanently amnesiac. The rule that came out of it: do not detect an environment by name, probe it. A path check is a guess, a real write probe is a fact.
2026-08-08. The package manifest referenced one 32x32 image for every tile slot, including the 150x150 one. Windows stretched it. This is exactly what had been rejected as blurry two months earlier and it survived every release in between, because a manifest references files by name and nothing verifies they exist at the size the name implies.
Scaling Laws. Object.Destroy is a no-op outside play mode. A texture cache that correctly released memory at runtime released nothing in the editor, and only a soak test building every screen in edit mode ever noticed.

Fix

One test that resolves the writable directory using a real write probe, including the read-only redirect path. A CI matrix across every supported runtime. And for anything visual, render the frame and look at it, because no assertion has ever replaced a screenshot.

SF-07Two sources of one truth

Symptom

The product states two different values for the same quantity, in two different places, and both look authoritative. Users notice before you do, because they see both screens and you wrote both formulas.

Why tests pass

Each calculation has a test. Each test passes, because each calculation is internally correct. Nothing in the codebase compares the two results, so the disagreement has no owner and no assertion.

Detection

Look for the same constant, unit or concept appearing in more than one module. Duplicated helper names are the cheapest signal.

grep -rhoE "^\s*def (_?[a-z_]+)\(" --include=*.py . \ | sed -E 's/.*def (_?[a-z_]+)\(/\1/' | sort | uniq -c \ | awk '$1 > 1 {print "DUPLICATE DEF x"$1" "$2}'

Real case

2026-08-19 · Scaling Laws A model creator quoted the player eleven days for a training run. The run finished in one. Both numbers came out of my own code. The projection divided by a precision throughput multiplier. The daily tick never multiplied by it, and applied two bonuses the projection had never heard of. Two formulas producing one quantity, for weeks, with nothing comparing them.

More cases

2026-07-17. The version number lived in eight files and was already wrong in three. The main window titled itself 1.8.1 while the single-instance check searched for a 1.8.2 window title, so launching the application twice stopped focusing the running copy. That shipped for two full releases and no test saw it.
2026-07-04 and 2026-07-15. The same base-directory helper existed in six copies. An administrator check existed in five.
2026-07-18. Per-process CPU load was read on a per-core scale. One busy thread on a twelve-thread machine displayed 100 percent while the machine idled at 8. Top-process lists ranked noise and the proactive alert fired at nothing.

Fix

If the same quantity is computed in two places, that is not duplication. It is a future disagreement with a date on it. The fix is never to keep the copies in sync. It is one function both sides call, so they cannot drift, plus a ratchet test that fails the build when a second copy appears.

SF-08The rule that only exists in the interface

Symptom

A limit holds perfectly until someone reaches the same action a different way: another screen, a loaded save file, an edited config, a keyboard shortcut, an API call. Then it does not exist and never did.

Why tests pass

The tests drive the interface, and in the interface the rule is real. The domain layer, which is where the rule has to live for it to mean anything, has never been asked to enforce it.

Detection

Write the test that skips your interface entirely. Construct the illegal state directly against the domain layer and assert that it refuses.

// deliberately does not touch any screen var blueprint = new Blueprint { Parameters = ceiling * 10 }; var result = company.TryStartTraining(blueprint); Assert.IsFalse(result.Accepted, "parameter ceiling is enforced in the UI only");

Real case

Scaling Laws · model creator A parameter ceiling was enforced inside TryStartTraining. But Project(), the function the creator screen calls on every slider movement, did not check the ceiling or the precision gates at all. The creator happily priced a run that the START button then refused, with no warning attached to anything the player had touched. I wrote a test file that never opens a panel: it builds blueprints above the ceiling directly and asserts the company says no.

Variant

Order of operations is the same class wearing different clothes. In one Reprice() the blueprint was built from the raw slider value and the ceiling was applied after. The slider handle snapped back and every number on screen kept the value the player had dragged to. Validation that runs after the read is not validation.

Fix

Every constraint that matters is enforced in the domain layer. The interface displays it. Tests verify it with the interface bypassed. A limit that lives in a screen is a suggestion with a nice font, and it expires the moment a second entry point to that action exists.

Running all of this as one pre-release check

None of these sweeps is worth much as a thing you remember to do. Vigilance does not scale past a few thousand lines, and it fails exactly when you are tired, which is exactly when this class of bug ships.

What works is making them boring. Every sweep on this page runs in a single script that I fire before tagging a release, and the whole thing finishes in under a minute on a repository of about sixty thousand lines.

#!/usr/bin/env bash # preflight.sh - run before tagging. Exit code is advisory, not a gate. set -u fail=0 echo "== SF-01 unreachable domain methods ==" ./tools/caller_audit.sh || fail=1 echo "== SF-03 undeclared imports ==" python tools/import_audit.py || fail=1 echo "== SF-04 success strings with no write ==" grep -rniE "(applied|saved|success)" ui/ --include=*.py -l \ | xargs -r grep -LiE "(open\(|json\.dump|\.write\(|commit\()" || true echo "== SF-05 bare excepts ==" grep -rn "except:" --include=*.py . && fail=1 echo "== SF-06 supported runtime matrix ==" for v in 3.9 3.10 3.11 3.12 3.13 3.14; do py -$v -m unittest discover tests -q >/dev/null 2>&1 \ || { echo "FAILS ON $v"; fail=1; } done echo "== SF-07 duplicate definitions ==" ./tools/duplicate_defs.sh exit $fail

The important detail is the last line of the comment. This script does not block a release. It prints a list a human reads, because half of what it finds is legitimate: internal plumbing with no user-facing caller, a helper that genuinely should exist twice, a success message on an operation that really is read-only. A check that cries wolf gets disabled within a week, and then it protects nothing.

The ones that must actually fail a build are narrower and there are only a handful: no bare excepts, no dead imports, no hardcoded version literals, no second copy of a helper that is supposed to be single-source. Those are ratchets. Everything else on this page is a report.

Why one person misses every one of these

There is a reason this catalogue comes from a solo project rather than a team one, and it is not that I am careless, although in several of these entries I clearly was.

The failures here share a shape: they are all invisible from the inside. Nothing crashes. Nothing logs. Nothing looks wrong on the screen you happen to be looking at. Every single one of them requires somebody to ask a question nobody thought to ask, and when you build alone, the set of questions that get asked is exactly the set you already know to ask.

A reviewer supplies that for free. Not because they are smarter, but because they arrive without your assumptions. The reviewer who reads your test file has never been told that the test calls the function directly and that this is fine. They just see a test that does not resemble what a user does, and they say so.

Without one, you need a substitute, and the substitute has to be mechanical. That is the entire argument for the sweeps above. A grep does not get tired, does not assume, and does not politely skip the file it read last week. It is a worse reviewer than a person in every respect except the one that matters here: it is not you, and it does not share your blind spots.

The second substitute is a real user, and it is uncomfortable how much better they are at this than any tooling. Three testers spent five hours each on a release I considered finished and found six stability issues, seventeen intents that matched at full confidence and returned nothing, and a function calling the system snapshot nineteen times per message. None of that was in my test suite. All of it was in their first evening.

Where AI assistance actually changes this

I build with an assistant, openly, and have done for over a year. The honest account of what that changed is narrower than either side of the current argument suggests.

It did not lower the quality of individual functions. Most of the twenty-four cases in my log are ordinary mistakes with ordinary causes, and several predate any assistant.

What it changed is where the standard has to be enforced. An assistant is extremely effective at producing a function that satisfies a description. It has no view of whether a path exists from a human hand to that function, because that path lives in another file, in another layer, usually in another conversation. Writing got faster. Verifying that anything reaches the result did not. Every entry above lives in the gap between those two speeds.

The incident I think about most. This project keeps a written rule against generated-sounding prose and a test that fires every assistant response and fails the build on any em dash. On 17 July an automated cleanup pass removed 312 em dashes across 42 files. It also rewrote the em dash literal inside the test guarding against them, quietly converting that test into "does this answer contain a hyphen". The guard against automated damage was eaten by an automated pass. The fix was to write the character as chr(0x2014), which no future text sweep can match, then verify with a negative control: inject an em dash into a real response and watch the test fail.

That is the thesis in one incident. The tool did exactly what I asked, quickly and correctly on its own terms, and disabled the only thing standing between me and the problem it was hired to solve.

Limits, stated out loud

This is one person and two projects. I have no data on whether it generalises to a team, and a team has review, which removes some of these entirely and creates others I have never met.

I also cannot cleanly separate "failures caused by AI assistance" from "failures I would have shipped anyway". Anyone claiming they can, in either direction, is selling something. What I can say is what the log shows, with dates, in public repositories.

What the app does about it

PC Workman is a Windows monitor with a local assistant, and it is built under the rules on this page rather than merely describing them. The suite went from 21 tests in June to 331 in August, and the growth that mattered was not the count. It was that every silent failure which shipped once now has a ratchet test that fails the build if the class returns: hardcoded version literals, dead imports, bare excepts, duplicate helpers, estimated sensor readings entering history, and em dashes in generated text.

None of that makes the software correct. It makes one specific category of wrongness unable to ship twice.

SWEEP caller audit · Simulation to UI

public mutators found 46

no caller anywhere 21

user actions unreachable 5

tests green before sweep 526

tests after 531

time to fix one afternoon

A green suite proves your code ran.
It does not prove anyone can reach it.

This catalogue is free, has no signup and no email wall, and it is a living document. If you have a class of silent failure that is not here, the fastest way to reach me is GitHub.

The narrative version of several of these entries, with more of the story around each, is in the bug that does not crash. If you want to see the rules applied to a real Windows application, PC Workman is open source and on the Microsoft Store.

Sources: PEP 649, Deferred Evaluation Of Annotations · PEP 8 · psutil API reference · Unity, Object.Destroy · Microsoft, AppUserModelIDs

Marcin Firmuga

Solo developer · HCK_Labs · building PC Workman in public

I write about what I actually shipped, with real numbers and real code, including the parts that did nothing. More: my story.

GitHub LinkedIn X Buy me a coffee ☕