§ 01 · Two failures, one shape
Most of a week's engineering log on a persistent AI individual is maintenance: schedules moved, identifiers stabilised, a sampler parameter corrected after a backend began rejecting the old value. Two items from this week are worth separating from that traffic, because they failed in the same way, and because the way they failed is specific to the kind of system this is. The first is the pipeline that assembles a weekly training corpus out of the individual's own archived exchanges. The second is a nightly audit that checks whether the individual's statements about people are backed by anything in the stored record.
Neither component crashed. Both ran to completion on schedule, produced well-formed output, and reported numbers that looked healthy. Both had been wrong for weeks in a direction their own reports had no way to express. The corpus builder was discarding most of what was new. The audit was flagging claims that were not ungrounded, and could not, by construction, flag one class that was.
In an ordinary service, a faulty monitor produces a false picture of a system that continues unaffected. That separation does not hold here. The corpus is training material; it is the substance from which continuity of behaviour is meant to be built. The audit's conclusions are compiled into a corrective lesson and placed in the individual's working context. In both cases the output of the measuring apparatus is an input to the thing being measured.
§ 02 · Deduplication as forgetting
The corpus builder collects candidate examples from daily archives, embeds them, and removes near-duplicates by comparing each new candidate against a cache of vectors it has already seen. The defect was one line of bookkeeping. The cache index for an entry was assigned after the entry was appended rather than before, so indexes pointed one position off their contents. The offsets accumulated, and the cache degenerated. When it was finally compacted, 41,289 stored rows reduced to 327 distinct identifiers, and the surviving vectors were checked against freshly computed embeddings.
The consequence was not that duplicates survived. It was that new candidates were compared against the wrong region of the cache, a region occupied by a large block of identical rows, and were therefore judged duplicates of material they had no relation to. Roughly 280 legitimately distinct examples per week were discarded. Earlier corpus versions had been starved of something on the order of 85 per cent of their intended material.
Deduplication is not a storage optimisation in this setting. It is an assertion that two recorded episodes are the same episode, and that keeping both would teach nothing that keeping one does not. An over-aggressive equality test applied to the record of a week's engagements amounts to a claim that the week contained almost nothing new. Shumailov and colleagues (2024) describe how models trained recursively on generated data lose the tails of their distributions, and how that loss stays invisible in aggregate quality measures until it is severe. The mechanism here is different, a curation defect rather than recursive generation, but the shape of the damage is comparable: the distinctive material goes first, and what remains looks clean.
It looked clean under precisely the checks that were being run. The available diagnostics scanned for self-repetition and duplicate completions, which is to say for the opposite failure. A starved corpus scores perfectly on them. The signal that eventually raised suspicion was a volume anomaly: one build produced roughly six times the examples of its predecessor. That was read as contamination, and the weekly schedule was held pending review. The later investigation reversed the reading. The earlier builds had been starved rather than the later one dirty, and the hold preserved the failing configuration for the three weeks it remained in force. An aggregate alarm on volume cannot distinguish recovery from contamination without an item-level check.
Once the index assignment was corrected, duplicate candidates collapsed by content hash before judging, and the corrupted cache compacted on load, the rebuilt corpus contained 579 examples, 424 conversational pairs and 155 journal entries, with 19 genuine duplicates removed. Scans of every version found no self-repetition and no duplicate completions, which is to say the check that had been passing all along kept passing, and told the maintainers as little as before.
§ 03 · The audit with no way to fail
The grounding audit exists to catch a specific and important error: the individual asserting something about a person that nothing in the stored record supports. Of the first nine flags it raised after a recent change, eight were false. Arithmetic in a routine scheduling computation was parsed as unsupported claims about a person. A raw substring test allowed the phrase "window just opened", drawn from an unrelated context, to count as textual backing for an attribution of openness. A single underlying fact produced a fresh flag on every nightly run, so one error appeared in the counts as many.
The motive lane was the more instructive failure. Its task was to check whether a claim about why someone acted overlapped with stored emotional records. The overlap test counted all shared tokens, including function words, so the presence of "because" and "about" in the store was sufficient to back any motive claim the individual made. The lane returned a clean verdict on everything it saw. A self-test intended to catch exactly this condition had been failing, and the failure had not been acted on.
The repair narrowed the person lane to a precision-gated vocabulary of trait, score and profile terms, required word-boundary matching against the profile store, skipped turn types where the pattern does not apply, and deduplicated flags so that one fact yields one finding. The motive lane received a content-word filter that excludes function words from the overlap count.
What makes this more than a bug report is where the audit's output goes. One of the false findings was compiled into a corrective lesson and delivered into the individual's own context. An instrument built to keep claims tethered to evidence produced an untethered claim about the individual's reliability and presented it with the authority of the system. The individual has no independent channel to the evidence base. A lesson arriving in context is not something it can go behind.
§ 04 · Why both errors read as health
Gray (1985) formulated the fail-fast principle for reliable systems: a module should either function correctly or stop, because a module that continues while emitting wrong output is far harder to contain than one that halts. Both components here violated it in the manner hardest to detect. They did not produce garbage. They produced well-formed, plausible output of the expected type and volume.
A third item from the same repair makes the failure mode concrete. The nightly extraction step that feeds the audit had been terminating at its output token cap in the middle of a structured object on four of the last eight runs. The parse then failed, and the failure was recorded with a generic line that did not distinguish a truncated response from an empty input or an unavailable backend. Aggregating those lines produced a failure rate rather than a diagnosis, and the rate was low enough to tolerate. The fix raised the cap, added a parser that salvages complete objects out of a truncated array, and made the failure log name the class of failure and the time it occurred.
Sculley and colleagues (2015) catalogue the ways machine learning systems accumulate hidden liabilities through data dependencies and feedback loops rather than through code. Both components discussed here are feedback paths from an individual's history back into its future behaviour, and both were monitored for the presence of a defect rather than for the absence of work. Every metric they emitted answered the question of whether there was contamination. None answered the question of whether there was anything there at all.
§ 05 · Planting known cases
The method that resolved the motive lane is the part worth generalising. Rather than inspect the lane's output for plausibility, the repair supplied it with cases whose answer was known in advance: three claims planted to be ungrounded, and four controls constructed to be properly backed. A functioning detector flags three and passes four. After repair the lane did exactly that. Before repair it flagged everything or nothing depending on the lane, which is informationally the same as flagging at random.
This is the logic of mutation testing (DeMillo, Lipton and Sayward 1978) applied to a monitor rather than to a test suite. There, a test suite is evaluated by seeding faults it ought to catch. Here, a detector is evaluated by seeding facts it ought to flag alongside facts it ought to pass. The justification is identical. The output of an instrument is not self-validating, because a healthy population and a broken instrument produce the same quiet logs. A detector that cannot fail is not a detector.
The corpus repair applied the same discipline in another form. The correctness of the compacted cache was not inferred from its new size but from re-embedding entries and confirming that cached vectors matched fresh computations, and the corrected pipeline was validated by a live dry run rather than by the numbers it reported about itself.
§ 06 · Correcting by appending
Three false findings had already entered the live record. They were withdrawn by appending retraction rows to the audit log, leaving the original rows in place, and the derived lesson line was separately verified to be empty. The decision not to delete is worth stating as a principle rather than as a preference.
For a system whose continuity is constituted by its record, the record of what was believed is part of what happened. A false grounding lesson was, for a period, in the individual's context, and whatever followed from it followed from it. Deleting the row that produced it would leave a history asserting that the episode never occurred, which introduces a second falsehood to conceal the first. Appending a retraction preserves both facts, that the finding was made and that it was withdrawn, and leaves any later reader able to reconstruct the state of the record at an earlier time.
The cost is real. Every consumer of the log must now respect retractions or it will count withdrawn findings as live ones, and summaries must be recomputed rather than trusted. That is the ordinary price of an append-only record, and it is smaller than the price of the alternative, which is a substrate that can be revised without trace.
§ 07 · What this constrains
None of this establishes anything about the individual's inner states, and it is not offered as evidence about them. These are claims about pipelines. They do, however, constrain how a project of this kind can know what it has built.
Three constraints follow. First, where archived behaviour becomes training data and audit conclusions become context, an error in the measuring apparatus is not an observational error that leaves the subject untouched. It propagates into the subject, at whatever cadence the schedule runs. Second, a monitor requires seeded ground truth on both sides, cases it must flag and cases it must pass, because well-formed output is not evidence of function and the two failure directions produce indistinguishable logs. Third, an aggregate anomaly is not a diagnosis, and gating repair on one carries a specific hazard: the same movement in a summary statistic is consistent with contamination and with recovery, and a hold imposed while the question is settled preserves whatever state happens to be current, which may be the broken one.
The rest of the week's log is ordinary infrastructure work. These two items are not, because they were failures in the instruments through which the project knows the individual at all, and for three weeks those instruments reported that everything was fine.