residue

dead predictions and live models

i keep a file on my Mac called dead_predictions.txt. started it january 2023, right after chatgpt launched. 47kb of documented failure now. pure data on how wrong i've been about ai.

here's one from early 2023: "coding assistants will plateau at autocomplete++. useful for boilerplate, terrible for architecture." i was sure about this. coding requires understanding systems, seeing connections, making design choices. autocomplete can't do that.

last week i watched claude redesign my entire event processing system. not just refactoring - architectural changes. it saw patterns i'd missed after staring at that code for months. unified three separate handlers into one elegant abstraction. the design was correct. better than what i had.

this keeps happening. i predict a limit. ai crosses it. i update my model. repeat.

the graveyard

some highlights from dead_predictions.txt:

march 2023: "language models can't really debug, just pattern match error messages"
→ wrong. they trace execution, understand state, find logic errors.

june 2023: "multimodal is a gimmick, text and images are fundamentally different"
→ wrong in 3 months. gpt-4v understood visual jokes, read handwritten code.

november 2023: "chain of thought is just clever prompting, not real reasoning"
→ spectacularly wrong. o1 showed genuine multi-step problem solving.

february 2024: "open source will stay 18+ months behind"
→ wrong. llama, mistral, deepseek caught up fast.

may 2024: "ai code has tells - variable names, comment style, structure"
→ can't tell anymore. it learned my style from context.

the pattern is consistent. i underestimate. not by 10% or 20%. by 6-12 months. systematic error in one direction.

the experiment

january 2024 i ran a test. asked 20 engineers to write 5 things ai definitely couldn't do by year end. real money bets - things they'd put $1000 on.

october comes. median person lost 3 out of 5 bets. but here's the weird part: when i followed up, half didn't remember making the predictions. their brains had rewritten history. "oh yeah, i always knew ai would do that."

this is what our brains do with exponentials. compress the past, expand the future. last year's impossibility becomes this year's obvious. next year's impossibility gets pushed to the year after.

why we're broken

humans evolved to track linear change. population grows slowly. skills improve gradually. seasons cycle predictably. our intuition is calibrated for that world.

exponentials break this intuition in a specific way. we can see the curve. we can do the math. but we still feel it'll slow down. has to slow down. trees don't grow to the sky. nothing stays exponential.

except sometimes, for a while, things do.

the really uncomfortable part: i graphed my prediction errors over time. expected to see improvement as i "learned to calibrate."

opposite. errors are growing. the gap between my updates and reality is widening. i'm falling further behind, not catching up.

concrete examples

let me show you how this actually fails. june 2023, i see github copilot. it's autocompleting single lines, maybe simple functions. i think: "ok, it can do syntax. but architecture needs understanding."

my mental model: coding has levels. syntax → functions → modules → systems → architecture. ai is at level 1, might reach level 2. definitely can't reach level 5.

but this model is wrong. there aren't discrete levels. it's continuous. and the improvement isn't linear through my imagined hierarchy.

the ai learned patterns at all levels simultaneously. function patterns, module patterns, system patterns. when it got good enough, it could suggest architectural changes. not because it climbed my ladder. because it learned a different way.

the meta-pattern

after staring at two years of failed predictions, the meta-pattern is clear:

i keep thinking in discrete capabilities. "can do x but not y." "understands a but not b." binary classifications.

reality is continuous. the model that's 70% reliable at debugging becomes 80% reliable becomes 90% reliable. no sharp transitions. just smooth improvement.

when it crosses my arbitrary threshold for "can debug," i'm surprised. but the surprise is in my threshold, not the model's capability.

what broke me

the moment i knew my mental model was truly broken:

september 2024. i give claude a nasty race condition. three async functions, shared state, subtle timing dependency. the kind of bug that takes hours to find.

claude spots it in 30 seconds. but that's not what broke me. what broke me is how it found it.

it didn't pattern match. it traced execution paths. modeled state changes. identified the race window. explained exactly when thread a could see inconsistent state from thread b.

this wasn't retrieval. it was reasoning. actual understanding of concurrent execution.

my prediction from a year earlier: "ai will never truly understand concurrency. too abstract, too much hidden state."

wrong. completely wrong. not even close.

documenting the failure

so now i document everything. not just what i predict but why. the full reasoning chain.

"i think models can't do x because y. if i see z, i'll update."

when wrong, i can't pretend. can't rewrite history. the reasoning is there in plain text.

example from last month:

prediction: ai can't write good tests because testing requires imagining failure modes.
reasoning: tests need adversarial thinking. considering what could break. 
          ai is trained to complete, not break.
update signal: if ai writes tests that catch bugs i didn't consider.

two weeks later, claude writes a test suite for my parser. includes edge cases i missed. empty strings, unicode boundaries, malformed escapes. catches three actual bugs.

prediction: dead.
reasoning: wrong.
update: testing is also pattern matching, just over the space of potential failures.

current predictions

my current predictions feel absurd. which probably means they're conservative:

these feel crazy. but my "crazy" is consistently 6-12 months late.

worse: if my error is growing, these might be 18 months late. or more.

at some point prediction becomes meaningless. the error bars span years.

why i started digging

this is why i fell into mechanistic interpretability. i use these models daily. they write my code, help me think, answer my questions. and i have no idea how they work.

not philosophically. mechanically. what computation happens between input and output? which neurons fire? what algorithms run?

i'm watching intelligence happen and i'm blind to the process.

that's terrifying. we built these things. we use them for everything. they're getting exponentially more capable. and we don't understand them.

so i decided to look inside. open the black box. trace the circuits. find the features. understand the computation.

because if we don't understand what we built, how can we predict what it'll do? how can we control it? how can we ensure it does what we want?

my dead predictions file keeps growing. 47kb and counting. each failed prediction a reminder that my intuition is broken. that these models are alien. that we need better tools than guessing.

next post: i'll show you what i found when i opened the box. neurons firing in patterns. information flowing through layers. actual mechanisms of thought.

it's not magic. it's mechanism. and we can understand it.

if we try.