Your AI assistant remembers what you said. It doesn't notice when it stops being true.
Memory systems are good at recalling what you said and bad at noticing when it changed. A small local model can do better.
Imagine telling your assistant, early in the year, that you've lived in Portland for ages and the rain doesn't bother you anymore. Months later you mention, in passing, that you shook a bark scorpion out of your boot this morning and the dry heat is driving them indoors.
You never said "I moved." But you did. And an assistant that still suggests rainy-day activities in Portland has quietly stopped knowing you.
This is the problem I spent the last few weeks on, and it turns out to be a surprisingly big gap.
Memory is good at recall, bad at change
Long-term memory for chat assistants has mostly been judged on one skill: finding what you said. Ask which city you mentioned three hundred messages ago and modern memory systems will usually find it.
Keeping memory current is a different skill. A recent benchmark, STALE, tests exactly that: 400 scenarios where a later conversation overturns an earlier one, sometimes openly ("I moved to Austin") and sometimes only by implication (the scorpion). The published memory systems everyone uses scored between 5 and 18 percent. The best frontier model, reading the entire history, managed about 55.
I wanted to know how far a small model running on my own GPU could get, and what actually makes the difference.
What worked
Write down what a message implies, not just what it says. When you mention the scorpion and the dry heat, the memory records "the user lives in a hot, dry region" alongside the literal fact. That one change turns an invisible conflict into one that a simple similarity search and a cheap check can catch. It was the biggest single gain: about 17 points.
Check for conflicts in two stages. Asking the model "does this new fact overturn that old one?" with a one-word answer is fast, but small models are badly calibrated at it. Depending on the wording, mine flagged almost everything or almost nothing. So the first stage is deliberately permissive and cheap, and only the pairs it flags get a second look where the model reasons for a few sentences.
Always show the whole current state. A person's current situation is small: where they live, work, who's in their family, what they eat. It fits in the prompt. So instead of hoping a retrieval step finds the right fact, the assistant always sees everything that's true now, plus a short list of what used to be true and what replaced it.
Guard against outdated premises. If you ask "since I live in Portland, what should I do this weekend?", the assistant checks whether your question relies on something that's no longer true, and if so, tells you what changed.
The result
With an open 8B model, the whole system scores 55 percent on 300 scenarios I never looked at while building it. That's roughly where the frontier model reading your entire history lands, from a model that runs on one consumer GPU.
Two problems with how this is measured
While working on this I found two blind spots in the benchmark itself.
First, it rewards forgetting. It checks that outdated facts are dropped, but never checks that facts which are still true survive. A memory that forgets everything would look great. So I added a "keeps the new fact" check, and it shows a real trade-off: the premise guard that adds 36 points on trick questions also makes the system more willing to doubt things that are still true.
Second, it accepts the right answer for the wrong reason. "You may not live there anymore" counts as correct even if the reason given has nothing to do with what actually changed. I added a check that only counts answers pointing at the real change.
Both are in the paper, with the numbers.
Why this matters
If assistants are going to remember us, they have to notice when we change. People move, change jobs, go vegetarian, have kids. An assistant that confidently works from last year's version of you isn't just unhelpful, it's a little unsettling.
Everything is open: the code, the prompts, and every judged answer, so anyone can check the numbers or build on them.
Ali Ashter is a senior software engineer who builds and measures AI systems. hello@ashter.dev
Paper: PDF
Code and data: github.com/alleyxd/LastKnown