Compass audits how AI agents navigate websites — so I trained a small 2.6B model to make the same navigation mistakes a frontier model makes. Same failures, same conclusions, roughly 85% agreement on a holdout it never trained on.
A couple of months ago I relaunched Compass as a desktop app in beta, and then went totally silent. The reason: while everyone's trying to squeeze frontier intelligence into small models, I was doing something weirder — training a small model to be wrong. Because that's what my product needs.
Compass audits how AI agents navigate a website. You point it at a site, it runs through the site the way an agent would, and it tells you what the agent finds, where it gets lost, and where it gives up.
I want it to be like "Screaming Frog for AI search", which means you download it, press go, and that's it. Which means the tool needs to be able to run with no "Paste your OpenAI API key here" config that nobody will bother with. That means it needs to run on your laptop, which if it is anything like mine, is shit. Which means the model must be small. Really small. But it needs to reason through a site structure like a model that costs billions in compute to train.
So the research question became: can I take a small local model and teach it to behave like a much bigger one on one specific task (navigating a website) and can I do that in a way that survives contact with real sites?
It turns out it's very easy to make a small model succeed at that task. Easier than you'd think. But a specialist single-purpose model that outperforms ChatGPT in this niche area is no use for auditing how ChatGPT navigates. What I needed was a doppelganger. A model that would make the same sort of mistakes a frontier agent would.
I'd built navigation tooling before, but I hadn't actually trained a model, so I read up on how people do this properly. LoRA, QLoRA, RLHF, SFT, those are just some of the acronyms I read and did not fully understand. Eventually, though, I came across Andrej Karpathy's autoresearch, a repository from March 2026 that turned out to be exactly the shape I was looking for.
The idea of autoresearch is simple. You give an AI agent a small but real LLM training setup and let it experiment autonomously overnight. It modifies the training code, trains for a fixed five-minute budget, checks whether the result improved, keeps or discards, and repeats. You wake up in the morning to a log of experiments and, hopefully, a better model. The human never writes the training code. Given I do not know how to write training code, this was a critical selling point for me. Instead, you write the program.md file that sets up the agent, and you iterate on that instead. Essentially, you give an AI the instructions and tools it needs to train another AI.
That's the design I borrowed, pointed at a different problem. My loop has the same shape: a strong local model, the teacher, generates traces of navigation tasks through the real engine; a small student model gets fine-tuned on those traces; the student is served and replayed on a fixed test matrix; the results are scored against a frozen consensus of bigger models; and a supervisor agent writes up what it predicted versus what happened, in plain English, and asks what I want to do about it.
One iteration, step by step:
The kit I started with was a MacBook and an Asus GX10 with a lot of memory. It was great for dabbling with local models, and it was not enough for this project. So I needed a way to build a rig that would work, cheaply.
I can't afford brand-new kit, so I went with second-hand workstation gear: a second-hand tower, second-hand GPUs, a lot of fiddling. It gets the job done rather than being something I threw money at. It still cost a lot, but less than you imagine, cheaper than a new Mac Studio of any kind. I called it the poverty rig.
The loop only works if the training data is trustworthy, so the first project was to build a gold standard and certify the teachers against it.
The gold standard was two frontier models, GPT-5.6 Luna and Gemini 3.5 Flash-Lite, run through the real engine on roughly 600 tasks each, frozen at the start of the project. Why these? Because Luna powers ChatGPT free, and Gemini 3.5 Flash-Lite powers AI overviews. This is what literally billions of people experience as "AI Search" today, not the bleeding edge models that grab headlines.
Then the certification: six candidate teacher models, each generating traces on the test set, scored against the gold standard. The goal was to find a model that made similar navigation choices to those two models, to 90% agreement.
Of the six, technically none passed.
GLM-5.3-Flash came in at 89.2% agreement with the gold consensus and became the "silver standard", a cheap bulk generator that produced a sizable chunk of the training corpus. I decided it was close enough, and that 0.8% was a sufficiently small deviation that it would still work.
Gemma 4 26B made the threshold overall, with 91.9%, and still failed, because of eight systematic skew cells: it declared success where the gold models failed. Ironically, it was too good, and in a model that would drive an audit whose job is to find failure points, that's a liability.
But I am a cheapskate, so as well as GLM-5.3-Flash, I also used Qwen3.8-27B, a dense 27B model that fits on a single GPU to produce free training data, whenever I needed it. That let me train, test, and generate new data simultaneously. This was often loosely referred to as "the teacher" because it hit 84.3% agreement with the gold standard, which was good enough to push the student upwards.
The student was the excellent LiquidAI-produced LFM2.5-2.6B, an open-licenced model designed to bring frontier instruction-following and tool usage to edge computing and potato hardware. It scored around 27% on its first try in the test set.
The dataset rules, locked early:
This is the part I need to be straight about.
The loop was designed as an autoresearch-style semi-autonomous...thing. The supervisor agent runs the iteration, writes up the hypothesis for each failure bucket, runs the eval, and scores the result against the prediction. On paper, that's a research org that runs itself. I was supposed to be able to kick it off and go do something else.
In practice, I didn't really know how to make it work without human intervention, so I started and stopped each run myself, with the AI agent doing the supervision. The agent kept the bookkeeping, the scoring, and the hypothesis tracking. I made the calls on which dataset change to make next, which hardware layout to run, and when to trust a number and when to throw it away. It's likely that I did it wrong and someone will tell me so, but I learned a lot by being more involved along the way.
That division of labour is what actually made it work in the end. Both sides were necessary, and I'd be lying if I said the loop just ran.
The stock student, out of the box, scored 26.9% agreement with the gold consensus on the full matrix. Eight fleet repeats gave me the determinism number and confirmed the student had everything to learn. The gap to the certified corpus was about 62 points, and that was the mandate.
I was trying to work out what bigger change the data needed to move the needle harder. And then the coding agent found my mistake.
The thing you need to understand is that the bottleneck in this kind of work isn't GPU inference, it's the browser and the network. Browsers cost CPU and RAM, not GPU. I'd noticed my costly GPUs were sat at minimal utilisation most of the time, so I'd had the bright idea, and in fairness it was a good idea at the time, to parallelise the runs. One runner became 8, split across the massive RAM and twin-CPU setup. The GPUs hit 90% utilisation, I was a genius.
Unfortunately I am also an idiot, and I never changed the amount of context allocation the student model had on the GPU. 8,192 tokens for one runner became 1,000 for each of 8.
The symptom was a fleet failing en masse, with only 21 of 599 runs ever making a navigation decision. Fixed, iteration-2 wasn't 39.2%. It was 86.9%. The weak bucket I'd been diagnosing for days wasn't in the data. It was an artefact of the bug, or put another way, my stupidity.
That's the part I'll remember. I was on the edge of walking away from the project, small deltas and no clear path, and in the space of a day or two, "holy shit, we were almost done the whole time."
Then the last hurdle. Iteration-3: 91.8% on the 300 never-trained holdout cells, 167 of 182 scorable, zero false findings. Ship gate passed.
That 91.8% deserves a caveat: It's the high point of a single, lucky run. A three-way stability test across runs puts the average at 84.9%. So the number to quote is roughly 85% agreement with the frontier consensus, across multiple runs and multiple sites, on a holdout the model never trained on.
That's a good result. What it is, precisely: a small local model, trained on observed behaviour, reproduces most of the bigger model's navigation behaviour, successes and failures both, well enough to be a useful stand-in. What it doesn't do is prove the model generalises across all the kinds of sites you'd actually audit. The 300-cell matrix is only enough to know it works well. Generalisation is the next gate, and it's what I'm still working on.
No scraping. No reasoning distillation. What the student does see is the model's output reasoning, which it was asked to provide alongside its link picks while navigating.
The model has learned to read an extracted set of HTML and links and pick the link most likely to get it closer to its goal, based on what the smarter models said their selection criteria were. The distinction is important because it means the student isn't smarter in any meaningful sense. It's a behavioural mirror of frontier navigation, failures included, that runs on your machine, no API key, no cloud, fast enough and cheap enough to be the default.
The model is going to be released. I'll probably upload it to Hugging Face, but it's largely useless without the harness that the Compass software provides, so the model and the tool are coming together. First it gets a bit more training on a dataset that aims for breadth rather than depth, so it generalises well across site types, and then it launches as Pathfinder1.0, coupled with a new Pro version of Compass.
I didn't know what I was doing when I started this. Arguably I still don't, but I know more than I expected when I began, and I now have the tools and the knowledge to do further experiments and improvements, which is the actual point of the loop.
And the result still stands: you can teach a small model to be as stupid as a large one, and that turns out to be the hard part.
