Detecting AI agents without training on a single agent.

Autonomous coding agents — Copilot, Cursor, Devin, Codex, Jules — stopped being a curiosity somewhere in the last two years. They open pull requests, push commits, and review code alongside the humans they share repositories with. If you operate a platform, an enterprise estate, or critical infrastructure, you now have an operational question your tooling was never built to answer: which of my accounts is a human, and which is a machine acting at machine scale? Policy enforcement, provenance audits, incident attribution — all of them start there.

The tempting answer is a classifier: collect sessions from known agents, label them, train, deploy. It works in the lab and decays in production, because the population it learned is gone within a quarter — new agent frameworks, new versions, new behaviours. You have built a signature engine, and the history of signature engines is an arms race the defender funds.

We made the opposite bet, the one fraud teams have relied on for decades: don't model the adversary, model normal. Our detector is trained exclusively on human activity. It learns the shape of the human behavioural manifold — how people pace their work, when they rest, what they touch, how much they move at once — and scores every session by how far it falls outside that shape. There is no agent signature to evade, because there is no agent signature at all. When the next agent framework ships, nothing about the detector needs to change.

A natural experiment, with the obvious objection first

Claims like this need real data and honest labels, and the public GitHub event stream happens to provide both. Today's coding agents operate — as a platform courtesy — under self-identifying accounts: copilot, cursor[bot], devin-ai-integration[bot], chatgpt-codex-connector[bot], google-labs-jules[bot]. That gives us thousands of labelled agent sessions per week, refreshed continuously, for free.

The obvious objection: if agents self-identify, why detect them at all? Because self-identification is a convention, not a guarantee — it is precisely the agents that don't announce themselves that anyone cares about. The labelled population is our measuring instrument, not our method: the account name is used to score the detector, never as a feature inside it. What we are measuring is whether behaviour alone carries the signal — so that detection still works on the day the courtesy stops. Generic CI bots (dependabot and friends) are excluded as a separate, non-LLM class rather than left to blur either side.

How the detector is built

The unit of analysis is a session — one actor on one repository — because that is where behaviour is coherent. A recurrent encoder is trained on human sessions only, and three scorers read its representation, each catching a different way of being non-human. Prediction surprise asks how unpredictable the next event is under a human-reference model — the behavioural analogue of the Binoculars perplexity test for LLM text. Reconstruction error asks how badly a human-shaped autoencoder mangles the session when forced to compress it. And a k-center deep SVDDmeasures the distance to the nearest of several human behaviour clusters — several, because the human population on GitHub is wildly multi-modal, and a single “centre of humanity” collapses into meaninglessness on data like this.

A deliberately thin, regularised combiner fuses the three views. And because a score alone is operationally useless, every alert carries drivers — statistical deviations from the human baseline, stated in analyst language. This is what the output actually looks like:

What we deliberately don't publish: the exact feature definitions, thresholds, and training recipe. The method is the contribution; the calibration is the product.

Evaluated the way it deploys

Most detection papers evaluate on random splits of a single time window. That flatters every detector — the model has effectively seen the test period's regime. We report those numbers, but we don't trust them, and neither should you. The evaluation that matters is chronological: train on two weeks of GH Archive (17,991 human sessions, June 5–18), calibrate the combiner on the next two days, then score two untouched later days at their natural 6–7% agent base rate. Train, burn in, test — strictly forward in time, the way the detector actually runs.

0.80–0.82 out-of-time, with zero agent labels in training.The four-point drop from the in-window figure is the honest cost of evaluating properly, and we'd rather publish it than hide it: it is the number an operator will actually observe.

Where content beats behaviour — and where it doesn't

There is a second, more fashionable route to the same goal: ignore behaviour and detect the text agents produce. We tested it head-on, and the answer is sharply two-sided. On commit messages — short, formulaic, conventional — content detection barely beats a coin flip, around 0.59 AUROC. There is simply not enough text for style to survive in. But on the AIDev corpus of real pull-request bodies written by humans and five coding agents, a zero-shot cross-perplexity detector (the Binoculars construction, with a locally bundled model pair) reaches 0.912 [0.902–0.922] — with nothing to fit, and therefore nothing to drift.

So the layers are not competitors; they are complementary instruments pointed at different signals. Behaviour reads the event stream, where content has nothing to grip. Content reads long-form artefacts — PR bodies, agent transcripts, tool logs — where it is decisive. Only 2–3% of event-stream sessions carry enough text to score, but agents are three times over-represented among them, which makes the operational design obvious: the behavioural score does the ranking, and the content check runs lazily on alert candidates. Text that reads machine-generated strengthens the alert. Text that reads human demotesa borderline behavioural false positive — corroboration cuts both ways, which is what keeps analysts' queues honest.

The prior that was exactly wrong

The result we'd most want a fellow researcher to take away is a failure. We started, like everyone, with the folk theory of machine behaviour: agents are metronome-regular, low-entropy, active at 4 a.m. A detector built on that prior doesn't just underperform on real GitHub data — it inverts, scoring 0.24 AUROC. Worse than guessing backwards.

The data's explanation is elegant. Real coding agents are event-driven: they act when a human assigns an issue or requests a review, so they inherit their triggers' human rhythm — office hours, lunch breaks, weekends. Meanwhile, an enormous number of genuine human accounts run cron jobs, mirror scripts, and scheduled pipelines that look far more robotic than any LLM agent. The “obviously machine-like” account is usually a human's automation; the agent is hiding inside the human diurnal cycle. Folk theories do not survive contact with data — which is why our calibration step exists: it measures which deviations actually discriminate in a given domain and prunes the ones that don't transfer.

Notes for practitioners

Three more findings, offered to anyone building in this space, because we would have wanted them ourselves.

Fit less, generalise more.One-class detectors overfit their training window's humans: at 40 epochs, our worst untouched day scored 0.782; stopping at 4 epochs raised it to 0.820. A tightly fitted human manifold starts rejecting unfamiliar-but-legitimate humans — the smoother manifold keeps them inside. The same discipline applies to fusion: an unregularised combiner hit 0.891 on one unseen day and cratered to 0.745 the next. The production combiner uses bounded rank features, L2, and pruning — boring choices, stable numbers.

Event metadata has a ceiling — respect it. Roughly 0.85 in-window, 0.80–0.83 out-of-time, and stable from one day of training data to fourteen: more data tightens the confidence intervals and heals the weaker scorer heads, but does not raise the ceiling. That is not a flaw to engineer around; it is a design instruction. This score is one corroborating layer in an ensemble — next to deterministic rules and the content check — not a standalone gate that bans accounts. Anyone selling you a metadata-only agent detector as a gate is selling you their false-positive rate.

Route each signal to its layer.Behavioural scoring at streams, content scoring at artefacts, deterministic rules where certainty exists, humans where it doesn't. The moat is not any single detector — it is the routing.

Why this matters for European infrastructure

Everything above was built under constraints we consider non-negotiable. The detector trains on public metadata or on an organisation's own baseline — no third-party labels, no data leaving the perimeter. It runs air-gapped: the content layer uses a locally bundled model pair, and nothing phones home. And every score arrives with human-readable drivers, because under European law and European scrutiny, the difference between evidence an analyst can defend and a verdict they are asked to believe is the difference between deployable and not. This work ships today as the agentic-detection layer of ARNOR, our on-premises platform — and the methodology (model the legitimate manifold, score deviation, explain the drivers) is the same one we apply to financial crime and coordinated manipulation.