We measure, we don't guess

Small models. Local hardware. Measured against frontier systems.

Every number here comes from a benchmark I can reproduce on machines I own — and the failed approaches are reported next to the results, because a record that only lists wins is marketing. Same instrumentation habit behind a study of chain-of-thought faithfulness across 19 frontier models.

01 — The record

Four claims, each measured against the system it's up against. Open one for the method, the head-to-head, and what failed on the way.

Legal reasoning · MBE bar exam · Harvey AI has never published a score

Cadence

MBE bar exam under an adversarial protocol — a ~14B open model on one 4090.

98.3%

A ~14B open model wrapped in retrieval, deterministic citation extraction, and an adversarial checker, over a million-opinion, 50-state corpus — running locally on one RTX 4090, not on rented frontier inference. The 28-point lift over the base model came from pipeline architecture, not parameters.

98.3%

MBE bar exam, adversarial protocol (115/117)

100%

statute recall, 50-query benchmark

1,002,364

court opinions in corpus, all 50 states

14,889

jury verdicts extracted & structured

System	MBE score	Hardware	Funding
Cadence (local)	98.3%	1× RTX 4090 ($2K)	$0
DescrybeLM *	100%	Cloud API	startup VC
ChatGPT 5.2	93.4%	Azure datacenters	$13B+
Gemini 3 Pro	91.5%	Google TPU fleet	Google
Claude Opus 4.5	89.0%	AWS clusters	$8B+
Harvey AI	no published score	Cloud API	$1B+ · $11B val
CoCounsel	no published score	Cloud API	$200M+ acq.
Lexis+AI	no published score	Cloud API	LexisNexis
GPT-4 (2023)	75.7%	Azure datacenters	$13B+
Kimi K2.5 (1T params)	~70%	240GB+ RAM min	Moonshot AI
Human avg (law grads)	68%	a brain	$200K+ debt

Cadence, ChatGPT 5.2, Gemini 3 Pro and Opus 4.5 are my own run on one 117-question MBE set (March 2026). Independently, a March-2026 study put the same general-purpose models at 88.5–93.5%. * DescrybeLM's 100% in that study was self-evaluated and AI-judged on its own question set, with training-data overlap not ruled out. GPT-4 scored 75.7% on the same exam in 2023; the human average is 68%. Harvey / CoCounsel / Lexis have never published an MBE score.

DisclosedThree intermediate architectures scored 28%, 63%, and 26% before the final one hit 98.3%. The progression is documented, not hidden.

Confidently wrongOf the 52 answers the three frontier models got wrong, 49 were confidently wrong — fluent, assertive, no signal of doubt. That specific failure is the one the adversarial checker is built to catch.

The gapHarvey AI has raised over $1B at an $11B valuation (March 2026), and Stanford RegLab found the incumbents' tools hallucinate 17–33% of the time. This ran on hardware I own, for a few dollars a month.

Execution latency · on-chain verified against competitor marketing claims

Same-slot execution engine

of 1,817 trades land in the leader's own block — at a $0.001 fee, over a home connection.

40.2%

A Solana copy-execution pipeline built and instrumented end-to-end — signal, decision, transaction build, network — running on consumer hardware over a home connection. No VPS, no colocation. A slot is ~400ms; a same-slot fill lands in the very block of the trade it copies — the fastest fill physically possible. Every figure below settles on a public ledger; verification requires a block explorer, not my word.

40.2%

same-block execution (731 of 1,817 trades)

95.3%

within one slot of the leader (<400ms)

190ms

avg buy pipeline (119ms median)

$0.001

priority fee per trade — vs $2–5 typical

Bot	Effective cost	Priority fee	Latency	vs this engine
GMGN AI	~3–5%	~$1.20	1–3s (claim)	5.3× slower
Trojan	~4–5%	~$3.50	<500ms (claim)	2.6× slower
Axiom	~3–6%	~$2–10	≤400ms (claim)	2.1× slower
This engine	0%	$0.001	190ms (on-chain)	40.2% same-slot

Platform fees run ~1% on top of inflated priority fees and Jito tips — 3–6% effective on small trades. Competitor latencies are the bots' own marketing claims; mine settle on-chain. No Telegram bot reliably lands in the leader's slot — this one does, 40.2% of the time.

Bought vs builtOn Solana, landing position is normally bought: you tip validators through Jito, inflate the priority fee, or run a private validator on colocated hardware to jump the queue — that's what the competitors' $1–3.50 fees pay for. This engine takes the same position by optimization instead — 40.2% same-slot at a $0.001 tip, no VPS, no private validator, on a home connection for about $1,000/month all-in. It does with code what the field does with money.

DisclosedIncumbent bots advertise 400ms–3s as marketing claims; these numbers are on-chain over a 26-day window (Dec 2025–Jan 2026). Primary bottleneck identified and kept in the data: queue/setup phase, 50–90ms. Worst recorded pipeline outlier: 5.7s.

Poker decision quality · solver-graded head-to-head against frontier models, identical spots

Qwen3-4B exploit adapters

same-spot record vs Gemini 3 Flash — every model graded on the same hands by the same solver.

72–5

Per-street LoRA adapters on a 4B open model, trained on solver-graded contested spots — 856 converged postflop decisions, every model graded on the same hands by the same solver. Trained on one consumer 4090.

72–5

same-spot wins vs Gemini 3 Flash (119 ties)

128–93

vs Claude Opus 4.8 — and a winning record vs every big lab

+398 bb/100

live heads-up vs Llama-3.1-8B, 1,000 hands (CI +248 to +547; replicated +306)

6.7M

frontier-model decisions with reasoning traces in the training corpus

DisclosedOn mean EV-loss vs GTO this bot ranks 2nd-worst of 20 — by construction: the metric punishes deviation from equilibrium, and deviation is the exploit strategy. Its only head-to-head losses are to the other exploiters. Live vs gemini-2.5-flash: a statistical tie over 3,000 hands (+53 bb/100, CI −7 to +117). The pre-heads-up version lost at −121 bb/100 — that number stays in the record too.

DownstreamThe corpus built to train this became the instrument for The Say/Do Gap — 6.62M model-vs-model decisions where the hole cards, the reasoning trace, and the committed action are all visible at once.

Animal pose estimation · benchmarked against DeepLabCut SuperAnimal (EPFL)

26-keypoint canine pose model

mAP50 on the domain benchmark — a specialist beating the award-winning generalist.

.858

A custom 26-keypoint canine pose model, trained for a production behavior-telemetry pipeline that tracks dog and handler skeletons in the same frame — the interaction, not just the animal.

.858

mAP50, domain benchmark

26 kp

custom canine keypoint schema

skeletons tracked per frame: dog + human

DisclosedOutperforms the award-winning generalist model on this domain task — a specialist beating a generalist at its own specialty, which is exactly what specialists are for.

02 — Reported failures

28 → 63 → 26

Three legal-AI architectures that didn't work, scored and kept, before the 98.3% run.

~60%

Custom soil-probe detector precision — training-data limited (227 images, 80 labeled), and diagnosed as such rather than excused.

10 of 19

Frontier models that lose money at poker on the same public leaderboard — context for what "beats frontier" means and doesn't.

2019

A breakfast robot that worked until anything moved — months spent engineering the environment because the machine couldn't see. The failure that produced the worldview.

03 — Systems in operation

Pawly behavior telemetry

dual-skeleton · live

Consumer perception pipeline: pose, segmentation, and bark-event detection over ordinary phone video, with a certified trainer supplying ground-truth labels in the loop. Demand side grounded in a published 228k-discussion study.

Self-hosted semantic memory

86,942 chunks · sub-second recall

Semantic memory over two years of working sessions — local 4090 inference, queryable from any session. The continuity layer everything else runs on.

Garden rover

<200ms glass-to-glass

Garage-built garden rover: open-vocabulary detection, tracked segmentation, custom-trained probe detector, geometric behavior states — perception to display in under 200 milliseconds on a Raspberry Pi camera feed.

Unified inference daemon

owned GPUs · one HTTP surface

Pose, segmentation, depth, 3D body, face analysis, and embedding models consolidated behind a single local endpoint — the reason marginal inference cost across every project above rounds to zero.

04 — Writing

Empirical study · DOI 10.5281/zenodo.21495752 · 6.62M decisions

The Say/Do Gap of Frontier Language Models

How much a model's visible reasoning predicts its committed action — measured judge-free across 19 models and 27,750 traces, with match-grouped splits, permutation nulls, and bootstrap intervals. The link is strong for a few models, weak for many, and it is a per-model number nobody publishes.

Essay · DOI 10.5281/zenodo.20768819

Robots Aren't Ten Years Away

The bottleneck was never the body — it was perception, and it broke. Deployment timelines offered as falsifiable predictions, with a stated note on what the numbers are and aren't.

Empirical study · 228,753 discussions

What Dog Owners Are Actually Dealing With

A frozen-instruction LLM read over 87 public communities, with per-batch grounding audits (85–96% exact-match, reported not assumed) and a full statement of limitations.

salvo.systems · pawlytechnologies.com