# LAN Brain benchmarks

Measured performance, published method. Two experiments, run 2026-08-23.

## Experiment 1: MEME (memory that changes)

Public benchmark, dataset HF meme-benchmark/MEME, config nofiller, 100 episodes, 694 questions. One BM25 retriever for both arms, deterministic scoring, no LLM judge. The only variable is how the content is stored and how it is read. LAN Brain is read by descent, the way the product operates: an index or area hub that lists the entities with their value in force, and from there to the entity page.

| Category | Raw memory | LAN Brain (descent with index) |
| --- | ---: | ---: |
| Cascade | 1% | 100% |
| Absence | 2% | 74% |
| Tracking | 32% | 77% |
| Deletion | 83% | 99% |
| Aggregation | 22% | 100% |
| Exact Recall | 100% | 100% |
| Overall (694 questions) | 35% | 92% |

Without the index, the same curated pages read by flat search score: Cascade 85%, Absence 68%, Tracking 77%, Deletion 91%, Aggregation 1%, Exact Recall 100%, overall 72%. The index is what takes Aggregation from 1% to 100%, because the hub already lists the three entities of the question with their value in force.

The two limits that remain, stated. Tracking stays at 77% for a lexical reason: the question uses the plural and the entity is singular, so the retriever scores zero and the page never opens. With plural normalisation it goes to 98% and the total reaches 95%, the exact detection ceiling. And at scale the descent has to be able to open more than one hub: with the most pessimistic partition and a single hub Aggregation falls back to 0%, opening 3 hubs the total lands at 89%.

Change detection measured on the raw text: Cascade 100%, deletions 100% recall and precision, conditionals 100% precision, Absence 74%.

### Against what the market publishes

| System | Cascade | Absence | Deletion | Condition |
| --- | ---: | ---: | ---: | --- |
| LAN Brain (measured by us) | 100% | 74% | 99% | 100 cases, nofiller, descent with an index, deterministic reader, public harness |
| Sentra (self-reported) | 40% | 43% | 50% | no methodology, harness or config disclosed; sentra.app, consulted 2026-08-23 |
| Field average (MEME paper) | 3% | 1% | n/p | the paper's default config (filler noise, LLM judge) |

These rows are NOT directly comparable: each ran under different conditions, and only ours has a reproducible harness and a declared config. The MEME paper is under review and has not published its per-system table.

## Experiment 2: the same model, alone, on its raw sources, and with the Brain

50 real company questions: 45 with a verifiable answer in the Brain, plus 5 planted traps that have no answer anywhere. The pool spans decisions, dates, technical facts, facts that changed over time, people and organisations, commercial processes, and questions that need two hops. One model, answering three times: on general knowledge alone, with tools over the raw sources (the product wiki repo read the way a Drive connector reads it, plus the Notion CRM, read-only Gmail and the meeting notes, and no LAN Brain of any kind), and connected to the Company Brain over MCP.

| Metric | Model alone | Model + tools over raw sources | Model + LAN Brain | Honest reading |
| --- | ---: | ---: | ---: | --- |
| Accuracy (all 50) | 10% | 88% | 98% | 9.8x the model alone; against the raw sources the gap is 10 points, not a multiple |
| Accuracy on the 45 questions that have an answer | 0 / 45 | 40 / 45 | 44 / 45 | the raw sources are the real bar the Brain has to clear |
| The 5 traps, with no answer in the Brain | 5 / 5 | 4 / 5 | 5 / 5 | the raw arm lost one to a client email carrying discount bands the Brain never received |
| Hallucination | 0% | 8% | 0% | all 4 raw-arm inventions are faithful readings of a stale source, correctly cited |
| Output quality (0-3) | 1.16 | 2.84 | 2.98 | correct, specific and cited in the two arms that read something |
| Tool calls per question | 0 | 2.04 | 1.24 | the raw sources spend the first call deciding which of four systems holds the fact |
| Speed per answer | 2.0 s | 15.3 s | 7.1 s | only 1.2 s of the Brain's 7.1 s is our server |
| Total tokens | 4.3k | 88k | 160k | measured call by call for the Brain arm; the raw arm's connector reads are estimated |
| Tokens per correct answer | 857 | 2,004 | 3,260 | the model alone spends less only because it answers nothing real |

What the raw-sources arm changes about the claim: the honest statement is NOT "without a Brain your AI does not know". The raw sources also know, and they score 88%. What the Brain buys is 8% hallucination against 0%, and all four of the raw arm's inventions are FAITHFUL readings of a source that had gone stale (a role in the CRM, a country field, a title in an email signature), each one cited correctly. Curation is the delta, not access. It also buys 2.2x the clock, 7.1 s per answer against 15.3 s, at 1.6x the tokens per correct answer.

Two caveats that run the other way, stated. Fairness: these questions are answered out of a tidy markdown repo where the filenames are already the answer, which is friendly ground for a raw connector and is not what a typical corporate Drive looks like, so the raw arm is read at its best. Scoring: one of its four inventions is a trap where a client email carried discount bands that never reached the Brain; the strict rule of this series counts any concrete claim on a trap as an invention, and scored the other way the raw arm is 90% with 6% hallucination. The strict figure is the one published, so the criterion never changes mid-series.

The cost framing, plainly: the model alone spends 857 tokens per correct answer, and every one of those correct answers is a trap where "I do not know" was the right response. On the 45 questions that do have an answer it scored zero, so its cost per useful company fact is not 857, it is infinite. Against the raw sources the trade is real and small: 3,260 tokens per correct and cited answer versus 2,004, in exchange for half the clock and none of the inventions.

Where the extra time goes: of the Brain's 7.1 s per answer, 1.2 s is the LAN server and 5.9 s is the provider model inference plus the client network. Server time is measured in production call by call (62 calls over the 50 questions, 62,119 ms total, median 872 ms, p90 1,468 ms). The rest is obtained by subtraction and is a decomposition, not a measurement: provider inference and client network never reach our logs. The raw-sources arm's 15.3 s gets no decomposition at all, because none of it passes through our server.

Five runs, not one, all on the same day, with the server improving between them. Run 1: 20 questions against the pre-optimisation server, 70%, tokens estimated from character counts. Run 2: the pool rebuilt at 50 harder questions, 36%, which exposed the retriever rather than the Brain, because the content was there and search kept handing back index and department pages instead of the page that answers. Run 3: search returns 8 results instead of 20 and includes the matched excerpt, the generated map pages are demoted in the ranking and kept out of the excerpt budget, and the tools teach the navigation route of listing the folder and opening the page by path; 94%. Run 4: the Brain injects its context pack at the start of the session, halving the calls and the cost; 98%. Run 5 is the one published here: the same 50 questions with the index in router form, which is what every tenant runs today, so the pack carries departments with rules instead of a catalogue of page names. Accuracy did not move a point and the cost went up 13%, because what the catalogue gave away for free now has to be bought back by listing folders. Run 1 estimated its tokens; every run since measures them call by call.

The raw-sources arm ran on the same 50 questions with the same scoring rule: no MCP tool, no hosted Brain page, no web. Client-sized budget of about five or six calls per question, 2.04 used on average, 102 in total. It answered 32 of 32 on everything that lives in the repo (decisions, ship dates, architecture, two-hop questions, commercial processes) with no inventions among them, and broke on people, 2 of 6, with 3 of its 4 inventions there. Its filesystem reads are exact character counts; its Notion, Gmail and meeting-note reads are estimated, and its clock is 527 s measured plus about 240 s estimated.

Caveats, stated: internal run, N=50, one model, rubric applied by the same system with mechanical criteria rather than an independent judge (the mechanical fact recall of 95.7% for the Brain arm and 82.0% for the raw sources does not depend on it), the "model alone" arm is a controlled blindness simulation rather than an API call with an empty context, frozen from the second run because no server change can move it, the raw-sources arm is read at its best on friendly ground and part of its token count is estimated, and one specific to the Brain run: the client had read the write ups of the earlier runs, so which page it chose to open may have been informed by them. That bias runs in favour of run 5, so the cost figure published here is a floor, not a ceiling.

Full page with charts and the complete method: https://www.lanbrain.ai/en/benchmarks
