What we actually did, day by day — compiled from the project’s session history and commit log, not just the experiment record. The research log and model designs pages show what the loop tried; this page is the surrounding story.
Bootstrap — the pipeline, the harness, and the first real experiments
The repo is bootstrapped end-to-end in one
session. Against the project spec, the frozen bbox-generic data
pipeline, a naive baseline model, SQLite lineage, an HTML gallery, and
loop.sh are scaffolded in one pass and proven with a real
train→score→log run, including confirming the Hamburg holdout
stays untouched.
Two open questions get resolved before more code is written. The deployment hardware: an ESP32-P4-Function-EV-Board paired with the SmartSens SC2336 sensor turns out to be Espressif’s own documented reference pairing. The reference imagery: 10 m/px Sentinel-2 is swapped for 1 m/px open state orthophotos, matching a real ~100 m-altitude camera footprint, and the relighting pipeline is rebuilt for the larger rasters. A v2 baseline scores 2765.67 m.
The dashboard is rebuilt in the project’s house
style. It’s restyled to the same Tufte research-log language used
across earlier projects, then pushed through three more rounds of feedback
toward plain-language framing and precise miss-distance wording.
loop.sh is hardened for safe interruption — a clean stop
on Ctrl-C or a typed exit — and the train/eval split is
rebalanced to 360 m blocks after the original 144 m blocks were
starving some areas of eval crops.
A clean-slate reset breaks the project’s first real plateau. The lineage resets to a clean baseline, and the first post-reset experiment breaks the ~3.2 km “predicts the map center” floor that nine prior designs had all converged on — a soft-argmax probability field, kept at 2489 m. A concrete plateau rule follows: after three consecutive reverts, a design must leave the refuted mechanism family entirely.
Every experiment now gets its own hand-drawn architecture diagram. The design agent begins drawing one per experiment, under a visual contract distinguishing frozen, current, and just-changed stages. Experiment 8 pivots to a residual encoder and is kept at 2326.54 m, closing out the day.
Infrastructure moves to a rented GPU; the public site takes shape
An overnight diagnosis finds a real blind spot in the model’s field head. It has only ever seen a spatially-averaged descriptor with no positional information, and the loop tests a layout-aware fix; a pretrained MobileNetV3-Small backbone is explored in parallel, resolving the from-scratch-vs-pretrained question directly. Progress is slow — the loop repeatedly hits a connection error and restarts the same iteration from scratch several times before anything lands.
The pod’s database becomes the one production record. Now that compute costs money on a rented RunPod RTX 4090, the policy is set explicitly: the laptop only ever pulls from it. The rule is encoded in the sync scripts themselves, not left as convention.
A new gallery page ships for every proposed architecture. “Inference Paths” is matched to the typography of the researcher’s existing personal site, along with a frozen-contract figure showing the fixed camera-frame input and output every design shares, its coordinates anchored so every architecture diagram scroll-compares cleanly against it.
Four experiments land in a row, each a real improvement. 1856 m → 1638 m → 1369 m → 1095 m → 1055 m. The loop is also restructured around two agent roles — one designs an experiment, a second implements it — with per-phase timing logged for each.
The project goes public. The site and a matching entry on the researcher’s personal homepage ship together, with a tagline chosen after two rounds of brainstorming, and an airloom-style “view on GitHub” ribbon added to every page.
A plateau-enforcement mechanism that took two real fixes to actually hold
An overnight disk leak crashes the loop. A training run has been silently caching several gigabytes of rendered images per experiment into a directory the loop was committing, filling the pod’s disk. Fixed by excluding scratch renders from persistence, and the loop is relaunched.
The pivot check itself turns out to be broken. The loop runs through the night with no net progress, and by midday the pivot check turns out to be testing the wrong signal entirely — a declared-category heuristic rather than whether the frozen architecture actually changed — so seven consecutive “pivot” experiments have in fact left the same pretrained backbone untouched.
Two real bugs, not the prompt wording, were blocking every pivot. A first fix — a diagnostic naming which stages hadn’t changed — proves insufficient: its patience threshold turns out to be hardcoded inside the checker script, ignoring the loop’s actual setting, so the intended stricter check silently doesn’t apply at the moment it’s needed. A second, code-level gate replaces it, checking the real backbone-construction diff before training runs at all. Both root causes are found and fixed together; this is the version that holds.
Monitoring and budget expectations get set explicitly. An hourly automated health check posts to a GitHub issue only when something is actually broken. Budget context is set: roughly 100 experiments is the real ceiling, and the loop should not self-stop at the target metric — only when budget runs out or the work is stopped by hand.
Two infrastructure incidents get resolved in one day. A disk-exhaustion failure in the build pipeline, and a git-history divergence between the pod and GitHub after both independently applied the same fix. Policy tightens: every repository change now happens on the pod exclusively; the laptop becomes read-only.
Nineteen reverted pivots, a forced redirect, and compute moves off the rented pod
A ten-hour GitHub outage gets root-caused and fixed. An oversized quarantine folder of rejected renders had been accidentally committed; excluding the path structurally fixes it for good. Loop vocabulary is also overhauled — “iteration” becomes “experiment” throughout — and every experiment row gains a deterministic, plain-language reason for its outcome.
The comparison baseline resets to a faster, fairer standard. The champion architecture, retrained on Berlin alone at a fast budget, scores 1176.5 m, retiring the older, heavier four-area 743 m champion as historical so quick local experiments are judged fairly. RunPod branding is dropped from the site as compute moves back to local hardware.
Four forced pivots in a row all fail the same way. Each is required to avoid the champion’s banned backbone entirely — a residual VQ location codebook, a Gaussian-mixture regression, a pretrained ShuffleNetV2 trunk, a memory-safe learned-relighting redesign — all reverted. Across nineteen straight reverted pivots, one pattern holds: decode-mechanism rewrites keep losing, while the one round that changed the trunk came closest. The encoder, not the decode head, is the real bottleneck.
A human-forced redirect finally lands a genuinely new architecture. A from-scratch dense scene-coordinate regression design is forced directly into the next experiment, and a one-shot forced-direction mechanism is added to the loop so a specific redirect can be handed to a single run without further intervention. It scores 2360 m — architecturally novel, still reverted.
Compute strategy pivots twice in one evening. Off the rented GPU pod back to local hardware, then toward a scale-to-zero serverless GPU provider alongside a self-hosted always-on worker option — removing the recurring overhead of a persistent rented pod.
A four-experiment losing streak, still short of the 743 m champion
Figure-checking gets stricter. Mechanical checks for the architecture-diagram style contract land — palette, font, stroke-width, gradients/icons/emoji, red-consistency — then the SVG arcs get flattened so the line-crosses-label check can actually see curved elements.
Experiment 60 retries exp 38’s illumination-invariant channel with a from-scratch trunk and a cross-lighting consistency loss. 963.42 m — closer than most recent attempts, still reverted against the 743.07 m champion.
The forced-pivot rule fires twice more, both
gated failures. A quantization-freed depthwise-separable trunk with a
unified single-head field (experiment 61) fails outright. Patience
spent again, the champion’s mobilenet_v3_small trunk is
banned outright and a Fire-module SqueezeNet1.1 + FiLM-conditioned field
(experiment 62) is forced under that ban — also a gated
failure.
A real training-speed bug gets fixed mid-streak. Bucket/realization images were being re-decoded from disk every epoch — profiling shows this was ~78% of per-area wall-clock. Caching the decode once per process is the whole fix.
Experiment 63 — another forced pivot under the mobilenet ban, this time a from-scratch trunk with domain-contrastive pretraining, a fixed retinex channel, D4 symmetry, and top-K-masked decode — scores 1074.76 m. Four straight reverts; the day closes with the same 743.07 m champion still standing.
Three measurement bugs found in one day — the evaluation, the score, and what the harness kept — and everything measured before today invalidated
The human proposes deliberately narrowing scope. After weeks of ~50–65-minute iterations optimizing a worst-case across 4 areas × 6 lighting buckets, still far from the 20 m target, the idea: one locale (Berlin), daytime imagery only, treat “overfitting to one place” as a feature rather than a failure mode, in exchange for much faster rounds — and re-derive a fair accuracy milestone from what the actual image resolution can support, rather than assuming the original number still applies.
The plan gets grounded in real numbers before any code changes. The champion’s own Berlin-only daytime score (~650–720 m) sets a realistic near-term milestone at 100 m, not the original 20 m (~35× off). Per-experiment timing data shows the real speed levers aren’t what they first seem: the agent-drawn architecture figure (~15–20 min, the single largest non-training phase) and the six lighting buckets concatenated into one training set are the actual costs — dropping to one area is not, since areas already train in parallel.
The berlin-slim branch is created and the whole slice implemented in one pass. Relighting collapses to a single as-is daytime bucket (no synthetic ambient/gain/noise simulation), areas scope to Berlin only, and the pivot-gate, holdout check, and figure-drawing step are disabled as reversible one-line toggles, not deletions. The target milestone moves to 100 m everywhere it’s referenced in the gallery. Design and implementation are reassigned to Fable and Opus 5 — a deliberate reversal of both models’ standing bans, flagged before proceeding rather than silently applied.
Two smoke tests catch real bugs before a live
run. A hardcoded midday.png heatmap background (missed by
a substring grep) and a training-time realization generator that
would’ve kept applying synthetic sensor noise despite relighting
being “disabled” both get fixed and reverified.
A clean restart, at explicit request. The
champion architecture — shaped by 63 rounds against the old
4-area/6-bucket problem, including machinery like a dark-lighting expert
head that no longer applies — is retired in favor of the original
from-scratch TinyLocNet baseline. The full experiment lineage
is wiped, not just the comparison metric, so berlin-slim’s research
trail starts at Experiment 1 with nothing to live up to; the old
62-row history is archived, not deleted.
A question about one roundabout unravels the whole evaluation. Looking at Berlin’s error map, the human asks why the Großer Stern — a seven-way star roundabout that exists exactly once in the city, about as visually unmistakable as a landmark gets — is surrounded entirely by red dots. If anything should be memorable, that should be. Checking the frozen split directly: zero training crops contain that roundabout, and twelve evaluation questions do. It sits three pixels from the edge of its 360 m block, and the block next door is an eval block — so the anti-leakage buffer had deleted every single training view of it.
The evaluation had been asking the model to locate places it was never shown. The one-in-five block holdout, plus a 92 px no-leakage buffer around every eval block, means 28.2% of Berlin never appears inside any training crop — and 100% of the 15,585 eval questions are centred on ground the model has never seen. Worse, for 65% of them the entire 128 m frame contains not one familiar pixel; the median eval frame is 0% familiar ground. Two independent derivations agree: from the geometry alone, a train crop’s content stops 28 px short of any eval block, so only eval centres within 36 px of a block edge can see anything familiar at all — predicting 64.0% blind, against 65.1% measured. This is the likely reason roughly sixty experiments plateaued between 700 and 1000 m: for most of the test set the question is not hard, it is unanswerable.
The mismatch, stated plainly. This branch was founded on treating overfitting as a feature — build a visual memory of one city. But the metric was built to defeat memorisation: withhold regions, then grade the model on those regions. The architecture is a memory system; the ruler was grading it as an extrapolator. It is also stricter than the prior art the spec cites: DSAC and ACE evaluate on held-out camera trajectories through a fully mapped scene, not on unmapped parts of it. And in deployment the aircraft only ever flies over the box the model was trained on, so the metric measured a capability the product never needs while never measuring the one it does. The loop is stopped mid-experiment; the split is being redesigned to hold out viewpoints rather than regions.
The split is rebuilt around viewpoints. Training now covers every lattice position of the whole city — the model is supposed to memorise its one box, so it is shown all of it. Evaluation moves off-lattice: each test frame sits 11–17 m from the nearest training vantage (17 m being the largest offset a 24 m lattice permits) and carries its own rotation. Mapped ground, unseen view — which is what an aircraft over a mapped area actually faces. A small one-in-thirty-two-block region stays genuinely untrained as a logged-only diagnostic, never allowed to touch the metric, to catch a model that is a lookup table with no spatial structure.
Measured again, on the same instruments. Berlin ever shown in training rises from 71.8% to 93.5% (the remaining gap is the deliberate diagnostic region and its buffer). Eval questions centred on ground the model has seen: 0% → 100%. Eval frames containing no familiar pixel at all: 65.1% → 0%. Distance from each test frame to the nearest training vantage: 11.3–16.3 m. The lineage resets a second time in one day — scores from the old ruler cannot be compared to the new one — and the five experiments run that morning are archived rather than deleted.
A second measurement bug, found the same way as the first — by looking at the map. The human asks how an experiment whose error map is dotted with green hits can be discarded, while one with a single green dot in five hundred is kept. It is not the map that is wrong. A model that guesses near the map centre produces a tight, mediocre, unimodal error distribution — a decent median and no good tail. A model that genuinely memorises places produces a bimodal one: some spots nailed, the rest badly wrong when it misidentifies. The median rewards the guesser and punishes the learner. Experiment 2 located 8% of Berlin to within 100 m — the baseline managed 0.0%, experiment 3 managed 0.2% — and was reverted for a slightly worse median, while experiment 3, a marginally better centre-guesser, was kept. Tested against every candidate statistic, the median was the only one that ranked experiment 2 last; p25, p10, geometric mean and hit-rate all ranked it first.
The score becomes the geometric mean. Chosen over a percentile because it reads every frame rather than one cutoff, and because on an error spanning two orders of magnitude the log scale is the natural one: pulling a single spot from 2 km to 50 m — actually memorising it — now moves the number meaningfully. Median, mean, p10, p25 and a product-facing hit-rate are all still recorded, just no longer optimised. This time no lineage reset was needed: the evaluation data never changed, only its summary, so all three experiments were rescored from their saved weights. Corrected, the history inverts — experiment 2 becomes the champion at 1323.6 m, experiment 3 the revert.
And the correction exposes a hole in the harness.
Experiment 2 was the best implementation the project had — and it no
longer existed. Reverting an experiment ran git checkout -- model/,
which deleted its source outright; only kept experiments ever entered the git
trail. That is safe only if the scoring metric is correct and permanent, an
assumption nobody had written down and which had just been proven false twice
in one day. Of the 60 archived experiments, the 13 kept ones survive in git;
the 47 reverted ones do not, though their design briefs and trained weights
remain, which is how experiment 2 was rebuilt. Four separate code paths
could destroy an implementation; all four now snapshot it first.
“There is ONLY the product requirement.” Pressed on why the score was a statistic about errors at all, the answer does not survive contact: the geometric mean had been chosen to make research progress visible, which is the same species of mistake as the two bugs already found today — optimising something adjacent to the goal instead of the goal. So the metric becomes the goal. The aircraft takes a vision fix every 5–10 s and it is its only drift correction, so per frame exactly three things can happen: confident and within 100 m is a usable fix; not confident is an abstention, which is safe because it simply waits; confident and wrong is a false fix, which feeds a wrong position into navigation and is worse than saying nothing. mission score = (1 − usable-fix rate) + false-fix rate: zero is perfect, 1.0 is abstaining on everything, 2.0 is being confidently wrong on everything. §6 had asserted since day one that honest abstention beats confident guessing; no metric until now actually encoded it.
The baseline’s new score is the argument for the change. The same naive model that the old rulers reported as “1993 m median” — a number that sounds like a measurement — scores 2.001: it is confident on every single frame and wrong on every single one. Zero usable fixes, a 100% false-fix rate, the worst value the scale can produce. Nothing about the model changed; only the question being asked of it. Error statistics stay logged as diagnostics, and the standing check is written into the scorer: if the score improves while the usable-fix rate does not, the metric is wrong again. The lineage is wiped a third time and the metric propagated through every prompt, document and page.
What changed, and what it costs. Three separate things were found wrong today, and all three were found the same way — by a human looking at a picture and saying that makes no sense, not by any check the harness ran.
1. The evaluation asked an impossible question. Holding out one map block in five, plus a no-leakage buffer, left 28.2% of Berlin in no training crop at all and put 100% of test questions on ground the model had never seen; 65% of test frames contained not one familiar pixel. For a system whose entire job is to memorise one city, that is unanswerable by construction. Fixed by holding out viewpoints instead of regions: train on all of Berlin, test on frames 11–17 m off the nearest training vantage.
2. The score rewarded the wrong behaviour — twice. The median favours a model that guesses the map centre (tight, mediocre, unimodal) over one that genuinely memorises (bimodal: some places nailed, others badly missed). It was caught reverting the only experiment that had begun to memorise anything. Replacing it with a geometric mean fixed that symptom but was still a research proxy chosen to make progress visible rather than to state what the aircraft needs — the same mistake in nicer clothes. The score is now the product requirement itself: usable fixes minus dangerous ones.
3. The harness was deleting its own work. A reverted experiment had its source discarded outright; only kept experiments entered the git trail. That is safe only if the metric is correct and permanent, an assumption nobody had written down and which had just been disproved twice in a day. Of 60 archived experiments, the 13 kept ones survive; the 47 reverted ones do not.
What this means for everything before today: the record is void. Every experiment up to this point — roughly sixty-three of them, including the 743 m champion the project chased for a week — was scored on an evaluation that was largely unanswerable, using a statistic that preferred guessing to learning. Their verdicts do not transfer. “Reverted” mostly means “did not help on an impossible test”, and “kept” may mean “guessed the centre more tidily”. The lineage was therefore wiped rather than carried forward, and the old databases archived; the design agent is deliberately not given the old conclusions, because importing invalid refutations would steer it away from ideas that were never actually refuted. The catalogue of approaches tried remains readable by a human, but nothing about which of them worked survives.
And the plateau finally has an explanation. Sixty-odd experiments sat between 700 m and 1 km, and it was read as the problem being hard. On the corrected evaluation the naive baseline scores 2.001 — confidently wrong on every single frame, the worst value the scale allows — while the one experiment the old metric threw away had already located 8% of Berlin to within 100 m. The work was not stalling because the task was hard. It was stalling because the instruments were pointed at the wrong thing.
The champion is pointed at rural Prignitz — a measurement, not an experiment. Experiment 5’s code, unchanged, trained and scored on the rural bounding box instead of Berlin: same architecture, same eight epochs, same hyperparameters, only the box is different. It needed no code change at all — the cell grid comes from the raster (Berlin 55×54 = 2,970 cells, Prignitz 55×55 = 3,025), which is the genericity requirement of §0 being exercised rather than asserted. Result: 0.113 against Berlin’s 0.040.
The interesting part is where the 2.8× went. The prediction beforehand was that fields would be indistinguishable and the model would abstain more — the safe failure. It did not. Abstention barely moved (3.0% → 3.8%) and usable fixes barely moved (96.5% → 92.5%); almost the entire loss is false fixes, 0.5% → 3.75%, a 7.5× rise in exactly the outcome the mission score treats as worse than silence. Median error is nearly unchanged at 29.5 m, so when it is right it is just as sharp: the gap is calibration, not precision. Belief pooling protects against mass split across adjacent cells and against mass scattered across distant ones, but not against mass landing confidently on one wrong field that looks exactly like the right one — a case Berlin never presents and Prignitz presents constantly. The unseen-region diagnostic says the same thing from the other side: coverage rises to 64.2% (Berlin 38.2%) while staying 0% usable, i.e. on ground it has never seen the rural model answers more readily and is wrong every time.
Nothing from this run was written to the experiment
database, state/best.json or the git trail. Scored against a
Berlin champion it would have been logged “reverted”, which would be
a meaningless verdict — the two areas are different questions, not
competing answers to one. It is recorded here and as a branch on the
evolution graph instead. The research
question it names is sharper than Berlin’s was: cut false fixes
without spending usable ones. Sharp enough that within the hour it had an
era of its own.
One model over both boxes: proposed, examined, dropped — all before an experiment ran. The idea was to stop letting each area find its own setup and instead iterate toward a single model covering Berlin and Prignitz, amending §1’s one-model-per-bounding-box rule. It survived about an hour. The arithmetic was encouraging: the boxes sit 90 km apart, so their union is 88×71 km of which only 1.6% is imaged — a union grid would be 384,196 cells, nearly all of it ground no camera has seen — while a grid over just the two imaged boxes is 5,995 cells, barely twice Berlin’s. It also needed no frozen file changed: the scorer loads one ONNX per area, so a shared trunk exported twice, each framed to its own raster, fits the existing interface. Dropped anyway, on the grounds that unifying is a different question from the one Prignitz actually poses, and running both at once would confound them.
That hour turned up a real measurement bug, left open
until 1 November (issue #8). Checking whether a shared model would fit the ESP32‑P4
gave 4.88 MB against a 4 MB ceiling — an apparent blocker. It
wasn’t. MODEL_MAX_BYTES was set in the bootstrap
commit and never revisited — git log -S returns exactly
one commit — with no derivation behind it. And it weighs the fp32
ONNX, while the deployment path this repo documents is esp‑dl, which
ingests that ONNX for int8 quantisation. Measured: the Berlin champion
is 2.94 MB fp32 and 0.78 MB int8. The gate overstates every
model by about 3.8×, and the shared model that “failed” is
1.29 MB quantised. Nothing in the pipeline quantises anything, so
§3’s “quantization-aware training” is a line in the spec
no experiment can currently act on. Deliberately not changed on the
spot: it is a frozen file, and the re-scoring that puts every past era on one
ruler runs the current scorer, so relaxing the gate could flip several
of the 17 archived gated-fails at once. That deserves its own step, with the
effect measured first.
A rural era opens — Prignitz alone.
AREAS=prignitz, everything else about the narrow configuration
unchanged. The era’s question is not Berlin’s. Berlin had to prove
the approach worked at all; Prignitz already half works at 0.113, and the
entire gap is in one place, so the question is exact: cut false fixes
without spending usable ones.
Experiment 1 — the seed. The champion architecture, no design agent, trained and scored on Prignitz natively so the era starts from a number on its own board rather than a probe’s. 0.113, reproducing the probe exactly: 92.5% usable, 3.75% false, 3.8% abstaining, median 29.5 m.
Experiment 2 — rotation-consensus decode. Refuted, and informatively. Training draws a fresh random heading per position per epoch, so a genuinely memorised place ought to be heading-invariant while a false match to a look-alike field should ride view-specific micro-texture that does not survive being turned. The decode was changed to run the trunk over the frame plus its 90/180/270° rotations, average the four belief maps, and decode the average — a wrong field would have to win from at least two headings. Predicted false fixes 3.75% → ≤1.5%. Measured: false fixes went up, 3.75% → 4.25%, usable 92.5% → 92.0%, mission 0.113 → 0.123. Reverted. The negative is worth more than the change would have been: the rural confusions are rotation-stable. Two fields that fool the model fool it from every heading, so this is not a viewing-angle artefact and consensus at decode time cannot reach it. Whatever fixes it has to act during training.
Experiment 3 was designed and implemented on exactly that reading — continuous-vantage training: jitter each training framing by up to ±12 px so vantages tile the ground instead of standing on the 24 m lattice, since every eval view (and every real flight frame) stands 8–16 px off-lattice, and a model that has only ever seen lattice framings can tell fields apart by accidents of framing that vanish a few metres away. It never trained. Recorded as the era’s open move, not as a result.
The rural era is rewound after four hours and three experiments; the trunk goes back to berlin-slim
The era is closed and the repository rewound to the
commit before it opened. model/, state/best.json
and the experiment database all return to the Berlin champion at 0.040,
and the two Prignitz rows are removed from the lineage rather than archived as
an era — four hours and three experiments is a detour, not a chapter, and
carrying it as one would put a band on every chart in this site for work that
produced no kept result. Not a verdict on the rural problem: nothing about it
failed on its merits, and the 0.113 measurement and the rotation-stability
finding both stand. The reason is simpler — one question at a time, and
the question is still Berlin.
What survives the rewind, deliberately: this entry; the
detour drawn as a branch that leaves the berlin-slim line and stops, on the
evolution graph; and the open finding
about the size gate above. The raw artifacts — run directories, exported
models, agent transcripts for all three experiments — are not deleted
either; they stay reachable at the git tag
prignitz-detour-record. The rule the project already had for
wiped eras applies here too: re-measure or record, never quietly
erase.
The flight test — a virtual UAV flies the champion from a to b
The champion is put to the product’s own
test. A flight simulator (sim/flightsim.py, outside the
frozen pipeline, reading it untouched) flies a point-mass fixed-wing
through wind, gusts and gyro bias it cannot see, with the exported
champion’s vision fixes as the only position correction —
camera frames are real orthophoto crops at the true pose, thresholds are
the frozen scorer’s own, false fixes are accepted like any other,
and arrival is declared on the navigator’s estimate but scored
against the truth. The commitment, fixed before the first run: whatever
the number is, it ships.
The first Monte Carlo scores 53/100 — and the bug is in the simulator, not the model. Flights with perfect position fixes still spiral off the map: the hand-built navigator had no heading reference, so gyro bias integrated unbounded and guidance chased a believed heading 100°+ from the true one. Every real flight controller carries a magnetometer; adding one (yaw aiding only — vision stays the sole position source) takes the result to 91/100 flights arriving within 100 m, median declared miss 45 m. The control run — same flights, vision disabled — arrives 0/100.
The failures decompose cleanly. 167 of 207 in-flight false fixes stand on the ~3% of Berlin the harness deliberately never trained (the region-holdout diagnostic blocks); all three catastrophic misses had their target on or beside that ground. Excluding it, the in-flight false-fix rate is 0.77% — consistent with the frozen eval’s 0.5%. A deployed model trains on 100% of its box, so the catastrophes are an artifact of flying over the research configuration; they are reported as flown on the new flight path page, which replays a corner-to-corner 9 km flight — one 6.5 km false fix survived, arrival 25.5 m — over the Berlin orthophoto.
The champion knew a photograph, not the city — and a sixth era opens on a testing branch
A lunchtime question becomes a measurement. Are Google’s satellite tiles different from the ones we trained on? They are — a different survey, season, sun angle and colour pipeline — so the champion was scored on them, warped onto its own grid as a local diagnostic (never data: Google’s terms forbid it). Mission score 1.971 against 0.040: usable fixes 0.2%, confidently wrong on 97%, median miss 3.5 km, which is what a random guess over the box scores. Coverage stayed at 97%: the model did not know it was lost. Its 390 confident answers landed on 40 map cells, two of them absorbing 225.
It is the model, not Google. Perturbing the champion’s own training raster and re-scoring: shifted 3 px → 0.040, unchanged; JPEG at quality 60 → 0.498; a 1 px blur → 1.271; +20 added to every pixel value → 1.453. Rotation was the only augmentation it ever saw, so it had memorised pixel intensities and fine texture — a fingerprint of one file. Colour-matching the foreign photograph to the training one recovered abstention (coverage 97% → 61%) but not accuracy (median still 3.1 km): intensity is what the confidence head keys on; the fix itself rides texture and shadow, which change between surveys.
The legitimate second photographs exist, and are open. Berlin’s geoportal serves TrueDOP 2022, 2023 and 2024 as separate services under Datenlizenz Deutschland Zero. Fetched onto the same grid (alignment ≤ 1 px) they break the champion just as Google did: 1.866, 1.776, 1.873. Every photograph of Berlin other than the one it memorised gives the same result. The same-photograph evaluation could never see this, and the loop only fixes what its ruler measures — a photometric-augmentation experiment scored the old way would have come in a hair above 0.040 and been reverted.
The capture-holdout era opens, on a testing
branch. The frozen pipeline gains captures: several independent
photographs of one bounding box on one grid. Berlin trains on three (the
Brandenburg DOP20 the champion knew, TrueDOP 2022 leaf-off, TrueDOP 2023
early spring) and is scored on TrueDOP 2024, which it never sees in any
form. Same viewpoint holdout, same product metric; the new axis is “the
photograph is new” on top of “the view is new” — the
aircraft’s camera being a photograph nobody has. The training
photographs are scored too, logged only, so every experiment records its
memorisation-vs-ground gap. The lineage is wiped, the mission era archived,
and all 83 earlier experiments re-scored on the new ruler: best 1.231,
reached by abstaining on 77% of frames. main and the published
site stay untouched until the era beats its own baseline.
Baseline: 1.651 — and a story already in the diagnostics. The champion’s code, trained on three photographs, scores 1.651 on the held-out one (3.3% usable, 68% confidently wrong). On the three it was shown: 0.015 on TrueDOP 2023, 1.401 on 2022, 1.533 on the Brandenburg raster. It memorised only the last photograph of each epoch: the training loop makes one full pass per raster in turn, so each pass overwrites the previous one. Left for the loop to find.
The loop restarts, locally, and teaches the model three photographs at once
A month on hold, then go. The laptop was running hot and needed; the Modal credits had gone to another project. The design agent moves to Fable 5.1 and the implementer to Opus 5.5, both verified headless. The design brief now reads the re-scored history as a map of the search space with the old verdicts deliberately omitted — every idea refuted on the old ruler is open again on this one. Five experiments targeted, about an hour each on the M1.
Four pages had not noticed the era. The chart, the lineage, the overview and the model-designs page drew the current era from a record rebuilt only when history is re-scored, so the new era had a band of zero width while the research log moved on; the log itself numbered the new baseline with the previous era’s prefix. Every page now reads the live era from the experiment database on each render. Caught by the human looking at the pictures, not by a test — again.
Reuse is encouraged, and Berlin is the only customer. Two additions to the design brief from the human. Old ideas were already “open again”; now the agent is invited to lift them, told where every earlier design and its source survive, and asked to name what it builds on. And a technique that works for Berlin and nowhere else counts as complete success. A deep-research pass on JEPA-style world models came back the same morning with one usable lesson: not an architecture but a training discipline, predicting the representation of a related view — and the harness holds the richest related view any of those papers had, three pixel-aligned photographs of the same ground. Its ranked shortlist went into the design agent’s library.
The forgetting is fixed in one move. Experiment 2 trains every step on the same places in all three photographs at once, instead of one photograph per pass. The model now knows all three: 95.2%, 96.0% and 94.2% usable on the training photographs, where the baseline knew only the last one it saw. On the held-out 2024 photograph: 1.651 → 1.583, usable 3.3% → 7.0%, still confidently wrong on 65%. Knowing three photographs pixel by pixel does not give the fourth. The gap between those two numbers is now the whole problem.
The research shortlist’s first pick fails; the plainest idea works. Experiment 3 asked the trunk to describe a place the same way in every photograph (the shortlist’s first item) and scored 1.648, reverted. Experiment 4 re-photographs every training view at random — brightness, contrast, gamma, colour balance, blur, sensor noise — so the fingerprint of one survey stops being a usable solution: 1.271, kept, false fixes 65% → 39%. Experiment 5 widened the trunk’s view to the whole 128 m frame: 1.298, reverted, though it won back much of the memorisation on the training photographs that experiment 4 had cost. The held-out number did not care.
Two display rules. The chart’s dots stopped showing the keep/revert verdict made at the time: black now means the dot lowered the running best on today’s ruler, and the old verdict moves to the tooltip. And the architecture figures, switched off in July for cost, came back — but only for experiments that became the new best. That second rule lasted a day.
Every experiment gets its figure; one experiment takes three hours, and the code is the reason
Every experiment is illustrated, reverted ones
too. The human wanted all of them drawn; experiments 3, 4 and 5 were
backfilled and DRAW_FIGURES=1 became the default again. The
backfill found a hazard: a figure agent ran an archived reference script
whose database write was unconditional, and it silently replaced this
era’s figures for experiments 1–4 with mission-era drawings.
Restored from each run’s own record; the write now needs an explicit
flag. The chart also gained a thin dashed line per era, that era’s own
running best, kept “for now”.
Hours per experiment, and the M1 overheating. Three photographs per epoch tripled the work, and the human asked for a cheap way to train elsewhere. The research answered a different question first: the training code used about 3–4% of the M1’s GPU. Every crop was rotated in PIL on one CPU core, and each step stopped the GPU up to eight times to read numbers back. That was true of every experiment of the era, not one; the loop never noticed because it scores accuracy, not wall-clock time. Without fixing the code any cloud GPU would buy only 3–5×. Recommended service: AWS, a GPU machine created and destroyed per run, about $0.10–0.60 an experiment.
Experiment 6: blending the surveys. Every training view became a per-pixel mixture of the three photographs of the same place, so the model would learn the continuum between surveys. 1.288, reverted, after 174 minutes on the laptop.
The code fix lands between experiments. Training now keeps the three photographs on the GPU and cuts and rotates each batch there, matching the frozen pipeline’s crops within 1/255. A full retrain of the champion: 6.5 min per epoch against 12.8, and 1.246 against the original 1.271, inside run-to-run noise. Part of the old 12.8 was the fanless laptop throttling on long runs. The AWS account was set up the same evening; a new account has no GPU quota, so a request was filed at 18:14 and approved at 20:49.
Training moves to AWS, two frozen rules turn out to be vetoing the best models, and borrowed eyes take the score from 0.96 to 0.16
Ten times faster for fifteen cents. The first AWS run retrained the champion on an L4 GPU in 305 s, against 52 minutes on the M1: 9.6 minutes door to door, about $0.15, same score. Only the three training photographs are uploaded; the older Modal path, which had shipped all four, was fixed the same way and now refuses to run if the 2024 photograph is among them. A Modal A100 was faster still (151 s, $0.09), and three AWS GPUs were benchmarked: the biggest trained only 1.4× faster than the L4, so this small model is bound by per-step overhead, not the card. Decision (Alexis): AWS is the primary backend; Modal stays a backup inside its $30 monthly free credit.
A committee halves the false fixes; four ideas about invariance do not move the held-out photograph. Experiment 7 trains two localizers that must agree before a fix is trusted: 1.096, kept, false fixes 39% → 17%, but usable only 7.5%. Experiments 8–10 — distilling a frozen DINOv2 teacher across surveys, penalising a place’s spread across surveys, repainting a median of the three photographs — all reverted, one collapsing outright. A design-stage probe explained experiment 3 in passing: on the champion, features of two different places already had cosine 0.70, so asking the same place to match across surveys had demanded almost nothing. Held-out usable sat at 7–12.5% while training-photograph usable ranged 17–96%: the gap was survey, not fit. Meanwhile the laptop went to sleep at midday and stalled the agents, which still run on it; the loop now keeps it awake.
The era’s best model had been recorded as a failure. Experiment 11 replaced the cell classifier with a cosine head, one direction per tile: 0.963 — the first model of the era better than silence, with false fixes at 7%. The scorer recorded it as failed, because a rule from the bootstrap commit threw out any model confident on fewer than 20% of frames. That floor dates from when the score was an error statistic and abstaining could game it; the mission score already charges for silence. Alexis chose the replacement: a model fails only if it gives no usable fix at all. Experiment 11 was re-scored from its saved model and promoted; the history was re-measured for rows failed only by the floor.
A design bake-off, and the breakthrough it found by accident. Four models wrote a design for the same prompt; a fresh agent graded them blind. Opus 5.5 ranked first in 3.9 minutes; Fable 5.1, the incumbent, took 11–15 minutes for an equal or lower grade. Design moved to Opus. The reviewer also re-measured the loop’s own experiment 12 — an ImageNet-pretrained MobileNetV3 trunk in place of the from-scratch one — and found it over the 4 MiB size limit by about 150 KB. That limit had no derivation behind it and weighed the fp32 file, while the chip runs int8. Re-derived from current ESP32-P4 boards (16 MB flash): half the flash for the model, 8 MiB int8, written as 32 MiB fp32. Experiment 12, re-scored: 0.560, usable 10.75% → 49.5%. Two frozen rules written for an older question had each vetoed the era’s best model, within the same hour.
Figures: always, and better. At 15:54 the always-draw rule went into the project’s instructions as a hard rule, after it had been suggested away once more to save usage. The figure agent moved off the critical path, at the human’s prompting: it now draws in the background while the experiment trains, with retries instead of a silent miss. Then the human noticed the era’s figures had degraded — overlapping text, chaotic lines. Cause: an unset variable meant every one had been drawn by Sonnet 5, not the model intended, and the checker passed 11 of 13 messy figures. Opus now draws, renders its own figure and looks at it; the checker catches off-canvas content, labels on shapes and lines through labels, and holds the camera frame and the output on the centre line. All 14 were redrawn.
Borrowed eyes, then better borrowed eyes. With Opus designing, the loop kept four in a row: two deeper pretrained blocks (0.435), style mixing across surveys inside the trunk (0.395), the 128 px frame enlarged to 224 px, ImageNet’s own scale, before the trunk (0.293), and a stronger MobileNetV2 trunk (0.155, 86.75% usable, 2.25% false). Each step made training slower: 7–17 minutes on the GPU for the small and first pretrained models, 83 for MobileNetV2. Asked about it, Alexis ruled out a time budget: heavier models are what is paying off. Experiment 17, ring pooling, crashed 30 s into training on a missing import in the agent’s own change; the loop re-ran it as 18 (0.158, reverted). The loop’s target was raised to 100 experiments.
Two things planned for after the era, decided now. An altitude era: usable fixes from 25 to 100 m above ground, the model told its barometric height, altitude in the score and the worst altitude band as the primary — a new ruler, so a new era, opened only once this one reaches its target. And a simulated-flight phase on ArduPilot SITL with its simulated GPS switched off and the model as the only position source, the camera cut from the held-out 2024 photograph. No physical flying.
EfficientNet-B0 and weight averaging reach 93% usable on the held-out photograph; then seven ideas fail to beat it, and three of the failures were the cloud’s
The overnight champion. At 01:59 an EfficientNet-B0 trunk (Apache-2.0) replaced MobileNetV2: 0.085. At 04:02 experiment 20 exported the average of the weights visited over the second half of training instead of the last step: 0.078. On the 2024 photograph it never saw: 93.0% usable, 0.75% confidently wrong, median miss 36 m. On the three it trained on: 100% usable, no false fixes. The model is 31.7 MB fp32, inside the new 32 MiB limit. The era opened at 1.651.
A plateau. Experiments 21–27 all reverted: training the head on a frozen trunk first (0.090), continuous vantages (0.103), belief maps that must agree across surveys (0.090), Noisy Student weights for the same trunk (0.083), and patches of one survey pasted into another (0.110). Continuous vantages was the idea the rural detour had designed in July and never trained; trained at last, it did not help here. Every attempt landed between 90.75% and 92.5% usable. Each now trains for about two hours; one spot machine was reclaimed after 104 minutes and the run started over.
Three failures were not the ideas’. Of the three experiments recorded as failed overnight, none had tested its idea. Experiment 17 was a code bug. Experiment 21 finished training on AWS, but the laptop lost DNS while polling and gave up; scored by hand from the recovered model it came in at 0.090, so the revert was right for the wrong reason. Experiment 24 never reached a GPU: no capacity in any Frankfurt zone. The launcher now retries through network drops and waits up to 90 minutes for capacity, and a cloud failure is recorded as not tested, drawn as a grey ring, never as a refuted idea. The loop re-ran 24 as 25 on its own. Alexis kept the original records of 21 and 24.
The champion existed only on the laptop. Since experiment 19 every model is 31.7 MB, over the 25 MB guard the loop uses to keep runaway files out of git. Five models, the current best among them, had been moved to a folder that is never committed. The guard is now 40 MB, above the scorer’s own size limit and below the 100 MB files it exists to stop, and the models are committed.
The flight replay flies the new champion over the photograph it never saw. Same 100 routes and the same wind, the 2024 photograph as both camera and background map; the comparison is the July champion flown over its own training photograph. Arrivals within 100 m: 91 → 95. Worst flight: 7.4 km → 212 m. False fixes: 207 → 59. Median miss about the same, 45 → 47 m. Would 98% usable land closer? Probably not: the aircraft declares arrival on its own estimate, and each fix is typically 30–40 m off, which comes from the 128 m map cells, not from how often the model answers. The score counts any fix inside 100 m as fully usable, so the loop has no reason to sharpen it.
Ideas for a different loop, a second loop by accident, six kinds of mission, and an experiment cut off by its own safety switch
Seven reverts in a row suggest a different harness. Two tickets from the human, both about the loop rather than the model. Smaller bets: screen each hypothesis on a small piece of the map, iterate there while the signal is negative, and pay for a full-map run only when it is positive. Parallel lineages: several populations from different seeds or researcher personalities, each with its own champion, since one lineage climbs one hill. On 1 November a look at Weco’s published search tree had found that a tree of experiments need not mean parallel runs; these tickets ask whether it should. Both open, exploration first.
Two loops at once. Experiment 28 was training when a second launch of the loop started and, before it exited, committed the running experiment’s code as a checkpoint. Had experiment 28 been reverted, that commit would have made its change the new baseline in silence. Found because the dashboard showed a design phase while the GPU was training; repaired by hand, nothing pushed. A lock against a second launch is ticketed.
Six more missions in the simulator. Return home, a four-waypoint tour, a five-minute loiter, a survey grid, a cold start with no idea of position, and the Spree corridor: 40 flights each with the champion, each against a no-camera control in the same weather. Two findings. Error does not build up: the tour’s median miss per waypoint was 68, 56, 54 and 56 m; longer missions fail more only because they give one bad fix more chances. And false fixes cluster in places: 126 of the Spree’s 128 sit in two stretches, and 18% of fixes along the Spreebogen are confidently wrong. Of 58 failed flights across all missions, 51 had a false fix. Cold start located itself on the first frame (median), 98% success.
Experiment 28 was cut off by its own safety switch. It enlarged the frame to 288 px instead of 224 before the EfficientNet trunks, so small structures reach the borrowed filters closer to the scale they learned on. Training was healthy, at about 23 minutes an epoch, when the machine’s 150-minute shutdown fired at epoch 6 of 8. No model came back, and the run was recorded as a refuted idea — exactly what the previous day’s fix was meant to prevent. The shutdown is a forgotten-machine guard, not a training budget: raised to 480 minutes, and hitting it now counts as not tested. Retrained the same afternoon with its exact code, experiment 28 needed 181 minutes and scored 0.100: usable 90.75% against the champion’s 93.0%, the same 0.75% false fixes, more abstentions, and 1.7× slower. A real revert this time, and a measured one.
Not merging yet. The era has its answer to the
founding question — 93% usable on a photograph the model never saw,
against 0% for the July champion — but the next step is a new way of
running experiments, so capture-holdout stays a branch. The
published site still tells the July story; a ticket collects everything it
needs before the merge: the overview, the research evolution, this
notebook, the docs, and the pre-merge routine.
Compiled from git history and Claude Code session transcripts, 20 July – 3 November 2026 — condensed for readability; see the research log for the complete, machine-generated experiment record.