Lab notebook

The repo is bootstrapped end-to-end in one session. Against the project spec, the frozen bbox-generic data pipeline, a naive baseline model, SQLite lineage, an HTML gallery, and loop.sh are scaffolded in one pass and proven with a real train→score→log run, including confirming the Hamburg holdout stays untouched.

Two open questions get resolved before more code is written. The deployment hardware: an ESP32-P4-Function-EV-Board paired with the SmartSens SC2336 sensor turns out to be Espressif’s own documented reference pairing. The reference imagery: 10 m/px Sentinel-2 is swapped for 1 m/px open state orthophotos, matching a real ~100 m-altitude camera footprint, and the relighting pipeline is rebuilt for the larger rasters. A v2 baseline scores 2765.67 m.

The dashboard is rebuilt in the project’s house style. It’s restyled to the same Tufte research-log language used across earlier projects, then pushed through three more rounds of feedback toward plain-language framing and precise miss-distance wording. loop.sh is hardened for safe interruption — a clean stop on Ctrl-C or a typed exit — and the train/eval split is rebalanced to 360 m blocks after the original 144 m blocks were starving some areas of eval crops.

Every experiment now gets its own hand-drawn architecture diagram. The design agent begins drawing one per experiment, under a visual contract distinguishing frozen, current, and just-changed stages. Experiment 8 pivots to a residual encoder and is kept at 2326.54 m, closing out the day.

An overnight diagnosis finds a real blind spot in the model’s field head. It has only ever seen a spatially-averaged descriptor with no positional information, and the loop tests a layout-aware fix; a pretrained MobileNetV3-Small backbone is explored in parallel, resolving the from-scratch-vs-pretrained question directly. Progress is slow — the loop repeatedly hits a connection error and restarts the same iteration from scratch several times before anything lands.

A new gallery page ships for every proposed architecture. “Inference Paths” is matched to the typography of the researcher’s existing personal site, along with a frozen-contract figure showing the fixed camera-frame input and output every design shares, its coordinates anchored so every architecture diagram scroll-compares cleanly against it.

Four experiments land in a row, each a real improvement. 1856 m → 1638 m → 1369 m → 1095 m → 1055 m. The loop is also restructured around two agent roles — one designs an experiment, a second implements it — with per-phase timing logged for each.

The project goes public. The site and a matching entry on the researcher’s personal homepage ship together, with a tagline chosen after two rounds of brainstorming, and an airloom-style “view on GitHub” ribbon added to every page.

An overnight disk leak crashes the loop. A training run has been silently caching several gigabytes of rendered images per experiment into a directory the loop was committing, filling the pod’s disk. Fixed by excluding scratch renders from persistence, and the loop is relaunched.

The pivot check itself turns out to be broken. The loop runs through the night with no net progress, and by midday the pivot check turns out to be testing the wrong signal entirely — a declared-category heuristic rather than whether the frozen architecture actually changed — so seven consecutive “pivot” experiments have in fact left the same pretrained backbone untouched.

Monitoring and budget expectations get set explicitly. An hourly automated health check posts to a GitHub issue only when something is actually broken. Budget context is set: roughly 100 experiments is the real ceiling, and the loop should not self-stop at the target metric — only when budget runs out or the work is stopped by hand.

Two infrastructure incidents get resolved in one day. A disk-exhaustion failure in the build pipeline, and a git-history divergence between the pod and GitHub after both independently applied the same fix. Policy tightens: every repository change now happens on the pod exclusively; the laptop becomes read-only.

A ten-hour GitHub outage gets root-caused and fixed. An oversized quarantine folder of rejected renders had been accidentally committed; excluding the path structurally fixes it for good. Loop vocabulary is also overhauled — “iteration” becomes “experiment” throughout — and every experiment row gains a deterministic, plain-language reason for its outcome.

The comparison baseline resets to a faster, fairer standard. The champion architecture, retrained on Berlin alone at a fast budget, scores 1176.5 m, retiring the older, heavier four-area 743 m champion as historical so quick local experiments are judged fairly. RunPod branding is dropped from the site as compute moves back to local hardware.

A human-forced redirect finally lands a genuinely new architecture. A from-scratch dense scene-coordinate regression design is forced directly into the next experiment, and a one-shot forced-direction mechanism is added to the loop so a specific redirect can be handed to a single run without further intervention. It scores 2360 m — architecturally novel, still reverted.

Compute strategy pivots twice in one evening. Off the rented GPU pod back to local hardware, then toward a scale-to-zero serverless GPU provider alongside a self-hosted always-on worker option — removing the recurring overhead of a persistent rented pod.

Figure-checking gets stricter. Mechanical checks for the architecture-diagram style contract land — palette, font, stroke-width, gradients/icons/emoji, red-consistency — then the SVG arcs get flattened so the line-crosses-label check can actually see curved elements.

Experiment 60 retries exp 38’s illumination-invariant channel with a from-scratch trunk and a cross-lighting consistency loss. 963.42 m — closer than most recent attempts, still reverted against the 743.07 m champion.

A real training-speed bug gets fixed mid-streak. Bucket/realization images were being re-decoded from disk every epoch — profiling shows this was ~78% of per-area wall-clock. Caching the decode once per process is the whole fix.

Experiment 63 — another forced pivot under the mobilenet ban, this time a from-scratch trunk with domain-contrastive pretraining, a fixed retinex channel, D4 symmetry, and top-K-masked decode — scores 1074.76 m. Four straight reverts; the day closes with the same 743.07 m champion still standing.

The human proposes deliberately narrowing scope. After weeks of ~50–65-minute iterations optimizing a worst-case across 4 areas × 6 lighting buckets, still far from the 20 m target, the idea: one locale (Berlin), daytime imagery only, treat “overfitting to one place” as a feature rather than a failure mode, in exchange for much faster rounds — and re-derive a fair accuracy milestone from what the actual image resolution can support, rather than assuming the original number still applies.

The plan gets grounded in real numbers before any code changes. The champion’s own Berlin-only daytime score (~650–720 m) sets a realistic near-term milestone at 100 m, not the original 20 m (~35× off). Per-experiment timing data shows the real speed levers aren’t what they first seem: the agent-drawn architecture figure (~15–20 min, the single largest non-training phase) and the six lighting buckets concatenated into one training set are the actual costs — dropping to one area is not, since areas already train in parallel.

Two smoke tests catch real bugs before a live run. A hardcoded midday.png heatmap background (missed by a substring grep) and a training-time realization generator that would’ve kept applying synthetic sensor noise despite relighting being “disabled” both get fixed and reverified.

A clean restart, at explicit request. The champion architecture — shaped by 63 rounds against the old 4-area/6-bucket problem, including machinery like a dark-lighting expert head that no longer applies — is retired in favor of the original from-scratch TinyLocNet baseline. The full experiment lineage is wiped, not just the comparison metric, so berlin-slim’s research trail starts at Experiment 1 with nothing to live up to; the old 62-row history is archived, not deleted.

A question about one roundabout unravels the whole evaluation. Looking at Berlin’s error map, the human asks why the Großer Stern — a seven-way star roundabout that exists exactly once in the city, about as visually unmistakable as a landmark gets — is surrounded entirely by red dots. If anything should be memorable, that should be. Checking the frozen split directly: zero training crops contain that roundabout, and twelve evaluation questions do. It sits three pixels from the edge of its 360 m block, and the block next door is an eval block — so the anti-leakage buffer had deleted every single training view of it.

The mismatch, stated plainly. This branch was founded on treating overfitting as a feature — build a visual memory of one city. But the metric was built to defeat memorisation: withhold regions, then grade the model on those regions. The architecture is a memory system; the ruler was grading it as an extrapolator. It is also stricter than the prior art the spec cites: DSAC and ACE evaluate on held-out camera trajectories through a fully mapped scene, not on unmapped parts of it. And in deployment the aircraft only ever flies over the box the model was trained on, so the metric measured a capability the product never needs while never measuring the one it does. The loop is stopped mid-experiment; the split is being redesigned to hold out viewpoints rather than regions.

The split is rebuilt around viewpoints. Training now covers every lattice position of the whole city — the model is supposed to memorise its one box, so it is shown all of it. Evaluation moves off-lattice: each test frame sits 11–17 m from the nearest training vantage (17 m being the largest offset a 24 m lattice permits) and carries its own rotation. Mapped ground, unseen view — which is what an aircraft over a mapped area actually faces. A small one-in-thirty-two-block region stays genuinely untrained as a logged-only diagnostic, never allowed to touch the metric, to catch a model that is a lookup table with no spatial structure.

Measured again, on the same instruments. Berlin ever shown in training rises from 71.8% to 93.5% (the remaining gap is the deliberate diagnostic region and its buffer). Eval questions centred on ground the model has seen: 0% → 100%. Eval frames containing no familiar pixel at all: 65.1% → 0%. Distance from each test frame to the nearest training vantage: 11.3–16.3 m. The lineage resets a second time in one day — scores from the old ruler cannot be compared to the new one — and the five experiments run that morning are archived rather than deleted.

The score becomes the geometric mean. Chosen over a percentile because it reads every frame rather than one cutoff, and because on an error spanning two orders of magnitude the log scale is the natural one: pulling a single spot from 2 km to 50 m — actually memorising it — now moves the number meaningfully. Median, mean, p10, p25 and a product-facing hit-rate are all still recorded, just no longer optimised. This time no lineage reset was needed: the evaluation data never changed, only its summary, so all three experiments were rescored from their saved weights. Corrected, the history inverts — experiment 2 becomes the champion at 1323.6 m, experiment 3 the revert.

And the correction exposes a hole in the harness. Experiment 2 was the best implementation the project had — and it no longer existed. Reverting an experiment ran git checkout -- model/, which deleted its source outright; only kept experiments ever entered the git trail. That is safe only if the scoring metric is correct and permanent, an assumption nobody had written down and which had just been proven false twice in one day. Of the 60 archived experiments, the 13 kept ones survive in git; the 47 reverted ones do not, though their design briefs and trained weights remain, which is how experiment 2 was rebuilt. Four separate code paths could destroy an implementation; all four now snapshot it first.

The baseline’s new score is the argument for the change. The same naive model that the old rulers reported as “1993 m median” — a number that sounds like a measurement — scores 2.001: it is confident on every single frame and wrong on every single one. Zero usable fixes, a 100% false-fix rate, the worst value the scale can produce. Nothing about the model changed; only the question being asked of it. Error statistics stay logged as diagnostics, and the standing check is written into the scorer: if the score improves while the usable-fix rate does not, the metric is wrong again. The lineage is wiped a third time and the metric propagated through every prompt, document and page.

The champion is pointed at rural Prignitz — a measurement, not an experiment. Experiment 5’s code, unchanged, trained and scored on the rural bounding box instead of Berlin: same architecture, same eight epochs, same hyperparameters, only the box is different. It needed no code change at all — the cell grid comes from the raster (Berlin 55×54 = 2,970 cells, Prignitz 55×55 = 3,025), which is the genericity requirement of §0 being exercised rather than asserted. Result: 0.113 against Berlin’s 0.040.

The interesting part is where the 2.8× went. The prediction beforehand was that fields would be indistinguishable and the model would abstain more — the safe failure. It did not. Abstention barely moved (3.0% → 3.8%) and usable fixes barely moved (96.5% → 92.5%); almost the entire loss is false fixes, 0.5% → 3.75%, a 7.5× rise in exactly the outcome the mission score treats as worse than silence. Median error is nearly unchanged at 29.5 m, so when it is right it is just as sharp: the gap is calibration, not precision. Belief pooling protects against mass split across adjacent cells and against mass scattered across distant ones, but not against mass landing confidently on one wrong field that looks exactly like the right one — a case Berlin never presents and Prignitz presents constantly. The unseen-region diagnostic says the same thing from the other side: coverage rises to 64.2% (Berlin 38.2%) while staying 0% usable, i.e. on ground it has never seen the rural model answers more readily and is wrong every time.

Nothing from this run was written to the experiment database, state/best.json or the git trail. Scored against a Berlin champion it would have been logged “reverted”, which would be a meaningless verdict — the two areas are different questions, not competing answers to one. It is recorded here and as a branch on the evolution graph instead. The research question it names is sharper than Berlin’s was: cut false fixes without spending usable ones. Sharp enough that within the hour it had an era of its own.

One model over both boxes: proposed, examined, dropped — all before an experiment ran. The idea was to stop letting each area find its own setup and instead iterate toward a single model covering Berlin and Prignitz, amending §1’s one-model-per-bounding-box rule. It survived about an hour. The arithmetic was encouraging: the boxes sit 90 km apart, so their union is 88×71 km of which only 1.6% is imaged — a union grid would be 384,196 cells, nearly all of it ground no camera has seen — while a grid over just the two imaged boxes is 5,995 cells, barely twice Berlin’s. It also needed no frozen file changed: the scorer loads one ONNX per area, so a shared trunk exported twice, each framed to its own raster, fits the existing interface. Dropped anyway, on the grounds that unifying is a different question from the one Prignitz actually poses, and running both at once would confound them.

That hour turned up a real measurement bug, and it is still open. Checking whether a shared model would fit the ESP32‑P4 gave 4.88 MB against a 4 MB ceiling — an apparent blocker. It wasn’t. MODEL_MAX_BYTES was set in the bootstrap commit and never revisited — git log -S returns exactly one commit — with no derivation behind it. And it weighs the fp32 ONNX, while the deployment path this repo documents is esp‑dl, which ingests that ONNX for int8 quantisation. Measured: the Berlin champion is 2.94 MB fp32 and 0.78 MB int8. The gate overstates every model by about 3.8×, and the shared model that “failed” is 1.29 MB quantised. Nothing in the pipeline quantises anything, so §3’s “quantization-aware training” is a line in the spec no experiment can currently act on. Deliberately not changed on the spot: it is a frozen file, and the re-scoring that puts every past era on one ruler runs the current scorer, so relaxing the gate could flip several of the 17 archived gated-fails at once. That deserves its own step, with the effect measured first.

A rural era opens — Prignitz alone. AREAS=prignitz, everything else about the narrow configuration unchanged. The era’s question is not Berlin’s. Berlin had to prove the approach worked at all; Prignitz already half works at 0.113, and the entire gap is in one place, so the question is exact: cut false fixes without spending usable ones.

Experiment 1 — the seed. The champion architecture, no design agent, trained and scored on Prignitz natively so the era starts from a number on its own board rather than a probe’s. 0.113, reproducing the probe exactly: 92.5% usable, 3.75% false, 3.8% abstaining, median 29.5 m.

Experiment 2 — rotation-consensus decode. Refuted, and informatively. Training draws a fresh random heading per position per epoch, so a genuinely memorised place ought to be heading-invariant while a false match to a look-alike field should ride view-specific micro-texture that does not survive being turned. The decode was changed to run the trunk over the frame plus its 90/180/270° rotations, average the four belief maps, and decode the average — a wrong field would have to win from at least two headings. Predicted false fixes 3.75% → ≤1.5%. Measured: false fixes went up, 3.75% → 4.25%, usable 92.5% → 92.0%, mission 0.113 → 0.123. Reverted. The negative is worth more than the change would have been: the rural confusions are rotation-stable. Two fields that fool the model fool it from every heading, so this is not a viewing-angle artefact and consensus at decode time cannot reach it. Whatever fixes it has to act during training.

Experiment 3 was designed and implemented on exactly that reading — continuous-vantage training: jitter each training framing by up to ±12 px so vantages tile the ground instead of standing on the 24 m lattice, since every eval view (and every real flight frame) stands 8–16 px off-lattice, and a model that has only ever seen lattice framings can tell fields apart by accidents of framing that vanish a few metres away. It never trained. Recorded as the era’s open move, not as a result.

The era is closed and the repository rewound to the commit before it opened. model/, state/best.json and the experiment database all return to the Berlin champion at 0.040, and the two Prignitz rows are removed from the lineage rather than archived as an era — four hours and three experiments is a detour, not a chapter, and carrying it as one would put a band on every chart in this site for work that produced no kept result. Not a verdict on the rural problem: nothing about it failed on its merits, and the 0.113 measurement and the rotation-stability finding both stand. The reason is simpler — one question at a time, and the question is still Berlin.

What survives the rewind, deliberately: this entry; the detour drawn as a branch that leaves the berlin-slim line and stops, on the evolution graph; and the open finding about the size gate above. The raw artifacts — run directories, exported models, agent transcripts for all three experiments — are not deleted either; they stay reachable at the git tag prignitz-detour-record. The rule the project already had for wiped eras applies here too: re-measure or record, never quietly erase.