Where we are: The experiment record

Every experiment the autonomous loop has run, across all 6 evaluation eras — kept and discarded. Each row was pre-registered before training (hypothesis, method, expected outcome, architecture figure), then trained on a single GPU and scored. Every score on this page is measured by one frozen ruler: the worst mission score on held-out viewpoints of a photograph the model never trained on — (1 − usable-fix rate) + false-fix rate, the product requirement rather than an error statistic. A model must also fit the aircraft: the champion’s file is 31.7 MB against a hard limit of 32 MiB (≈ 8 MiB once quantised to int8 for the chip). The earlier eras were judged by different rulers at the time, and where those two disagree the row says so. One agent designs, one implements; failures stay on the record. New here? Start with the overview.

Where we are right now

Status: best mission score is 0.078 in its hardest cell. That number is (1 − usable-fix rate) + false-fix rate: a frame counts as a usable fix only when the model is confident and within 100 m; abstaining is safe; being confident and wrong is a false fix, which for a UAV is worse than saying nothing at all. 1.0 means it abstains on everything, 2.0 means it is confidently wrong on everything, 0 is the goal.

28 experiments in this era, 12 kept — and 81 before it, across 5 earlier evaluation eras. The chart below shows all 109. Cost of the whole search: $487.87 — $465.74 of agent tokens (equivalent API cost; the work ran on a Max subscription) plus $22.13 of rented GPU, recorded for 88 of 109 experiments.

New best Not a new best Running best Era's own best Goal (score 0) ×Gated fail Holdout check Pivot-directed Not on this ruler Evaluation era updated 2026-11-04 12:37 UTC
Mission score per experiment, all 109 of them — (1 − usable-fix rate) + false-fix rate, lower is better
bootstrap 4 areas · 6 lighting buckets · region holdout berlin only viewpoint holdout mission score held-out photograph 0 0.25 0.5 0.75 1 1.5 2 score 1 — no better than saying nothing 0 — every frame a usable fix × × × × × × × × × × × × × × × × × 1 4 1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 61 1 1 1 1 4 7 10 13 16 19 22 25 28 experiment № — restarting at 1 with each evaluation era
#Experiment Category Mission score Time Cost Status
▸ 6.28 Bigger magnifying glass: the 128 px frame is enlarged in-graph to 288 px instead of 224 px before the EfficientNet-B0 trunks, so Berlin's small structures (cars, roof edges, kerbs, tree crowns) reach the borrowed filters at a scale closer to the one their ImageNet invariance was learned on 2e90e35d
worse than champion: 0.100 vs 0.078
architecture pivot 0.100 172 m 23 s $2.85 discarded
▸ 6.27 Patchwork surveys: random rectangles of every training view are replaced by the SAME ground from another survey (or, sometimes, by unrelated ground), so a place must be recognised from whatever survives local change 1f8f06ad
worse than champion: 0.110 vs 0.078
augmentation pivot 0.110 133 m 19 s $3.03 discarded
▸ 6.26 Probe first, then fine-tune, actually trained: one extra opening epoch trains only the head on the FROZEN ImageNet trunk before the usual fine-tune (LP-FT; re-run of experiment 21, which never reached a GPU) a5c71e3e
worse than champion: 0.090 vs 0.078
training pivot 0.090 126 m 27 s $3.26 discarded
▸ 6.25 Same eyes, more worldly upbringing, actually trained: each member's EfficientNet-B0 trunk starts from Noisy Student weights instead of plain ImageNet weights (re-run of experiment 24, which never reached a GPU) f0d553b7
worse than champion: 0.083 vs 0.078
architecture pivot 0.083 126 m 01 s $2.65 discarded
▸ 6.24 Same eyes, more worldly upbringing: each member's EfficientNet-B0 trunk carries Noisy Student weights (semi-supervised on 300M photographs with injected noise, Apache-2.0) instead of plain ImageNet weights — identical architecture, identical size f55c8c48
training or scoring crashed — no result
architecture gated fail 9 m 59 s $4.07 gated fail
▸ 6.23 Same place, same belief: each member's tile-belief map for one place must agree across the three survey photographs (Jensen-Shannon consistency, AugMix-style), trained on batches that hold every place in all three surveys 4a3afb08
worse than champion: 0.090 vs 0.078
loss 0.090 121 m 20 s $3.20 discarded
▸ 6.22 Continuous vantages: every training framing is shifted by a fresh random offset of up to ±12 px, so over an epoch the training views cover the ground continuously instead of standing on the fixed 24 m lattice d1fb6f8a
worse than champion: 0.103 vs 0.078
augmentation 0.103 225 m 40 s $2.90 discarded
▸ 6.21 Probe first, then fine-tune: one extra opening epoch trains only the head on the FROZEN ImageNet trunk, so the random tile head's first gradients cannot distort the borrowed eyes (LP-FT) 3e9d902e
training or scoring crashed — no result
training gated fail 109 m 59 s $2.64 gated fail
▸ 6.20 Fly the average, not the last step: the exported model is the uniform average of the weights visited over the second half of training (SWAD/SWA-style dense weight averaging), with BatchNorm statistics recalibrated for the averaged weights 0c598205
new best: 0.085 → 0.078
training 0.078 121 m 52 s $2.76 kept
▸ 6.19 Sharper borrowed eyes: each member's MobileNetV2 trunk is replaced by an ImageNet-pretrained EfficientNet-B0 trunk (stem + all seven MBConv stages, Apache-2.0), read at the same two strides (112x14x14 and 320x7x7 at 224 px) f8ce7046
new best: 0.155 → 0.085
architecture 0.085 147 m 09 s $3.43 kept
▸ 6.18 Ring pooling, actually trained: the place description keeps WHERE in the frame things are (centre / middle ring / edge) instead of one average over the whole frame (re-run of experiment 17, which crashed before training) ec350b82
worse than champion: 0.158 vs 0.155
architecture 0.158 94 m 09 s $2.78 discarded
▸ 6.17 Ring pooling: the place description keeps WHERE in the frame things are (centre / middle ring / edge), not one average over the whole 128 m frame aba2697a
training or scoring crashed — no result
architecture gated fail 8 m 54 s $3.25 gated fail
▸ 6.16 Better borrowed eyes: each member's MobileNetV3-Small trunk is replaced by an ImageNet-pretrained MobileNetV2 trunk (stronger, wider; BSD-3), read at the same two depths (14x14 and 7x7 at 224 px) a4393f88
new best: 0.293 → 0.155
architecture 0.155 94 m 20 s $3.27 kept
▸ 6.15 Look at Berlin at the borrowed eyes' native scale: the 128 px frame is resized in-graph to 224 px (ImageNet's training resolution) before the pretrained trunk, so its 8x8/4x4 place maps become 14x14/7x7 0335e1ef
new best: 0.395 → 0.293
architecture 0.293 40 m 51 s $3.34 kept
▸ 6.14 Cross-survey style mixing inside the borrowed eyes: during training, the early trunk layers' per-channel feature statistics (the 'look' of a photograph) are randomly blended with another view's from a different survey, so the place description cannot key on any one survey's style 3c1f3700
new best: 0.435 → 0.395
augmentation 0.395 26 m 13 s $1.81 kept
▸ 6.13 Deeper borrowed eyes: each member's place description also reads two further ImageNet-pretrained MobileNetV3-Small blocks (96 channels, 4x4), alongside the 48-channel 8x8 level it reads today 07d19b92
new best: 0.560 → 0.435
architecture 0.435 26 m 22 s $4.91 kept
▸ 6.12 Borrowed eyes: each committee member's from-scratch trunk is replaced by ImageNet-pretrained MobileNetV3-Small blocks (BSD-3, baked in model/pretrained), fine-tuned at a tenth of the learning rate, so the place descriptor starts from invariances learned on a million photographs instead of three 64d06484
starting line: mission score 0.560
architecture 0.560 31 m 56 s $6.96 kept
▸ 6.11 One direction per tile: a normalised-prototype cosine head with an additive margin replaces the affine cell classifier, so every survey's picture of a place must point the same way 11a9de98
starting line: mission score 0.963
architecture 0.963 26 m 56 s $10.02 kept
▸ 6.10 Canonical-composite reconstruction: each member's 8x8 feature map must repaint the per-pixel MEDIAN of the three training surveys (a picture of Berlin in which anything seen in only one survey is gone), through a training-only decoder that never flies ed3332a7
worse than champion: 1.141 vs 1.096
loss 1.141 46 m 10 s $9.92 discarded
▸ 6.9 Within-place variance penalty: each member's pooled feature of one place, standardised against its spread across places, must be the same in all three surveys 0000746a
not one usable fix
loss gated fail 32 m 26 s $7.67 gated fail
▸ 6.8 Cross-survey DINOv2 distillation: each member's 8x8 feature map must reproduce a frozen foundation-model teacher's description of the SAME place photographed in a DIFFERENT survey (training-only; the teacher never flies) 5ee860df
worse than champion: 1.121 vs 1.096
loss 1.121 190 m 59 s $7.15 discarded
▸ 6.7 Two-member committee: two independently trained localizers must agree before a fix is trusted; confidence is the smaller of the two members' belief on the consensus block 5a0a4962
new best: 1.271 → 1.096
architecture 1.096 43 m 44 s $8.27 kept
▸ 6.6 Cross-survey blending: every training view becomes a per-pixel mixture of the three surveys' photographs of the same place, so the model learns the continuum between surveys instead of three fingerprints 87f5e046
worse than champion: 1.288 vs 1.271
augmentation 1.288 173 m 41 s $6.76 discarded
▸ 6.5 Whole-frame receptive field: two extra stride-2 stages take the trunk from a 31 m texture window to the full 128 m frame, so the classifier can read street-and-block layout instead of an average of local textures 0ccf089e
worse than champion: 1.298 vs 1.271
architecture 1.298 108 m 55 s $3.77 discarded
▸ 6.4 Exposure-and-sharpness nuisance randomisation: every training view is re-photographed at random (brightness, contrast, gamma, colour balance, saturation, blur, sensor noise) so a pixel fingerprint of one survey stops being a usable solution 937b4c32
new best: 1.583 → 1.271
augmentation 1.271 111 m 31 s $4.91 kept
▸ 6.3 Cross-survey latent consistency: the trunk feature of a place must be the same in all three photographs, not three fingerprints that happen to share a label 81aa79eb
worse than champion: 1.648 vs 1.583
loss 1.648 73 m 44 s $3.62 discarded
▸ 6.2 Paired-survey batches: every training step sees the same places in all three photographs, not one photograph at a time aa74176b
new best: 1.651 → 1.583
training 1.583 62 m 57 s $4.02 kept
▸ 6.1 Baseline seed: current model/ code, no design agent ea767139
starting line: mission score 1.651
other 1.651 46 m 46 s — kept
▸ 5.5 Decode-consistent training targets: bilinear tent cell labels + residual offset against the model's own pooled centroid 7a1314f9
The loop kept this — it beat the running best under the metric then in force. On today's ruler it scores 1.873 — worse than silence.
loss 1.873 22 m 58 s $3.90 KEPT
▸ 5.4 Local belief pooling decode: confidence = softmax mass on the 3x3 cell neighborhood, fix = its mass-weighted centroid 7958b03d
The loop kept this — it beat the running best under the metric then in force. On today's ruler it scores 1.893 — worse than silence.
architecture 1.893 20 m 36 s $2.75 KEPT
▸ 5.3 Full-lattice coverage training for the map-cell classifier: every epoch is a complete pass over all 72,712 vantages 014a6196
The loop kept this — it beat the running best under the metric then in force. On today's ruler it scores 1.871 — worse than silence.
training 1.871 22 m 28 s $3.61 KEPT
▸ 5.2 Map-cell classification decode: pick 1 of ~3,000 128 m tiles, confidence = softmax mass 9011afe6
The loop discarded this — no improvement under the metric then in force.
architecture 1.003 9 m 24 s $4.25 DISCARDED
▸ 5.1 Baseline seed: current model/ code, no design agent 92d1f12e
The loop kept this — it beat the running best under the metric then in force. On today's ruler it scores 2.001 — worse than silence.
other 2.001 47 s — KEPT
▸ 4.4 Re-implementation of lost experiment 2 — classify-then-refine: 32x32 map-cell softmax + within-cell offset replaces direct coordinate regression c0f3cd4e
The loop kept this — it beat the running best under the metric then in force. It cannot be placed on today's ruler at all — see the row for why.
architecture — 3 m 36 s — NO SCORE
▸ 4.3 Coverage over repetition: spend the same 48k-step budget on ~48k fresh-rotated distinct vantages instead of 6000 static crops replayed 8x f74a8666
The loop discarded this — no improvement under the metric then in force. It cannot be placed on today's ruler at all — see the row for why.
training — 6 m 36 s — NO SCORE
▸ 4.2 Classify-then-refine: 32x32 map-cell softmax + within-cell offset replaces direct coordinate regression 4affa746
The loop kept this — it beat the running best under the metric then in force. It cannot be placed on today's ruler at all — see the row for why.
architecture — 6 m 20 s — NO SCORE
▸ 4.1 Baseline seed on the corrected viewpoint-holdout split 8bdcc185
The loop kept this — it beat the running best under the metric then in force. It cannot be placed on today's ruler at all — see the row for why.
other — 44 s — NO SCORE
▸ 3.5 Per-epoch position resampling: expose all ~45k Berlin train vantages instead of a frozen 6k subset 0ede8d45
The loop kept this — it beat the running best under the metric then in force. It cannot be placed on today's ruler at all — see the row for why.
training — 16 m 23 s — NO SCORE
▸ 3.4 Hierarchical decode: 16x16 coarse-cell classification + per-cell continuous offset regression from a layout-preserving descriptor f625d8d7
The loop discarded this — no improvement under the metric then in force.
architecture 1.988 17 m 29 s $3.59 DISCARDED
▸ 3.3 Re-land the validated 64x64 grid-classification head with a convergence-scale training budget (x12 epochs, cosine LR, fresh per-epoch rotations) bea8a408
The loop kept this — it beat the running best under the metric then in force. On today's ruler it scores 1.953 — worse than silence.
training 1.953 17 m 28 s $3.70 KEPT
▸ 3.2 Reparameterize localization as 64x64 map-grid classification with in-graph soft-argmax d9805aa2
The loop discarded this — no improvement under the metric then in force.
architecture 1.483 6 m 37 s $3.15 DISCARDED
▸ 3.1 Baseline seed: original TinyLocNet, fresh berlin-slim lineage 28f2ebe1
The loop kept this — it beat the running best under the metric then in force. On today's ruler it scores 1.996 — worse than silence.
other 1.996 16 s — KEPT
▸ 2.65 Harness smoke test — SKIP_AGENT loop.sh iteration, 200 crops, 1 epoch 5e9419c3
The loop discarded this — no improvement under the metric then in force.
other 1.738 28 s — DISCARDED
▸ 2.64 Harness smoke test — SKIP_AGENT loop.sh iteration, 200 crops, 1 epoch 13657130
The loop discarded this — no improvement under the metric then in force.
other 1.738 34 s — DISCARDED
▸ 2.63 From-scratch trunk + domain-contrastive pretraining + fixed retinex channel + D4 symmetry + top-K-masked decode (forced pivot, mobilenet_v3_small banned) c4616251
The loop discarded this — no improvement under the metric then in force.
architecture 1.811 64 m 52 s $9.41 DISCARDED
▸ 2.62 Fire-module SqueezeNet1.1 trunk + FiLM-conditioned single field + D4 symmetry + adaptive-sharpening decode (forced pivot, mobilenet_v3_small banned) 0041192d
Failed outright: training or scoring crashed, it broke a deployment limit, or it never produced a single usable fix.
architecture — 26 m 11 s $6.86 GATED FAIL
▸ 2.61 Quantization-freed capacity: from-scratch depthwise-separable trunk + unified single-head field, weights trained for int8 export 9421cc39
Failed outright: training or scoring crashed, it broke a deployment limit, or it never produced a single usable fix.
architecture — 45 m 05 s $7.88 GATED FAIL
▸ 2.60 Corrected exp-38 retry: fixed illumination-invariant channel + from-scratch trunk + cross-lighting consistency loss, gate deleted 4add0860
The loop discarded this — no improvement under the metric then in force.
architecture 1.991 86 m 54 s $6.23 DISCARDED
▸ 2.59 Independently-trained day/night specialist twins with a trunk-free brightness dispatcher efa8f69b
The loop discarded this — no improvement under the metric then in force.
architecture 1.953 80 m 09 s $6.15 DISCARDED
▸ 2.55 Dense scene-coordinate voting with a differentiable IRLS robust consensus (from-scratch trunk) c998b079
The loop discarded this — no improvement under the metric then in force.
architecture 1.826 21 m 51 s $3.41 DISCARDED
▸ 2.54 Memory-safe flying delighting translator + from-scratch trunk: the never-fairly-tested canonicalization pivot, corrected d06f7d8d
The loop discarded this — no improvement under the metric then in force.
relighting 1.926 19 m 42 s $2.88 DISCARDED
▸ 2.53 ShuffleNetV2 trunk with FiLM-conditioned single field and top-K decode replaces the two-expert MobileNet blend 01ab7892
The loop discarded this — no improvement under the metric then in force.
architecture 1.836 21 m 36 s $2.96 DISCARDED
▸ 2.52 Continuous Gaussian-mixture location regression replaces the shared map-cell classification field 5361f941
The loop discarded this — no improvement under the metric then in force.
architecture 1.971 21 m 32 s $3.90 DISCARDED
▸ 2.51 Residual vector-quantized location codebook: hard nearest-neighbor coding replaces the shared-softmax cell field 4752c6c2
The loop discarded this — no improvement under the metric then in force.
quantization 1.263 28 m 14 s $3.73 DISCARDED
▸ 2.50 Spatial mixture-of-experts: a coarse macro-region classifier dispatches to disjoint local fine-cell experts e4a5f33d
The loop discarded this — no improvement under the metric then in force.
architecture 1.673 64 m 50 s $4.72 DISCARDED
▸ 2.49 Dense per-patch coordinate consensus (re-run after ENOSPC): from-scratch trunk, no map-cell field, error-driven sampler, disagreement-based confidence 7d7bea5d
The loop discarded this — no improvement under the metric then in force.
architecture 1.231 54 m 57 s $3.52 DISCARDED
▸ 2.48 Dense per-patch coordinate consensus (corrected re-attempt): from-scratch trunk, no map-cell field, error-driven sampler, disagreement-based confidence aedcce64
Failed outright: training or scoring crashed, it broke a deployment limit, or it never produced a single usable fix.
architecture — 14 m 08 s $3.25 GATED FAIL
▸ 2.47 Dense per-patch coordinate regression with learned-consensus pooling replaces the map-cell classification field aedcce64
Failed outright: training or scoring crashed, it broke a deployment limit, or it never produced a single usable fix.
architecture — 7 m 27 s $0.09 GATED FAIL
▸ 2.46 Two-expert lighting dispatcher with from-scratch trunks and factorized row/column coordinate fields f42a8309
Failed outright: training or scoring crashed, it broke a deployment limit, or it never produced a single usable fix.
architecture — 40 m 55 s $3.09 GATED FAIL
▸ 2.45 Retrieve-then-rerank: a coarse field proposes a shortlist, a pairwise comparator decides among it 0d258f6f
The loop discarded this — no improvement under the metric then in force.
architecture 1.816 60 m 53 s $4.24 DISCARDED
▸ 2.44 Sequential delighting translator + from-scratch trunk: canonicalize the pixels first, localize second f61b14fa
Failed outright: training or scoring crashed, it broke a deployment limit, or it never produced a single usable fix.
relighting — 75 m 59 s $5.09 GATED FAIL
▸ 2.43 Joint reconstruction-canonicalization trunk: a from-scratch encoder that must repaint the clean daytime crop, trained alongside localization every step 498f9a89
Failed outright: training or scoring crashed, it broke a deployment limit, or it never produced a single usable fix.
architecture — 77 m 03 s $8.41 GATED FAIL
▸ 2.42 Domain-native contrastive pretraining replaces the ImageNet trunk (corrected re-attempt) fed7c60a
The loop discarded this — no improvement under the metric then in force.
architecture 1.996 69 m 24 s $4.13 DISCARDED
▸ 2.41 Domain-native contrastive pretraining replaces the ImageNet trunk c0fda62e
Failed outright: training or scoring crashed, it broke a deployment limit, or it never produced a single usable fix.
architecture — 15 m 38 s $0.18 GATED FAIL
▸ 2.40 Coarse-to-fine geolocalization: a shared conv field ranks the region, a shared local head regresses the offset 9880938d
The loop discarded this — no improvement under the metric then in force.
architecture 1.956 81 m 04 s $8.40 DISCARDED
▸ 2.39 Coordinate-generated map field: replace 1,024 free-parameter cell templates with a shared position-conditioned embedding generator 898d8cb5
The loop discarded this — no improvement under the metric then in force.
architecture 1.726 71 m 34 s $7.28 DISCARDED
▸ 2.38 Illumination-invariant retinex channel joins raw RGB from the pixel level through the trunk 898d8cb5
Failed outright: training or scoring crashed, it broke a deployment limit, or it never produced a single usable fix.
architecture — 20 m 32 s $0.24 GATED FAIL
▸ 2.37 Multi-hypothesis coordinate regression replaces the 1024-cell field 5a6771be
The loop discarded this — no improvement under the metric then in force.
architecture 1.673 66 m 02 s $4.59 DISCARDED
▸ 2.36 Hardest-impostor margin hinge, retested on the confusability-weighted champion 3ee64311
The loop discarded this — no improvement under the metric then in force.
loss 1.986 59 m 54 s $3.14 DISCARDED
▸ 2.35 Confusability-weighted location sampling: oversample look-alike cell pairs, not just more places 3750ddb6
The loop kept this — it beat the running best under the metric then in force. On today's ruler it scores 1.928 — worse than silence.
training 1.928 62 m 59 s $4.20 KEPT
▸ 2.34 Hardest-impostor margin loss: the field must outrank its best lookalike, not just light up the truth bfbca30d
Failed outright: training or scoring crashed, it broke a deployment limit, or it never produced a single usable fix.
loss — 20 m 01 s $8.03 GATED FAIL
▸ 2.33 Sensor-sharpness nuisance randomization: per-crop PSF/resampling jitter breaks the blur-texture shortcut 0aaab0f5
Failed outright: training or scoring crashed, it broke a deployment limit, or it never produced a single usable fix.
augmentation — 59 m 18 s $7.04 GATED FAIL
▸ 2.32 Risk-controlled abstention: six per-lighting-regime thresholds calibrated on never-trained fenced blocks 4871a8c7
The loop discarded this — no improvement under the metric then in force.
training 1.773 108 m 15 s $19.07 DISCARDED
▸ 2.31 Neural-map correlation field: localization becomes sliding-window matching of a crop fingerprint against a learned 32 m map raster 1ee201ff
The loop discarded this — no improvement under the metric then in force.
architecture 1.991 73 m 46 s $12.11 DISCARDED
▸ 2.30 Supervised lighting dispatcher: day/dusk/night specialist field heads replace the two-expert blend, int8-paid 92404766
The loop discarded this — no improvement under the metric then in force.
architecture 1.953 73 m 20 s $11.12 DISCARDED
▸ 2.29 Stride-8 fine-layout tap: the field scorer reads an 8-m-resolution mid-trunk snapshot, int8-paid 7e6fb3ea
The loop discarded this — no improvement under the metric then in force.
architecture 1.951 70 m 11 s $11.05 DISCARDED
▸ 2.28 Rerun, never scored: unseen-ground confidence calibration + reinstated 3x trunk (iter 7 died on a full disk before training) 83bf925a
The loop discarded this — no improvement under the metric then in force.
training 1.663 57 m 43 s $5.51 DISCARDED
▸ 2.27 Unseen-ground confidence calibration unlocks the reinstated 3×-capacity trunk cc3faa2c
Failed outright: training or scoring crashed, it broke a deployment limit, or it never produced a single usable fix.
training — 14 m 55 s $6.71 GATED FAIL
▸ 2.26 Deployment-envelope capacity scaling: 3× pretrained trunk depth paid for by int8 FC-head storage 4fb65e2a
The loop discarded this — no improvement under the metric then in force.
architecture 1.273 69 m 41 s $10.18 DISCARDED
▸ 2.25 Convergence-scaled training: 3× optimizer steps with cosine LR decay under the kept fresh-draw sampler 7f8a2986
The loop kept this — it beat the running best under the metric then in force. On today's ruler it scores 1.911 — worse than silence.
training 1.911 63 m 08 s $9.15 KEPT
▸ 2.24 Daytime-redraw auxiliary, rerun: exp 23 was OOM-killed mid-harness, never scored — retest it with a memory-lean epoch sampler 5794e840
The loop discarded this — no improvement under the metric then in force.
loss 1.953 42 m 59 s $7.56 DISCARDED
▸ 2.23 Daytime-redraw auxiliary: a throwaway decoder must reconstruct the clean daytime crop from the encoder's features adfc8b91
Failed outright: training or scoring crashed, it broke a deployment limit, or it never produced a single usable fix.
loss — 42 m 08 s $6.88 GATED FAIL
▸ 2.22 Train-only per-patch place supervision: every feature cell places its own patch, the flying decode is untouched 9df5c839
The loop discarded this — no improvement under the metric then in force.
loss 1.916 40 m 44 s $8.67 DISCARDED
▸ 2.21 Periodic blind holdout check (hamburg) cac6ec61
Blind Hamburg holdout check. Logged for honesty, never allowed to drive keep/revert.
other — — — HOLDOUT
▸ 2.20 Per-epoch training-set resampling: a fresh 6,000-location draw per bucket every epoch replaces the one frozen 36k-crop tensor cac6ec61
The loop kept this — it beat the running best under the metric then in force. On today's ruler it scores 1.948 — worse than silence.
training 1.948 35 m 33 s $7.80 KEPT
▸ 2.19 Off-site distractor patching: half the training crops carry a pasted block of terrain from elsewhere, labels unchanged d5080d5d
The loop discarded this — no improvement under the metric then in force.
augmentation 1.911 42 m 51 s $9.87 DISCARDED
▸ 2.18 Dense per-patch field voting: 64 local place votes with learned fusion replace the global 1024-way template head 48e17da6
The loop discarded this — no improvement under the metric then in force.
architecture 1.901 45 m 05 s $10.24 DISCARDED
▸ 2.17 Nuisance-randomized training renders: each bucket's training crops drawn from three seeded realizations of the frozen relighting sim 95b4c3a0
The loop kept this — it beat the running best under the metric then in force. On today's ruler it scores 1.963 — worse than silence.
relighting 1.963 39 m 10 s $8.60 KEPT
▸ 2.16 C4 rotation-vote field: average field logits over the crop's four 90° turns 5c5373df
The loop kept this — it beat the running best under the metric then in force. On today's ruler it scores 1.956 — worse than silence.
architecture 1.956 36 m 53 s $7.58 KEPT
▸ 2.15 Selective prediction: field-shape confidence head with per-bucket calibrated abstention f3b87247
The loop kept this — it beat the running best under the metric then in force. On today's ruler it scores 1.848 — worse than silence.
architecture 1.848 40 m 01 s $11.51 KEPT
▸ 2.14 Peak-commit decode: β-sharpened softmax replaces mean-of-map soft-argmax d2a0a2b9
The loop kept this — it beat the running best under the metric then in force. On today's ruler it scores 1.996 — worse than silence.
architecture 1.996 31 m 13 s $7.30 KEPT
▸ 2.13 Periodic blind holdout check (hamburg) fe745eb4
Blind Hamburg holdout check. Logged for honesty, never allowed to drive keep/revert.
other — — — HOLDOUT
▸ 2.12 Luminance-gated dark-expert field head blended with the existing layout head fe745eb4
The loop kept this — it beat the running best under the metric then in force. On today's ruler it scores 1.993 — worse than silence.
architecture 1.993 42 m 10 s $4.95 KEPT
▸ 2.11 ImageNet-pretrained MobileNetV3-Small trunk replaces the from-scratch encoder d69862d9
The loop kept this — it beat the running best under the metric then in force. On today's ruler it scores 1.991 — worse than silence.
architecture 1.991 31 m 35 s $2.64 KEPT
▸ 2.10 Layout-aware field head: squeezed 8×8 spatial code ⊕ GAP replaces GAP-only head input ff271753
The loop kept this — it beat the running best under the metric then in force. On today's ruler it scores 2.001 — worse than silence.
architecture 2.001 38 m 08 s $4.27 KEPT
▸ 2.9 Cross-lighting contrastive pairs: NT-Xent metric learning on the place descriptor 76c45e3c
The loop discarded this — no improvement under the metric then in force.
loss 1.991 23 m 41 s $4.93 DISCARDED
▸ 2.8 Deployment-envelope residual encoder: 4.2× capacity (232k → 973k params), 11 conv layers with skips 76c45e3c
The loop discarded this — no improvement under the metric then in force.
architecture 1.996 48 m 45 s $5.33 DISCARDED
▸ 2.7 Scale training coverage 7.5x: 6,000 of ~45k available train locations per bucket (was 800) 76c45e3c
The loop kept this — it beat the running best under the metric then in force. On today's ruler it scores 2.001 — worse than silence.
training 2.001 21 m 44 s $3.83 KEPT
▸ 2.6 Per-epoch rotation resampling replaces the one-shot frozen training tensor de614d47
The loop discarded this — no improvement under the metric then in force.
augmentation 2.001 6 m 52 s $1.67 DISCARDED
▸ 2.5 Hierarchical coarse-to-fine supervision of the probability field via probability pooling 7d02e02d
The loop discarded this — no improvement under the metric then in force.
loss 1.996 6 m 20 s $1.91 DISCARDED
▸ 2.4 ACE-style dense per-patch scene-coordinate regression replaces global map-cell probability field 7d02e02d
The loop discarded this — no improvement under the metric then in force.
architecture 2.001 9 m 11 s $2.92 DISCARDED
▸ 2.3 Argmax-anchored local soft-argmax decode replaces global expected-coordinate ff65590e
The loop discarded this — no improvement under the metric then in force.
architecture 1.991 4 m 58 s $1.25 DISCARDED
▸ 2.2 DSNT-style spatial probability field over the map replaces direct (u,v) regression ff65590e
The loop kept this — it beat the running best under the metric then in force. On today's ruler it scores 2.001 — worse than silence.
architecture 2.001 7 m 08 s $2.06 KEPT
▸ 2.1 Starting baseline — naive TinyLocNet, from-scratch, frozen pipeline v1 b47d44a8
The loop kept this — it beat the running best under the metric then in force. On today's ruler it scores 2.001 — worse than silence.
architecture 2.001 — — KEPT
▸ 1.7 Relight v1 — ambient-compensated auto-exposure + golden/blue-hour grading (eval-set reset) 678dd1e7
The loop kept this — it beat the running best under the metric then in force. On today's ruler it scores 2.001 — worse than silence.
relighting 2.001 — — KEPT
▸ 1.6 Per-epoch crop resampling — train on fresh positions/rotations every epoch instead of a fixed 4,800-crop snapshot 4959f8fb
The loop discarded this — no improvement under the metric then in force.
training 2.001 — — DISCARDED
▸ 1.5 Grid-cell classification + within-cell offset regression replaces direct (u,v) MSE regression 7b7b491a
The loop discarded this — no improvement under the metric then in force.
loss 1.996 — — DISCARDED
▸ 1.4 Split rebalance — 360 m blocks so the no-leakage buffer stops eating the train area 2e5b3020
The loop kept this — it beat the running best under the metric then in force. On today's ruler it scores 2.001 — worse than silence.
other 2.001 — — KEPT
▸ 1.3 Data v2 baseline — same TinyLocNet on 1 m/px DOP imagery, all 4 dev areas c0a1bdfe
The loop kept this — it beat the running best under the metric then in force. On today's ruler it scores 1.996 — worse than silence.
other 1.996 — — KEPT
▸ 1.2 Harness smoke test (SKIP_AGENT loop.sh iteration, 1 epoch) 0687f20e
The loop kept this — it beat the running best under the metric then in force. It cannot be placed on today's ruler at all — see the row for why.
other — — — NO SCORE
▸ 1.1 Bootstrap baseline — naive TinyLocNet direct-coordinate regressor 15b43a25
The loop kept this — it beat the running best under the metric then in force. It cannot be placed on today's ruler at all — see the row for why.
architecture — — — NO SCORE
What do the columns and marks mean?
Mission score
The single number the loop optimizes, and deliberately the product requirement rather than a statistic about errors. The aircraft takes a vision fix every 5–10 s and it is its only drift correction, so per frame exactly three things can happen: it is confident and within 100 m (a usable fix — the product), not confident (it abstains — safe, it waits), or confident and wrong (a false fix — dangerous, it feeds a wrong position into navigation). The score is (1 − usable-fix rate) + false-fix rate: 0 means every frame is a usable fix, 1.0 means it abstains on everything, 2.0 means it is confidently wrong on everything — strictly worse than silence, which is the point. Error statistics (median, geometric mean, p10, p25) are all still recorded, but none is optimized: each was tried as the target and each rewarded something the aircraft does not want.
Region-holdout (diagnostic)
A small 1-in-32-block slice of Berlin is kept genuinely out of training and scored separately. It is logged and never optimized — it cannot move keep/revert. Its only job is to show whether the model has real spatial structure or is just a lookup table over memorized vantages, so expect it to read far worse than the headline number. That gap is information, not failure.
not tested (○)
Never measured. The design and code were fine as far as anyone knows, but the cloud run failed before a result came back (no GPU capacity, or the connection to the training machine dropped). It says nothing about the idea, which the loop is free to try again.
gated fail (×)
The §6 score also enforces the aircraft's hard limits: the exported model must fit the ESP32-P4 flight computer (≤ 32 MiB exported, about 8 MiB once compressed to 8-bit for the chip; 4 MiB until 1 November 2026) and answer within the latency budget (≤ 250 ms host proxy), and it must produce at least one usable fix (a model that only ever stays silent or is always wrong counts as failed; until 1 November 2026 this was a cruder rule that failed any cell abstaining on > 80% of frames). Any violation scores the whole experiment as failed regardless of accuracy.
rejected
Never trained at all. After a losing streak, the loop demands a genuine pivot — every non-frozen part of the design must change, not just the one piece that's gone stalest. If the proposed design doesn't clear that bar (checked against the actual code diff, not just what the agent claims it changed), it's rejected before spending any GPU time on it.
cov (coverage)
Share of test frames the model was confident enough to answer at all. Abstaining honestly on bad frames is allowed — down to the 20% floor above.
Kept / Discarded
Karpathy-style loop discipline: branch from the best code, run one focused experiment, keep the change (git commit) only if the worst-case error improves — otherwise revert it. The step line in the chart is the running best.
Eval-set reset (┆)
During the bootstrap phase the frozen evaluation data itself was revised twice (10 m satellite → 1 m orthophotos; then a split rebalance). Scores on different eval sets are measurements on different rulers and must not be compared — the dashed vertical rule marks the break, and the running-best line restarts there instead of pretending continuity. From Phase 2 on, the eval set does not change.
Holdout (○)
Hamburg is the blind fifth area: structurally different (port, river, spread-out), never seen by the loop, scored only as a periodic read-only check. If its error diverges from the four development areas, the pipeline has learned their quirks rather than a general method. Its result never influences keep/revert.
Pivot-directed (▽)
After 4 consecutive experiments in a row fail to beat the running best, the harness injects a mandatory pivot preamble into the next design prompt: do not refine the champion's current mechanism again — propose from a design family absent from the recent history. Marked ones ran under that directive; the streak resets every time an experiment is kept.
Cost
Estimated cloud-GPU spend for that one experiment: its measured wall time × a $0.69/hr rate, billed continuously by the clock (not per phase), so it covers the whole experiment — design, implementation, training, scoring, publishing — not GPU time alone. Shown only for the middle, rented-GPU window; the bootstrap and later local eras run at ~$0 marginal compute and show —.
Category
Which lever the experiment pulls: architecture, loss, augmentation, relighting, training procedure, or quantization.
Where the deployment numbers went
Exported model size, weight initialization and single-frame latency each used to have a column. All three were dropped: nearly every row read the same, and the size and latency gates are already folded into the mission score — a model that misses either scores as a gated fail, so the column was restating the Status column. All three remain in the expanded record, and init also on the model designs page.
Training set
Each area offers ~45,000 distinct training positions (1 m/px, 128 m crops, times random rotation); how many crops an experiment actually samples per lighting condition is its own choice and is shown in the detail view. Training crops never overlap the eval blocks — enforced by the frozen split, not by convention.
Agent prompt
Every loop experiment records the exact prompt its headless agent received — expandable in the detail view, so any experiment can be re-run or audited later.
In plain words
Each experiment's own jargon-free explanation of what it tried, pre-registered alongside the technical design — read this first if the title looks like alphabet soup.
One real test
The figure at the top of each detail view is not an illustration: the same held-out Berlin viewpoint is fed to that experiment's actual exported model, and the red glow on the map is the probability field recovered from the deployed ONNX artifact itself — with the true location (○), the model's answer (×), and the real miss distance. Because every experiment sees the identical crop, any difference between two figures is the mechanism change, not the example. The pipeline sentence beneath names the stages; red marks what that experiment changed (a “+” stage is training-only and never flies).
Click any row — or any point in the chart — to see the experiment's pre-registered hypothesis, method and expected outcome, the measured result, the per-area × lighting scoreboard, and what the model actually looked at.

Every experiment the project has run, across 6 evaluation eras. The lineage was wiped at each era boundary, so the numbering restarts inside every band — but the vertical scale is one ruler throughout. 61 of these were re‑measured for this chart by running their exported model through today’s frozen scorer on today’s held‑out viewpoints; 24 are native to the current era; 0 had their run artifacts deleted and have their usable- and false-fix rates recovered arithmetically from the logged record; 17 failed a deployment gate and had no working model to score, then or now. Nothing was converted by a fudge factor. Black means a new best on today’s ruler — the dot that lowered the step line. The step line is the best mission score anyone had actually achieved, and it is unbroken across the boundaries because the ruler no longer changes at them. What the loop decided at the time, under whatever metric was then in force, is in each dot’s tooltip; the two disagree often in the early bands, because the old metrics rewarded the wrong thing.

scroll to zoom · drag to pan · double-click to reset
view on GitHub