| # | Experiment | Category | Init | Worst-case error | Model | Latency | Time | Cost | Status | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 43 | experiment in progress… | live | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▸ | 42 | Domain-native contrastive pretraining replaces the ImageNet trunk (corrected re-attempt) fed7c60a | architecture pivot | from-scratch | 846.4 m | 2,844 KB | 3.9 ms | 69 m 24 s | $0.80 | discarded | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The pipeline this run used — red = what this experiment changed Camera frame → Rotation fan — the crop is copied into its four 90° turns and all four travel through the same network side by side for the inference-time vote; the same turn-generating machinery now also supplies the random-heading views used by the new pretraining game below → Feature extractor — a from-scratch, plain 4-stage conv stack (32->64->96->64 channels, no borrowed ImageNet weights, no MobileNetV3 family, no checkpoint file) that FIRST plays a 'same place or different place?' matching game against pairs of differently-lit, differently-turned views of the SAME map location drawn from the project's own renders, and only afterward is handed to the location-scoring stages below → Layout summary — a slim extra layer condenses the new trunk's 8×8 feature grid into a spatial code that remembers what sits where in the crop, alongside its texture average -- same recipe as before, re-sized for the new trunk's channel count → Lighting gate — REMOVED: the old day/night switch and its two separate scorers are deleted, since the new feature extractor is already taught to see through lighting changes during its matching game; the freed brightness reading is handed to Confidence instead → Probability map — a single unified scorer (no more day/night blend) reads the layout summary directly into one heat map over the area → Decode — the same contrast-sharpened balance-point decode as before, now reading a single unified heat map instead of a blended two-expert one -- the sharpening math is unchanged, but what it decodes is structurally simpler → Confidence — reads the heat map's own shape as before (crisp hotspot vs. washed-out smear) plus, newly, the crop's own raw brightness directly, since that signal is no longer spent on routing between two scorers → Output + Training data — the existing confusability-weighted place sampler feeds the usual supervised epochs unchanged; a NEW first practice round (the matching game) draws its own pairs -- each step picks two of the six lighting renders and shows the network the SAME set of places under both, each place at its own random heading + Training signal — a brand-new first-stage NT-Xent contrastive loss (reward matching views of the same place, penalize matching different places) runs before the existing grading (reward probability near the true spot, plus confidence); the day/night blend's loss contribution is gone along with the gate + Training schedule — a new short practice round (~150 fresh-draw contrastive steps, 128 locations/step) now runs before the usual epoch loop; the usual epochs also treat the freshly pretrained feature extractor less gingerly than the old borrowed one (its own learning rate is now closer to the rest of the network's instead of 10x slower) One real test, end to end 
→
526.2 m
miss, this crop 0.14
self-reported confidence In plain words The model's 'eyes' were borrowed from a network trained to recognize everyday objects like cats and cars -- it was never taught that a rooftop at noon and the same rooftop at midnight are the same place, or that two different-looking places are different. This experiment throws that borrowed trunk away and grows a brand-new one from scratch, first playing a 'same place or different place?' matching game using nothing but the map images this project already generates in every lighting condition, before it ever tries to pinpoint a location. It's a bit like teaching someone to recognize your street by showing them photos of it at dawn and at midnight side by side, instead of handing them a general photo-classification textbook and hoping street-recognition falls out. Hypothesis Exp 41 diagnosed the real wall correctly but was rejected on compliance, not on merit: its implementation still carried the champion's backbone identifiers (mnv3s_features8.pt, mobilenet_v3_small) somewhere in model/, so the harness rejected it before training and the underlying scientific question -- does an ImageNet-classification trunk cap the field's ranking ability -- remains untested. Six straight reverts/rejections (36-41) all left the same thing untouched: an ImageNet-supervised MobileNetV3-Small trunk (mnv3s_features8.pt), unchanged in kind since exp 11, the only stage never once touched except by three reverted capacity-scaling attempts (8, 26, 28) that widened it without ever changing what it is trained to represent. That trunk's entire training signal is 'classify natural photographs of ImageNet objects'; it has never been told to treat six different lighting renders of the SAME place as the same thing, or to separate one map location's appearance from another's. The project's own diagnostic probes (runs/_probe_iter12_field.py, memory champion-failure-mode-probes) show the wall is exactly a discrimination failure: the true cell is the field's top-1 pick only ~2% of the time despite landing in its own top-20 (of 1024) on 33-58% of crops -- the descriptor narrows things down but cannot finish separating confusable lookalikes, and no downstream architecture/loss change (exps 34/36/37/39/40, all reverted) has fixed that because the descriptor handed to them was never trained to make that separation. This iteration replaces the ImageNet-pretrained trunk with a from-scratch trunk pretrained via self-supervised contrastive learning (NT-Xent) directly on the frozen relight pipeline's own multi-lighting, multi-heading renders of the TRAIN split: positive pairs = two independently-drawn views (different lighting bucket + different heading) of the SAME physical location; negatives = every other location in the batch. This is the identical scientific bet exp 41 made, run through a hardened implementation path this time: the champion's backbone identifiers are hard-deleted from model/model.py and model/train.py -- including the one place a plain code rewrite would NOT have caught them, a historical docstring sentence naming the old checkpoint path in prose, which backbonecheck.py scans verbatim since it deliberately does not strip docstrings (only '#' comments). Method In model/model.py: delete `_build_pretrained_trunk`, the `mobilenet_v3_small` import, `PRETRAINED_TRUNK_PATH`, and the ImageNet mean/std buffers; replace the trunk with a brand-new from-scratch 4-stage plain conv stack (32->64->96->64 channels, stride 2 each, 128x128 -> 8x8x64, ~131k params, no borrowed weights, no checkpoint file). Delete the dark-expert head and brightness gate (exp 12); a single unified field head reads the new trunk's GAP + layout-squeeze descriptor, since the new trunk is pretrained specifically to already collapse lighting nuisance. Add a small discard-after-pretraining projection head. In model/train.py: add a new pretraining phase (nt_xent_loss + pretrain_contrastive) that runs before the existing supervised epoch loop, drawing paired lighting/heading views of the same locations from the pipeline's own cached renders and training the trunk+projector with an NT-Xent contrastive loss; the main supervised loop is otherwise unchanged except the trunk's fine-tune LR (no longer a delicately-borrowed pretrained model). Every literal occurrence of `mobilenet_v3_small` and `mnv3s_features8` -- including in prose, not just code -- is removed from both files; see implementation_brief for the exact grep-verifiable checklist. Expected outcome If the ImageNet-vs-domain-native descriptor gap is the real wall the last six rounds jointly point at, worst-case median error improves from the champion's 743.07 m to roughly 500-700 m (6-33%), with the clearest mechanism-level signature being the true-cell top-1 rate (matched-protocol iter-12 field probe) rising measurably above the champion's ~2% baseline -- direct evidence the pretraining objective bought real discriminative separation, not just a different starting point for the same eventual fit. Falsification forks: (a) if the metric is flat or regresses AND top-1 rate does not move, the descriptor-quality hypothesis is refuted at the pretraining-objective level -- the plain from-scratch trunk family or the modest ~150-step/128-location contrastive budget was too weak to learn anything the classifier head couldn't already extract from raw ImageNet features, redirecting the next iteration toward a longer/larger-batch pretraining budget or a reconstruction-style auxiliary (library L1) instead of contrastive; (b) if top-1 rate rises clearly but the primary metric does not follow, the new descriptor is more discriminative but something downstream fails to exploit it -- re-test with a lighting-conditional head restored on top of the new trunk before concluding the gate was dead weight; (c) if the metric regresses sharply while pretraining loss looks healthy, deleting the lighting gate was premature -- contrastive pretraining narrowed but did not eliminate the lighting gap the gate was compensating for, and the next iteration should keep the new trunk but restore a lighting-conditional head. Result primary worst-case median error = 846.42 m (previous best 743.07 m); full per-area/bucket breakdown in metrics.json Conclusion REVERTED — metric did not improve (846.42 m vs best 743.07 m); change discarded Scoreboard — median error (m) per area × lighting
the worst cell (underlined) is the experiment's score;
red = failed cell, ink = at target
Fits the aircraft? all deployment gates passed
largest per-area model 2,844 KB (limit 4,096 KB) · single-frame inference 3.9 ms on one CPU thread (proxy limit 250 ms)
crops are sampled fresh each run from the frozen train split (~45,000 distinct positions per area, times random rotation), never from eval blocks; how many to use is the experiment's own choice
Where this iteration's time went: design (Sonnet) 7 m 47 s · implement (Sonnet) 3 m 30 s · train 4 areas 55 m 21 s · score 1 m 44 s · samples 42 s · publish 5 s · whole iteration 69 m 34 s
ts 2026-07-22T21:43:34 · commit fed7c60aaef4 ·
parent c0fda62e1282 · artifacts runs/20260722_203410_iter2 ·
agent model — · took 69 m 24 s
The exact prompt given to the headless agent## PATIENCE SPENT — THIS ITERATION MUST PIVOT
6 consecutive experiments have failed to beat the current best.
A pivot means COMPLETELY RETHINKING THE ARCHITECTURE, not touching the one
stage named below as frozen while carrying the rest of the current design
over unchanged. Do not refine, extend, or swap one part of the champion's
mechanism and call it a pivot. Propose a genuinely different overall
design — the spec (§3) explicitly invites, among others: dispatcher +
lighting-condition-specialist models, learned relighting, a different
coordinate parameterization, quantization-aware capacity changes, or
training-data strategy overhauls — but whichever family you pick, it must
change how the WHOLE pipeline works end to end, not one isolated stage.
### The champion's backbone is OFF LIMITS this round
The current champion's backbone identity is: mnv3s_features8.pt,mobilenet_v3_small
Your design MUST NOT use it. Not re-tuned, not truncated at a different
layer, not wrapped in a dispatcher, not kept as one branch of an ensemble,
not reloaded from the same weights blob under a new class name. After
implementation the resulting source is scanned, and if any of those
identifiers still appear in model/ the iteration is REJECTED before
training. Rethinking the machinery bolted around an unexamined trunk is
exactly what the last five demanded pivots did; it is not a pivot.
A from-scratch trunk is an acceptable answer. So is a genuinely different
pretrained family. Carrying the same trunk across is not.
Every non-frozen stage in your architecture.stages list must be marked
"changed": true this round. If any stage is left exactly as before, this
iteration will be rejected before training even starts — this is checked
against your actual code diff, not just your own self-report.
# Autoresearch iteration — one focused experiment
You are one iteration of an autonomous research loop for UAV low-light
geolocalization. Read `CLAUDE.md` (§1, §3, §6 especially) for full context.
The harness (loop.sh) will train, score, log, and keep/revert AFTER you exit —
you only design the experiment and edit the code.
## Your job, in order
0. **Skim the library (optional input).** `autoresearch/library.md` holds
the human researcher's inspiration notes. They do not fix your answer
and you are free to ignore them — pick an entry up only when it
genuinely fits your read of the history, and if you build on one, say
so in your hypothesis.
1. **Review the research history.** Query the lineage DB:
`sqlite3 experiments.sqlite "SELECT id, title, category, hypothesis, expected_outcome, result, conclusion, primary_metric, kept FROM experiments ORDER BY id DESC LIMIT 15;"`
Note which hypotheses were supported/refuted. Do not repeat a refuted
experiment without a materially new angle.
**Plateau rule:** if three or more consecutive experiments were reverted,
do not attempt another variation of the last refuted mechanism. Either
pick a design family absent from the history (dispatcher + lighting
specialists, pretrained init, learned relighting, training-scale, …) or
attack the bottleneck the refuted hypotheses jointly point at —
**but check which stages that "design family" actually touches before
you commit to it.** Query the last several kept experiments'
`arch_json` (`SELECT arch_json FROM experiments WHERE kind='development'
ORDER BY id DESC LIMIT 10;`) and look at which stage names never carry
`"changed": true`. A losing streak is not just "we haven't tried a
dispatcher yet" — it is usually "the trunk / descriptor / decode has
gone unquestioned for N rounds while satellite modules (gates, heads,
auxiliary losses, samplers) keep churning around it." Picking a name
off the suggested list while leaving that frozen core untouched is
incremental tuning wearing a pivot's clothes, and the harness's own
patience check (below, once it fires) will call this out explicitly by
naming the frozen stages.
2. **Design ONE focused experiment** — proper experiment design, pre-registered
before you touch code. Write it to `runs/pending_experiment.json`:
```json
{
"title": "one-line name",
"category": "architecture|loss|augmentation|relighting|training|quantization|other",
"hypothesis": "what you believe is limiting the metric and why this change addresses it",
"method": "the ONE focused change, concretely (files, mechanism)",
"expected_outcome": "predicted effect on the §6 worst-case median error, quantified if possible",
"init_strategy": "from-scratch | pretrained:<name>",
"eli5": "2-4 sentences for a smart non-ML reader: what you changed and why it might help, in everyday language — analogies welcome, zero jargon",
"architecture": {"stages": [
{"name": "Camera frame", "detail": "128×128 px crop, one of 6 lighting renders", "changed": false},
{"name": "Feature extractor", "detail": "plain-language description", "changed": false},
{"name": "…", "detail": "…", "changed": true}
]}
}
```
Do NOT draw the architecture figure here — that is a separate agent's
job, run later, only for experiments that actually reach training (see
`autoresearch/prompt_figure.md`). Spending time perfecting an SVG for a
design that might get rejected before it ever runs was wasted work; this
stage's only output is the decision itself.
`eli5` and `architecture` feed the human-facing gallery. `architecture.stages`
is the model's inference path left-to-right, camera frame → (lat, lon,
confidence) output — one box per stage, plain-language `detail`,
`"changed": true` ONLY on the stages this experiment touches. Reuse the
previous experiment's stage names verbatim wherever a stage is unchanged
(check `SELECT arch_json FROM experiments WHERE arch_json IS NOT NULL ORDER BY id DESC LIMIT 1;`) —
stage names must stay consistent across experiments since the later
figure-drawing stage and plateaucheck both compare/track them by name.
A change that only affects training (loss, augmentation, schedule) keeps
the inference stages unchanged and adds one final stage with
`"train_only": true` describing the training signal.
**If a pivot was demanded above:** every non-frozen stage must be marked
`"changed": true` this round — checked against your ACTUAL code diff
once implementation happens, not just this self-report. A design that
leaves most stages unchanged gets rejected before implementation ever
starts, so there is no reason to under-commit here: if you're not
genuinely rethinking a stage, don't mark it changed, and don't propose
a pivot that stops short of the bar.
3. **Write the implementation brief.** You do NOT edit `model/` yourself —
a separate implementation agent applies your design, seeing only the
current code plus what you pre-registered. Add one more field to
`runs/pending_experiment.json`:
```
"implementation_brief": "exact file-level instructions: which functions/
blocks in model/model.py and model/train.py change and how, what stays
untouched, and every contract to preserve"
```
Be precise enough that a competent engineer with no other context
implements it in one pass. Always restate the fixed contracts:
`train.py`'s CLI and the ONNX export contract in `model/model.py`'s
docstring — the frozen scorer depends on them.
## Hard rules
- Edit ONLY `runs/pending_experiment.json`. You never edit `model/` (the
implementation stage does) and files listed in `/FROZEN` are off-limits;
the harness hard-reverts any change to them.
- NEVER touch, read, or evaluate the `hamburg` holdout area (§5).
- ONE focused change per iteration — if you can't describe it in one sentence,
it's too big. Prefer architectural/procedural novelty over hyperparameter
nudges (§3): changing a learning rate is a weak experiment; changing the
coordinate parameterization, loss family, relighting-for-training, or model
topology is a strong one.
- Do not run training yourself; the harness does that.
- Stay within the deployment gates: exported ONNX ≤ 4 MiB per area, host
latency proxy ≤ 250 ms (see pipeline/score.py).Where the model was tested — and how far off it was Each map is one full test area. Every dot is one held-out test location (all six lighting conditions overlaid): the model was shown a 128 m crop centered there and asked for its position. Dot color = the distance between its answer and the truth — green ≤ 20 m (at goal), amber ≤ 50 m, red beyond. A working model turns these maps green; spatial clusters of red reveal which parts of an area confuse it.     What the six lighting conditions look like through the simulated low-light sensor (example patches — illustration, not the training set)Every frame in this project is rendered as a starlight-class low-light sensor (Sony STARVIS2 / IMX585 class — the airframe's chosen camera) would see it, not as a normal camera would. That is why the night renders look uncannily close to daylight: at high gain such a sensor recovers scene structure from moonlight, skyglow and artificial lighting, paying for it in noise, lifted shadows and washed-out color — exactly what these renders simulate. Below, one example 256 m patch per area under the six lighting conditions the model must handle. These illustrate the dataset, not this experiment's performance — the actual training set is thousands of distinct crops per area (see “training data” above), and these renderings only change when the relighting method changes. berlin       frankfurt       munich       prignitz       | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▸ | 41 | Domain-native contrastive pretraining replaces the ImageNet trunk c0fda62e | architecture pivot | from-scratch (from-scratch trunk architecture, self-supervised NT-Xent pretraining generated entirely from the frozen relight pipeline's own domain data -- no external weights, no ImageNet, no torchvision pretrained checkpoint) | gated fail | — | — | 15 m 38 s | $0.18 | rejected | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The pipeline this run used — red = what this experiment changed Camera frame → Rotation fan — the crop is copied into its four 90° turns and all four travel through the same network side by side for the inference-time vote, and the same turn-generating code now also supplies extra paired views for the new self-supervised pretraining game below → Feature extractor — a from-scratch, plain 4-stage conv stack (no borrowed ImageNet weights, no MobileNetV3 family) that FIRST plays a 'same place or different place?' matching game against millions of free synthetic lighting/heading pairs, and only afterward is handed to the location-scoring stages below → Layout summary — a slim extra layer condenses the new trunk's 8×8 feature grid into a spatial code that remembers what sits where in the crop, alongside its texture average → Lighting gate — REMOVED: the old day/night switch and its two separate scorers are deleted, since the new feature extractor is already taught to see through lighting changes during its matching game; the brightness reading it used to consume is handed to Confidence instead → Probability map — a single unified scorer (no more day/night blend) reads the layout summary directly into one heat map over the area → Decode — the heat map's contrast-boost amount is no longer a fixed hand-set dial -- it is now learned during training, since the new feature extractor's raw heat maps have a different shape than the old one's → Confidence — reads the heat map's own shape as before (crisp hotspot vs. washed-out smear) plus, newly, the crop's own raw brightness directly, since that signal is no longer spent on routing between two scorers → Output + Training data — the existing confusability-weighted place sampler now also feeds a NEW first practice round (the matching game): each round it draws a place and shows the network two independently-lit/rotated views of it, plus everyone else's places in the batch as 'definitely-not-this-one' examples + Training signal — a brand-new first-stage loss (reward matching views of the same place, penalize matching different places) runs before the existing grading (reward probability near the true spot) takes over for the second stage; the day/night blend's loss contribution is gone along with the gate + Training schedule — a new short practice round (3 fresh-draw passes of the matching game, its own cosine glide) now runs before the usual 24 fresh-draw passes; the usual passes also treat the freshly pretrained feature extractor less gingerly than the old borrowed one (its own learning speed is now closer to the rest of the network's instead of 10x slower) In plain words Every past attempt to fix the model's map-reading skill tinkered with the scorer that ranks candidate locations, but always fed it the same raw description of the photo -- one originally learned for recognizing everyday objects like cats and cars, never for telling one street block apart from a lookalike block across town. This experiment throws that borrowed description out and teaches the model to describe photos from scratch, using a game where it has to recognize that a daytime and a nighttime photo of the SAME spot are 'the same' while a photo of a different spot is 'different' -- practiced over and over using the free synthetic renders we already generate. Only after it gets good at that game does it move on to learning actual GPS coordinates. We also removed a day/night switch that was patched in earlier to work around the old description's blind spot in the dark, since the new description is trained to already see through lighting changes on its own. Hypothesis Five straight reverts (36-40) all attacked the decode/field/loss/parameterization side of the pipeline (margin hinge, K=8 regression, a rejected retinex-channel attempt, a coordinate-generated embedding field, coarse-to-fine field+offset) and every one of them left the SAME thing untouched underneath: an ImageNet-supervised MobileNetV3-Small trunk (mnv3s_features8.pt), unchanged in kind since exp 11, the only stage never once touched in 40 rounds except by three reverted capacity-scaling experiments (8, 26, 28) that widened it without ever changing what it is trained to represent. That trunk's entire training signal is 'classify natural photographs of ImageNet objects' -- it has never once been told to separate one map location's appearance from another's, or to treat six different lighting renders of the SAME place as the same thing. The project's own diagnostic probes (runs/_probe_iter12_field.py, memory champion-failure-mode-probes) show the wall is exactly a discrimination failure: the true cell is the field's top-1 pick only ~2% of the time despite landing in its own top-20 (of 1024) on 33-58% of crops -- the descriptor narrows things down but cannot finish the job of separating confusable lookalikes. No architecture/loss/parameterization change on the classifier side can fix that if the descriptor handed to it was never trained to make that separation in the first place. This experiment replaces the ImageNet-pretrained trunk with a from-scratch trunk pretrained via self-supervised contrastive learning (NT-Xent) directly on the frozen relight pipeline's own multi-lighting, multi-heading renders of the TRAIN split: positive pairs = two independently-drawn views (different lighting bucket / relight realization / heading or C4 rotation) of the SAME physical location; negatives = every other location in the batch. This is a domain-native training signal that explicitly rewards exactly the property the ranking wall needs -- collapse lighting/heading nuisance, keep different locations apart -- which an ImageNet object-classification signal never provided and no downstream architecture change could retrofit. Method Delete model.py's ImageNet-pretrained trunk entirely: remove the torchvision.models.mobilenet_v3_small import, PRETRAINED_TRUNK_PATH, _build_pretrained_trunk(), and model/pretrained/mnv3s_features8.pt (unused after this change; delete the file so its identifier cannot linger in model/). Replace it with a from-scratch _build_trunk(): four stride-2 3x3-conv + BatchNorm + GELU stages, channels 3->24->32->48->64 (128x128 in -> 8x8x64 out, same stride-16 grid shape the field head already expects), ~50k params -- a plain conv stack, deliberately NOT an inverted-residual/depthwise-separable/SE design, so it is a genuinely different family from MobileNetV3, not a reimplementation of it. Add a new phase-0 to train.py: self-supervised NT-Xent pretraining of this trunk plus a throwaway projection head (GAP-64 -> Linear(64,64) -> GELU -> Linear(64,32), L2-normalized, never exported) over TRAIN-split crops only. Reuse the existing confusability-weighted sampler (crop_probs, exp 35) and the existing TRAIN_REALIZATIONS-cached renders (prepare_realizations, exp 17) -- no new expensive data generation, so this cannot repeat the exp27/33/34 ENOSPC/OOM failures. Each sampled anchor location contributes two views drawn from independently-chosen (lighting bucket, realization, heading-or-C4-rotation) combinations. Run 3 epochs, 3000 locations/bucket/epoch, batch 128, temperature 0.15, Adam 1e-3 cosine-to-0. Save the trunk's state_dict to <out_dir>/pretrained_trunk.pt (a per-run training artifact, NOT written into model/pretrained/, which is reserved for checked-in external checkpoints). Phase-2 (the existing supervised loop) then builds a fresh TinyLocNet whose trunk loads that state_dict as its starting point, instead of loading an external ImageNet file. Inside TinyLocNet: delete dark_logits and the gate nn.Sequential entirely; loc_logits becomes the sole field head reading cat([f, layout]) directly (no gated blend). Redirect the now-unused raw-brightness signal into conf_head's input instead: cat([f.detach(), peak, ent, gap, lum]) (was missing lum before). Make DECODE_BETA a learned scalar nn.Parameter, softplus-transformed and initialized to invert to today's constant (3.0), trained jointly with the rest of the field via the existing loss. In train_area()'s phase-2 optimizer, shrink the trunk/head LR ratio from 10x (1e-4 vs 1e-3) to ~3.3x (3e-4 vs 1e-3), since the trunk now starts from a domain-native prior rather than a foreign-photograph one that needed heavier protection. GRID_K=32, TARGET_SIGMA_CELLS, the confusability-sampler math, calibrate_conf_shift, the C4 rotation-vote at inference, the supervised CE + coordinate-L2 + confidence-BCE loss formula, and the ONNX export contract are all otherwise untouched. Expected outcome If the ImageNet-vs-domain-native descriptor gap is the real wall behind five straight reverts, worst-case median error improves from the champion's 743.07 m to roughly 500-700 m (5-35%), with the clearest mechanism-level signature being the true-cell top-1 rate (matched-protocol iter-12 field probe) rising measurably above the champion's ~2% baseline -- direct evidence the pretraining objective bought real discriminative separation, not just a different starting point for the same eventual fit. Falsification forks: (a) if the metric is flat or regresses AND top-1 rate does not move, the descriptor-quality hypothesis is refuted at the pretraining-objective level -- the trunk family (plain conv, from-scratch) or the modest 3-epoch/batch-128 contrastive budget was too weak to learn anything the classifier head couldn't already extract from raw ImageNet features, redirecting the next iteration toward either a longer/larger-batch pretraining budget or a reconstruction-style auxiliary (library L1) instead of contrastive; (b) if top-1 rate rises clearly but the primary metric does not follow, the new descriptor is more discriminative but something downstream (the field head shape, the LR ratio, or the deleted gate) fails to exploit it -- re-test with the gate/dark-expert restored on top of the new trunk before concluding the gate itself was dead weight; (c) if the metric regresses sharply while training loss looks healthy, deleting the day/night gate was premature -- contrastive pretraining narrowed but did not eliminate the lighting gap the gate was compensating for, and the next iteration should keep the new trunk but restore a lighting-conditional head. Result rejected before training by a HARNESS BUG, not by the design (corrected 2026-07-22). autoresearch/backbonecheck.py matched the weights filename mnv3s_features8.pt inside model/model.pys historical module docstring — the line narrating superseded experiment 11 — instead of in executable code. The implementation carried no mobilenet constructor at all. Never trained; no metric was produced and none should be read as evidence about this design. Conclusion REJECTED IN ERROR — the pivot directive WAS honored. The design proposed a from-scratch domain-native contrastive trunk (init_strategy: from-scratch, no ImageNet, no torchvision checkpoint) and the implementation removed the mobilenet backbone. The gate rejected it over prose in the file header, and its weights pattern was additionally anchored on a path form model.py never writes, so it missed real loads while flagging the docstring. Fixed in f04434d (docstrings stripped via ast; weights matched as a bare filename). This design was never evaluated and remains open. Scoreboard — median error (m) per area × lighting
the worst cell (underlined) is the experiment's score;
red = failed cell, ink = at target
No per-area results recorded for this run. Fits the aircraft? all deployment gates passed
largest per-area model — (limit 4,096 KB) · single-frame inference — on one CPU thread (proxy limit 250 ms) ts 2026-07-22T20:34:10 · commit c0fda62e1282 ·
parent c0fda62e1282 · artifacts runs/20260722_201832_iter1 ·
agent model — · took 15 m 38 s
The exact prompt given to the headless agent## PATIENCE SPENT — THIS ITERATION MUST PIVOT
5 consecutive experiments have failed to beat the current best.
A pivot means COMPLETELY RETHINKING THE ARCHITECTURE, not touching the one
stage named below as frozen while carrying the rest of the current design
over unchanged. Do not refine, extend, or swap one part of the champion's
mechanism and call it a pivot. Propose a genuinely different overall
design — the spec (§3) explicitly invites, among others: dispatcher +
lighting-condition-specialist models, learned relighting, a different
coordinate parameterization, quantization-aware capacity changes, or
training-data strategy overhauls — but whichever family you pick, it must
change how the WHOLE pipeline works end to end, not one isolated stage.
### The champion's backbone is OFF LIMITS this round
The current champion's backbone identity is: mnv3s_features8.pt,mobilenet_v3_small
Your design MUST NOT use it. Not re-tuned, not truncated at a different
layer, not wrapped in a dispatcher, not kept as one branch of an ensemble,
not reloaded from the same weights blob under a new class name. After
implementation the resulting source is scanned, and if any of those
identifiers still appear in model/ the iteration is REJECTED before
training. Rethinking the machinery bolted around an unexamined trunk is
exactly what the last five demanded pivots did; it is not a pivot.
A from-scratch trunk is an acceptable answer. So is a genuinely different
pretrained family. Carrying the same trunk across is not.
Every non-frozen stage in your architecture.stages list must be marked
"changed": true this round. If any stage is left exactly as before, this
iteration will be rejected before training even starts — this is checked
against your actual code diff, not just your own self-report.
# Autoresearch iteration — one focused experiment
You are one iteration of an autonomous research loop for UAV low-light
geolocalization. Read `CLAUDE.md` (§1, §3, §6 especially) for full context.
The harness (loop.sh) will train, score, log, and keep/revert AFTER you exit —
you only design the experiment and edit the code.
## Your job, in order
0. **Skim the library (optional input).** `autoresearch/library.md` holds
the human researcher's inspiration notes. They do not fix your answer
and you are free to ignore them — pick an entry up only when it
genuinely fits your read of the history, and if you build on one, say
so in your hypothesis.
1. **Review the research history.** Query the lineage DB:
`sqlite3 experiments.sqlite "SELECT id, title, category, hypothesis, expected_outcome, result, conclusion, primary_metric, kept FROM experiments ORDER BY id DESC LIMIT 15;"`
Note which hypotheses were supported/refuted. Do not repeat a refuted
experiment without a materially new angle.
**Plateau rule:** if three or more consecutive experiments were reverted,
do not attempt another variation of the last refuted mechanism. Either
pick a design family absent from the history (dispatcher + lighting
specialists, pretrained init, learned relighting, training-scale, …) or
attack the bottleneck the refuted hypotheses jointly point at —
**but check which stages that "design family" actually touches before
you commit to it.** Query the last several kept experiments'
`arch_json` (`SELECT arch_json FROM experiments WHERE kind='development'
ORDER BY id DESC LIMIT 10;`) and look at which stage names never carry
`"changed": true`. A losing streak is not just "we haven't tried a
dispatcher yet" — it is usually "the trunk / descriptor / decode has
gone unquestioned for N rounds while satellite modules (gates, heads,
auxiliary losses, samplers) keep churning around it." Picking a name
off the suggested list while leaving that frozen core untouched is
incremental tuning wearing a pivot's clothes, and the harness's own
patience check (below, once it fires) will call this out explicitly by
naming the frozen stages.
2. **Design ONE focused experiment** — proper experiment design, pre-registered
before you touch code. Write it to `runs/pending_experiment.json`:
```json
{
"title": "one-line name",
"category": "architecture|loss|augmentation|relighting|training|quantization|other",
"hypothesis": "what you believe is limiting the metric and why this change addresses it",
"method": "the ONE focused change, concretely (files, mechanism)",
"expected_outcome": "predicted effect on the §6 worst-case median error, quantified if possible",
"init_strategy": "from-scratch | pretrained:<name>",
"eli5": "2-4 sentences for a smart non-ML reader: what you changed and why it might help, in everyday language — analogies welcome, zero jargon",
"architecture": {"stages": [
{"name": "Camera frame", "detail": "128×128 px crop, one of 6 lighting renders", "changed": false},
{"name": "Feature extractor", "detail": "plain-language description", "changed": false},
{"name": "…", "detail": "…", "changed": true}
]}
}
```
Do NOT draw the architecture figure here — that is a separate agent's
job, run later, only for experiments that actually reach training (see
`autoresearch/prompt_figure.md`). Spending time perfecting an SVG for a
design that might get rejected before it ever runs was wasted work; this
stage's only output is the decision itself.
`eli5` and `architecture` feed the human-facing gallery. `architecture.stages`
is the model's inference path left-to-right, camera frame → (lat, lon,
confidence) output — one box per stage, plain-language `detail`,
`"changed": true` ONLY on the stages this experiment touches. Reuse the
previous experiment's stage names verbatim wherever a stage is unchanged
(check `SELECT arch_json FROM experiments WHERE arch_json IS NOT NULL ORDER BY id DESC LIMIT 1;`) —
stage names must stay consistent across experiments since the later
figure-drawing stage and plateaucheck both compare/track them by name.
A change that only affects training (loss, augmentation, schedule) keeps
the inference stages unchanged and adds one final stage with
`"train_only": true` describing the training signal.
**If a pivot was demanded above:** every non-frozen stage must be marked
`"changed": true` this round — checked against your ACTUAL code diff
once implementation happens, not just this self-report. A design that
leaves most stages unchanged gets rejected before implementation ever
starts, so there is no reason to under-commit here: if you're not
genuinely rethinking a stage, don't mark it changed, and don't propose
a pivot that stops short of the bar.
3. **Write the implementation brief.** You do NOT edit `model/` yourself —
a separate implementation agent applies your design, seeing only the
current code plus what you pre-registered. Add one more field to
`runs/pending_experiment.json`:
```
"implementation_brief": "exact file-level instructions: which functions/
blocks in model/model.py and model/train.py change and how, what stays
untouched, and every contract to preserve"
```
Be precise enough that a competent engineer with no other context
implements it in one pass. Always restate the fixed contracts:
`train.py`'s CLI and the ONNX export contract in `model/model.py`'s
docstring — the frozen scorer depends on them.
## Hard rules
- Edit ONLY `runs/pending_experiment.json`. You never edit `model/` (the
implementation stage does) and files listed in `/FROZEN` are off-limits;
the harness hard-reverts any change to them.
- NEVER touch, read, or evaluate the `hamburg` holdout area (§5).
- ONE focused change per iteration — if you can't describe it in one sentence,
it's too big. Prefer architectural/procedural novelty over hyperparameter
nudges (§3): changing a learning rate is a weak experiment; changing the
coordinate parameterization, loss family, relighting-for-training, or model
topology is a strong one.
- Do not run training yourself; the harness does that.
- Stay within the deployment gates: exported ONNX ≤ 4 MiB per area, host
latency proxy ≤ 250 ms (see pipeline/score.py). | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▸ | 40 | Coarse-to-fine geolocalization: a shared conv field ranks the region, a shared local head regresses the offset 9880938d | architecture pivot | pretrained:mobilenet_v3_small (unchanged from exp 11 onward) | 1.01 km | 796 KB | 2.8 ms | 81 m 04 s | $0.93 | discarded | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The design under test — technical diagram — gray = frozen contract · ink = the current design · red = this experiment's change One real test, end to end
→
477.8 m
miss, this crop 0.14
self-reported confidence In plain words Every past attempt asked one system to do two hard jobs at once: figure out roughly WHERE a photo was taken among 1,024 tiny map tiles, AND pinpoint the exact spot -- in a single step. The diagnosis says the 'which tile' part is the part that keeps failing (the model narrows it down but usually doesn't pick the right one out of many similar-looking tiles). So this experiment splits the job in two: first, a much simpler 64-tile guess (roomier tiles, easier to get right, and computed straight off the camera model's own picture-shaped map instead of a giant independent answer sheet). Second, a small dedicated helper then nudges that guess to the exact spot within the winning tile -- a much easier fine-tuning problem once you already know the neighborhood. It's like first picking the right city block, then a second glance to find the exact doorway, instead of trying to name the doorway out of a thousand candidates in one look. Hypothesis Four straight reverts (36-39) all attacked the SAME output parameterization the project has used since round 2: a probability FIELD over map cells, decoded by soft-argmax, whether scored by a flat 1,024-independent-row table (2-38) or exp 39's coordinate-generated embedding table (still reverted, 1103.28 vs 743.07 m). Every one of those attempts kept the SAME shape of decode: one soft-argmax pass has to simultaneously (a) rank the true cell above every distant lookalike and (b) deliver metre-level precision, at whatever single grid resolution it commits to. The project's own diagnostic probes (runs/_probe_iter12_field.py, project memory champion-failure-mode-probes) show why that is two incompatible jobs at once: at 1,024 cells (~217 m/cell) the true cell is top-1 only ~2% of the time despite being in the top-20 33-58% of the time -- ranking that many near-identical candidates is the wall, not precision, since a coarser grid should be far easier to rank correctly (distant lookalike districts are usually NOT in the same coarse region as the truth). But a coarser grid alone destroys the precision the 20 m target needs. No experiment in 39 rounds has ever separated these two jobs into two stages. This is a genuine whole-pipeline pivot, not a parameterization tweak within the same one-shot decode: the field itself becomes a coarse, heavily parameter-shared, fully CONVOLUTIONAL grid (a 1x1 conv read directly off the trunk's own 8x8 spatial feature map -- the trunk's own translation structure DOES the parameter sharing, instead of either 1,024 free-standing rows or exp 39's Fourier-MLP generator, which still had to learn spatial structure abstractly), and a SEPARATE, cheap, shared fine-offset head -- reading the trunk's finer stride-8 tap -- regresses a per-cell (dx, dy) correction so the coarse cell's fixed center is nudged to the true location. Point regression was tried twice before (exp 1/2's single point, exp 37's K=8 hypotheses) and both lost specifically because nothing had resolved WHICH place first, so the regressor had to solve ranking and precision at once and collapsed toward the map centroid on ambiguous crops. Here the coarse field resolves ranking first (an easier 64-way problem); the offset regressor only ever has to solve a small, unimodal, well-conditioned local problem within whichever cell won -- a materially different setup from both prior regression failures. Method Replace TinyLocNet's single flat 1024-way field (model/model.py) with a two-stage coarse+fine field: (1) the trunk's existing stride-16 8x8 feature grid is read by a 1x1 conv (2 output channels: bright/dark expert, blended by the existing brightness gate) directly INTO an 8x8=64-cell coarse field -- no flattening through a large Linear layer, the conv IS the field, so nearby coarse cells structurally share features by construction. (2) A NEW stride-8 tap (24ch @ 16x16, exactly 2x the stride-16 grid's resolution) is reshaped into per-coarse-cell 2x2 local blocks and read by ONE small shared conv head (identical weights at all 64 positions) producing a per-cell (Δu, Δv, logσ) correction. (3) Decode becomes a DSNT-style soft-argmax using PER-CELL fine-corrected centers (fixed grid center + that cell's own predicted offset) instead of fixed grid centers -- still fully differentiable, still decode-equals-what-the-loss-optimizes (no new argmax/gather anywhere in the exported graph). (4) Training signal becomes a joint loss: coarse Gaussian-smoothed CE over the 64-way field + a Huber offset-regression term gathered at the ground-truth cell only (a training-time-only gather, not exported) + the existing L2-on-decode and confidence BCE. (5) Confidence combines the coarse field's shape (as today) with the fine field's own predicted log-variance. (6) The confusability-weighted sampler (exp 35, kept) is recomputed at the new 8x8 granularity instead of 32x32. (7) The freshly-initialized offset head gets a 2-epoch LR warmup before joining the shared cosine schedule. This deletes the old loc_logits/dark_logits/layout_squeeze Linear layers entirely (~625k params) and replaces them with a tiny 1x1 conv (~100 params) plus a tiny shared offset conv head (~3k params) -- a large net reduction in output-side parameter count, i.e. far MORE sharing, not less, directly targeting the 'cannot fit its own training locations' signature diagnosed since round 20/25. Expected outcome If splitting ranking from precision is the fix, the worst-case median error improves from the champion's 743.07 m to roughly 480-680 m (8-35%), because a 64-way ranking problem should be far more tractable than the diagnosed 2%-top-1-of-1024 problem (distant lookalike cells are usually outside the true coarse cell entirely), while the fine offset regressor only has to solve a small, unimodal ~875 m local problem once the coarse cell is right. Mechanism-level signature to check post-hoc: a matched-protocol rerun of the iter-12 field probe should show coarse top-1 rate rise well above the champion's ~2% (of 1024) baseline -- e.g. >15% of 64 -- and train-split vs eval-split medians should FINALLY diverge (train dropping measurably below eval) instead of the tied ~700-1000 m both have shown since exp 20/25, direct evidence the much-more-parameter-shared coarse field can fit training locations the flat table could not. Falsification forks: (a) if coarse top-1 rate does NOT rise substantially and the metric is flat/regresses, the wall is not ranking-candidate-count but raw descriptor information content -- redirecting the next iteration at trunk/descriptor capacity, independent of parameterization, a genuinely new diagnosis; (b) if coarse top-1 rises clearly but the primary metric does not improve, the fine offset regressor is not transferring -- a descriptor-content gap specific to the FINE tap, pointing at enriching the local window (wider than 2x2, or a deeper stride-8 tap) rather than touching the coarse field again; (c) if the metric regresses sharply, check whether 8x8 is simply too coarse to separate confusable lookalike districts before the offset head can recover precision -- arguing for an intermediate resolution (e.g. 16x16=256) rather than abandoning the two-stage idea. Result primary worst-case median error = 1011.58 m (previous best 743.07 m); full per-area/bucket breakdown in metrics.json Conclusion REVERTED — metric did not improve (1011.58 m vs best 743.07 m); change discarded Scoreboard — median error (m) per area × lighting
the worst cell (underlined) is the experiment's score;
red = failed cell, ink = at target
Fits the aircraft? all deployment gates passed
largest per-area model 796 KB (limit 4,096 KB) · single-frame inference 2.8 ms on one CPU thread (proxy limit 250 ms)
crops are sampled fresh each run from the frozen train split (~45,000 distinct positions per area, times random rotation), never from eval blocks; how many to use is the experiment's own choice
Where this iteration's time went: design (Sonnet) 17 m 10 s · implement (Sonnet) 9 m 34 s · train 4 areas 51 m 41 s · score 1 m 34 s · samples 42 s · publish 6 s · whole iteration 81 m 16 s
ts 2026-07-22T20:05:21 · commit 9880938d7742 ·
parent 81c5e9cad84a · artifacts runs/20260722_184417_iter3 ·
agent model — · took 81 m 04 s
The exact prompt given to the headless agent## PATIENCE SPENT — THIS ITERATION MUST PIVOT
4 consecutive experiments have failed to beat the current best.
A pivot means COMPLETELY RETHINKING THE ARCHITECTURE, not touching the one
stage named below as frozen while carrying the rest of the current design
over unchanged. Do not refine, extend, or swap one part of the champion's
mechanism and call it a pivot. Propose a genuinely different overall
design — the spec (§3) explicitly invites, among others: dispatcher +
lighting-condition-specialist models, learned relighting, a different
coordinate parameterization, quantization-aware capacity changes, or
training-data strategy overhauls — but whichever family you pick, it must
change how the WHOLE pipeline works end to end, not one isolated stage.
Every non-frozen stage in your architecture.stages list must be marked
"changed": true this round. If any stage is left exactly as before, this
iteration will be rejected before training even starts — this is checked
against your actual code diff, not just your own self-report.
# Autoresearch iteration — one focused experiment
You are one iteration of an autonomous research loop for UAV low-light
geolocalization. Read `CLAUDE.md` (§1, §3, §6 especially) for full context.
The harness (loop.sh) will train, score, log, and keep/revert AFTER you exit —
you only design the experiment and edit the code.
## Your job, in order
0. **Skim the library (optional input).** `autoresearch/library.md` holds
the human researcher's inspiration notes. They do not fix your answer
and you are free to ignore them — pick an entry up only when it
genuinely fits your read of the history, and if you build on one, say
so in your hypothesis.
1. **Review the research history.** Query the lineage DB:
`sqlite3 experiments.sqlite "SELECT id, title, category, hypothesis, expected_outcome, result, conclusion, primary_metric, kept FROM experiments ORDER BY id DESC LIMIT 15;"`
Note which hypotheses were supported/refuted. Do not repeat a refuted
experiment without a materially new angle.
**Plateau rule:** if three or more consecutive experiments were reverted,
do not attempt another variation of the last refuted mechanism. Either
pick a design family absent from the history (dispatcher + lighting
specialists, pretrained init, learned relighting, training-scale, …) or
attack the bottleneck the refuted hypotheses jointly point at —
**but check which stages that "design family" actually touches before
you commit to it.** Query the last several kept experiments'
`arch_json` (`SELECT arch_json FROM experiments WHERE kind='development'
ORDER BY id DESC LIMIT 10;`) and look at which stage names never carry
`"changed": true`. A losing streak is not just "we haven't tried a
dispatcher yet" — it is usually "the trunk / descriptor / decode has
gone unquestioned for N rounds while satellite modules (gates, heads,
auxiliary losses, samplers) keep churning around it." Picking a name
off the suggested list while leaving that frozen core untouched is
incremental tuning wearing a pivot's clothes, and the harness's own
patience check (below, once it fires) will call this out explicitly by
naming the frozen stages.
2. **Design ONE focused experiment** — proper experiment design, pre-registered
before you touch code. Write it to `runs/pending_experiment.json`:
```json
{
"title": "one-line name",
"category": "architecture|loss|augmentation|relighting|training|quantization|other",
"hypothesis": "what you believe is limiting the metric and why this change addresses it",
"method": "the ONE focused change, concretely (files, mechanism)",
"expected_outcome": "predicted effect on the §6 worst-case median error, quantified if possible",
"init_strategy": "from-scratch | pretrained:<name>",
"eli5": "2-4 sentences for a smart non-ML reader: what you changed and why it might help, in everyday language — analogies welcome, zero jargon",
"architecture": {"stages": [
{"name": "Camera frame", "detail": "128×128 px crop, one of 6 lighting renders", "changed": false},
{"name": "Feature extractor", "detail": "plain-language description", "changed": false},
{"name": "…", "detail": "…", "changed": true}
]},
"architecture_svg": "<svg viewBox='0 0 980 300' …>…</svg>"
}
```
`architecture_svg` is the **technical architecture diagram** of the model
you are testing — a proper ML-paper figure of the design, drawn by you,
shown at the top of this experiment's gallery entry. **Draw tensors and
operations as the things they are — NOT as labeled boxes:** the input
image as a pixel-textured square; conv feature maps as pseudo-3D slabs
that shrink spatially and deepen in channels; 1-D vectors as thin
vertical tick-bars; fully-connected layers as fans of thin crossing
lines between bars; spatial fields/grids as actually-drawn grids with
shaded cells; decodes/aggregations as thin lines converging from cells
onto a point; samplers/targets as small pictorial glyphs. Captions sit
UNDER each element (name ≤10.5px weight 600, sub-note ≤9.5px); tensor
shapes annotate the flow in small italics (128×128×3 → 8²×128 → 1024).
Losses/targets/samplers live in a lane below the inference path,
annotated with slanted dashed leader lines to small text — no boxes
around them. Draw the two FROZEN endpoints — the camera-frame input and
the (lat, lon, confidence) output — entirely in #9b998c with a small
italic “frozen contract” tag: they are fixed by the harness and outside
your search space. Everything between them is your design: draw it in ink,
with red only on what THIS experiment changes. Style contract, so all experiments' figures read as one
paper: viewBox width 980 (height ~300–360); transparent background; ink
#111111, captions #6b6a60, annotations/arrows #9b998c; **#8c2f1f (red)
reserved for exactly what this experiment changed**; training-only
elements in #8a6a1e (or red if changed); font-family
Palatino,Georgia,serif; stroke-width ≈1.2 elements, 1 arrows; fills only
faint tints (opacity ≤ .12); no gradients, no icons, no emoji. Inference
flows left → right from camera frame to (lat, lon, confidence).
**The camera frame is a fixed glyph — copy it verbatim.** It must look
like an actual nadir frame of terrain (streets, building footprints),
not abstract pixels, and be identical in every figure. Use exactly this
snippet, substituting Y for the frame's top edge (pick Y so the frame
centers on your inference lane; captions go under it as usual — note
128²×3 is the model input crop, ~1 m/px, not the sensor's native
resolution):
```svg
<g id='cam-terrain'><rect id='frozen-input' x='26' y='Y' width='76'
height='76' fill='#f6f4ea' stroke='#9b998c' stroke-width='1.6'/>
<path d='M31 Y+47 L97 Y+27' stroke='#e6e3d4' stroke-width='5' fill='none'/>
<path d='M59 Y+6 L49 Y+70' stroke='#e6e3d4' stroke-width='3.5' fill='none'/>
<rect x='34' y='Y+9' width='12' height='8' fill='#d9d5c3' transform='rotate(-8 40 Y+13)'/>
<rect x='78' y='Y+8' width='10' height='11' fill='#cfccbd'/>
<rect x='35' y='Y+57' width='13' height='8' fill='#d9d5c3'/>
<rect x='75' y='Y+50' width='10' height='9' fill='#cfccbd' transform='rotate(6 80 Y+54)'/>
<rect x='54' y='Y+31' width='9' height='8' fill='#d9d5c3' opacity='.85'/>
<ellipse cx='86' cy='Y+63' rx='8' ry='6' fill='#8a6a1e' opacity='.12'/>
<circle cx='40' cy='Y+28' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='70' cy='Y+14' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='92' cy='Y+36' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='52' cy='Y+52' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='82' cy='Y+70' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='31' cy='Y+42' r='.7' fill='#6b6a60' opacity='.5'/></g>
```
(Y+n means the literal number Y plus n — compute the values. Every glyph
element stays strictly INSIDE the frame border — figcheck rejects a
figure without the `cam-terrain` group.) Draw your
receptive-field square and kernel-projection lines on top of it as
usual.
**Anchor the frozen endpoints identically in every figure** so all
experiments' figures line up when compared down the gallery page: the
camera-frame square starts at x=26 (its captions centered on x=53), and
the output is right-anchored — decode crosshair at x=812, the
(lat, lon, confidence) text block text-anchor=start at x=828. Mark the
anchors machine-checkably: the camera-frame rect carries
`id='frozen-input'`, the output text block carries `id='frozen-output'`.
Lay the elements between them out on a running x-cursor with generous
spacing so nothing overlaps.
**Readability contract** (figcheck verifies this geometrically): no
text may overlap other text, and no line (arrows, converge fans,
leader lines) may pass through a label — a line may END at a label,
never cross one. Rules distilled from review rounds: dashed leaders
run as orthogonal L-routes through empty lanes, never diagonally
across components; converge/vote lines originate at their true
sources (the actual dots or window cells, not generic positions);
every element owns its caption — adjacent columns stagger their
caption rows instead of colliding; prefer moving a label into empty
horizontal space over stacking it against a neighbor. Width is fixed
at 980, but **height is yours**: grow the viewBox (240–640) whenever
more vertical room makes the layout cleaner rather than cramming.
**Before finishing, validate:**
`.venv/bin/python -m autoresearch.figcheck` — it checks the anchor
contract on runs/pending_experiment.json; revise until it prints PASS.
Consecutive conv layers must be tied by kernel-projection lines (small
kernel square on one face, faint lines converging to a cell on the
next) so the encoder reads as one computation, not boxes in a row.
`archive/arch_svg_reference.py` is the concrete reference implementation
of this entire visual language — read it and reuse its helper geometry
for any stage that already has one.
`eli5` and `architecture` feed the human-facing gallery. `architecture.stages`
is the model's inference path left-to-right, camera frame → (lat, lon,
confidence) output — one box per stage, plain-language `detail`,
`"changed": true` ONLY on the stages this experiment touches. Reuse the
previous experiment's stage names verbatim wherever a stage is unchanged
(check `SELECT arch_json FROM experiments WHERE arch_json IS NOT NULL ORDER BY id DESC LIMIT 1;`) —
the diagrams are compared box-by-box across experiments. A change that only
affects training (loss, augmentation, schedule) keeps the inference stages
unchanged and adds one final stage with `"train_only": true` describing the
training signal.
3. **Write the implementation brief.** You do NOT edit `model/` yourself —
a separate implementation agent applies your design, seeing only the
current code plus what you pre-registered. Add one more field to
`runs/pending_experiment.json`:
```
"implementation_brief": "exact file-level instructions: which functions/
blocks in model/model.py and model/train.py change and how, what stays
untouched, and every contract to preserve"
```
Be precise enough that a competent engineer with no other context
implements it in one pass. Always restate the fixed contracts:
`train.py`'s CLI and the ONNX export contract in `model/model.py`'s
docstring — the frozen scorer depends on them.
## Hard rules
- Edit ONLY `runs/pending_experiment.json`. You never edit `model/` (the
implementation stage does) and files listed in `/FROZEN` are off-limits;
the harness hard-reverts any change to them.
- NEVER touch, read, or evaluate the `hamburg` holdout area (§5).
- ONE focused change per iteration — if you can't describe it in one sentence,
it's too big. Prefer architectural/procedural novelty over hyperparameter
nudges (§3): changing a learning rate is a weak experiment; changing the
coordinate parameterization, loss family, relighting-for-training, or model
topology is a strong one.
- Do not run training yourself; the harness does that.
- Stay within the deployment gates: exported ONNX ≤ 4 MiB per area, host
latency proxy ≤ 250 ms (see pipeline/score.py).Where the model was tested — and how far off it was Each map is one full test area. Every dot is one held-out test location (all six lighting conditions overlaid): the model was shown a 128 m crop centered there and asked for its position. Dot color = the distance between its answer and the truth — green ≤ 20 m (at goal), amber ≤ 50 m, red beyond. A working model turns these maps green; spatial clusters of red reveal which parts of an area confuse it.     What the six lighting conditions look like through the simulated low-light sensor (example patches — illustration, not the training set)Every frame in this project is rendered as a starlight-class low-light sensor (Sony STARVIS2 / IMX585 class — the airframe's chosen camera) would see it, not as a normal camera would. That is why the night renders look uncannily close to daylight: at high gain such a sensor recovers scene structure from moonlight, skyglow and artificial lighting, paying for it in noise, lifted shadows and washed-out color — exactly what these renders simulate. Below, one example 256 m patch per area under the six lighting conditions the model must handle. These illustrate the dataset, not this experiment's performance — the actual training set is thousands of distinct crops per area (see “training data” above), and these renderings only change when the relighting method changes. berlin frankfurt munich prignitz | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▸ | 39 | Coordinate-generated map field: replace 1,024 free-parameter cell templates with a shared position-conditioned embedding generator 898d8cb5 | architecture | pretrained:mobilenet_v3_small IMAGENET1K_V1 features[0..8] (torchvision, BSD-3) — unchanged | 1.10 km | 1,280 KB | 2.7 ms | 71 m 34 s | $0.82 | discarded | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The design under test — technical diagram — gray = frozen contract · ink = the current design · red = this experiment's change One real test, end to end
→
1.17 km
miss, this crop 0.15
self-reported confidence In plain words For 38 rounds the model has picked a location by keeping a giant multiple-choice answer sheet: 1,024 separate, totally independent answers, one per map tile, that it must learn from scratch with no help from each other. That's like memorizing 1,024 unrelated flashcards instead of learning a map. This experiment throws the flashcards away and replaces them with a tiny function that GENERATES an answer for any tile just from its map coordinates -- so learning about one tile automatically improves nearby and similar tiles too, instead of each one starting from zero. The photo still gets turned into its own compact fingerprint, and the guess is just 'which generated tile-fingerprint looks most like mine' -- same idea as before, just with a far smaller, far smarter answer sheet. Hypothesis Three consecutive reverts (36: hardest-impostor margin hinge on the 1024-way field; 37: K=8 multi-hypothesis regression replacing the field; 38: rejected outright for touching only the trunk's input channels while leaving the field, decode, gate, and confidence self-reported unchanged) all left one thing untouched across the entire 38-round history: the map-cell scorer itself is a flat nn.Linear(560, 1024) -- exactly 1,024 completely independent weight rows, unchanged in kind since exp 2/10, each fit from scratch with zero structural relationship to its neighbors. Exp 26 named the resulting signature precisely: since exp 20/25, train-split and eval-split unfiltered medians are EQUAL (~1 km in every area/bucket) -- the model cannot even fit its OWN training locations, which is the textbook signature of a statistically weak output parameterization, not of missing capacity (exps 26/28 tripled trunk depth three times, all reverted, two after coverage was honestly measured), missing optimization (exp 25 trained to a genuine loss plateau and moved the metric only 5%), or a weak objective (exps 34/36's hardest-impostor margin hinge, twice refuted). The one experiment that DID attack the parameterization directly -- exp 31's neural-map correlation field -- was reverted, but its own pre-registered failure mode was orthogonal to weak-parameterization itself: a spatial 4x4 crop kernel cross-correlated against a map raster is sensitive to the residual +/-45 degree heading error left after the C4 vote, which smears correlation peaks regardless of how good the underlying map representation is. That failure mode does not apply to a HOLISTIC embedding comparison (a single 64-d crop vector dotted against a single 64-d cell vector, exactly the same shape of comparison the current flat head already makes) -- so the parameterization hypothesis is genuinely untested in a form that avoids exp 31's specific breakage. This experiment replaces the flat table with a coordinate-conditioned generator: a fixed Fourier positional encoding of each cell's (u, v) center feeds a tiny shared 2-layer MLP that outputs all 1,024 cell embeddings, so a gradient at one map location updates the SAME ~6k shared parameters that determine every other cell's embedding -- an explicit smoothness/parameter-sharing prior across map positions that a bank of 1,024 independent rows structurally cannot have, directly targeting the diagnosed 'cannot fit own training locations' signature. Per the pivot directive this also forces (not as an afterthought, but as a structural consequence of deleting the flat head) a rewrite of every downstream stage that read it: the two-expert brightness gate (rd 12) is replaced by continuous FiLM conditioning of the embedding MLP; the feature extractor now taps the trunk's stride-8 block (16x16x24) as a second GAP summary alongside the stride-16 block, since a metric-learning embedding benefits from a multi-scale descriptor in a way a single flat classifier never got to exploit; decode and confidence are recomputed over the new cosine-similarity field, with confidence now reading a peak-vs-second-peak embedding margin instead of the old hand-crafted shape detector. Method model/model.py: delete loc_logits, dark_logits, and gate (the flat linear head, the dark expert, and the brightness blend -- roughly 625k of the model's 823k parameters). Split the pretrained trunk into features_early = full[:4] (stride-8, 24ch @ 16x16) and features_late = full[4:] (stride-16, 48ch @ 8x8) -- same loaded weights, just referenced as two slices. Add: an embed_hidden/embed_out MLP (584-d descriptor [late GAP 48 + skip GAP 24 + layout 512] -> 128 -> 64, L2-normalized) producing the crop embedding; a film_mlp ([brightness, late GAP] -> gamma/beta, 128-d each) FiLM-modulating the embedding MLP's hidden layer; a fixed Fourier positional encoding of the 1,024 grid-cell centers (registered buffer, L=6 frequencies) feeding a small map_gen MLP (26 -> 64 -> 64, L2-normalized) that generates all 1,024 cell embeddings fresh every forward call; a learned scalar logit_scale (init 10.0) temperature-scaling the cosine-similarity matmul (crop_embedding @ map_embeddings.T) into the same-shaped [N, 1024] logits the rest of the pipeline (C4 vote, Gaussian-CE, sharpened soft-argmax decode) already consumes unchanged. Confidence head's input swaps the old decode-gap feature for a peak-vs-second-peak cosine-similarity margin (torch.topk(2)), kept alongside peak mass and entropy. model/train.py: exactly one line changes -- `trunk_params = list(model.features.parameters())` becomes `list(model.features_early.parameters()) + list(model.features_late.parameters())` -- sampler, calibration, schedule, and loss_fn's math are byte-identical. Expected outcome If the flat table's independent-row parameterization is the binding constraint the last three refutations jointly point at, the worst-case median improves from 743.07 m to roughly 620-720 m (3-16%), with the clearest mechanism-level signature being train-split and eval-split unfiltered medians FINALLY diverging (train dropping measurably below eval) instead of the tied ~1 km both have shown since exp 20/25 -- direct evidence the shared-parameter generator can fit training locations the independent-row table could not. A matched-protocol rerun of the iter-12 field probe should show the true-cell top-1 rate rise measurably above its ~2% baseline. Falsification forks: (a) if the metric is flat/regresses AND train-split/eval-split medians stay tied, the weak-parameterization hypothesis is cleanly refuted -- the wall is descriptor information content (crop-embedding capacity), not how the map side is parameterized, redirecting the next iteration at richer crop features rather than any further map-side reparameterization; (b) if the primary metric improves but top-1 rate barely moves, parameter sharing bought regularization/generalization without fixing the specific confusable-lookalike ranking failure -- a real, bankable win whose natural next step is a margin term reapplied in THIS embedding space (materially different context from exps 34/36's refuted flat-table margin hinge); (c) if the metric regresses sharply, check via the same iter-12-style probes whether the failure mode is a globally over-smoothed field (parameter sharing collapsing spatial resolution) rather than exp 31's heading-misalignment smearing, to keep the two parameterization attempts' failure modes cleanly distinguished. Result primary worst-case median error = 1103.28 m (previous best 743.07 m); full per-area/bucket breakdown in metrics.json Conclusion REVERTED — metric did not improve (1103.28 m vs best 743.07 m); change discarded Scoreboard — median error (m) per area × lighting
the worst cell (underlined) is the experiment's score;
red = failed cell, ink = at target
Fits the aircraft? all deployment gates passed
largest per-area model 1,280 KB (limit 4,096 KB) · single-frame inference 2.7 ms on one CPU thread (proxy limit 250 ms)
crops are sampled fresh each run from the frozen train split (~45,000 distinct positions per area, times random rotation), never from eval blocks; how many to use is the experiment's own choice
Where this iteration's time went: design (Sonnet) 18 m 30 s · implement (Sonnet) 2 m 36 s · train 4 areas 47 m 46 s · score 1 m 33 s · samples 41 s · publish 5 s · whole iteration 71 m 44 s
ts 2026-07-22T18:43:41 · commit 898d8cb5b18b ·
parent 898d8cb5b18b · artifacts runs/20260722_173207_iter2 ·
agent model — · took 71 m 34 s
The exact prompt given to the headless agent## PATIENCE SPENT — THIS ITERATION MUST PIVOT
3 consecutive experiments have failed to beat the current best.
A pivot means COMPLETELY RETHINKING THE ARCHITECTURE, not touching the one
stage named below as frozen while carrying the rest of the current design
over unchanged. Do not refine, extend, or swap one part of the champion's
mechanism and call it a pivot. Propose a genuinely different overall
design — the spec (§3) explicitly invites, among others: dispatcher +
lighting-condition-specialist models, learned relighting, a different
coordinate parameterization, quantization-aware capacity changes, or
training-data strategy overhauls — but whichever family you pick, it must
change how the WHOLE pipeline works end to end, not one isolated stage.
These parts of the design have not changed in ANY of the last 3 experiments:
- Lighting gate (unchanged 3 rounds running)
- Probability map (unchanged 3 rounds running)
- Decode (unchanged 3 rounds running)
- Training signal (unchanged 1 rounds running)
- Confidence (unchanged 1 rounds running)
A genuine pivot MUST change at least one of these — the ones closest to the model's actual representational capacity (feature extraction, descriptor construction, decode) count far more than a training-signal or output-adjacent tweak. Adding a new satellite module (a gate, a head, an auxiliary loss, a sampler) around an unquestioned backbone is NOT a pivot, even if it comes from the suggested list below and even if it trains a new component from scratch — the backbone itself is the thing that has gone unexamined.
Every non-frozen stage in your architecture.stages list must be marked
"changed": true this round. If any stage is left exactly as before, this
iteration will be rejected before training even starts — this is checked
against your actual code diff, not just your own self-report.
# Autoresearch iteration — one focused experiment
You are one iteration of an autonomous research loop for UAV low-light
geolocalization. Read `CLAUDE.md` (§1, §3, §6 especially) for full context.
The harness (loop.sh) will train, score, log, and keep/revert AFTER you exit —
you only design the experiment and edit the code.
## Your job, in order
0. **Skim the library (optional input).** `autoresearch/library.md` holds
the human researcher's inspiration notes. They do not fix your answer
and you are free to ignore them — pick an entry up only when it
genuinely fits your read of the history, and if you build on one, say
so in your hypothesis.
1. **Review the research history.** Query the lineage DB:
`sqlite3 experiments.sqlite "SELECT id, title, category, hypothesis, expected_outcome, result, conclusion, primary_metric, kept FROM experiments ORDER BY id DESC LIMIT 15;"`
Note which hypotheses were supported/refuted. Do not repeat a refuted
experiment without a materially new angle.
**Plateau rule:** if three or more consecutive experiments were reverted,
do not attempt another variation of the last refuted mechanism. Either
pick a design family absent from the history (dispatcher + lighting
specialists, pretrained init, learned relighting, training-scale, …) or
attack the bottleneck the refuted hypotheses jointly point at —
**but check which stages that "design family" actually touches before
you commit to it.** Query the last several kept experiments'
`arch_json` (`SELECT arch_json FROM experiments WHERE kind='development'
ORDER BY id DESC LIMIT 10;`) and look at which stage names never carry
`"changed": true`. A losing streak is not just "we haven't tried a
dispatcher yet" — it is usually "the trunk / descriptor / decode has
gone unquestioned for N rounds while satellite modules (gates, heads,
auxiliary losses, samplers) keep churning around it." Picking a name
off the suggested list while leaving that frozen core untouched is
incremental tuning wearing a pivot's clothes, and the harness's own
patience check (below, once it fires) will call this out explicitly by
naming the frozen stages.
2. **Design ONE focused experiment** — proper experiment design, pre-registered
before you touch code. Write it to `runs/pending_experiment.json`:
```json
{
"title": "one-line name",
"category": "architecture|loss|augmentation|relighting|training|quantization|other",
"hypothesis": "what you believe is limiting the metric and why this change addresses it",
"method": "the ONE focused change, concretely (files, mechanism)",
"expected_outcome": "predicted effect on the §6 worst-case median error, quantified if possible",
"init_strategy": "from-scratch | pretrained:<name>",
"eli5": "2-4 sentences for a smart non-ML reader: what you changed and why it might help, in everyday language — analogies welcome, zero jargon",
"architecture": {"stages": [
{"name": "Camera frame", "detail": "128×128 px crop, one of 6 lighting renders", "changed": false},
{"name": "Feature extractor", "detail": "plain-language description", "changed": false},
{"name": "…", "detail": "…", "changed": true}
]},
"architecture_svg": "<svg viewBox='0 0 980 300' …>…</svg>"
}
```
`architecture_svg` is the **technical architecture diagram** of the model
you are testing — a proper ML-paper figure of the design, drawn by you,
shown at the top of this experiment's gallery entry. **Draw tensors and
operations as the things they are — NOT as labeled boxes:** the input
image as a pixel-textured square; conv feature maps as pseudo-3D slabs
that shrink spatially and deepen in channels; 1-D vectors as thin
vertical tick-bars; fully-connected layers as fans of thin crossing
lines between bars; spatial fields/grids as actually-drawn grids with
shaded cells; decodes/aggregations as thin lines converging from cells
onto a point; samplers/targets as small pictorial glyphs. Captions sit
UNDER each element (name ≤10.5px weight 600, sub-note ≤9.5px); tensor
shapes annotate the flow in small italics (128×128×3 → 8²×128 → 1024).
Losses/targets/samplers live in a lane below the inference path,
annotated with slanted dashed leader lines to small text — no boxes
around them. Draw the two FROZEN endpoints — the camera-frame input and
the (lat, lon, confidence) output — entirely in #9b998c with a small
italic “frozen contract” tag: they are fixed by the harness and outside
your search space. Everything between them is your design: draw it in ink,
with red only on what THIS experiment changes. Style contract, so all experiments' figures read as one
paper: viewBox width 980 (height ~300–360); transparent background; ink
#111111, captions #6b6a60, annotations/arrows #9b998c; **#8c2f1f (red)
reserved for exactly what this experiment changed**; training-only
elements in #8a6a1e (or red if changed); font-family
Palatino,Georgia,serif; stroke-width ≈1.2 elements, 1 arrows; fills only
faint tints (opacity ≤ .12); no gradients, no icons, no emoji. Inference
flows left → right from camera frame to (lat, lon, confidence).
**The camera frame is a fixed glyph — copy it verbatim.** It must look
like an actual nadir frame of terrain (streets, building footprints),
not abstract pixels, and be identical in every figure. Use exactly this
snippet, substituting Y for the frame's top edge (pick Y so the frame
centers on your inference lane; captions go under it as usual — note
128²×3 is the model input crop, ~1 m/px, not the sensor's native
resolution):
```svg
<g id='cam-terrain'><rect id='frozen-input' x='26' y='Y' width='76'
height='76' fill='#f6f4ea' stroke='#9b998c' stroke-width='1.6'/>
<path d='M31 Y+47 L97 Y+27' stroke='#e6e3d4' stroke-width='5' fill='none'/>
<path d='M59 Y+6 L49 Y+70' stroke='#e6e3d4' stroke-width='3.5' fill='none'/>
<rect x='34' y='Y+9' width='12' height='8' fill='#d9d5c3' transform='rotate(-8 40 Y+13)'/>
<rect x='78' y='Y+8' width='10' height='11' fill='#cfccbd'/>
<rect x='35' y='Y+57' width='13' height='8' fill='#d9d5c3'/>
<rect x='75' y='Y+50' width='10' height='9' fill='#cfccbd' transform='rotate(6 80 Y+54)'/>
<rect x='54' y='Y+31' width='9' height='8' fill='#d9d5c3' opacity='.85'/>
<ellipse cx='86' cy='Y+63' rx='8' ry='6' fill='#8a6a1e' opacity='.12'/>
<circle cx='40' cy='Y+28' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='70' cy='Y+14' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='92' cy='Y+36' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='52' cy='Y+52' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='82' cy='Y+70' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='31' cy='Y+42' r='.7' fill='#6b6a60' opacity='.5'/></g>
```
(Y+n means the literal number Y plus n — compute the values. Every glyph
element stays strictly INSIDE the frame border — figcheck rejects a
figure without the `cam-terrain` group.) Draw your
receptive-field square and kernel-projection lines on top of it as
usual.
**Anchor the frozen endpoints identically in every figure** so all
experiments' figures line up when compared down the gallery page: the
camera-frame square starts at x=26 (its captions centered on x=53), and
the output is right-anchored — decode crosshair at x=812, the
(lat, lon, confidence) text block text-anchor=start at x=828. Mark the
anchors machine-checkably: the camera-frame rect carries
`id='frozen-input'`, the output text block carries `id='frozen-output'`.
Lay the elements between them out on a running x-cursor with generous
spacing so nothing overlaps.
**Readability contract** (figcheck verifies this geometrically): no
text may overlap other text, and no line (arrows, converge fans,
leader lines) may pass through a label — a line may END at a label,
never cross one. Rules distilled from review rounds: dashed leaders
run as orthogonal L-routes through empty lanes, never diagonally
across components; converge/vote lines originate at their true
sources (the actual dots or window cells, not generic positions);
every element owns its caption — adjacent columns stagger their
caption rows instead of colliding; prefer moving a label into empty
horizontal space over stacking it against a neighbor. Width is fixed
at 980, but **height is yours**: grow the viewBox (240–640) whenever
more vertical room makes the layout cleaner rather than cramming.
**Before finishing, validate:**
`.venv/bin/python -m autoresearch.figcheck` — it checks the anchor
contract on runs/pending_experiment.json; revise until it prints PASS.
Consecutive conv layers must be tied by kernel-projection lines (small
kernel square on one face, faint lines converging to a cell on the
next) so the encoder reads as one computation, not boxes in a row.
`archive/arch_svg_reference.py` is the concrete reference implementation
of this entire visual language — read it and reuse its helper geometry
for any stage that already has one.
`eli5` and `architecture` feed the human-facing gallery. `architecture.stages`
is the model's inference path left-to-right, camera frame → (lat, lon,
confidence) output — one box per stage, plain-language `detail`,
`"changed": true` ONLY on the stages this experiment touches. Reuse the
previous experiment's stage names verbatim wherever a stage is unchanged
(check `SELECT arch_json FROM experiments WHERE arch_json IS NOT NULL ORDER BY id DESC LIMIT 1;`) —
the diagrams are compared box-by-box across experiments. A change that only
affects training (loss, augmentation, schedule) keeps the inference stages
unchanged and adds one final stage with `"train_only": true` describing the
training signal.
3. **Write the implementation brief.** You do NOT edit `model/` yourself —
a separate implementation agent applies your design, seeing only the
current code plus what you pre-registered. Add one more field to
`runs/pending_experiment.json`:
```
"implementation_brief": "exact file-level instructions: which functions/
blocks in model/model.py and model/train.py change and how, what stays
untouched, and every contract to preserve"
```
Be precise enough that a competent engineer with no other context
implements it in one pass. Always restate the fixed contracts:
`train.py`'s CLI and the ONNX export contract in `model/model.py`'s
docstring — the frozen scorer depends on them.
## Hard rules
- Edit ONLY `runs/pending_experiment.json`. You never edit `model/` (the
implementation stage does) and files listed in `/FROZEN` are off-limits;
the harness hard-reverts any change to them.
- NEVER touch, read, or evaluate the `hamburg` holdout area (§5).
- ONE focused change per iteration — if you can't describe it in one sentence,
it's too big. Prefer architectural/procedural novelty over hyperparameter
nudges (§3): changing a learning rate is a weak experiment; changing the
coordinate parameterization, loss family, relighting-for-training, or model
topology is a strong one.
- Do not run training yourself; the harness does that.
- Stay within the deployment gates: exported ONNX ≤ 4 MiB per area, host
latency proxy ≤ 250 ms (see pipeline/score.py).Where the model was tested — and how far off it was Each map is one full test area. Every dot is one held-out test location (all six lighting conditions overlaid): the model was shown a 128 m crop centered there and asked for its position. Dot color = the distance between its answer and the truth — green ≤ 20 m (at goal), amber ≤ 50 m, red beyond. A working model turns these maps green; spatial clusters of red reveal which parts of an area confuse it.     What the six lighting conditions look like through the simulated low-light sensor (example patches — illustration, not the training set)Every frame in this project is rendered as a starlight-class low-light sensor (Sony STARVIS2 / IMX585 class — the airframe's chosen camera) would see it, not as a normal camera would. That is why the night renders look uncannily close to daylight: at high gain such a sensor recovers scene structure from moonlight, skyglow and artificial lighting, paying for it in noise, lifted shadows and washed-out color — exactly what these renders simulate. Below, one example 256 m patch per area under the six lighting conditions the model must handle. These illustrate the dataset, not this experiment's performance — the actual training set is thousands of distinct crops per area (see “training data” above), and these renderings only change when the relighting method changes. berlin frankfurt munich prignitz | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▸ | 38 | Illumination-invariant retinex channel joins raw RGB from the pixel level through the trunk 898d8cb5 | architecture | pretrained:mobilenet_v3_small IMAGENET1K_V1 features[0..8], stem conv expanded 3->6 input channels (RGB weights copied verbatim + 0.1x-scaled duplicate for the 3 new channels), all other pretrained blocks loaded strict and unchanged | gated fail | — | — | 20 m 32 s | $0.24 | rejected | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The design under test — technical diagram — gray = frozen contract · ink = the current design · red = this experiment's change In plain words Every version of this model so far has looked at the photo exactly as the camera sees it -- raw colors and brightness. But the same street can look totally different depending on how dark it is or how many lights are on, and we've measured that the model's mental fingerprint of a place gets thrown off by that brightness noise, not just by the actual layout of streets and buildings. This experiment hands the model a second picture alongside the first: a version mathematically flattened to remove slow brightness changes and keep only the sharp local edges -- like squinting to ignore how sunny or dim it is and see only shapes. Both pictures are fed in together from the very first layer, so the network can lean on the shape-only picture whenever plain color is confounded by lighting. We also sprinkle a bit of random blur or sharpening onto training photos so the model can't cheat by reading blur artifacts as if they were a location clue. Hypothesis Two consecutive reverts (exp 36: hardest-impostor margin hinge on the same 1024-way field; exp 37: multi-hypothesis K=8 regression replacing the field) both left the feature extractor, layout summary, rotation fan, training data, and training schedule byte-identical to round 35 (see arch_json for ids 36/37) -- confirming those five stages are the frozen core, not the decode/loss/head churn happening around them. Every change to that core across the project's 37-round history has been a CAPACITY change within the same RGB-only pretrained-CNN family: exp 8 scaled a from-scratch encoder 4.2x (reverted), exp 26/27/28 tripled the pretrained trunk's depth three separate times (all reverted) -- exp 26/28 even confirmed real like-for-like accuracy gains (munich morning 536->347m, frankfurt evening 798->466m) that were wiped out once coverage was measured honestly, or reversed once it was. No experiment in the entire history has ever changed WHAT the trunk's first layer sees: all 37 rounds fed it raw RGB and nothing else. Two independent live probes point at that exact gap. runs/_probe_iter12_heading.py found the champion's descriptor rides low-level resampling-blur statistics that carry zero location information: eval crops that skip PIL's bilinear rotation-resampling blur nearly DOUBLE the error (983->1668m, berlin midday) from a blur artifact alone, not layout. runs/_probe_iter12_field.py found the true map cell is the field's top-1 pick only 2% of the time despite landing in its own top-20 (of 1024) on 33-58% of crops in every bucket -- the descriptor already narrows the field down, but something confounds fine discrimination among the candidates, exactly what lighting/blur nuisance riding through every learned channel would do (it blurs classifier rows for visually similar districts without erasing their coarse similarity). This experiment gives the network, for the first time, an explicit non-learned illumination-invariant geometric signal -- a Retinex-style channel (log(pixel) minus its own large-radius Gaussian blur, per RGB channel, fixed kernel, no gradient path) concatenated with raw RGB before the very first conv layer -- so the trunk, layout summary, and every downstream stage can draw on a signal invariant to ambient brightness/gain by construction, instead of relying on fine-tuning alone to discover that invariance, which the last three capacity experiments show it has not done. Method Concatenate a fixed, non-learned Retinex-style illumination-invariant channel (log(pixel) minus its own large-radius Gaussian blur, per RGB channel, sigma=10px, no gradient path) with raw RGB before the first conv layer; widen the pretrained MobileNetV3-Small stem from 3 to 6 input channels (RGB weights kept verbatim, new channels a 0.1x-scaled copy); L2-normalize the layout-squeeze code per spatial cell before flattening; add exp 33's never-trained per-crop sharpness-jitter augmentation and a short LR warm-up for the newly-initialized stem weights. Files: model/model.py (_retinex, _build_pretrained_trunk, TinyLocNet.__init__/forward), model/train.py (_jitter_sharpness, sample_epoch, train_area's scheduler). Expected outcome If the descriptor's lighting/blur-riding is a real tax on ranking, the worst-case median improves from the champion's 743.07m to roughly 620-720m (3-15%), with the improvement most visible in the three self-similar metro areas (berlin/munich/frankfurt, where lookalike-district confusion is the dominant failure mode) and in the daytime-to-night gap narrowing, since the retinex channel should matter most exactly where raw-RGB appearance varies most across lighting buckets for the same place. Mechanism-level check: re-running the iter-12 heading probe on the new export should show the crisp-vs-blurred-crop error gap (983 vs 1668m) collapse toward parity, since the raw-RGB blur-riding shortcut now has a parallel invariant channel available and the added per-crop sharpness jitter makes raw-channel blur an uninformative training axis by design. Falsification forks: (a) if the metric is flat (+/-3%) with a healthy training loss, the descriptor's failure is not about missing invariance but about classifier capacity/objective for the ranking task itself -- redirecting cleanly to an embedding/metric-learning head, a family never attempted; (b) if the metric regresses, the retinex transform likely discarded a genuinely informative photometric cue (CLAUDE.md itself names active artificial lighting -- streetlamp/window position -- as real location signal, not noise) at a normalization bandwidth too coarse to spare it, pointing the next iteration at a smaller retinex sigma or a learned (rather than fixed) illumination-decomposition front end. Result rejected before training: a complete architecture rethink was required, but these stages are still self-reported unchanged: Lighting gate, Probability map, Decode, Confidence, Training signal Conclusion REJECTED — pivot directive was not honored (partial change, not a complete rethink); iteration discarded without training Scoreboard — median error (m) per area × lighting
the worst cell (underlined) is the experiment's score;
red = failed cell, ink = at target
No per-area results recorded for this run. Fits the aircraft? all deployment gates passed
largest per-area model — (limit 4,096 KB) · single-frame inference — on one CPU thread (proxy limit 250 ms) ts 2026-07-22T17:32:07 · commit 898d8cb5b18b ·
parent 898d8cb5b18b · artifacts runs/20260722_171135_iter1 ·
agent model — · took 20 m 32 s
The exact prompt given to the headless agent## PATIENCE SPENT — THIS ITERATION MUST PIVOT
2 consecutive experiments have failed to beat the current best.
A pivot means COMPLETELY RETHINKING THE ARCHITECTURE, not touching the one
stage named below as frozen while carrying the rest of the current design
over unchanged. Do not refine, extend, or swap one part of the champion's
mechanism and call it a pivot. Propose a genuinely different overall
design — the spec (§3) explicitly invites, among others: dispatcher +
lighting-condition-specialist models, learned relighting, a different
coordinate parameterization, quantization-aware capacity changes, or
training-data strategy overhauls — but whichever family you pick, it must
change how the WHOLE pipeline works end to end, not one isolated stage.
These parts of the design have not changed in ANY of the last 2 experiments:
- Layout summary (unchanged 2 rounds running)
- Training data (unchanged 2 rounds running)
- Feature extractor (unchanged 2 rounds running)
- Rotation fan (unchanged 2 rounds running)
- Training schedule (unchanged 2 rounds running)
A genuine pivot MUST change at least one of these — the ones closest to the model's actual representational capacity (feature extraction, descriptor construction, decode) count far more than a training-signal or output-adjacent tweak. Adding a new satellite module (a gate, a head, an auxiliary loss, a sampler) around an unquestioned backbone is NOT a pivot, even if it comes from the suggested list below and even if it trains a new component from scratch — the backbone itself is the thing that has gone unexamined.
Every non-frozen stage in your architecture.stages list must be marked
"changed": true this round. If any stage is left exactly as before, this
iteration will be rejected before training even starts — this is checked
against your actual code diff, not just your own self-report.
# Autoresearch iteration — one focused experiment
You are one iteration of an autonomous research loop for UAV low-light
geolocalization. Read `CLAUDE.md` (§1, §3, §6 especially) for full context.
The harness (loop.sh) will train, score, log, and keep/revert AFTER you exit —
you only design the experiment and edit the code.
## Your job, in order
0. **Skim the library (optional input).** `autoresearch/library.md` holds
the human researcher's inspiration notes. They do not fix your answer
and you are free to ignore them — pick an entry up only when it
genuinely fits your read of the history, and if you build on one, say
so in your hypothesis.
1. **Review the research history.** Query the lineage DB:
`sqlite3 experiments.sqlite "SELECT id, title, category, hypothesis, expected_outcome, result, conclusion, primary_metric, kept FROM experiments ORDER BY id DESC LIMIT 15;"`
Note which hypotheses were supported/refuted. Do not repeat a refuted
experiment without a materially new angle.
**Plateau rule:** if three or more consecutive experiments were reverted,
do not attempt another variation of the last refuted mechanism. Either
pick a design family absent from the history (dispatcher + lighting
specialists, pretrained init, learned relighting, training-scale, …) or
attack the bottleneck the refuted hypotheses jointly point at —
**but check which stages that "design family" actually touches before
you commit to it.** Query the last several kept experiments'
`arch_json` (`SELECT arch_json FROM experiments WHERE kind='development'
ORDER BY id DESC LIMIT 10;`) and look at which stage names never carry
`"changed": true`. A losing streak is not just "we haven't tried a
dispatcher yet" — it is usually "the trunk / descriptor / decode has
gone unquestioned for N rounds while satellite modules (gates, heads,
auxiliary losses, samplers) keep churning around it." Picking a name
off the suggested list while leaving that frozen core untouched is
incremental tuning wearing a pivot's clothes, and the harness's own
patience check (below, once it fires) will call this out explicitly by
naming the frozen stages.
2. **Design ONE focused experiment** — proper experiment design, pre-registered
before you touch code. Write it to `runs/pending_experiment.json`:
```json
{
"title": "one-line name",
"category": "architecture|loss|augmentation|relighting|training|quantization|other",
"hypothesis": "what you believe is limiting the metric and why this change addresses it",
"method": "the ONE focused change, concretely (files, mechanism)",
"expected_outcome": "predicted effect on the §6 worst-case median error, quantified if possible",
"init_strategy": "from-scratch | pretrained:<name>",
"eli5": "2-4 sentences for a smart non-ML reader: what you changed and why it might help, in everyday language — analogies welcome, zero jargon",
"architecture": {"stages": [
{"name": "Camera frame", "detail": "128×128 px crop, one of 6 lighting renders", "changed": false},
{"name": "Feature extractor", "detail": "plain-language description", "changed": false},
{"name": "…", "detail": "…", "changed": true}
]},
"architecture_svg": "<svg viewBox='0 0 980 300' …>…</svg>"
}
```
`architecture_svg` is the **technical architecture diagram** of the model
you are testing — a proper ML-paper figure of the design, drawn by you,
shown at the top of this experiment's gallery entry. **Draw tensors and
operations as the things they are — NOT as labeled boxes:** the input
image as a pixel-textured square; conv feature maps as pseudo-3D slabs
that shrink spatially and deepen in channels; 1-D vectors as thin
vertical tick-bars; fully-connected layers as fans of thin crossing
lines between bars; spatial fields/grids as actually-drawn grids with
shaded cells; decodes/aggregations as thin lines converging from cells
onto a point; samplers/targets as small pictorial glyphs. Captions sit
UNDER each element (name ≤10.5px weight 600, sub-note ≤9.5px); tensor
shapes annotate the flow in small italics (128×128×3 → 8²×128 → 1024).
Losses/targets/samplers live in a lane below the inference path,
annotated with slanted dashed leader lines to small text — no boxes
around them. Draw the two FROZEN endpoints — the camera-frame input and
the (lat, lon, confidence) output — entirely in #9b998c with a small
italic “frozen contract” tag: they are fixed by the harness and outside
your search space. Everything between them is your design: draw it in ink,
with red only on what THIS experiment changes. Style contract, so all experiments' figures read as one
paper: viewBox width 980 (height ~300–360); transparent background; ink
#111111, captions #6b6a60, annotations/arrows #9b998c; **#8c2f1f (red)
reserved for exactly what this experiment changed**; training-only
elements in #8a6a1e (or red if changed); font-family
Palatino,Georgia,serif; stroke-width ≈1.2 elements, 1 arrows; fills only
faint tints (opacity ≤ .12); no gradients, no icons, no emoji. Inference
flows left → right from camera frame to (lat, lon, confidence).
**The camera frame is a fixed glyph — copy it verbatim.** It must look
like an actual nadir frame of terrain (streets, building footprints),
not abstract pixels, and be identical in every figure. Use exactly this
snippet, substituting Y for the frame's top edge (pick Y so the frame
centers on your inference lane; captions go under it as usual — note
128²×3 is the model input crop, ~1 m/px, not the sensor's native
resolution):
```svg
<g id='cam-terrain'><rect id='frozen-input' x='26' y='Y' width='76'
height='76' fill='#f6f4ea' stroke='#9b998c' stroke-width='1.6'/>
<path d='M31 Y+47 L97 Y+27' stroke='#e6e3d4' stroke-width='5' fill='none'/>
<path d='M59 Y+6 L49 Y+70' stroke='#e6e3d4' stroke-width='3.5' fill='none'/>
<rect x='34' y='Y+9' width='12' height='8' fill='#d9d5c3' transform='rotate(-8 40 Y+13)'/>
<rect x='78' y='Y+8' width='10' height='11' fill='#cfccbd'/>
<rect x='35' y='Y+57' width='13' height='8' fill='#d9d5c3'/>
<rect x='75' y='Y+50' width='10' height='9' fill='#cfccbd' transform='rotate(6 80 Y+54)'/>
<rect x='54' y='Y+31' width='9' height='8' fill='#d9d5c3' opacity='.85'/>
<ellipse cx='86' cy='Y+63' rx='8' ry='6' fill='#8a6a1e' opacity='.12'/>
<circle cx='40' cy='Y+28' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='70' cy='Y+14' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='92' cy='Y+36' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='52' cy='Y+52' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='82' cy='Y+70' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='31' cy='Y+42' r='.7' fill='#6b6a60' opacity='.5'/></g>
```
(Y+n means the literal number Y plus n — compute the values. Every glyph
element stays strictly INSIDE the frame border — figcheck rejects a
figure without the `cam-terrain` group.) Draw your
receptive-field square and kernel-projection lines on top of it as
usual.
**Anchor the frozen endpoints identically in every figure** so all
experiments' figures line up when compared down the gallery page: the
camera-frame square starts at x=26 (its captions centered on x=53), and
the output is right-anchored — decode crosshair at x=812, the
(lat, lon, confidence) text block text-anchor=start at x=828. Mark the
anchors machine-checkably: the camera-frame rect carries
`id='frozen-input'`, the output text block carries `id='frozen-output'`.
Lay the elements between them out on a running x-cursor with generous
spacing so nothing overlaps.
**Readability contract** (figcheck verifies this geometrically): no
text may overlap other text, and no line (arrows, converge fans,
leader lines) may pass through a label — a line may END at a label,
never cross one. Rules distilled from review rounds: dashed leaders
run as orthogonal L-routes through empty lanes, never diagonally
across components; converge/vote lines originate at their true
sources (the actual dots or window cells, not generic positions);
every element owns its caption — adjacent columns stagger their
caption rows instead of colliding; prefer moving a label into empty
horizontal space over stacking it against a neighbor. Width is fixed
at 980, but **height is yours**: grow the viewBox (240–640) whenever
more vertical room makes the layout cleaner rather than cramming.
**Before finishing, validate:**
`.venv/bin/python -m autoresearch.figcheck` — it checks the anchor
contract on runs/pending_experiment.json; revise until it prints PASS.
Consecutive conv layers must be tied by kernel-projection lines (small
kernel square on one face, faint lines converging to a cell on the
next) so the encoder reads as one computation, not boxes in a row.
`archive/arch_svg_reference.py` is the concrete reference implementation
of this entire visual language — read it and reuse its helper geometry
for any stage that already has one.
`eli5` and `architecture` feed the human-facing gallery. `architecture.stages`
is the model's inference path left-to-right, camera frame → (lat, lon,
confidence) output — one box per stage, plain-language `detail`,
`"changed": true` ONLY on the stages this experiment touches. Reuse the
previous experiment's stage names verbatim wherever a stage is unchanged
(check `SELECT arch_json FROM experiments WHERE arch_json IS NOT NULL ORDER BY id DESC LIMIT 1;`) —
the diagrams are compared box-by-box across experiments. A change that only
affects training (loss, augmentation, schedule) keeps the inference stages
unchanged and adds one final stage with `"train_only": true` describing the
training signal.
3. **Write the implementation brief.** You do NOT edit `model/` yourself —
a separate implementation agent applies your design, seeing only the
current code plus what you pre-registered. Add one more field to
`runs/pending_experiment.json`:
```
"implementation_brief": "exact file-level instructions: which functions/
blocks in model/model.py and model/train.py change and how, what stays
untouched, and every contract to preserve"
```
Be precise enough that a competent engineer with no other context
implements it in one pass. Always restate the fixed contracts:
`train.py`'s CLI and the ONNX export contract in `model/model.py`'s
docstring — the frozen scorer depends on them.
## Hard rules
- Edit ONLY `runs/pending_experiment.json`. You never edit `model/` (the
implementation stage does) and files listed in `/FROZEN` are off-limits;
the harness hard-reverts any change to them.
- NEVER touch, read, or evaluate the `hamburg` holdout area (§5).
- ONE focused change per iteration — if you can't describe it in one sentence,
it's too big. Prefer architectural/procedural novelty over hyperparameter
nudges (§3): changing a learning rate is a weak experiment; changing the
coordinate parameterization, loss family, relighting-for-training, or model
topology is a strong one.
- Do not run training yourself; the harness does that.
- Stay within the deployment gates: exported ONNX ≤ 4 MiB per area, host
latency proxy ≤ 250 ms (see pipeline/score.py). | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▸ | 37 | Multi-hypothesis coordinate regression replaces the 1024-cell field 5a6771be | architecture | pretrained:mobilenet_v3_small | 1.10 km | 1,069 KB | 2.6 ms | 66 m 02 s | $0.76 | discarded | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The design under test — technical diagram — gray = frozen contract · ink = the current design · red = this experiment's change One real test, end to end
→
1.44 km
miss, this crop 0.18
self-reported confidence In plain words For twenty-some rounds straight, the model has guessed your location the same way: score 1,024 fixed map tiles and pick whichever one scores highest -- like a multiple-choice test with 1,024 options. We measured that the right tile is in the model's own top-20 guesses most of the time, but wins outright only 2% of the time -- so the model roughly knows the neighborhood but keeps losing the final vote to a lookalike tile. This round throws out the fixed 1,024-tile menu and lets the model instead point to 8 of its own candidate spots anywhere on the map, then simply picks whichever one it's most sure about -- like narrowing that multiple-choice test from 1,024 options down to 8, which should be a much easier pick. Each of the four camera headings picks its own best spot, and the four picks get blended by how confident each heading was. Hypothesis The champion's field/decode mechanism (32x32-cell classification field -> beta-sharpened soft-argmax) has been completely unchanged since round 16 -- twenty rounds (17-36) have only churned satellite modules around it (samplers, auxiliary losses, gates, a margin hinge) while the frozen core keeps producing the exact failure mode diagnosed in runs/_probe_iter12_field.py and recorded in project memory 'champion-failure-mode-probes': the true cell is the field's top-1 pick on only ~2% of eval crops, yet lands in the field's own top-20 (of 1024) on 33-58% of crops in every lighting bucket. That gap means the descriptor already carries enough signal to narrow the answer down to a small neighborhood -- the failure is specifically that a 1024-way ranking problem is too hard for the head to solve, not that the features are uninformative. Round 34's and round 36's margin-hinge losses attacked this by pushing individual impostor cells down within the SAME 1024-way field and both failed to beat the champion (1e9 gated-fail, then 754.22 vs 743.07) -- evidence the bottleneck is the discretization itself, not which cells get extra gradient. Shrinking the ranking problem from 1-of-1024 fixed grid cells to 1-of-8 data-dependent candidate points should make correct-hypothesis selection tractable while a small K also sidesteps exp 5's diagnosed sparse-supervision problem (a rigid 1024-cell grid needs many training locations per cell to learn anything; 8 free-floating points just need to drift toward wherever they're useful, no fixed-cell occupancy requirement). Method In model/model.py, delete the 1024-way classification field entirely (loc_logits, dark_logits, gate, cell_u/cell_v grid, GRID_K, DECODE_BETA, the beta-sharpened soft-argmax decode) and replace it with a K=8 multi-hypothesis regression head: a small MLP reads [GAP descriptor, layout code, raw brightness] and outputs 8 candidate (u,v) points (each an offset added to one of 8 fixed, non-trainable anchor positions spread across the map, purely to break symmetry at init) plus 8 softmax mixture weights. Each of the 4 C4-rotation views independently picks its own highest-weight hypothesis; the 4 views' winning points are blended into the final (u,v) by a softmax over their own weights (replacing the old logit-averaging C4 vote with a point-averaging C4 vote). Confidence is rebuilt from the spread of the 4 views' winning points (tight agreement = confident) and their mean weight, replacing the old field-peak/entropy signal. Training signal changes from Gaussian-smoothed 1024-way cross-entropy + coordinate L2 to a winner-take-all regression loss (only the hypothesis nearest the true (u,v) gets the full L2 gradient each step; the other 7 get an epsilon=0.05 trickle to avoid dead hypotheses) plus an 8-way cross-entropy that trains the weight head to identify which hypothesis was nearest. The exp-35 confusability-weighted sampler (Training data, kept, unchanged) still needs a 32x32 grid to define 'look-alike cells' for oversampling -- since model.py no longer has any grid, that constant moves into model/train.py as a standalone CONFUSABILITY_GRID_K, with identical value and math, so the sampler's actual behavior is byte-for-byte unchanged. Expected outcome Worst-case median error improves from the champion's 743.07 m to roughly 450-650 m, on the reasoning that an 8-way ranking problem should be far more tractable than the diagnosed 1024-way one given the descriptor already narrows the true location into a small candidate set most of the time. Genuine risk: pure point regression is exactly what exp 1/2 (the project's very first rounds) showed loses badly to a probability-field approach (3216.74 m vs 2489.03 m) because a single regressed point collapses toward the conditional mean on ambiguous/multimodal crops -- this experiment is a bet that K=8 *learned, data-dependent* candidate points with winner-take-all training avoids that collapse (each hypothesis can specialize and only the closest one is graded, so ambiguity is represented by disagreement among the 8 points rather than smeared into one shrunken average), unlike the single-point regression that round 2 correctly moved away from. If this round regresses toward round-1/2 territory instead of improving, that is informative: it means the shrinkage failure mode is about point-estimate training generally, not specifically about the fixed grid, and would argue for reintroducing an explicit multimodal representation (e.g. mixture density output kept for uncertainty at inference) rather than any point-based decode. Result primary worst-case median error = 1100.35 m (previous best 743.07 m); full per-area/bucket breakdown in metrics.json Conclusion REVERTED — metric did not improve (1100.35 m vs best 743.07 m); change discarded Scoreboard — median error (m) per area × lighting
the worst cell (underlined) is the experiment's score;
red = failed cell, ink = at target
Fits the aircraft? all deployment gates passed
largest per-area model 1,069 KB (limit 4,096 KB) · single-frame inference 2.6 ms on one CPU thread (proxy limit 250 ms)
crops are sampled fresh each run from the frozen train split (~45,000 distinct positions per area, times random rotation), never from eval blocks; how many to use is the experiment's own choice
Where this iteration's time went: design (Sonnet) 13 m 48 s · implement (Sonnet) 2 m 08 s · train 4 areas 47 m 21 s · score 1 m 34 s · samples 42 s · publish 5 s · whole iteration 66 m 12 s
ts 2026-07-22T16:01:49 · commit 5a6771be65eb ·
parent a8fb2c8b3906 · artifacts runs/20260722_145547_iter1 ·
agent model — · took 66 m 02 s
The exact prompt given to the headless agent## PATIENCE SPENT — THIS ITERATION MUST PIVOT
1 consecutive experiments have failed to beat the current best.
Do NOT refine the champion's current mechanism again this round. Step back
and propose from a design family ABSENT from the last 1 experiments
— the spec (§3) explicitly invites, among others: dispatcher + lighting-
condition-specialist models, learned relighting, a different coordinate
parameterization, quantization-aware capacity changes, or training-data
strategy overhauls. Genuine novelty over incremental tuning.
# Autoresearch iteration — one focused experiment
You are one iteration of an autonomous research loop for UAV low-light
geolocalization. Read `CLAUDE.md` (§1, §3, §6 especially) for full context.
The harness (loop.sh) will train, score, log, and keep/revert AFTER you exit —
you only design the experiment and edit the code.
## Your job, in order
0. **Skim the library (optional input).** `autoresearch/library.md` holds
the human researcher's inspiration notes. They do not fix your answer
and you are free to ignore them — pick an entry up only when it
genuinely fits your read of the history, and if you build on one, say
so in your hypothesis.
1. **Review the research history.** Query the lineage DB:
`sqlite3 experiments.sqlite "SELECT id, title, category, hypothesis, expected_outcome, result, conclusion, primary_metric, kept FROM experiments ORDER BY id DESC LIMIT 15;"`
Note which hypotheses were supported/refuted. Do not repeat a refuted
experiment without a materially new angle.
**Plateau rule:** if three or more consecutive experiments were reverted,
do not attempt another variation of the last refuted mechanism. Either
pick a design family absent from the history (dispatcher + lighting
specialists, pretrained init, learned relighting, training-scale, …) or
attack the bottleneck the refuted hypotheses jointly point at —
**but check which stages that "design family" actually touches before
you commit to it.** Query the last several kept experiments'
`arch_json` (`SELECT arch_json FROM experiments WHERE kind='development'
ORDER BY id DESC LIMIT 10;`) and look at which stage names never carry
`"changed": true`. A losing streak is not just "we haven't tried a
dispatcher yet" — it is usually "the trunk / descriptor / decode has
gone unquestioned for N rounds while satellite modules (gates, heads,
auxiliary losses, samplers) keep churning around it." Picking a name
off the suggested list while leaving that frozen core untouched is
incremental tuning wearing a pivot's clothes, and the harness's own
patience check (below, once it fires) will call this out explicitly by
naming the frozen stages.
2. **Design ONE focused experiment** — proper experiment design, pre-registered
before you touch code. Write it to `runs/pending_experiment.json`:
```json
{
"title": "one-line name",
"category": "architecture|loss|augmentation|relighting|training|quantization|other",
"hypothesis": "what you believe is limiting the metric and why this change addresses it",
"method": "the ONE focused change, concretely (files, mechanism)",
"expected_outcome": "predicted effect on the §6 worst-case median error, quantified if possible",
"init_strategy": "from-scratch | pretrained:<name>",
"eli5": "2-4 sentences for a smart non-ML reader: what you changed and why it might help, in everyday language — analogies welcome, zero jargon",
"architecture": {"stages": [
{"name": "Camera frame", "detail": "128×128 px crop, one of 6 lighting renders", "changed": false},
{"name": "Feature extractor", "detail": "plain-language description", "changed": false},
{"name": "…", "detail": "…", "changed": true}
]},
"architecture_svg": "<svg viewBox='0 0 980 300' …>…</svg>"
}
```
`architecture_svg` is the **technical architecture diagram** of the model
you are testing — a proper ML-paper figure of the design, drawn by you,
shown at the top of this experiment's gallery entry. **Draw tensors and
operations as the things they are — NOT as labeled boxes:** the input
image as a pixel-textured square; conv feature maps as pseudo-3D slabs
that shrink spatially and deepen in channels; 1-D vectors as thin
vertical tick-bars; fully-connected layers as fans of thin crossing
lines between bars; spatial fields/grids as actually-drawn grids with
shaded cells; decodes/aggregations as thin lines converging from cells
onto a point; samplers/targets as small pictorial glyphs. Captions sit
UNDER each element (name ≤10.5px weight 600, sub-note ≤9.5px); tensor
shapes annotate the flow in small italics (128×128×3 → 8²×128 → 1024).
Losses/targets/samplers live in a lane below the inference path,
annotated with slanted dashed leader lines to small text — no boxes
around them. Draw the two FROZEN endpoints — the camera-frame input and
the (lat, lon, confidence) output — entirely in #9b998c with a small
italic “frozen contract” tag: they are fixed by the harness and outside
your search space. Everything between them is your design: draw it in ink,
with red only on what THIS experiment changes. Style contract, so all experiments' figures read as one
paper: viewBox width 980 (height ~300–360); transparent background; ink
#111111, captions #6b6a60, annotations/arrows #9b998c; **#8c2f1f (red)
reserved for exactly what this experiment changed**; training-only
elements in #8a6a1e (or red if changed); font-family
Palatino,Georgia,serif; stroke-width ≈1.2 elements, 1 arrows; fills only
faint tints (opacity ≤ .12); no gradients, no icons, no emoji. Inference
flows left → right from camera frame to (lat, lon, confidence).
**The camera frame is a fixed glyph — copy it verbatim.** It must look
like an actual nadir frame of terrain (streets, building footprints),
not abstract pixels, and be identical in every figure. Use exactly this
snippet, substituting Y for the frame's top edge (pick Y so the frame
centers on your inference lane; captions go under it as usual — note
128²×3 is the model input crop, ~1 m/px, not the sensor's native
resolution):
```svg
<g id='cam-terrain'><rect id='frozen-input' x='26' y='Y' width='76'
height='76' fill='#f6f4ea' stroke='#9b998c' stroke-width='1.6'/>
<path d='M31 Y+47 L97 Y+27' stroke='#e6e3d4' stroke-width='5' fill='none'/>
<path d='M59 Y+6 L49 Y+70' stroke='#e6e3d4' stroke-width='3.5' fill='none'/>
<rect x='34' y='Y+9' width='12' height='8' fill='#d9d5c3' transform='rotate(-8 40 Y+13)'/>
<rect x='78' y='Y+8' width='10' height='11' fill='#cfccbd'/>
<rect x='35' y='Y+57' width='13' height='8' fill='#d9d5c3'/>
<rect x='75' y='Y+50' width='10' height='9' fill='#cfccbd' transform='rotate(6 80 Y+54)'/>
<rect x='54' y='Y+31' width='9' height='8' fill='#d9d5c3' opacity='.85'/>
<ellipse cx='86' cy='Y+63' rx='8' ry='6' fill='#8a6a1e' opacity='.12'/>
<circle cx='40' cy='Y+28' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='70' cy='Y+14' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='92' cy='Y+36' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='52' cy='Y+52' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='82' cy='Y+70' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='31' cy='Y+42' r='.7' fill='#6b6a60' opacity='.5'/></g>
```
(Y+n means the literal number Y plus n — compute the values. Every glyph
element stays strictly INSIDE the frame border — figcheck rejects a
figure without the `cam-terrain` group.) Draw your
receptive-field square and kernel-projection lines on top of it as
usual.
**Anchor the frozen endpoints identically in every figure** so all
experiments' figures line up when compared down the gallery page: the
camera-frame square starts at x=26 (its captions centered on x=53), and
the output is right-anchored — decode crosshair at x=812, the
(lat, lon, confidence) text block text-anchor=start at x=828. Mark the
anchors machine-checkably: the camera-frame rect carries
`id='frozen-input'`, the output text block carries `id='frozen-output'`.
Lay the elements between them out on a running x-cursor with generous
spacing so nothing overlaps.
**Readability contract** (figcheck verifies this geometrically): no
text may overlap other text, and no line (arrows, converge fans,
leader lines) may pass through a label — a line may END at a label,
never cross one. Rules distilled from review rounds: dashed leaders
run as orthogonal L-routes through empty lanes, never diagonally
across components; converge/vote lines originate at their true
sources (the actual dots or window cells, not generic positions);
every element owns its caption — adjacent columns stagger their
caption rows instead of colliding; prefer moving a label into empty
horizontal space over stacking it against a neighbor. Width is fixed
at 980, but **height is yours**: grow the viewBox (240–640) whenever
more vertical room makes the layout cleaner rather than cramming.
**Before finishing, validate:**
`.venv/bin/python -m autoresearch.figcheck` — it checks the anchor
contract on runs/pending_experiment.json; revise until it prints PASS.
Consecutive conv layers must be tied by kernel-projection lines (small
kernel square on one face, faint lines converging to a cell on the
next) so the encoder reads as one computation, not boxes in a row.
`archive/arch_svg_reference.py` is the concrete reference implementation
of this entire visual language — read it and reuse its helper geometry
for any stage that already has one.
`eli5` and `architecture` feed the human-facing gallery. `architecture.stages`
is the model's inference path left-to-right, camera frame → (lat, lon,
confidence) output — one box per stage, plain-language `detail`,
`"changed": true` ONLY on the stages this experiment touches. Reuse the
previous experiment's stage names verbatim wherever a stage is unchanged
(check `SELECT arch_json FROM experiments WHERE arch_json IS NOT NULL ORDER BY id DESC LIMIT 1;`) —
the diagrams are compared box-by-box across experiments. A change that only
affects training (loss, augmentation, schedule) keeps the inference stages
unchanged and adds one final stage with `"train_only": true` describing the
training signal.
3. **Write the implementation brief.** You do NOT edit `model/` yourself —
a separate implementation agent applies your design, seeing only the
current code plus what you pre-registered. Add one more field to
`runs/pending_experiment.json`:
```
"implementation_brief": "exact file-level instructions: which functions/
blocks in model/model.py and model/train.py change and how, what stays
untouched, and every contract to preserve"
```
Be precise enough that a competent engineer with no other context
implements it in one pass. Always restate the fixed contracts:
`train.py`'s CLI and the ONNX export contract in `model/model.py`'s
docstring — the frozen scorer depends on them.
## Hard rules
- Edit ONLY `runs/pending_experiment.json`. You never edit `model/` (the
implementation stage does) and files listed in `/FROZEN` are off-limits;
the harness hard-reverts any change to them.
- NEVER touch, read, or evaluate the `hamburg` holdout area (§5).
- ONE focused change per iteration — if you can't describe it in one sentence,
it's too big. Prefer architectural/procedural novelty over hyperparameter
nudges (§3): changing a learning rate is a weak experiment; changing the
coordinate parameterization, loss family, relighting-for-training, or model
topology is a strong one.
- Do not run training yourself; the harness does that.
- Stay within the deployment gates: exported ONNX ≤ 4 MiB per area, host
latency proxy ≤ 250 ms (see pipeline/score.py).Where the model was tested — and how far off it was Each map is one full test area. Every dot is one held-out test location (all six lighting conditions overlaid): the model was shown a 128 m crop centered there and asked for its position. Dot color = the distance between its answer and the truth — green ≤ 20 m (at goal), amber ≤ 50 m, red beyond. A working model turns these maps green; spatial clusters of red reveal which parts of an area confuse it.     What the six lighting conditions look like through the simulated low-light sensor (example patches — illustration, not the training set)Every frame in this project is rendered as a starlight-class low-light sensor (Sony STARVIS2 / IMX585 class — the airframe's chosen camera) would see it, not as a normal camera would. That is why the night renders look uncannily close to daylight: at high gain such a sensor recovers scene structure from moonlight, skyglow and artificial lighting, paying for it in noise, lifted shadows and washed-out color — exactly what these renders simulate. Below, one example 256 m patch per area under the six lighting conditions the model must handle. These illustrate the dataset, not this experiment's performance — the actual training set is thousands of distinct crops per area (see “training data” above), and these renderings only change when the relighting method changes. berlin frankfurt munich prignitz | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▸ | 36 | Hardest-impostor margin hinge, retested on the confusability-weighted champion 3ee64311 | loss | pretrained:mobilenet_v3_small | 754.2 m | 3,225 KB | 2.8 ms | 59 m 54 s | $0.69 | discarded | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The design under test — technical diagram — gray = frozen contract · ink = the current design · red = this experiment's change One real test, end to end
→
1.61 km
miss, this crop 0.44
self-reported confidence In plain words Right now the model's best two guesses for ‘where is this’ are often a real place and a look-alike place somewhere else on the map, and it only barely favors the real one. Last round we made it practice more on exactly those confusing pairs; this round we also change how hard it gets corrected when it still ranks a look-alike above the truth. Instead of a gentle nudge spread thinly over the whole map, it now gets a direct penalty aimed specifically at its single worst rival guess, every time, until the true spot clearly wins. Hypothesis The §6 metric is bound by ranking quality, not descriptor content or the operating point: iter-12 probes of the exp-25 champion (runs/_probe_iter12_field.py) found the true cell is the field's top-1 on only 2% of eval crops while making the top-20 (of 1024) on 33-58% in every lighting bucket, with decode, rotation, and operating-point mechanisms all ruled out as the wall (see project memory champion-failure-mode-probes). Exp 35 (kept, 797.91 -> 743.07 m) improved this by changing WHICH locations get training exposure (confusability-weighted sampling over cell-descriptor lookalikes) but never touched the training SIGNAL itself: loss_fn's Gaussian-smeared CE (sigma=1.5 cells) still spreads its wrong-cell gradient across all 1024 cells in proportion to softmax mass, so no gradient concentrates specifically on the one distant lookalike currently outranking the truth -- exactly the objective-family gap exp 34 pre-registered (hardest-impostor margin hinge) and never got to test: its run died with ENOSPC inside prepare_realizations before a single gradient step (timings.json score_s: 0, train_wall_s: 268 -- see project memory exp34-enospc-disk-lifecycle). The hypothesis is UNTESTED, not refuted, and the mechanism it targets (the training signal) is a materially different lever from exp 35's (the sampler). The two should compound rather than compete: exp 35 already brings confusable cell pairs to the model more often; the margin hinge is the loss term that actually forces their scores apart once they arrive, rather than relying on CE's diffuse likelihood gradient to do it incidentally. Disk is currently clean (33 GB free per df -h) so this run should not repeat exp 34's infrastructure failure. Method ONE change, entirely inside model/model.py's loss_fn (training-only; forward(), export_onnx(), estimate_position(), and all of train.py -- sampler, prepare_realizations, schedule, calibrate_conf_shift -- are untouched): add a hardest-impostor margin hinge on the same C4-voted logits the decode reads. Per crop: s_pos = max logit within Chebyshev radius 1 of the true cell (the 3x3 block a 128 m crop legitimately straddles across a ~220 m cell boundary); s_neg = max logit over cells at Chebyshev radius > 3 (outside the 7x7 ring the sigma=1.5-cell Gaussian CE bump already covers, so the hinge never fights the CE term over near-misses -- same exclusion radius convention train.py's compute_cell_weights already uses for its confusability lookup, NEIGHBOR_EXCLUDE_R=1, just applied at a wider ring for the negative). loss += 0.5 * relu(1.0 + s_neg - s_pos), added to loss_fn's existing returned sum (ce + coord_loss + 0.3 * conf_loss). Margin 1.0 logit is roughly a 20x mass ratio after the beta=3 decode sharpening already in forward(). CE, coord L2, conf BCE, the sampler, the epoch schedule, and the calibration procedure are all unchanged, so the deployment gates and operating-point comparison to the champion stay apples-to-apples. Expected outcome Worst-case median kept error improves from the champion's 743.07 m to roughly 550-700 m, with the improvement spread across area x bucket cells rather than concentrated in one -- a real ranking fix, not an operating-point artifact, since the calibration procedure is unchanged and coverage should stay comparable to the champion's (see coverage-median-coupling memory: any large coverage shift would make the comparison unmeasured, not a genuine win). Mechanism-level prediction, matched to the iter-12 probe protocol: true-cell top-1 rate should rise measurably above the champion's ~2% baseline, and median sharpened peak mass should rise above ~0.14. Falsification fork: if primary metric doesn't improve even though ranking/peak-mass probes do, the wall has moved from the objective to descriptor information content itself -- pointing future iterations back at the trunk/feature capacity rather than the loss or sampler. Result primary worst-case median error = 754.22 m (previous best 743.07 m); full per-area/bucket breakdown in metrics.json Conclusion REVERTED — metric did not improve (754.22 m vs best 743.07 m); change discarded Scoreboard — median error (m) per area × lighting
the worst cell (underlined) is the experiment's score;
red = failed cell, ink = at target
Fits the aircraft? all deployment gates passed
largest per-area model 3,225 KB (limit 4,096 KB) · single-frame inference 2.8 ms on one CPU thread (proxy limit 250 ms)
crops are sampled fresh each run from the frozen train split (~45,000 distinct positions per area, times random rotation), never from eval blocks; how many to use is the experiment's own choice
Where this iteration's time went: design (Sonnet) 5 m 58 s · implement (Sonnet) 1 m 31 s · train 4 areas 49 m 47 s · score 1 m 37 s · samples 42 s · publish 4 s · whole iteration 60 m 03 s
ts 2026-07-22T12:30:57 · commit 3ee64311291b ·
parent 42f65c0a1cb8 · artifacts runs/20260722_113103_iter2 ·
agent model — · took 59 m 54 s
The exact prompt given to the headless agent# Autoresearch iteration — one focused experiment
You are one iteration of an autonomous research loop for UAV low-light
geolocalization. Read `CLAUDE.md` (§1, §3, §6 especially) for full context.
The harness (loop.sh) will train, score, log, and keep/revert AFTER you exit —
you only design the experiment and edit the code.
## Your job, in order
0. **Skim the library (optional input).** `autoresearch/library.md` holds
the human researcher's inspiration notes. They do not fix your answer
and you are free to ignore them — pick an entry up only when it
genuinely fits your read of the history, and if you build on one, say
so in your hypothesis.
1. **Review the research history.** Query the lineage DB:
`sqlite3 experiments.sqlite "SELECT id, title, category, hypothesis, expected_outcome, result, conclusion, primary_metric, kept FROM experiments ORDER BY id DESC LIMIT 15;"`
Note which hypotheses were supported/refuted. Do not repeat a refuted
experiment without a materially new angle.
**Plateau rule:** if three or more consecutive experiments were reverted,
do not attempt another variation of the last refuted mechanism. Either
pick a design family absent from the history (dispatcher + lighting
specialists, pretrained init, learned relighting, training-scale, …) or
attack the bottleneck the refuted hypotheses jointly point at.
2. **Design ONE focused experiment** — proper experiment design, pre-registered
before you touch code. Write it to `runs/pending_experiment.json`:
```json
{
"title": "one-line name",
"category": "architecture|loss|augmentation|relighting|training|quantization|other",
"hypothesis": "what you believe is limiting the metric and why this change addresses it",
"method": "the ONE focused change, concretely (files, mechanism)",
"expected_outcome": "predicted effect on the §6 worst-case median error, quantified if possible",
"init_strategy": "from-scratch | pretrained:<name>",
"eli5": "2-4 sentences for a smart non-ML reader: what you changed and why it might help, in everyday language — analogies welcome, zero jargon",
"architecture": {"stages": [
{"name": "Camera frame", "detail": "128×128 px crop, one of 6 lighting renders", "changed": false},
{"name": "Feature extractor", "detail": "plain-language description", "changed": false},
{"name": "…", "detail": "…", "changed": true}
]},
"architecture_svg": "<svg viewBox='0 0 980 300' …>…</svg>"
}
```
`architecture_svg` is the **technical architecture diagram** of the model
you are testing — a proper ML-paper figure of the design, drawn by you,
shown at the top of this experiment's gallery entry. **Draw tensors and
operations as the things they are — NOT as labeled boxes:** the input
image as a pixel-textured square; conv feature maps as pseudo-3D slabs
that shrink spatially and deepen in channels; 1-D vectors as thin
vertical tick-bars; fully-connected layers as fans of thin crossing
lines between bars; spatial fields/grids as actually-drawn grids with
shaded cells; decodes/aggregations as thin lines converging from cells
onto a point; samplers/targets as small pictorial glyphs. Captions sit
UNDER each element (name ≤10.5px weight 600, sub-note ≤9.5px); tensor
shapes annotate the flow in small italics (128×128×3 → 8²×128 → 1024).
Losses/targets/samplers live in a lane below the inference path,
annotated with slanted dashed leader lines to small text — no boxes
around them. Draw the two FROZEN endpoints — the camera-frame input and
the (lat, lon, confidence) output — entirely in #9b998c with a small
italic “frozen contract” tag: they are fixed by the harness and outside
your search space. Everything between them is your design: draw it in ink,
with red only on what THIS experiment changes. Style contract, so all experiments' figures read as one
paper: viewBox width 980 (height ~300–360); transparent background; ink
#111111, captions #6b6a60, annotations/arrows #9b998c; **#8c2f1f (red)
reserved for exactly what this experiment changed**; training-only
elements in #8a6a1e (or red if changed); font-family
Palatino,Georgia,serif; stroke-width ≈1.2 elements, 1 arrows; fills only
faint tints (opacity ≤ .12); no gradients, no icons, no emoji. Inference
flows left → right from camera frame to (lat, lon, confidence).
**The camera frame is a fixed glyph — copy it verbatim.** It must look
like an actual nadir frame of terrain (streets, building footprints),
not abstract pixels, and be identical in every figure. Use exactly this
snippet, substituting Y for the frame's top edge (pick Y so the frame
centers on your inference lane; captions go under it as usual — note
128²×3 is the model input crop, ~1 m/px, not the sensor's native
resolution):
```svg
<g id='cam-terrain'><rect id='frozen-input' x='26' y='Y' width='76'
height='76' fill='#f6f4ea' stroke='#9b998c' stroke-width='1.6'/>
<path d='M31 Y+47 L97 Y+27' stroke='#e6e3d4' stroke-width='5' fill='none'/>
<path d='M59 Y+6 L49 Y+70' stroke='#e6e3d4' stroke-width='3.5' fill='none'/>
<rect x='34' y='Y+9' width='12' height='8' fill='#d9d5c3' transform='rotate(-8 40 Y+13)'/>
<rect x='78' y='Y+8' width='10' height='11' fill='#cfccbd'/>
<rect x='35' y='Y+57' width='13' height='8' fill='#d9d5c3'/>
<rect x='75' y='Y+50' width='10' height='9' fill='#cfccbd' transform='rotate(6 80 Y+54)'/>
<rect x='54' y='Y+31' width='9' height='8' fill='#d9d5c3' opacity='.85'/>
<ellipse cx='86' cy='Y+63' rx='8' ry='6' fill='#8a6a1e' opacity='.12'/>
<circle cx='40' cy='Y+28' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='70' cy='Y+14' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='92' cy='Y+36' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='52' cy='Y+52' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='82' cy='Y+70' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='31' cy='Y+42' r='.7' fill='#6b6a60' opacity='.5'/></g>
```
(Y+n means the literal number Y plus n — compute the values. Every glyph
element stays strictly INSIDE the frame border — figcheck rejects a
figure without the `cam-terrain` group.) Draw your
receptive-field square and kernel-projection lines on top of it as
usual.
**Anchor the frozen endpoints identically in every figure** so all
experiments' figures line up when compared down the gallery page: the
camera-frame square starts at x=26 (its captions centered on x=53), and
the output is right-anchored — decode crosshair at x=812, the
(lat, lon, confidence) text block text-anchor=start at x=828. Mark the
anchors machine-checkably: the camera-frame rect carries
`id='frozen-input'`, the output text block carries `id='frozen-output'`.
Lay the elements between them out on a running x-cursor with generous
spacing so nothing overlaps.
**Readability contract** (figcheck verifies this geometrically): no
text may overlap other text, and no line (arrows, converge fans,
leader lines) may pass through a label — a line may END at a label,
never cross one. Rules distilled from review rounds: dashed leaders
run as orthogonal L-routes through empty lanes, never diagonally
across components; converge/vote lines originate at their true
sources (the actual dots or window cells, not generic positions);
every element owns its caption — adjacent columns stagger their
caption rows instead of colliding; prefer moving a label into empty
horizontal space over stacking it against a neighbor. Width is fixed
at 980, but **height is yours**: grow the viewBox (240–640) whenever
more vertical room makes the layout cleaner rather than cramming.
**Before finishing, validate:**
`.venv/bin/python -m autoresearch.figcheck` — it checks the anchor
contract on runs/pending_experiment.json; revise until it prints PASS.
Consecutive conv layers must be tied by kernel-projection lines (small
kernel square on one face, faint lines converging to a cell on the
next) so the encoder reads as one computation, not boxes in a row.
`archive/arch_svg_reference.py` is the concrete reference implementation
of this entire visual language — read it and reuse its helper geometry
for any stage that already has one.
`eli5` and `architecture` feed the human-facing gallery. `architecture.stages`
is the model's inference path left-to-right, camera frame → (lat, lon,
confidence) output — one box per stage, plain-language `detail`,
`"changed": true` ONLY on the stages this experiment touches. Reuse the
previous experiment's stage names verbatim wherever a stage is unchanged
(check `SELECT arch_json FROM experiments WHERE arch_json IS NOT NULL ORDER BY id DESC LIMIT 1;`) —
the diagrams are compared box-by-box across experiments. A change that only
affects training (loss, augmentation, schedule) keeps the inference stages
unchanged and adds one final stage with `"train_only": true` describing the
training signal.
3. **Write the implementation brief.** You do NOT edit `model/` yourself —
a separate implementation agent applies your design, seeing only the
current code plus what you pre-registered. Add one more field to
`runs/pending_experiment.json`:
```
"implementation_brief": "exact file-level instructions: which functions/
blocks in model/model.py and model/train.py change and how, what stays
untouched, and every contract to preserve"
```
Be precise enough that a competent engineer with no other context
implements it in one pass. Always restate the fixed contracts:
`train.py`'s CLI and the ONNX export contract in `model/model.py`'s
docstring — the frozen scorer depends on them.
## Hard rules
- Edit ONLY `runs/pending_experiment.json`. You never edit `model/` (the
implementation stage does) and files listed in `/FROZEN` are off-limits;
the harness hard-reverts any change to them.
- NEVER touch, read, or evaluate the `hamburg` holdout area (§5).
- ONE focused change per iteration — if you can't describe it in one sentence,
it's too big. Prefer architectural/procedural novelty over hyperparameter
nudges (§3): changing a learning rate is a weak experiment; changing the
coordinate parameterization, loss family, relighting-for-training, or model
topology is a strong one.
- Do not run training yourself; the harness does that.
- Stay within the deployment gates: exported ONNX ≤ 4 MiB per area, host
latency proxy ≤ 250 ms (see pipeline/score.py).Where the model was tested — and how far off it was Each map is one full test area. Every dot is one held-out test location (all six lighting conditions overlaid): the model was shown a 128 m crop centered there and asked for its position. Dot color = the distance between its answer and the truth — green ≤ 20 m (at goal), amber ≤ 50 m, red beyond. A working model turns these maps green; spatial clusters of red reveal which parts of an area confuse it.     What the six lighting conditions look like through the simulated low-light sensor (example patches — illustration, not the training set)Every frame in this project is rendered as a starlight-class low-light sensor (Sony STARVIS2 / IMX585 class — the airframe's chosen camera) would see it, not as a normal camera would. That is why the night renders look uncannily close to daylight: at high gain such a sensor recovers scene structure from moonlight, skyglow and artificial lighting, paying for it in noise, lifted shadows and washed-out color — exactly what these renders simulate. Below, one example 256 m patch per area under the six lighting conditions the model must handle. These illustrate the dataset, not this experiment's performance — the actual training set is thousands of distinct crops per area (see “training data” above), and these renderings only change when the relighting method changes. berlin frankfurt munich prignitz | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▸ | 35 | Confusability-weighted location sampling: oversample look-alike cell pairs, not just more places 3750ddb6 | training pivot | pretrained:mobilenet_v3_small IMAGENET1K_V1 features[0..8] (torchvision, BSD-3) — unchanged | 743.1 m | 3,225 KB | 2.7 ms | 62 m 59 s | $0.72 | kept | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The design under test — technical diagram — gray = frozen contract · ink = the current design · red = this experiment's change One real test, end to end
→
1.75 km
miss, this crop 0.42
self-reported confidence In plain words The model has to recognize 1,024 map tiles from a single photo, and we already know its biggest problem: some far-away tile that just happens to look similar often beats the true tile in the ranking. Until now, training showed the model every tile equally often, whether or not it has a lookalike twin somewhere else on the map. This experiment changes nothing about the model or the grading -- only which practice photos get shown more often. Before training starts, we compare the average color and texture of every tile against every other (a one-time, cheap lookup from the daytime map image itself) and flag pairs of far-apart tiles that look alike. During training, places with a lookalike twin get shown about twice as often as places that look nothing like anywhere else -- half the practice still covers the whole map evenly, so nothing is forgotten, but the confusing pairs get extra practice specifically at telling themselves apart. Hypothesis Per the plateau rule this iteration leaves the last nine mechanisms entirely (26-34: trunk capacity x3, stride-8 readout tap, lighting dispatcher, neural-map coordinate reparameterization, fenced/risk calibration, sensor-sharpness augmentation, and a hardest-impostor margin loss -- the last two never actually trained to completion, ENOSPC/OOM-killed before a single score, per runs/_probe_iter12_*.py forensics logged against exps 33/34) and opens a design family CLAUDE.md SS3 invites but the loop has never tried: a training-DATA strategy overhaul, distinct from every loss/architecture/ calibration change already attempted. The iter-12 probes of the exp-25 champion (runs/_probe_iter12_field.py) established the wall precisely: the true cell is the field's top-1 on only 2% of eval crops while making the top-20 (of 1024) on 33-58% in every bucket -- a lookalike cell elsewhere on the map usually outranks the truth. Since exp 20, every cell's classifier row gets the SAME training exposure regardless of how confusable it is (uniform fresh location draw, 6,000/bucket/epoch): a cell with a distant near-duplicate twin gets no more corrective gradient than an entirely distinctive one, even though it is precisely the confusable cells whose classifier rows must be pushed hardest apart to fix the measured top-1 failure. Rural Prignitz (distinctive macro-structure, few self-similar districts) already beats all three self-similar metros in every bucket under this exact uniform scheme -- the metros' worse scores are concentrated in exactly the cells uniform sampling under-serves. Rather than touch the loss (margin family, attempted but crashed untested in exp 34) or the architecture (capacity/dispatcher/coordinate families, all attempted and refuted for real in exps 26-31), this experiment changes WHICH locations the sampler draws: a cheap, one-time-per-area map of each classifier cell's nearest visually-similar DISTANT cell (computed straight from the frozen reference.tif raster, no network, no per-epoch cost) sets a sampling weight, so confusable cells are oversampled relative to distinctive ones -- concentrating the fixed per-epoch gradient budget where the ranking wall actually is. Method ONE change, entirely inside model/train.py (model/model.py untouched -- no architecture or loss change): (1) a new one-time-per-area function computes a GRID_K x GRID_K (32x32=1024) cell-descriptor map by average-pooling reference.tif's RGB into the same grid the field head already scores (mean R,G,B + luminance std per cell, 4 numbers), then for every cell finds its nearest OTHER cell in descriptor space excluding itself and its 8 spatial neighbors (so the weight targets genuine distant lookalikes, not trivial local blur-driven near-duplicates), giving each cell a confusability distance d_i and weight w_i = 1/(d_i + eps); (2) every training crop is mapped to its ground-truth cell and assigned sampling probability p = 0.5 x uniform + 0.5 x normalize(w) over that cell's crops; (3) sample_epoch's uniform rng.choice(..., replace=False) is replaced with the weighted rng.choice(..., p=crop_probs, replace=False) for the TRAINING draw only. calibrate_conf_shift's sampling stays uniform/unchanged (coverage-median-coupling memory: do not let a data-strategy change also alter the calibration distribution, or a metric shift can't be attributed to either mechanism cleanly). Model architecture, loss_fn, epoch count, LR schedule, and the C4/gate/decode/conf machinery are byte-identical. Expected outcome If concentrating supervision on confusable cell pairs sharpens the shared classifier's separating margins where the measured top-1 failure lives, the worst-case median error improves modestly from 797.91 m to roughly 720-780 m (2-10%), with gains concentrated in the three self-similar metro areas (berlin/munich/frankfurt) where confusable cell pairs are common, and prignitz roughly unchanged (its distinctive macro-structure yields near-uniform weights, since few cells have a close descriptor twin). Coverage per cell should stay near the champion's 0.31-0.52 profile (calibration path is untouched, so a median win here is not a coverage-median-coupling artifact). Falsification forks: (a) if the computed weights are nearly flat (w_i ratio top-to-bottom decile < 1.5x) -- meaning 220 m-cell-average RGB/luminance is too coarse a descriptor to separate confusable from distinctive cells -- the mechanism cannot have fired and a finer per-cell descriptor (or a learned one) is the next lever, not oversampling harder; (b) if weights are clearly non-flat but the metric does not move (within +-3%), oversampling frequency is exonerated and the wall is in the classifier's capacity to separate cells given equal exposure -- redirecting back to the loss family (the untested margin/hinge mechanism from exp 34) with a clean conscience, since this experiment addresses exposure while leaving the objective alone; (c) if the metric regresses, concentrating draws too heavily on a cell subset reduced effective location diversity per epoch below what the 1024-way head needs -- rerun at a lower ALPHA (blend weight) rather than abandoning the family. Result primary worst-case median error = 743.07 m (previous best 797.91 m); full per-area/bucket breakdown in metrics.json Conclusion KEPT — metric improved (797.91 -> 743.07 m); change committed as 3750ddb Scoreboard — median error (m) per area × lighting
the worst cell (underlined) is the experiment's score;
red = failed cell, ink = at target
Fits the aircraft? all deployment gates passed
largest per-area model 3,225 KB (limit 4,096 KB) · single-frame inference 2.7 ms on one CPU thread (proxy limit 250 ms)
crops are sampled fresh each run from the frozen train split (~45,000 distinct positions per area, times random rotation), never from eval blocks; how many to use is the experiment's own choice
Where this iteration's time went: design (Sonnet) 9 m 34 s · implement (Sonnet) 1 m 32 s · train 4 areas 49 m 17 s · score 1 m 34 s · samples 42 s · publish 6 s · whole iteration 63 m 09 s
ts 2026-07-22T11:30:46 · commit 3750ddb61f7a ·
parent 959e874fc3ca · artifacts runs/20260722_102747_iter1 ·
agent model — · took 62 m 59 s
The exact prompt given to the headless agent## PATIENCE SPENT — THIS ITERATION MUST PIVOT
9 consecutive experiments have failed to beat the current best.
Do NOT refine the champion's current mechanism again this round. Step back
and propose from a design family ABSENT from the last 9 experiments
— the spec (§3) explicitly invites, among others: dispatcher + lighting-
condition-specialist models, learned relighting, a different coordinate
parameterization, quantization-aware capacity changes, or training-data
strategy overhauls. Genuine novelty over incremental tuning.
# Autoresearch iteration — one focused experiment
You are one iteration of an autonomous research loop for UAV low-light
geolocalization. Read `CLAUDE.md` (§1, §3, §6 especially) for full context.
The harness (loop.sh) will train, score, log, and keep/revert AFTER you exit —
you only design the experiment and edit the code.
## Your job, in order
0. **Skim the library (optional input).** `autoresearch/library.md` holds
the human researcher's inspiration notes. They do not fix your answer
and you are free to ignore them — pick an entry up only when it
genuinely fits your read of the history, and if you build on one, say
so in your hypothesis.
1. **Review the research history.** Query the lineage DB:
`sqlite3 experiments.sqlite "SELECT id, title, category, hypothesis, expected_outcome, result, conclusion, primary_metric, kept FROM experiments ORDER BY id DESC LIMIT 15;"`
Note which hypotheses were supported/refuted. Do not repeat a refuted
experiment without a materially new angle.
**Plateau rule:** if three or more consecutive experiments were reverted,
do not attempt another variation of the last refuted mechanism. Either
pick a design family absent from the history (dispatcher + lighting
specialists, pretrained init, learned relighting, training-scale, …) or
attack the bottleneck the refuted hypotheses jointly point at.
2. **Design ONE focused experiment** — proper experiment design, pre-registered
before you touch code. Write it to `runs/pending_experiment.json`:
```json
{
"title": "one-line name",
"category": "architecture|loss|augmentation|relighting|training|quantization|other",
"hypothesis": "what you believe is limiting the metric and why this change addresses it",
"method": "the ONE focused change, concretely (files, mechanism)",
"expected_outcome": "predicted effect on the §6 worst-case median error, quantified if possible",
"init_strategy": "from-scratch | pretrained:<name>",
"eli5": "2-4 sentences for a smart non-ML reader: what you changed and why it might help, in everyday language — analogies welcome, zero jargon",
"architecture": {"stages": [
{"name": "Camera frame", "detail": "128×128 px crop, one of 6 lighting renders", "changed": false},
{"name": "Feature extractor", "detail": "plain-language description", "changed": false},
{"name": "…", "detail": "…", "changed": true}
]},
"architecture_svg": "<svg viewBox='0 0 980 300' …>…</svg>"
}
```
`architecture_svg` is the **technical architecture diagram** of the model
you are testing — a proper ML-paper figure of the design, drawn by you,
shown at the top of this experiment's gallery entry. **Draw tensors and
operations as the things they are — NOT as labeled boxes:** the input
image as a pixel-textured square; conv feature maps as pseudo-3D slabs
that shrink spatially and deepen in channels; 1-D vectors as thin
vertical tick-bars; fully-connected layers as fans of thin crossing
lines between bars; spatial fields/grids as actually-drawn grids with
shaded cells; decodes/aggregations as thin lines converging from cells
onto a point; samplers/targets as small pictorial glyphs. Captions sit
UNDER each element (name ≤10.5px weight 600, sub-note ≤9.5px); tensor
shapes annotate the flow in small italics (128×128×3 → 8²×128 → 1024).
Losses/targets/samplers live in a lane below the inference path,
annotated with slanted dashed leader lines to small text — no boxes
around them. Draw the two FROZEN endpoints — the camera-frame input and
the (lat, lon, confidence) output — entirely in #9b998c with a small
italic “frozen contract” tag: they are fixed by the harness and outside
your search space. Everything between them is your design: draw it in ink,
with red only on what THIS experiment changes. Style contract, so all experiments' figures read as one
paper: viewBox width 980 (height ~300–360); transparent background; ink
#111111, captions #6b6a60, annotations/arrows #9b998c; **#8c2f1f (red)
reserved for exactly what this experiment changed**; training-only
elements in #8a6a1e (or red if changed); font-family
Palatino,Georgia,serif; stroke-width ≈1.2 elements, 1 arrows; fills only
faint tints (opacity ≤ .12); no gradients, no icons, no emoji. Inference
flows left → right from camera frame to (lat, lon, confidence).
**The camera frame is a fixed glyph — copy it verbatim.** It must look
like an actual nadir frame of terrain (streets, building footprints),
not abstract pixels, and be identical in every figure. Use exactly this
snippet, substituting Y for the frame's top edge (pick Y so the frame
centers on your inference lane; captions go under it as usual — note
128²×3 is the model input crop, ~1 m/px, not the sensor's native
resolution):
```svg
<g id='cam-terrain'><rect id='frozen-input' x='26' y='Y' width='76'
height='76' fill='#f6f4ea' stroke='#9b998c' stroke-width='1.6'/>
<path d='M31 Y+47 L97 Y+27' stroke='#e6e3d4' stroke-width='5' fill='none'/>
<path d='M59 Y+6 L49 Y+70' stroke='#e6e3d4' stroke-width='3.5' fill='none'/>
<rect x='34' y='Y+9' width='12' height='8' fill='#d9d5c3' transform='rotate(-8 40 Y+13)'/>
<rect x='78' y='Y+8' width='10' height='11' fill='#cfccbd'/>
<rect x='35' y='Y+57' width='13' height='8' fill='#d9d5c3'/>
<rect x='75' y='Y+50' width='10' height='9' fill='#cfccbd' transform='rotate(6 80 Y+54)'/>
<rect x='54' y='Y+31' width='9' height='8' fill='#d9d5c3' opacity='.85'/>
<ellipse cx='86' cy='Y+63' rx='8' ry='6' fill='#8a6a1e' opacity='.12'/>
<circle cx='40' cy='Y+28' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='70' cy='Y+14' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='92' cy='Y+36' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='52' cy='Y+52' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='82' cy='Y+70' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='31' cy='Y+42' r='.7' fill='#6b6a60' opacity='.5'/></g>
```
(Y+n means the literal number Y plus n — compute the values. Every glyph
element stays strictly INSIDE the frame border — figcheck rejects a
figure without the `cam-terrain` group.) Draw your
receptive-field square and kernel-projection lines on top of it as
usual.
**Anchor the frozen endpoints identically in every figure** so all
experiments' figures line up when compared down the gallery page: the
camera-frame square starts at x=26 (its captions centered on x=53), and
the output is right-anchored — decode crosshair at x=812, the
(lat, lon, confidence) text block text-anchor=start at x=828. Mark the
anchors machine-checkably: the camera-frame rect carries
`id='frozen-input'`, the output text block carries `id='frozen-output'`.
Lay the elements between them out on a running x-cursor with generous
spacing so nothing overlaps.
**Readability contract** (figcheck verifies this geometrically): no
text may overlap other text, and no line (arrows, converge fans,
leader lines) may pass through a label — a line may END at a label,
never cross one. Rules distilled from review rounds: dashed leaders
run as orthogonal L-routes through empty lanes, never diagonally
across components; converge/vote lines originate at their true
sources (the actual dots or window cells, not generic positions);
every element owns its caption — adjacent columns stagger their
caption rows instead of colliding; prefer moving a label into empty
horizontal space over stacking it against a neighbor. Width is fixed
at 980, but **height is yours**: grow the viewBox (240–640) whenever
more vertical room makes the layout cleaner rather than cramming.
**Before finishing, validate:**
`.venv/bin/python -m autoresearch.figcheck` — it checks the anchor
contract on runs/pending_experiment.json; revise until it prints PASS.
Consecutive conv layers must be tied by kernel-projection lines (small
kernel square on one face, faint lines converging to a cell on the
next) so the encoder reads as one computation, not boxes in a row.
`archive/arch_svg_reference.py` is the concrete reference implementation
of this entire visual language — read it and reuse its helper geometry
for any stage that already has one.
`eli5` and `architecture` feed the human-facing gallery. `architecture.stages`
is the model's inference path left-to-right, camera frame → (lat, lon,
confidence) output — one box per stage, plain-language `detail`,
`"changed": true` ONLY on the stages this experiment touches. Reuse the
previous experiment's stage names verbatim wherever a stage is unchanged
(check `SELECT arch_json FROM experiments WHERE arch_json IS NOT NULL ORDER BY id DESC LIMIT 1;`) —
the diagrams are compared box-by-box across experiments. A change that only
affects training (loss, augmentation, schedule) keeps the inference stages
unchanged and adds one final stage with `"train_only": true` describing the
training signal.
3. **Write the implementation brief.** You do NOT edit `model/` yourself —
a separate implementation agent applies your design, seeing only the
current code plus what you pre-registered. Add one more field to
`runs/pending_experiment.json`:
```
"implementation_brief": "exact file-level instructions: which functions/
blocks in model/model.py and model/train.py change and how, what stays
untouched, and every contract to preserve"
```
Be precise enough that a competent engineer with no other context
implements it in one pass. Always restate the fixed contracts:
`train.py`'s CLI and the ONNX export contract in `model/model.py`'s
docstring — the frozen scorer depends on them.
## Hard rules
- Edit ONLY `runs/pending_experiment.json`. You never edit `model/` (the
implementation stage does) and files listed in `/FROZEN` are off-limits;
the harness hard-reverts any change to them.
- NEVER touch, read, or evaluate the `hamburg` holdout area (§5).
- ONE focused change per iteration — if you can't describe it in one sentence,
it's too big. Prefer architectural/procedural novelty over hyperparameter
nudges (§3): changing a learning rate is a weak experiment; changing the
coordinate parameterization, loss family, relighting-for-training, or model
topology is a strong one.
- Do not run training yourself; the harness does that.
- Stay within the deployment gates: exported ONNX ≤ 4 MiB per area, host
latency proxy ≤ 250 ms (see pipeline/score.py).Where the model was tested — and how far off it was Each map is one full test area. Every dot is one held-out test location (all six lighting conditions overlaid): the model was shown a 128 m crop centered there and asked for its position. Dot color = the distance between its answer and the truth — green ≤ 20 m (at goal), amber ≤ 50 m, red beyond. A working model turns these maps green; spatial clusters of red reveal which parts of an area confuse it.     What the six lighting conditions look like through the simulated low-light sensor (example patches — illustration, not the training set)Every frame in this project is rendered as a starlight-class low-light sensor (Sony STARVIS2 / IMX585 class — the airframe's chosen camera) would see it, not as a normal camera would. That is why the night renders look uncannily close to daylight: at high gain such a sensor recovers scene structure from moonlight, skyglow and artificial lighting, paying for it in noise, lifted shadows and washed-out color — exactly what these renders simulate. Below, one example 256 m patch per area under the six lighting conditions the model must handle. These illustrate the dataset, not this experiment's performance — the actual training set is thousands of distinct crops per area (see “training data” above), and these renderings only change when the relighting method changes. berlin frankfurt munich prignitz | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▸ | 34 | Hardest-impostor margin loss: the field must outrank its best lookalike, not just light up the truth bfbca30d | loss pivot | pretrained:mobilenet_v3_small IMAGENET1K_V1 features[0..8] (torchvision, BSD-3) — unchanged | gated fail | — | — | 20 m 01 s | $0.23 | gated fail | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The design under test — technical diagram — gray = frozen contract · ink = the current design · red = this experiment's change In plain words The model scores 1,024 map tiles for each camera frame and we fly toward the highest-scoring one. Until now training only rewarded giving the right tile a decent score — like a teacher handing out partial credit but never checking who came FIRST. Measurements show the right tile usually makes the model's top-20 list, yet almost never ranks #1: some lookalike district across town keeps beating it. The new rule is a police-lineup rule: on every training example, find the one impostor tile the model currently likes best and demand the true tile's neighborhood beat it by a clear gap. The specific impostor that keeps winning gets pushed down, the truth pushed up. Hypothesis The §6 metric is bound by ranking quality, and the objective family is the one lever nobody has pulled. The iter-12 probes of the exp-25 champion show the true cell is the field's top-1 on only 2% of eval crops while making the top-20 (of 1024) on 33–58% — in every lighting bucket including midday — with median peak mass 0.14 and ~2 separated modes; decode, rotation, and operating-point mechanisms are ruled out. The eight consecutive reverts (26–33) all changed what FEEDS the ranking (3× trunk capacity, stride-8 readout tap, lighting dispatcher, learned map raster) or how its output is thresholded (fenced/risk calibration), or the input statistics (exp 33 — whose 1e9 is an infrastructure artifact: munich training was OOM-killed and the run was never scored). The training signal that CREATES the ranking has been pure likelihood since exp 2: Gaussian-smeared CE spreads its wrong-cell gradient across 1024 cells in proportion to softmax mass, and the σ=1.5-cell smearing makes near-misses cheap, so no gradient ever concentrates on the single distant lookalike that currently outranks the truth — which is precisely the measured failure (truth present in the candidate set, outranked at the top). Visual place recognition trains this exact problem with hardest-negative margin objectives (NetVLAD/FaceNet family). A per-crop hinge that demotes the current best-scoring impostor cell should convert top-20 presence into top-1 rank and raise peak mass, improving median kept error across all 24 area×bucket cells rather than gaming one operating point. Method ONE change, entirely inside model/model.py's loss_fn (training-only; forward, export, train.py untouched): add a hardest-impostor margin hinge on the same C4-voted logits the decode uses. Per crop: s_pos = max logit within Chebyshev radius 1 of the true cell (3×3 block — a 128 m crop legitimately straddles ~220 m cells), s_neg = max logit over cells at Chebyshev radius > 3 (outside the 7×7 ring, so the hinge never fights the σ=1.5 CE bump), loss += 0.5 · relu(1.0 + s_neg − s_pos). Margin 1.0 logit ≈ 20× mass ratio after the β=3 decode sharpening. CE, coord L2, conf BCE, sampler, schedule, calibration, and the ONNX graph are unchanged, so the deployment gates and the operating-point calibration procedure are identical to the champion's. Expected outcome Worst-case median kept error improves from 797.91 m to 600–750 m, with the improvement spread across all area×bucket cells (same calibration procedure → comparable coverage, so the comparison is not an operating-point fluke). Mechanism-level predictions: probe-comparable top-1 rate (berlin midday, matched protocol) rises from ~2% to ≥8%, median sharpened peak mass from ~0.14 to ≥0.25. Falsification fork: if ranking metrics improve but kept medians do not, the residual wall is descriptor information content, not the objective — redirecting future iterations at the trunk/descriptor rather than the loss or the operating point. Result primary worst-case median error = 1000000000.0 m (previous best 797.91 m); full per-area/bucket breakdown in metrics.json Conclusion REVERTED — metric did not improve (1000000000.0 m vs best 797.91 m); change discarded Scoreboard — median error (m) per area × lighting
the worst cell (underlined) is the experiment's score;
red = failed cell, ink = at target
No per-area results recorded for this run. Fits the aircraft? all deployment gates passed
largest per-area model — (limit 4,096 KB) · single-frame inference — on one CPU thread (proxy limit 250 ms) Where this iteration's time went: design (Haiku) 13 m 45 s · implement (Sonnet) 44 s · train 4 areas 4 m 28 s · samples 42 s · publish 1 s · whole iteration 20 m 07 s
ts 2026-07-22T08:53:43 · commit bfbca30d5c51 ·
parent bfbca30d5c51 · artifacts runs/20260722_083342_iter7 ·
agent model — · took 20 m 01 s
The exact prompt given to the headless agent## PATIENCE SPENT — THIS ITERATION MUST PIVOT
8 consecutive experiments have failed to beat the current best.
Do NOT refine the champion's current mechanism again this round. Step back
and propose from a design family ABSENT from the last 8 experiments
— the spec (§3) explicitly invites, among others: dispatcher + lighting-
condition-specialist models, learned relighting, a different coordinate
parameterization, quantization-aware capacity changes, or training-data
strategy overhauls. Genuine novelty over incremental tuning.
# Autoresearch iteration — one focused experiment
You are one iteration of an autonomous research loop for UAV low-light
geolocalization. Read `CLAUDE.md` (§1, §3, §6 especially) for full context.
The harness (loop.sh) will train, score, log, and keep/revert AFTER you exit —
you only design the experiment and edit the code.
## Your job, in order
0. **Skim the library (optional input).** `autoresearch/library.md` holds
the human researcher's inspiration notes. They do not fix your answer
and you are free to ignore them — pick an entry up only when it
genuinely fits your read of the history, and if you build on one, say
so in your hypothesis.
1. **Review the research history.** Query the lineage DB:
`sqlite3 experiments.sqlite "SELECT id, title, category, hypothesis, expected_outcome, result, conclusion, primary_metric, kept FROM experiments ORDER BY id DESC LIMIT 15;"`
Note which hypotheses were supported/refuted. Do not repeat a refuted
experiment without a materially new angle.
**Plateau rule:** if three or more consecutive experiments were reverted,
do not attempt another variation of the last refuted mechanism. Either
pick a design family absent from the history (dispatcher + lighting
specialists, pretrained init, learned relighting, training-scale, …) or
attack the bottleneck the refuted hypotheses jointly point at.
2. **Design ONE focused experiment** — proper experiment design, pre-registered
before you touch code. Write it to `runs/pending_experiment.json`:
```json
{
"title": "one-line name",
"category": "architecture|loss|augmentation|relighting|training|quantization|other",
"hypothesis": "what you believe is limiting the metric and why this change addresses it",
"method": "the ONE focused change, concretely (files, mechanism)",
"expected_outcome": "predicted effect on the §6 worst-case median error, quantified if possible",
"init_strategy": "from-scratch | pretrained:<name>",
"eli5": "2-4 sentences for a smart non-ML reader: what you changed and why it might help, in everyday language — analogies welcome, zero jargon",
"architecture": {"stages": [
{"name": "Camera frame", "detail": "128×128 px crop, one of 6 lighting renders", "changed": false},
{"name": "Feature extractor", "detail": "plain-language description", "changed": false},
{"name": "…", "detail": "…", "changed": true}
]},
"architecture_svg": "<svg viewBox='0 0 980 300' …>…</svg>"
}
```
`architecture_svg` is the **technical architecture diagram** of the model
you are testing — a proper ML-paper figure of the design, drawn by you,
shown at the top of this experiment's gallery entry. **Draw tensors and
operations as the things they are — NOT as labeled boxes:** the input
image as a pixel-textured square; conv feature maps as pseudo-3D slabs
that shrink spatially and deepen in channels; 1-D vectors as thin
vertical tick-bars; fully-connected layers as fans of thin crossing
lines between bars; spatial fields/grids as actually-drawn grids with
shaded cells; decodes/aggregations as thin lines converging from cells
onto a point; samplers/targets as small pictorial glyphs. Captions sit
UNDER each element (name ≤10.5px weight 600, sub-note ≤9.5px); tensor
shapes annotate the flow in small italics (128×128×3 → 8²×128 → 1024).
Losses/targets/samplers live in a lane below the inference path,
annotated with slanted dashed leader lines to small text — no boxes
around them. Draw the two FROZEN endpoints — the camera-frame input and
the (lat, lon, confidence) output — entirely in #9b998c with a small
italic “frozen contract” tag: they are fixed by the harness and outside
your search space. Everything between them is your design: draw it in ink,
with red only on what THIS experiment changes. Style contract, so all experiments' figures read as one
paper: viewBox width 980 (height ~300–360); transparent background; ink
#111111, captions #6b6a60, annotations/arrows #9b998c; **#8c2f1f (red)
reserved for exactly what this experiment changed**; training-only
elements in #8a6a1e (or red if changed); font-family
Palatino,Georgia,serif; stroke-width ≈1.2 elements, 1 arrows; fills only
faint tints (opacity ≤ .12); no gradients, no icons, no emoji. Inference
flows left → right from camera frame to (lat, lon, confidence).
**The camera frame is a fixed glyph — copy it verbatim.** It must look
like an actual nadir frame of terrain (streets, building footprints),
not abstract pixels, and be identical in every figure. Use exactly this
snippet, substituting Y for the frame's top edge (pick Y so the frame
centers on your inference lane; captions go under it as usual — note
128²×3 is the model input crop, ~1 m/px, not the sensor's native
resolution):
```svg
<g id='cam-terrain'><rect id='frozen-input' x='26' y='Y' width='76'
height='76' fill='#f6f4ea' stroke='#9b998c' stroke-width='1.6'/>
<path d='M31 Y+47 L97 Y+27' stroke='#e6e3d4' stroke-width='5' fill='none'/>
<path d='M59 Y+6 L49 Y+70' stroke='#e6e3d4' stroke-width='3.5' fill='none'/>
<rect x='34' y='Y+9' width='12' height='8' fill='#d9d5c3' transform='rotate(-8 40 Y+13)'/>
<rect x='78' y='Y+8' width='10' height='11' fill='#cfccbd'/>
<rect x='35' y='Y+57' width='13' height='8' fill='#d9d5c3'/>
<rect x='75' y='Y+50' width='10' height='9' fill='#cfccbd' transform='rotate(6 80 Y+54)'/>
<rect x='54' y='Y+31' width='9' height='8' fill='#d9d5c3' opacity='.85'/>
<ellipse cx='86' cy='Y+63' rx='8' ry='6' fill='#8a6a1e' opacity='.12'/>
<circle cx='40' cy='Y+28' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='70' cy='Y+14' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='92' cy='Y+36' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='52' cy='Y+52' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='82' cy='Y+70' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='31' cy='Y+42' r='.7' fill='#6b6a60' opacity='.5'/></g>
```
(Y+n means the literal number Y plus n — compute the values. Every glyph
element stays strictly INSIDE the frame border — figcheck rejects a
figure without the `cam-terrain` group.) Draw your
receptive-field square and kernel-projection lines on top of it as
usual.
**Anchor the frozen endpoints identically in every figure** so all
experiments' figures line up when compared down the gallery page: the
camera-frame square starts at x=26 (its captions centered on x=53), and
the output is right-anchored — decode crosshair at x=812, the
(lat, lon, confidence) text block text-anchor=start at x=828. Mark the
anchors machine-checkably: the camera-frame rect carries
`id='frozen-input'`, the output text block carries `id='frozen-output'`.
Lay the elements between them out on a running x-cursor with generous
spacing so nothing overlaps.
**Readability contract** (figcheck verifies this geometrically): no
text may overlap other text, and no line (arrows, converge fans,
leader lines) may pass through a label — a line may END at a label,
never cross one. Rules distilled from review rounds: dashed leaders
run as orthogonal L-routes through empty lanes, never diagonally
across components; converge/vote lines originate at their true
sources (the actual dots or window cells, not generic positions);
every element owns its caption — adjacent columns stagger their
caption rows instead of colliding; prefer moving a label into empty
horizontal space over stacking it against a neighbor. Width is fixed
at 980, but **height is yours**: grow the viewBox (240–640) whenever
more vertical room makes the layout cleaner rather than cramming.
**Before finishing, validate:**
`.venv/bin/python -m autoresearch.figcheck` — it checks the anchor
contract on runs/pending_experiment.json; revise until it prints PASS.
Consecutive conv layers must be tied by kernel-projection lines (small
kernel square on one face, faint lines converging to a cell on the
next) so the encoder reads as one computation, not boxes in a row.
`archive/arch_svg_reference.py` is the concrete reference implementation
of this entire visual language — read it and reuse its helper geometry
for any stage that already has one.
`eli5` and `architecture` feed the human-facing gallery. `architecture.stages`
is the model's inference path left-to-right, camera frame → (lat, lon,
confidence) output — one box per stage, plain-language `detail`,
`"changed": true` ONLY on the stages this experiment touches. Reuse the
previous experiment's stage names verbatim wherever a stage is unchanged
(check `SELECT arch_json FROM experiments WHERE arch_json IS NOT NULL ORDER BY id DESC LIMIT 1;`) —
the diagrams are compared box-by-box across experiments. A change that only
affects training (loss, augmentation, schedule) keeps the inference stages
unchanged and adds one final stage with `"train_only": true` describing the
training signal.
3. **Write the implementation brief.** You do NOT edit `model/` yourself —
a separate implementation agent applies your design, seeing only the
current code plus what you pre-registered. Add one more field to
`runs/pending_experiment.json`:
```
"implementation_brief": "exact file-level instructions: which functions/
blocks in model/model.py and model/train.py change and how, what stays
untouched, and every contract to preserve"
```
Be precise enough that a competent engineer with no other context
implements it in one pass. Always restate the fixed contracts:
`train.py`'s CLI and the ONNX export contract in `model/model.py`'s
docstring — the frozen scorer depends on them.
## Hard rules
- Edit ONLY `runs/pending_experiment.json`. You never edit `model/` (the
implementation stage does) and files listed in `/FROZEN` are off-limits;
the harness hard-reverts any change to them.
- NEVER touch, read, or evaluate the `hamburg` holdout area (§5).
- ONE focused change per iteration — if you can't describe it in one sentence,
it's too big. Prefer architectural/procedural novelty over hyperparameter
nudges (§3): changing a learning rate is a weak experiment; changing the
coordinate parameterization, loss family, relighting-for-training, or model
topology is a strong one.
- Do not run training yourself; the harness does that.
- Stay within the deployment gates: exported ONNX ≤ 4 MiB per area, host
latency proxy ≤ 250 ms (see pipeline/score.py).What the six lighting conditions look like through the simulated low-light sensor (example patches — illustration, not the training set)Every frame in this project is rendered as a starlight-class low-light sensor (Sony STARVIS2 / IMX585 class — the airframe's chosen camera) would see it, not as a normal camera would. That is why the night renders look uncannily close to daylight: at high gain such a sensor recovers scene structure from moonlight, skyglow and artificial lighting, paying for it in noise, lifted shadows and washed-out color — exactly what these renders simulate. Below, one example 256 m patch per area under the six lighting conditions the model must handle. These illustrate the dataset, not this experiment's performance — the actual training set is thousands of distinct crops per area (see “training data” above), and these renderings only change when the relighting method changes. berlin frankfurt munich prignitz | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▸ | 33 | Sensor-sharpness nuisance randomization: per-crop PSF/resampling jitter breaks the blur-texture shortcut 0aaab0f5 | augmentation pivot | pretrained:mobilenet_v3_small IMAGENET1K_V1 features[0..8] (unchanged from champion) | gated fail | — | — | 59 m 18 s | $0.68 | gated fail | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The design under test — technical diagram — gray = frozen contract · ink = the current design · red = this experiment's change In plain words When a computer rotates a photo, the result comes out a tiny bit softer — like a photocopy of a photocopy. Nearly every training picture we generate goes through one such rotation, so they all share that same subtle softness, and the model quietly learned to lean on it as a crutch: when we handed it a perfectly sharp picture, its guesses got twice as bad — even though softness says nothing about WHERE the picture was taken. This round we make the crutch useless: during training, some pictures now arrive pin-sharp and others extra blurry, completely at random, so the only cue that reliably pays off is the actual arrangement of streets and buildings. It's like training a wine taster with the bottles wrapped in paper: hide the label and they have to learn the taste. Hypothesis The §6 metric is bound by ranking quality, not decode or operating point: the iter-12 probes on the exp-25 champion show the true cell is field top-1 on only 2% of eval crops while argmax, local-window, and exported decodes agree within 3% (runs/_probe_iter12_field.py), and exp 32's per-regime abstention fix — the operating-point attack — was just refuted. The same probe round measured an untested nuisance axis: eval crops that skip PIL bilinear rotation resampling (headings at exact 90° multiples) nearly DOUBLE the champion's median error, 983→1668 m on berlin midday (runs/_probe_iter12_heading.py), so the descriptor demonstrably rides on low-level resampling-blur statistics. That band is identical everywhere on the map — it carries zero location information — yet it varies with heading, so every unit of descriptor sensitivity spent on it both jitters the place code across eval headings and steals encoder capacity from layout, which is exactly the ranking signal that must improve. Exp 17 already validated the mechanism (randomize a train-time nuisance → only nuisance-stable structure stays discriminative between locations) and the training-data family is 4-for-4 kept (exps 7, 17, 20, 25) while the last seven architecture/operating-point experiments (26–32) all reverted. Per-crop randomization of the imaging chain's sharpness — from crisper-than-eval to softer-than-eval — should force the encoder off the blur band and onto ground layout, improving field ranking in every area×lighting cell. Not built on a library entry; built on the iter-12 probe forensics. Method ONE change, entirely inside model/train.py's sample_epoch (the per-epoch fresh-draw sampler): each training crop rolls a random sharpness before entering the tensor. (a) With p=0.25 its heading is snapped to 0° so extract_crop skips rotation resampling entirely and the crop arrives pin-sharp — the model's C4 rotation fan then exposes all four crisp orientations for free. (b) Independently, with p=0.5, the extracted 128×128 uint8 crop gets an extra PIL GaussianBlur with σ ~ U(0.4, 1.4) px. The eval-matched statistic (continuous-heading resample, no extra blur) remains the single most common case (~37.5%), and calibrate_conf_shift stays exactly eval-matched (no jitter there). Model, losses, decode, calibration, EPOCH_MULT schedule, and the frozen pipeline are untouched. Expected outcome If the blur shortcut is taxing ranking, the §6 worst-case median (champion 797.91 m, binding cell frankfurt/evening) drops ~5–15% to ≈680–760 m, with per-cell coverage roughly unchanged (calibration path untouched); confirming signature: the crisp-crop pathology (983→1668 m on resample-free crops) collapses to near parity. If sharpness invariance only makes the fit harder without transferring (exp-6 failure mode), expect ≈800–850 m and revert. Result primary worst-case median error = 1000000000.0 m (previous best 797.91 m); full per-area/bucket breakdown in metrics.json Conclusion REVERTED — metric did not improve (1000000000.0 m vs best 797.91 m); change discarded Scoreboard — median error (m) per area × lighting
the worst cell (underlined) is the experiment's score;
red = failed cell, ink = at target
No per-area results recorded for this run. Fits the aircraft? all deployment gates passed
largest per-area model — (limit 4,096 KB) · single-frame inference — on one CPU thread (proxy limit 250 ms) Where this iteration's time went: design (Haiku) 13 m 19 s · implement (Sonnet) 47 s · train 4 areas 44 m 09 s · samples 42 s · whole iteration 59 m 23 s
ts 2026-07-22T08:33:30 · commit 0aaab0f53186 ·
parent 0aaab0f53186 · artifacts runs/20260722_073412_iter6 ·
agent model — · took 59 m 18 s
The exact prompt given to the headless agent## PATIENCE SPENT — THIS ITERATION MUST PIVOT
7 consecutive experiments have failed to beat the current best.
Do NOT refine the champion's current mechanism again this round. Step back
and propose from a design family ABSENT from the last 7 experiments
— the spec (§3) explicitly invites, among others: dispatcher + lighting-
condition-specialist models, learned relighting, a different coordinate
parameterization, quantization-aware capacity changes, or training-data
strategy overhauls. Genuine novelty over incremental tuning.
# Autoresearch iteration — one focused experiment
You are one iteration of an autonomous research loop for UAV low-light
geolocalization. Read `CLAUDE.md` (§1, §3, §6 especially) for full context.
The harness (loop.sh) will train, score, log, and keep/revert AFTER you exit —
you only design the experiment and edit the code.
## Your job, in order
0. **Skim the library (optional input).** `autoresearch/library.md` holds
the human researcher's inspiration notes. They do not fix your answer
and you are free to ignore them — pick an entry up only when it
genuinely fits your read of the history, and if you build on one, say
so in your hypothesis.
1. **Review the research history.** Query the lineage DB:
`sqlite3 experiments.sqlite "SELECT id, title, category, hypothesis, expected_outcome, result, conclusion, primary_metric, kept FROM experiments ORDER BY id DESC LIMIT 15;"`
Note which hypotheses were supported/refuted. Do not repeat a refuted
experiment without a materially new angle.
**Plateau rule:** if three or more consecutive experiments were reverted,
do not attempt another variation of the last refuted mechanism. Either
pick a design family absent from the history (dispatcher + lighting
specialists, pretrained init, learned relighting, training-scale, …) or
attack the bottleneck the refuted hypotheses jointly point at.
2. **Design ONE focused experiment** — proper experiment design, pre-registered
before you touch code. Write it to `runs/pending_experiment.json`:
```json
{
"title": "one-line name",
"category": "architecture|loss|augmentation|relighting|training|quantization|other",
"hypothesis": "what you believe is limiting the metric and why this change addresses it",
"method": "the ONE focused change, concretely (files, mechanism)",
"expected_outcome": "predicted effect on the §6 worst-case median error, quantified if possible",
"init_strategy": "from-scratch | pretrained:<name>",
"eli5": "2-4 sentences for a smart non-ML reader: what you changed and why it might help, in everyday language — analogies welcome, zero jargon",
"architecture": {"stages": [
{"name": "Camera frame", "detail": "128×128 px crop, one of 6 lighting renders", "changed": false},
{"name": "Feature extractor", "detail": "plain-language description", "changed": false},
{"name": "…", "detail": "…", "changed": true}
]},
"architecture_svg": "<svg viewBox='0 0 980 300' …>…</svg>"
}
```
`architecture_svg` is the **technical architecture diagram** of the model
you are testing — a proper ML-paper figure of the design, drawn by you,
shown at the top of this experiment's gallery entry. **Draw tensors and
operations as the things they are — NOT as labeled boxes:** the input
image as a pixel-textured square; conv feature maps as pseudo-3D slabs
that shrink spatially and deepen in channels; 1-D vectors as thin
vertical tick-bars; fully-connected layers as fans of thin crossing
lines between bars; spatial fields/grids as actually-drawn grids with
shaded cells; decodes/aggregations as thin lines converging from cells
onto a point; samplers/targets as small pictorial glyphs. Captions sit
UNDER each element (name ≤10.5px weight 600, sub-note ≤9.5px); tensor
shapes annotate the flow in small italics (128×128×3 → 8²×128 → 1024).
Losses/targets/samplers live in a lane below the inference path,
annotated with slanted dashed leader lines to small text — no boxes
around them. Draw the two FROZEN endpoints — the camera-frame input and
the (lat, lon, confidence) output — entirely in #9b998c with a small
italic “frozen contract” tag: they are fixed by the harness and outside
your search space. Everything between them is your design: draw it in ink,
with red only on what THIS experiment changes. Style contract, so all experiments' figures read as one
paper: viewBox width 980 (height ~300–360); transparent background; ink
#111111, captions #6b6a60, annotations/arrows #9b998c; **#8c2f1f (red)
reserved for exactly what this experiment changed**; training-only
elements in #8a6a1e (or red if changed); font-family
Palatino,Georgia,serif; stroke-width ≈1.2 elements, 1 arrows; fills only
faint tints (opacity ≤ .12); no gradients, no icons, no emoji. Inference
flows left → right from camera frame to (lat, lon, confidence).
**The camera frame is a fixed glyph — copy it verbatim.** It must look
like an actual nadir frame of terrain (streets, building footprints),
not abstract pixels, and be identical in every figure. Use exactly this
snippet, substituting Y for the frame's top edge (pick Y so the frame
centers on your inference lane; captions go under it as usual — note
128²×3 is the model input crop, ~1 m/px, not the sensor's native
resolution):
```svg
<g id='cam-terrain'><rect id='frozen-input' x='26' y='Y' width='76'
height='76' fill='#f6f4ea' stroke='#9b998c' stroke-width='1.6'/>
<path d='M31 Y+47 L97 Y+27' stroke='#e6e3d4' stroke-width='5' fill='none'/>
<path d='M59 Y+6 L49 Y+70' stroke='#e6e3d4' stroke-width='3.5' fill='none'/>
<rect x='34' y='Y+9' width='12' height='8' fill='#d9d5c3' transform='rotate(-8 40 Y+13)'/>
<rect x='78' y='Y+8' width='10' height='11' fill='#cfccbd'/>
<rect x='35' y='Y+57' width='13' height='8' fill='#d9d5c3'/>
<rect x='75' y='Y+50' width='10' height='9' fill='#cfccbd' transform='rotate(6 80 Y+54)'/>
<rect x='54' y='Y+31' width='9' height='8' fill='#d9d5c3' opacity='.85'/>
<ellipse cx='86' cy='Y+63' rx='8' ry='6' fill='#8a6a1e' opacity='.12'/>
<circle cx='40' cy='Y+28' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='70' cy='Y+14' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='92' cy='Y+36' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='52' cy='Y+52' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='82' cy='Y+70' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='31' cy='Y+42' r='.7' fill='#6b6a60' opacity='.5'/></g>
```
(Y+n means the literal number Y plus n — compute the values. Every glyph
element stays strictly INSIDE the frame border — figcheck rejects a
figure without the `cam-terrain` group.) Draw your
receptive-field square and kernel-projection lines on top of it as
usual.
**Anchor the frozen endpoints identically in every figure** so all
experiments' figures line up when compared down the gallery page: the
camera-frame square starts at x=26 (its captions centered on x=53), and
the output is right-anchored — decode crosshair at x=812, the
(lat, lon, confidence) text block text-anchor=start at x=828. Mark the
anchors machine-checkably: the camera-frame rect carries
`id='frozen-input'`, the output text block carries `id='frozen-output'`.
Lay the elements between them out on a running x-cursor with generous
spacing so nothing overlaps.
**Readability contract** (figcheck verifies this geometrically): no
text may overlap other text, and no line (arrows, converge fans,
leader lines) may pass through a label — a line may END at a label,
never cross one. Rules distilled from review rounds: dashed leaders
run as orthogonal L-routes through empty lanes, never diagonally
across components; converge/vote lines originate at their true
sources (the actual dots or window cells, not generic positions);
every element owns its caption — adjacent columns stagger their
caption rows instead of colliding; prefer moving a label into empty
horizontal space over stacking it against a neighbor. Width is fixed
at 980, but **height is yours**: grow the viewBox (240–640) whenever
more vertical room makes the layout cleaner rather than cramming.
**Before finishing, validate:**
`.venv/bin/python -m autoresearch.figcheck` — it checks the anchor
contract on runs/pending_experiment.json; revise until it prints PASS.
Consecutive conv layers must be tied by kernel-projection lines (small
kernel square on one face, faint lines converging to a cell on the
next) so the encoder reads as one computation, not boxes in a row.
`archive/arch_svg_reference.py` is the concrete reference implementation
of this entire visual language — read it and reuse its helper geometry
for any stage that already has one.
`eli5` and `architecture` feed the human-facing gallery. `architecture.stages`
is the model's inference path left-to-right, camera frame → (lat, lon,
confidence) output — one box per stage, plain-language `detail`,
`"changed": true` ONLY on the stages this experiment touches. Reuse the
previous experiment's stage names verbatim wherever a stage is unchanged
(check `SELECT arch_json FROM experiments WHERE arch_json IS NOT NULL ORDER BY id DESC LIMIT 1;`) —
the diagrams are compared box-by-box across experiments. A change that only
affects training (loss, augmentation, schedule) keeps the inference stages
unchanged and adds one final stage with `"train_only": true` describing the
training signal.
3. **Write the implementation brief.** You do NOT edit `model/` yourself —
a separate implementation agent applies your design, seeing only the
current code plus what you pre-registered. Add one more field to
`runs/pending_experiment.json`:
```
"implementation_brief": "exact file-level instructions: which functions/
blocks in model/model.py and model/train.py change and how, what stays
untouched, and every contract to preserve"
```
Be precise enough that a competent engineer with no other context
implements it in one pass. Always restate the fixed contracts:
`train.py`'s CLI and the ONNX export contract in `model/model.py`'s
docstring — the frozen scorer depends on them.
## Hard rules
- Edit ONLY `runs/pending_experiment.json`. You never edit `model/` (the
implementation stage does) and files listed in `/FROZEN` are off-limits;
the harness hard-reverts any change to them.
- NEVER touch, read, or evaluate the `hamburg` holdout area (§5).
- ONE focused change per iteration — if you can't describe it in one sentence,
it's too big. Prefer architectural/procedural novelty over hyperparameter
nudges (§3): changing a learning rate is a weak experiment; changing the
coordinate parameterization, loss family, relighting-for-training, or model
topology is a strong one.
- Do not run training yourself; the harness does that.
- Stay within the deployment gates: exported ONNX ≤ 4 MiB per area, host
latency proxy ≤ 250 ms (see pipeline/score.py).What the six lighting conditions look like through the simulated low-light sensor (example patches — illustration, not the training set)Every frame in this project is rendered as a starlight-class low-light sensor (Sony STARVIS2 / IMX585 class — the airframe's chosen camera) would see it, not as a normal camera would. That is why the night renders look uncannily close to daylight: at high gain such a sensor recovers scene structure from moonlight, skyglow and artificial lighting, paying for it in noise, lifted shadows and washed-out color — exactly what these renders simulate. Below, one example 256 m patch per area under the six lighting conditions the model must handle. These illustrate the dataset, not this experiment's performance — the actual training set is thousands of distinct crops per area (see “training data” above), and these renderings only change when the relighting method changes. berlin frankfurt munich prignitz | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▸ | 32 | Risk-controlled abstention: six per-lighting-regime thresholds calibrated on never-trained fenced blocks 4871a8c7 | training pivot | pretrained:mobilenet_v3_small IMAGENET1K_V1 features[0..8] (torchvision, BSD-3) — unchanged | 1.08 km | 3,226 KB | 2.7 ms | 108 m 15 s | $1.24 | discarded | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The design under test — technical diagram — gray = frozen contract · ink = the current design · red = this experiment's change One real test, end to end
→
1.83 km
miss, this crop 0.80
self-reported confidence In plain words The model already grades its own guesses and is allowed to answer 'no fix' on frames it can't place — like a student who may skip questions on a test that only grades the questions they answer, as long as they attempt at least 1 in 5. Until now the skip rule was one global bar, tuned on map areas the model had already flown over in training — so how picky it actually was at night versus midday, in Berlin versus Frankfurt, was essentially accidental. We now rope off a slice of the map the model never trains on, measure there — separately for each lighting condition — how accuracy trades against how often it answers, and bake six lighting-specific bars into the model (it senses the lighting from the frame's brightness). A telling detail from the forensics: the last four 'failed' experiments were actually answering MORE questions than the champion at similar accuracy — they lost on a technicality of where their bar happened to land, which this change fixes for every future round too. Hypothesis The §6 scalar is currently bound not by representation quality but by the selective-prediction operating point, which is uncontrolled per lighting regime and mis-calibrated by construction. The primary metric is the median over KEPT crops with a 0.2 coverage floor, yet the abstention bar is a single global conf_shift chosen as the MINIMUM of six per-bucket thresholds, calibrated on TRAIN-VISITED ground — so each (area, bucket) cell's eval keep rate is an accident of distribution shift (champion: 0.25–0.67 across cells). Forensics on the six-experiment plateau: exps 28–31 matched or beat the champion cell-by-cell while keeping far more crops — exp 28 was better or equal in 17 of 24 cells at coverage 0.52–0.79 vs the champion's 0.42–0.67 (frankfurt evening 649.5 m @ 0.58 vs 797.9 m @ 0.46) and was reverted on ONE cell (berlin night 1057.7 m @ 0.445 vs 632.1 m @ 0.3125); exp 31 kept 30–60% more crops everywhere (0.60–0.87) and landed within 6%. The keep/revert record since exp 26 has therefore mostly measured where each candidate's threshold happened to land, not which candidate localizes better. Two fresh read-only probes justify attacking the confidence path INSTEAD of the representation this round: (a) _probe_iter12_field.py shows the ranking failure is deep and decode-independent (true cell in top-1 only 2%, argmax/local-window/export decodes within 3% of each other, oracle mode selection still ~500 m) — no cheap representation fix is on the table; (b) _probe_iter12_riskcov.py shows the confidence signal RANKS errors in every cell (Spearman conf-vs-error −0.21 to −0.42) and quantifies the exploitable headroom: with each cell run at its own best keep rate the worst §6 cell falls from 797.9 m to 731.1 m oracle (binding cell moves to frankfurt/night; frankfurt evening itself drops 798→~562-590 m, and 20 of 24 cells improve). The same probe also explains exp 28's veto mechanistically: its one global threshold kept 44.5% at berlin night (1057.7 m), while a per-regime risk-optimal pick would have run that cell at ~0.26-0.32 keep (~650-750 m). Making the operating point a designed, per-regime, distribution-matched quantity should both lower the worst cell directly and remove the measurement distortion that has been vetoing genuine representation gains. Method One mechanism changes — the path from confidence score to abstention decision; trunk, heads, gate, field, decode, losses, sampler mechanics, and schedule stay byte-identical. (1) model/model.py: the scalar conf_shift buffer becomes a 6-vector paired with a new conf_lum_anchors buffer; forward() piecewise-linearly interpolates the shift over the frame's own mean brightness (the lum feature the lighting gate already computes), so the exported graph infers the lighting regime from the frame alone: conf = sigmoid(z − shift(lum)). (2) model/train.py: ~10% of train-split blocks (stable-hash selected, bbox-generic) are FENCED — excluded from every training draw with the same window-halo rule the frozen split applies to eval blocks — so their terrain is honestly unseen and per-crop errors are measurable there. (3) Calibration (replaces calibrate_conf_shift): per lighting bucket, run the trained model on fenced crops at fresh random headings, trace the risk–coverage curve (median kept error vs keep rate), pick the keep rate q* in [0.26, 0.58] that minimizes the fenced kept-median (floor 0.26 is safe against the frozen 0.2 coverage gate because exp 28 measured the fenced-to-eval keep transfer as upward-biased, +0.00 to +0.11 — unlike today's train-visited calibration, which transfers DOWN by up to 0.15), and bake the six (median bucket luminance, threshold) pairs into the exported buffers. Expected outcome Re-scoring the champion's own predictions at per-cell-optimal keep rates (read-only probe, oracle thresholds) bounds the gain: worst §6 cell 731.1 m vs 797.91 m shipped (−8.4%), with the binding cell moving from frankfurt/evening to frankfurt/night — night cells are where the risk–coverage curves flatten, so gating cannot fix them, only stop mismeasuring them. The deployed version picks thresholds on fenced blocks rather than on the eval set, so I predict the primary lands at ~745–795 m (0.5–7% improvement, point estimate ~765 m), with per-cell coverage settling in ~0.25–0.45 (coverage is reported, not optimized, per §6; operationally the UAV just uses the next frame). The equally important, pre-registered secondary claim: with operating points controlled, per-cell medians become comparable across experiments at matched coverage — the instrument repair that lets exp-28-style representation gains score. Refutation modes: conf misranks on honestly unseen fenced ground (fenced q* fails to transfer, worst cell unchanged), or a cell's eval coverage slips under 0.2 (scored 1e9) — the 0.26 floor is guarded by exp 28's measured upward-biased fenced-to-eval transfer. Result primary worst-case median error = 1078.31 m (previous best 797.91 m); full per-area/bucket breakdown in metrics.json Conclusion REVERTED — metric did not improve (1078.31 m vs best 797.91 m); change discarded Scoreboard — median error (m) per area × lighting
the worst cell (underlined) is the experiment's score;
red = failed cell, ink = at target
Fits the aircraft? all deployment gates passed
largest per-area model 3,226 KB (limit 4,096 KB) · single-frame inference 2.7 ms on one CPU thread (proxy limit 250 ms)
crops are sampled fresh each run from the frozen train split (~45,000 distinct positions per area, times random rotation), never from eval blocks; how many to use is the experiment's own choice
Where this iteration's time went: design (Haiku) 46 m 22 s · implement (Sonnet) 12 m 02 s · train 4 areas 47 m 13 s · score 1 m 34 s · samples 42 s · publish 5 s · whole iteration 108 m 25 s
ts 2026-07-22T07:33:54 · commit 4871a8c721c1 ·
parent 4871a8c721c1 · artifacts runs/20260722_054539_iter5 ·
agent model — · took 108 m 15 s
The exact prompt given to the headless agent## PATIENCE SPENT — THIS ITERATION MUST PIVOT
6 consecutive experiments have failed to beat the current best.
Do NOT refine the champion's current mechanism again this round. Step back
and propose from a design family ABSENT from the last 6 experiments
— the spec (§3) explicitly invites, among others: dispatcher + lighting-
condition-specialist models, learned relighting, a different coordinate
parameterization, quantization-aware capacity changes, or training-data
strategy overhauls. Genuine novelty over incremental tuning.
# Autoresearch iteration — one focused experiment
You are one iteration of an autonomous research loop for UAV low-light
geolocalization. Read `CLAUDE.md` (§1, §3, §6 especially) for full context.
The harness (loop.sh) will train, score, log, and keep/revert AFTER you exit —
you only design the experiment and edit the code.
## Your job, in order
0. **Skim the library (optional input).** `autoresearch/library.md` holds
the human researcher's inspiration notes. They do not fix your answer
and you are free to ignore them — pick an entry up only when it
genuinely fits your read of the history, and if you build on one, say
so in your hypothesis.
1. **Review the research history.** Query the lineage DB:
`sqlite3 experiments.sqlite "SELECT id, title, category, hypothesis, expected_outcome, result, conclusion, primary_metric, kept FROM experiments ORDER BY id DESC LIMIT 15;"`
Note which hypotheses were supported/refuted. Do not repeat a refuted
experiment without a materially new angle.
**Plateau rule:** if three or more consecutive experiments were reverted,
do not attempt another variation of the last refuted mechanism. Either
pick a design family absent from the history (dispatcher + lighting
specialists, pretrained init, learned relighting, training-scale, …) or
attack the bottleneck the refuted hypotheses jointly point at.
2. **Design ONE focused experiment** — proper experiment design, pre-registered
before you touch code. Write it to `runs/pending_experiment.json`:
```json
{
"title": "one-line name",
"category": "architecture|loss|augmentation|relighting|training|quantization|other",
"hypothesis": "what you believe is limiting the metric and why this change addresses it",
"method": "the ONE focused change, concretely (files, mechanism)",
"expected_outcome": "predicted effect on the §6 worst-case median error, quantified if possible",
"init_strategy": "from-scratch | pretrained:<name>",
"eli5": "2-4 sentences for a smart non-ML reader: what you changed and why it might help, in everyday language — analogies welcome, zero jargon",
"architecture": {"stages": [
{"name": "Camera frame", "detail": "128×128 px crop, one of 6 lighting renders", "changed": false},
{"name": "Feature extractor", "detail": "plain-language description", "changed": false},
{"name": "…", "detail": "…", "changed": true}
]},
"architecture_svg": "<svg viewBox='0 0 980 300' …>…</svg>"
}
```
`architecture_svg` is the **technical architecture diagram** of the model
you are testing — a proper ML-paper figure of the design, drawn by you,
shown at the top of this experiment's gallery entry. **Draw tensors and
operations as the things they are — NOT as labeled boxes:** the input
image as a pixel-textured square; conv feature maps as pseudo-3D slabs
that shrink spatially and deepen in channels; 1-D vectors as thin
vertical tick-bars; fully-connected layers as fans of thin crossing
lines between bars; spatial fields/grids as actually-drawn grids with
shaded cells; decodes/aggregations as thin lines converging from cells
onto a point; samplers/targets as small pictorial glyphs. Captions sit
UNDER each element (name ≤10.5px weight 600, sub-note ≤9.5px); tensor
shapes annotate the flow in small italics (128×128×3 → 8²×128 → 1024).
Losses/targets/samplers live in a lane below the inference path,
annotated with slanted dashed leader lines to small text — no boxes
around them. Draw the two FROZEN endpoints — the camera-frame input and
the (lat, lon, confidence) output — entirely in #9b998c with a small
italic “frozen contract” tag: they are fixed by the harness and outside
your search space. Everything between them is your design: draw it in ink,
with red only on what THIS experiment changes. Style contract, so all experiments' figures read as one
paper: viewBox width 980 (height ~300–360); transparent background; ink
#111111, captions #6b6a60, annotations/arrows #9b998c; **#8c2f1f (red)
reserved for exactly what this experiment changed**; training-only
elements in #8a6a1e (or red if changed); font-family
Palatino,Georgia,serif; stroke-width ≈1.2 elements, 1 arrows; fills only
faint tints (opacity ≤ .12); no gradients, no icons, no emoji. Inference
flows left → right from camera frame to (lat, lon, confidence).
**The camera frame is a fixed glyph — copy it verbatim.** It must look
like an actual nadir frame of terrain (streets, building footprints),
not abstract pixels, and be identical in every figure. Use exactly this
snippet, substituting Y for the frame's top edge (pick Y so the frame
centers on your inference lane; captions go under it as usual — note
128²×3 is the model input crop, ~1 m/px, not the sensor's native
resolution):
```svg
<g id='cam-terrain'><rect id='frozen-input' x='26' y='Y' width='76'
height='76' fill='#f6f4ea' stroke='#9b998c' stroke-width='1.6'/>
<path d='M31 Y+47 L97 Y+27' stroke='#e6e3d4' stroke-width='5' fill='none'/>
<path d='M59 Y+6 L49 Y+70' stroke='#e6e3d4' stroke-width='3.5' fill='none'/>
<rect x='34' y='Y+9' width='12' height='8' fill='#d9d5c3' transform='rotate(-8 40 Y+13)'/>
<rect x='78' y='Y+8' width='10' height='11' fill='#cfccbd'/>
<rect x='35' y='Y+57' width='13' height='8' fill='#d9d5c3'/>
<rect x='75' y='Y+50' width='10' height='9' fill='#cfccbd' transform='rotate(6 80 Y+54)'/>
<rect x='54' y='Y+31' width='9' height='8' fill='#d9d5c3' opacity='.85'/>
<ellipse cx='86' cy='Y+63' rx='8' ry='6' fill='#8a6a1e' opacity='.12'/>
<circle cx='40' cy='Y+28' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='70' cy='Y+14' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='92' cy='Y+36' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='52' cy='Y+52' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='82' cy='Y+70' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='31' cy='Y+42' r='.7' fill='#6b6a60' opacity='.5'/></g>
```
(Y+n means the literal number Y plus n — compute the values. Every glyph
element stays strictly INSIDE the frame border — figcheck rejects a
figure without the `cam-terrain` group.) Draw your
receptive-field square and kernel-projection lines on top of it as
usual.
**Anchor the frozen endpoints identically in every figure** so all
experiments' figures line up when compared down the gallery page: the
camera-frame square starts at x=26 (its captions centered on x=53), and
the output is right-anchored — decode crosshair at x=812, the
(lat, lon, confidence) text block text-anchor=start at x=828. Mark the
anchors machine-checkably: the camera-frame rect carries
`id='frozen-input'`, the output text block carries `id='frozen-output'`.
Lay the elements between them out on a running x-cursor with generous
spacing so nothing overlaps.
**Readability contract** (figcheck verifies this geometrically): no
text may overlap other text, and no line (arrows, converge fans,
leader lines) may pass through a label — a line may END at a label,
never cross one. Rules distilled from review rounds: dashed leaders
run as orthogonal L-routes through empty lanes, never diagonally
across components; converge/vote lines originate at their true
sources (the actual dots or window cells, not generic positions);
every element owns its caption — adjacent columns stagger their
caption rows instead of colliding; prefer moving a label into empty
horizontal space over stacking it against a neighbor. Width is fixed
at 980, but **height is yours**: grow the viewBox (240–640) whenever
more vertical room makes the layout cleaner rather than cramming.
**Before finishing, validate:**
`.venv/bin/python -m autoresearch.figcheck` — it checks the anchor
contract on runs/pending_experiment.json; revise until it prints PASS.
Consecutive conv layers must be tied by kernel-projection lines (small
kernel square on one face, faint lines converging to a cell on the
next) so the encoder reads as one computation, not boxes in a row.
`archive/arch_svg_reference.py` is the concrete reference implementation
of this entire visual language — read it and reuse its helper geometry
for any stage that already has one.
`eli5` and `architecture` feed the human-facing gallery. `architecture.stages`
is the model's inference path left-to-right, camera frame → (lat, lon,
confidence) output — one box per stage, plain-language `detail`,
`"changed": true` ONLY on the stages this experiment touches. Reuse the
previous experiment's stage names verbatim wherever a stage is unchanged
(check `SELECT arch_json FROM experiments WHERE arch_json IS NOT NULL ORDER BY id DESC LIMIT 1;`) —
the diagrams are compared box-by-box across experiments. A change that only
affects training (loss, augmentation, schedule) keeps the inference stages
unchanged and adds one final stage with `"train_only": true` describing the
training signal.
3. **Write the implementation brief.** You do NOT edit `model/` yourself —
a separate implementation agent applies your design, seeing only the
current code plus what you pre-registered. Add one more field to
`runs/pending_experiment.json`:
```
"implementation_brief": "exact file-level instructions: which functions/
blocks in model/model.py and model/train.py change and how, what stays
untouched, and every contract to preserve"
```
Be precise enough that a competent engineer with no other context
implements it in one pass. Always restate the fixed contracts:
`train.py`'s CLI and the ONNX export contract in `model/model.py`'s
docstring — the frozen scorer depends on them.
## Hard rules
- Edit ONLY `runs/pending_experiment.json`. You never edit `model/` (the
implementation stage does) and files listed in `/FROZEN` are off-limits;
the harness hard-reverts any change to them.
- NEVER touch, read, or evaluate the `hamburg` holdout area (§5).
- ONE focused change per iteration — if you can't describe it in one sentence,
it's too big. Prefer architectural/procedural novelty over hyperparameter
nudges (§3): changing a learning rate is a weak experiment; changing the
coordinate parameterization, loss family, relighting-for-training, or model
topology is a strong one.
- Do not run training yourself; the harness does that.
- Stay within the deployment gates: exported ONNX ≤ 4 MiB per area, host
latency proxy ≤ 250 ms (see pipeline/score.py).Where the model was tested — and how far off it was Each map is one full test area. Every dot is one held-out test location (all six lighting conditions overlaid): the model was shown a 128 m crop centered there and asked for its position. Dot color = the distance between its answer and the truth — green ≤ 20 m (at goal), amber ≤ 50 m, red beyond. A working model turns these maps green; spatial clusters of red reveal which parts of an area confuse it.     What the six lighting conditions look like through the simulated low-light sensor (example patches — illustration, not the training set)Every frame in this project is rendered as a starlight-class low-light sensor (Sony STARVIS2 / IMX585 class — the airframe's chosen camera) would see it, not as a normal camera would. That is why the night renders look uncannily close to daylight: at high gain such a sensor recovers scene structure from moonlight, skyglow and artificial lighting, paying for it in noise, lifted shadows and washed-out color — exactly what these renders simulate. Below, one example 256 m patch per area under the six lighting conditions the model must handle. These illustrate the dataset, not this experiment's performance — the actual training set is thousands of distinct crops per area (see “training data” above), and these renderings only change when the relighting method changes. berlin frankfurt munich prignitz | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▸ | 31 | Neural-map correlation field: localization becomes sliding-window matching of a crop fingerprint against a learned 32 m map raster 1ee201ff | architecture pivot | pretrained:mobilenet_v3_small IMAGENET1K_V1 features[0..8] (trunk unchanged); neural map + kernel heads from scratch | 846.1 m | 3,005 KB | 5.7 ms | 73 m 46 s | $0.85 | discarded | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The design under test — technical diagram — gray = frozen contract · ink = the current design · red = this experiment's change One real test, end to end
→
207.0 m
miss, this crop 0.18
self-reported confidence In plain words Until now the model answered 'where am I?' by comparing one overall impression of the camera frame against about a thousand memorized district flashcards -- and in a city full of lookalike districts, overall impressions collide. Now the model instead paints its own miniature map of the whole area into its weights while training. In flight it turns the camera frame into a small 4x4 fingerprint tile and slides that tile across its painted map to find where the pattern truly lines up, the way you find where a puzzle piece belongs. Because the whole arrangement inside the fingerprint has to line up -- not just the general vibe -- two neighborhoods that merely feel similar no longer fool it, and the answer grid becomes about seven times finer in each direction. Hypothesis The last five reverted experiments (26-30: 3x trunk capacity, coverage calibration, stride-8 readout tap, lighting-specialist dispatch) all attacked what feeds or routes around the SAME output parameterization -- a flat table of 1,024 independent ~217 m cell templates scored by dot product against one holistic 560-d crop vector, unchanged since exp 2 -- and were jointly refuted. Per the plateau rule this iteration replaces that parameterization itself, the one design family CLAUDE.md §3 invites ('a different coordinate parameterization') that the history has never touched. Three signatures say the template table is the binding constraint: (1) since exps 20/25 the converged model's train-split and eval-split medians are EQUAL (~600-800 m) -- it cannot even fit visited locations, the signature of a statistically weak output parameterization, not of missing capacity (exps 26/28 proved bigger trunks don't score) or missing optimization (exp 25); (2) rural prignitz, with distinctive macro-structure, beats all three self-similar metros in every bucket -- lookalike-cell confusion, which holistic-vector template scoring cannot resolve because it enforces no spatial-arrangement consistency between crop and cell; (3) mean/median ~= 2 everywhere -- a heavy tail of long-range aliasing errors, the same confusion. The change (OrienterNet-family matching, encoded entirely in weights -- no reference imagery ships): delete the 1,024-template tables and learn a NEURAL MAP, an 8-channel raster over the area at ~32 m/cell (dimensions derived from meta.json width/height -- no area-specific constants), scoring every raster position by cross-correlating a crop-derived 4x4x8 feature kernel whose taps each summarize 32 m of ground, matching the raster's cell size. One mechanism addresses all three signatures: (a) a position scores high only if the crop's whole 4x4 spatial arrangement matches the map there, so lookalike districts differing in sub-crop layout are suppressed geometrically; (b) the scoring function is shared across all ~45k positions (one matching kernel + one map) instead of 1,024 separately-learned discriminants, and overlapping training crops write coherently into the SAME map cells at their true offsets, multiplying effective supervision per output parameter; (c) the decode grid drops from 217 m to ~32 m pitch. The kept exp-12 bright/dark expert pair and gate survive as two kernel generators (layout-aware conv vs GAP-only linear) blended before a single correlation -- correlation is linear in the kernel, so blending kernels equals blending fields; the C4 vote (exp 16), beta-sharpened decode (exp 14), shape-feature confidence + calibration (exp 15), fresh-draw sampler (exp 20), 3-realization renders (exp 17) and 24-epoch cosine schedule (exp 25) are all byte-identical, and supervision smoothness is preserved exactly (Gaussian-CE sigma stays 1.5/32 in normalized map units). Method model/model.py: remove loc_logits (560->1024), dark_logits (48->1024) and the GRID_K template machinery. Add per-area parameters map_feat [8, mh, mw] (mh=round(height/32), mw=round(width/32), init 0.01*randn) and map_bias [(mh-3)*(mw-3)] (init 0). The trunk's 8x8x48 fmap is projected by a 1x1 conv (48->8) and 2x2 avg-pooled into a 4x4x8 'bright' kernel; a Linear(48->128) forms a 'dark' kernel from the GAP descriptor; the existing lum+GAP gate blends the two KERNELS per example; logits4 = F.conv2d(map_feat.unsqueeze(0), blended_kernels).squeeze(0).flatten(1) + map_bias -- valid cross-correlation, one logit per raster position, C4-averaged exactly as before. Cell centers: u=(j+2)/mw, v=(i+2)/mh, row-major. Decode, confidence features, and loss keep their exact form with TARGET_SIGMA_UV = 1.5/32. model/train.py: build_model(meta) passes area dims; loss_fn additionally receives the model's cell-center buffers. Sampler, calibration, schedule, CLI, export contract untouched. Expected outcome If arrangement-enforcing matching plus map-space parameter sharing is the fix the last five refutations jointly point at, worst-case median drops 10-25% below the 797.91 m champion (predict 600-720 m), with metros gaining more than prignitz (lookalike suppression) and mean/median falling toward 1.5 (fewer long-range aliases); the finer 32 m grid alone is worth little at current error levels. Confidence machinery is untouched, so per-bucket coverage should stay near the champion's 0.31-0.52 profile -- a median gain achieved at collapsed coverage would not count (coverage-median coupling). Named failure mode: after the C4 vote a residual heading mismatch up to +/-45 deg misaligns outer kernel taps by ~1 map cell; if learned features cannot absorb that, correlation peaks smear, fields flatten, and errors move toward the ~3.2 km center-guess floor with coverage collapse. Deployment: export ~2.8-3.0 MB fp32 (< 4 MiB gate, no int8 needed), host latency estimated < 40 ms (< 250 ms gate). Result primary worst-case median error = 846.09 m (previous best 797.91 m); full per-area/bucket breakdown in metrics.json Conclusion REVERTED — metric did not improve (846.09 m vs best 797.91 m); change discarded Scoreboard — median error (m) per area × lighting
the worst cell (underlined) is the experiment's score;
red = failed cell, ink = at target
Fits the aircraft? all deployment gates passed
largest per-area model 3,005 KB (limit 4,096 KB) · single-frame inference 5.7 ms on one CPU thread (proxy limit 250 ms)
crops are sampled fresh each run from the frozen train split (~45,000 distinct positions per area, times random rotation), never from eval blocks; how many to use is the experiment's own choice
Where this iteration's time went: design (Haiku) 20 m 26 s · implement (Sonnet) 3 m 42 s · train 4 areas 46 m 33 s · score 2 m 01 s · samples 42 s · publish 5 s · whole iteration 73 m 56 s
ts 2026-07-22T05:45:21 · commit 1ee201ffc513 ·
parent 1ee201ffc513 · artifacts runs/20260722_043135_iter4 ·
agent model — · took 73 m 46 s
The exact prompt given to the headless agent## PATIENCE SPENT — THIS ITERATION MUST PIVOT
5 consecutive experiments have failed to beat the current best.
Do NOT refine the champion's current mechanism again this round. Step back
and propose from a design family ABSENT from the last 5 experiments
— the spec (§3) explicitly invites, among others: dispatcher + lighting-
condition-specialist models, learned relighting, a different coordinate
parameterization, quantization-aware capacity changes, or training-data
strategy overhauls. Genuine novelty over incremental tuning.
# Autoresearch iteration — one focused experiment
You are one iteration of an autonomous research loop for UAV low-light
geolocalization. Read `CLAUDE.md` (§1, §3, §6 especially) for full context.
The harness (loop.sh) will train, score, log, and keep/revert AFTER you exit —
you only design the experiment and edit the code.
## Your job, in order
0. **Skim the library (optional input).** `autoresearch/library.md` holds
the human researcher's inspiration notes. They do not fix your answer
and you are free to ignore them — pick an entry up only when it
genuinely fits your read of the history, and if you build on one, say
so in your hypothesis.
1. **Review the research history.** Query the lineage DB:
`sqlite3 experiments.sqlite "SELECT id, title, category, hypothesis, expected_outcome, result, conclusion, primary_metric, kept FROM experiments ORDER BY id DESC LIMIT 15;"`
Note which hypotheses were supported/refuted. Do not repeat a refuted
experiment without a materially new angle.
**Plateau rule:** if three or more consecutive experiments were reverted,
do not attempt another variation of the last refuted mechanism. Either
pick a design family absent from the history (dispatcher + lighting
specialists, pretrained init, learned relighting, training-scale, …) or
attack the bottleneck the refuted hypotheses jointly point at.
2. **Design ONE focused experiment** — proper experiment design, pre-registered
before you touch code. Write it to `runs/pending_experiment.json`:
```json
{
"title": "one-line name",
"category": "architecture|loss|augmentation|relighting|training|quantization|other",
"hypothesis": "what you believe is limiting the metric and why this change addresses it",
"method": "the ONE focused change, concretely (files, mechanism)",
"expected_outcome": "predicted effect on the §6 worst-case median error, quantified if possible",
"init_strategy": "from-scratch | pretrained:<name>",
"eli5": "2-4 sentences for a smart non-ML reader: what you changed and why it might help, in everyday language — analogies welcome, zero jargon",
"architecture": {"stages": [
{"name": "Camera frame", "detail": "128×128 px crop, one of 6 lighting renders", "changed": false},
{"name": "Feature extractor", "detail": "plain-language description", "changed": false},
{"name": "…", "detail": "…", "changed": true}
]},
"architecture_svg": "<svg viewBox='0 0 980 300' …>…</svg>"
}
```
`architecture_svg` is the **technical architecture diagram** of the model
you are testing — a proper ML-paper figure of the design, drawn by you,
shown at the top of this experiment's gallery entry. **Draw tensors and
operations as the things they are — NOT as labeled boxes:** the input
image as a pixel-textured square; conv feature maps as pseudo-3D slabs
that shrink spatially and deepen in channels; 1-D vectors as thin
vertical tick-bars; fully-connected layers as fans of thin crossing
lines between bars; spatial fields/grids as actually-drawn grids with
shaded cells; decodes/aggregations as thin lines converging from cells
onto a point; samplers/targets as small pictorial glyphs. Captions sit
UNDER each element (name ≤10.5px weight 600, sub-note ≤9.5px); tensor
shapes annotate the flow in small italics (128×128×3 → 8²×128 → 1024).
Losses/targets/samplers live in a lane below the inference path,
annotated with slanted dashed leader lines to small text — no boxes
around them. Draw the two FROZEN endpoints — the camera-frame input and
the (lat, lon, confidence) output — entirely in #9b998c with a small
italic “frozen contract” tag: they are fixed by the harness and outside
your search space. Everything between them is your design: draw it in ink,
with red only on what THIS experiment changes. Style contract, so all experiments' figures read as one
paper: viewBox width 980 (height ~300–360); transparent background; ink
#111111, captions #6b6a60, annotations/arrows #9b998c; **#8c2f1f (red)
reserved for exactly what this experiment changed**; training-only
elements in #8a6a1e (or red if changed); font-family
Palatino,Georgia,serif; stroke-width ≈1.2 elements, 1 arrows; fills only
faint tints (opacity ≤ .12); no gradients, no icons, no emoji. Inference
flows left → right from camera frame to (lat, lon, confidence).
**The camera frame is a fixed glyph — copy it verbatim.** It must look
like an actual nadir frame of terrain (streets, building footprints),
not abstract pixels, and be identical in every figure. Use exactly this
snippet, substituting Y for the frame's top edge (pick Y so the frame
centers on your inference lane; captions go under it as usual — note
128²×3 is the model input crop, ~1 m/px, not the sensor's native
resolution):
```svg
<g id='cam-terrain'><rect id='frozen-input' x='26' y='Y' width='76'
height='76' fill='#f6f4ea' stroke='#9b998c' stroke-width='1.6'/>
<path d='M31 Y+47 L97 Y+27' stroke='#e6e3d4' stroke-width='5' fill='none'/>
<path d='M59 Y+6 L49 Y+70' stroke='#e6e3d4' stroke-width='3.5' fill='none'/>
<rect x='34' y='Y+9' width='12' height='8' fill='#d9d5c3' transform='rotate(-8 40 Y+13)'/>
<rect x='78' y='Y+8' width='10' height='11' fill='#cfccbd'/>
<rect x='35' y='Y+57' width='13' height='8' fill='#d9d5c3'/>
<rect x='75' y='Y+50' width='10' height='9' fill='#cfccbd' transform='rotate(6 80 Y+54)'/>
<rect x='54' y='Y+31' width='9' height='8' fill='#d9d5c3' opacity='.85'/>
<ellipse cx='86' cy='Y+63' rx='8' ry='6' fill='#8a6a1e' opacity='.12'/>
<circle cx='40' cy='Y+28' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='70' cy='Y+14' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='92' cy='Y+36' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='52' cy='Y+52' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='82' cy='Y+70' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='31' cy='Y+42' r='.7' fill='#6b6a60' opacity='.5'/></g>
```
(Y+n means the literal number Y plus n — compute the values. Every glyph
element stays strictly INSIDE the frame border — figcheck rejects a
figure without the `cam-terrain` group.) Draw your
receptive-field square and kernel-projection lines on top of it as
usual.
**Anchor the frozen endpoints identically in every figure** so all
experiments' figures line up when compared down the gallery page: the
camera-frame square starts at x=26 (its captions centered on x=53), and
the output is right-anchored — decode crosshair at x=812, the
(lat, lon, confidence) text block text-anchor=start at x=828. Mark the
anchors machine-checkably: the camera-frame rect carries
`id='frozen-input'`, the output text block carries `id='frozen-output'`.
Lay the elements between them out on a running x-cursor with generous
spacing so nothing overlaps.
**Readability contract** (figcheck verifies this geometrically): no
text may overlap other text, and no line (arrows, converge fans,
leader lines) may pass through a label — a line may END at a label,
never cross one. Rules distilled from review rounds: dashed leaders
run as orthogonal L-routes through empty lanes, never diagonally
across components; converge/vote lines originate at their true
sources (the actual dots or window cells, not generic positions);
every element owns its caption — adjacent columns stagger their
caption rows instead of colliding; prefer moving a label into empty
horizontal space over stacking it against a neighbor. Width is fixed
at 980, but **height is yours**: grow the viewBox (240–640) whenever
more vertical room makes the layout cleaner rather than cramming.
**Before finishing, validate:**
`.venv/bin/python -m autoresearch.figcheck` — it checks the anchor
contract on runs/pending_experiment.json; revise until it prints PASS.
Consecutive conv layers must be tied by kernel-projection lines (small
kernel square on one face, faint lines converging to a cell on the
next) so the encoder reads as one computation, not boxes in a row.
`archive/arch_svg_reference.py` is the concrete reference implementation
of this entire visual language — read it and reuse its helper geometry
for any stage that already has one.
`eli5` and `architecture` feed the human-facing gallery. `architecture.stages`
is the model's inference path left-to-right, camera frame → (lat, lon,
confidence) output — one box per stage, plain-language `detail`,
`"changed": true` ONLY on the stages this experiment touches. Reuse the
previous experiment's stage names verbatim wherever a stage is unchanged
(check `SELECT arch_json FROM experiments WHERE arch_json IS NOT NULL ORDER BY id DESC LIMIT 1;`) —
the diagrams are compared box-by-box across experiments. A change that only
affects training (loss, augmentation, schedule) keeps the inference stages
unchanged and adds one final stage with `"train_only": true` describing the
training signal.
3. **Write the implementation brief.** You do NOT edit `model/` yourself —
a separate implementation agent applies your design, seeing only the
current code plus what you pre-registered. Add one more field to
`runs/pending_experiment.json`:
```
"implementation_brief": "exact file-level instructions: which functions/
blocks in model/model.py and model/train.py change and how, what stays
untouched, and every contract to preserve"
```
Be precise enough that a competent engineer with no other context
implements it in one pass. Always restate the fixed contracts:
`train.py`'s CLI and the ONNX export contract in `model/model.py`'s
docstring — the frozen scorer depends on them.
## Hard rules
- Edit ONLY `runs/pending_experiment.json`. You never edit `model/` (the
implementation stage does) and files listed in `/FROZEN` are off-limits;
the harness hard-reverts any change to them.
- NEVER touch, read, or evaluate the `hamburg` holdout area (§5).
- ONE focused change per iteration — if you can't describe it in one sentence,
it's too big. Prefer architectural/procedural novelty over hyperparameter
nudges (§3): changing a learning rate is a weak experiment; changing the
coordinate parameterization, loss family, relighting-for-training, or model
topology is a strong one.
- Do not run training yourself; the harness does that.
- Stay within the deployment gates: exported ONNX ≤ 4 MiB per area, host
latency proxy ≤ 250 ms (see pipeline/score.py).Where the model was tested — and how far off it was Each map is one full test area. Every dot is one held-out test location (all six lighting conditions overlaid): the model was shown a 128 m crop centered there and asked for its position. Dot color = the distance between its answer and the truth — green ≤ 20 m (at goal), amber ≤ 50 m, red beyond. A working model turns these maps green; spatial clusters of red reveal which parts of an area confuse it.     What the six lighting conditions look like through the simulated low-light sensor (example patches — illustration, not the training set)Every frame in this project is rendered as a starlight-class low-light sensor (Sony STARVIS2 / IMX585 class — the airframe's chosen camera) would see it, not as a normal camera would. That is why the night renders look uncannily close to daylight: at high gain such a sensor recovers scene structure from moonlight, skyglow and artificial lighting, paying for it in noise, lifted shadows and washed-out color — exactly what these renders simulate. Below, one example 256 m patch per area under the six lighting conditions the model must handle. These illustrate the dataset, not this experiment's performance — the actual training set is thousands of distinct crops per area (see “training data” above), and these renderings only change when the relighting method changes. berlin frankfurt munich prignitz | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▸ | 30 | Supervised lighting dispatcher: day/dusk/night specialist field heads replace the two-expert blend, int8-paid 92404766 | architecture pivot | pretrained:mobilenet_v3_small IMAGENET1K_V1 features[0..8] (torchvision, BSD-3) — trunk unchanged; the three specialist heads and the dispatcher are fresh-initialized | 808.2 m | 2,490 KB | 2.8 ms | 73 m 20 s | $0.84 | discarded | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The design under test — technical diagram — gray = frozen contract · ink = the current design · red = this experiment's change One real test, end to end
→
912.7 m
miss, this crop 0.13
self-reported confidence In plain words The model recognizes places by comparing what it sees against a big internal catalog of how every map cell looks. Until now one shared catalog had to describe each place at noon AND at midnight — like one photo album trying to cover both — so every entry was a blurry compromise, and the model coped in the dark mostly by refusing to answer. We split it into three catalogs — day, dusk, night — and added a tiny receptionist that checks how bright the frame is and hands the image to the right catalog. During practice the receptionist is simply told the real time of day, so it can't learn the routing wrong. Three catalogs would normally be too heavy for the drone's tiny computer, so they're stored in a compact 1-byte number format we measured earlier to be essentially harmless. Hypothesis The last four reverted experiments (26-29) all attacked capacity or readout width and were jointly refuted; per the plateau rule this iteration leaves that family entirely and opens the one design family CLAUDE.md §3 explicitly invites but the history has never tried: dispatcher + lighting-condition specialist models. The evidence it targets: the champion's per-bucket medians LOOK flat (berlin 632-760 m), but that flatness is manufactured by the confidence gate — night keeps only 25-31% of crops vs 50-67% for daytime buckets, so the model abstains its way to parity in low light instead of localizing to parity, and the binding §6 cell is a low-ambient one (frankfurt EVENING, 797.91 m at 0.46 coverage). The mechanism: all six lighting regimes share ONE 560→1024 template table, plus exp 12's deliberately crippled texture-only 'dark expert' (48-d input, no layout access, unsupervised gate). The same map cell must therefore be recognized by one template averaged across appearance regimes with nearly disjoint cues — daytime shadow/texture vs night's active-light constellations — so every template is a compromise, worst exactly where cues are scarcest. Exp 12 is direct evidence the axis works: even its crippled version was kept (-8%). This experiment builds the full version: each regime gets its own complete layout-aware 560→1024 template table, routing is SUPERVISED (bucket labels are free at train time), and the 3x table storage is paid in int8 FC bytes, which exps 26/29 mechanically proved passes the frozen gates at ~7 m accuracy cost. Method model/model.py: delete dark_logits and the sigmoid gate; replace loc_logits with three identical specialist heads Linear(560→1024) in an nn.ModuleList (day/dusk/night) and add a 3-way dispatcher MLP Linear(49→16)-ReLU-Linear(16→3) reading [lum, GAP]; per turned copy, field logits = Σ_k softmax(dispatcher)_k · specialist_k(desc); C4 vote, decode, confidence all unchanged. loss_fn gains a supervised routing term: 0.3 × cross-entropy of the dispatcher logits vs the crop's regime (morning/midday/afternoon→day, early_evening/evening→dusk, night→night). model/train.py: sample_epoch also returns the per-crop regime index; the train step passes it to loss_fn. export_onnx: fp32 export to <area>_fp32_debug.onnx, then onnxruntime.quantization.quantize_dynamic(QInt8, op_types=['MatMul','Gemm']) to the contract path — the exact recipe exps 26/29 validated (convs stay fp32; no ConvInteger); assert size ≤ 4 MiB. Expected outcome If the lighting-shared readout is binding as hypothesized: night and evening cells improve most, night coverage rises above its current 0.25-0.31 at the unchanged 40% calibration floor, and the primary worst-case drops from 797.91 m to ~700-760 m, with the binding cell either shrinking in place (frankfurt evening) or migrating to a daytime bucket. Prignitz should stay the best area. Clean refutation: a statistical tie (±3%) with healthy routing (train-split dispatcher accuracy >95%) would show regime-pure templates add nothing over exp 12's blend — a different refutation than the capacity family (26-29), since parameters per APPLIED readout are unchanged here; only regime-conditioning is added. Confound pre-registered: the known coverage-median coupling — a median win bought by a coverage drop in the same cells is not a win; the per-bucket coverage log discriminates. Result primary worst-case median error = 808.22 m (previous best 797.91 m); full per-area/bucket breakdown in metrics.json Conclusion REVERTED — metric did not improve (808.22 m vs best 797.91 m); change discarded Scoreboard — median error (m) per area × lighting
the worst cell (underlined) is the experiment's score;
red = failed cell, ink = at target
Fits the aircraft? all deployment gates passed
largest per-area model 2,490 KB (limit 4,096 KB) · single-frame inference 2.8 ms on one CPU thread (proxy limit 250 ms)
crops are sampled fresh each run from the frozen train split (~45,000 distinct positions per area, times random rotation), never from eval blocks; how many to use is the experiment's own choice
Where this iteration's time went: design (Haiku) 19 m 54 s · implement (Sonnet) 2 m 35 s · train 4 areas 48 m 17 s · score 1 m 34 s · samples 43 s · publish 5 s · whole iteration 73 m 29 s
ts 2026-07-22T04:31:16 · commit 92404766e474 ·
parent 92404766e474 · artifacts runs/20260722_031756_iter3 ·
agent model — · took 73 m 20 s
The exact prompt given to the headless agent## PATIENCE SPENT — THIS ITERATION MUST PIVOT
4 consecutive experiments have failed to beat the current best.
Do NOT refine the champion's current mechanism again this round. Step back
and propose from a design family ABSENT from the last 4 experiments
— the spec (§3) explicitly invites, among others: dispatcher + lighting-
condition-specialist models, learned relighting, a different coordinate
parameterization, quantization-aware capacity changes, or training-data
strategy overhauls. Genuine novelty over incremental tuning.
# Autoresearch iteration — one focused experiment
You are one iteration of an autonomous research loop for UAV low-light
geolocalization. Read `CLAUDE.md` (§1, §3, §6 especially) for full context.
The harness (loop.sh) will train, score, log, and keep/revert AFTER you exit —
you only design the experiment and edit the code.
## Your job, in order
0. **Skim the library (optional input).** `autoresearch/library.md` holds
the human researcher's inspiration notes. They do not fix your answer
and you are free to ignore them — pick an entry up only when it
genuinely fits your read of the history, and if you build on one, say
so in your hypothesis.
1. **Review the research history.** Query the lineage DB:
`sqlite3 experiments.sqlite "SELECT id, title, category, hypothesis, expected_outcome, result, conclusion, primary_metric, kept FROM experiments ORDER BY id DESC LIMIT 15;"`
Note which hypotheses were supported/refuted. Do not repeat a refuted
experiment without a materially new angle.
**Plateau rule:** if three or more consecutive experiments were reverted,
do not attempt another variation of the last refuted mechanism. Either
pick a design family absent from the history (dispatcher + lighting
specialists, pretrained init, learned relighting, training-scale, …) or
attack the bottleneck the refuted hypotheses jointly point at.
2. **Design ONE focused experiment** — proper experiment design, pre-registered
before you touch code. Write it to `runs/pending_experiment.json`:
```json
{
"title": "one-line name",
"category": "architecture|loss|augmentation|relighting|training|quantization|other",
"hypothesis": "what you believe is limiting the metric and why this change addresses it",
"method": "the ONE focused change, concretely (files, mechanism)",
"expected_outcome": "predicted effect on the §6 worst-case median error, quantified if possible",
"init_strategy": "from-scratch | pretrained:<name>",
"eli5": "2-4 sentences for a smart non-ML reader: what you changed and why it might help, in everyday language — analogies welcome, zero jargon",
"architecture": {"stages": [
{"name": "Camera frame", "detail": "128×128 px crop, one of 6 lighting renders", "changed": false},
{"name": "Feature extractor", "detail": "plain-language description", "changed": false},
{"name": "…", "detail": "…", "changed": true}
]},
"architecture_svg": "<svg viewBox='0 0 980 300' …>…</svg>"
}
```
`architecture_svg` is the **technical architecture diagram** of the model
you are testing — a proper ML-paper figure of the design, drawn by you,
shown at the top of this experiment's gallery entry. **Draw tensors and
operations as the things they are — NOT as labeled boxes:** the input
image as a pixel-textured square; conv feature maps as pseudo-3D slabs
that shrink spatially and deepen in channels; 1-D vectors as thin
vertical tick-bars; fully-connected layers as fans of thin crossing
lines between bars; spatial fields/grids as actually-drawn grids with
shaded cells; decodes/aggregations as thin lines converging from cells
onto a point; samplers/targets as small pictorial glyphs. Captions sit
UNDER each element (name ≤10.5px weight 600, sub-note ≤9.5px); tensor
shapes annotate the flow in small italics (128×128×3 → 8²×128 → 1024).
Losses/targets/samplers live in a lane below the inference path,
annotated with slanted dashed leader lines to small text — no boxes
around them. Draw the two FROZEN endpoints — the camera-frame input and
the (lat, lon, confidence) output — entirely in #9b998c with a small
italic “frozen contract” tag: they are fixed by the harness and outside
your search space. Everything between them is your design: draw it in ink,
with red only on what THIS experiment changes. Style contract, so all experiments' figures read as one
paper: viewBox width 980 (height ~300–360); transparent background; ink
#111111, captions #6b6a60, annotations/arrows #9b998c; **#8c2f1f (red)
reserved for exactly what this experiment changed**; training-only
elements in #8a6a1e (or red if changed); font-family
Palatino,Georgia,serif; stroke-width ≈1.2 elements, 1 arrows; fills only
faint tints (opacity ≤ .12); no gradients, no icons, no emoji. Inference
flows left → right from camera frame to (lat, lon, confidence).
**The camera frame is a fixed glyph — copy it verbatim.** It must look
like an actual nadir frame of terrain (streets, building footprints),
not abstract pixels, and be identical in every figure. Use exactly this
snippet, substituting Y for the frame's top edge (pick Y so the frame
centers on your inference lane; captions go under it as usual — note
128²×3 is the model input crop, ~1 m/px, not the sensor's native
resolution):
```svg
<g id='cam-terrain'><rect id='frozen-input' x='26' y='Y' width='76'
height='76' fill='#f6f4ea' stroke='#9b998c' stroke-width='1.6'/>
<path d='M31 Y+47 L97 Y+27' stroke='#e6e3d4' stroke-width='5' fill='none'/>
<path d='M59 Y+6 L49 Y+70' stroke='#e6e3d4' stroke-width='3.5' fill='none'/>
<rect x='34' y='Y+9' width='12' height='8' fill='#d9d5c3' transform='rotate(-8 40 Y+13)'/>
<rect x='78' y='Y+8' width='10' height='11' fill='#cfccbd'/>
<rect x='35' y='Y+57' width='13' height='8' fill='#d9d5c3'/>
<rect x='75' y='Y+50' width='10' height='9' fill='#cfccbd' transform='rotate(6 80 Y+54)'/>
<rect x='54' y='Y+31' width='9' height='8' fill='#d9d5c3' opacity='.85'/>
<ellipse cx='86' cy='Y+63' rx='8' ry='6' fill='#8a6a1e' opacity='.12'/>
<circle cx='40' cy='Y+28' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='70' cy='Y+14' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='92' cy='Y+36' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='52' cy='Y+52' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='82' cy='Y+70' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='31' cy='Y+42' r='.7' fill='#6b6a60' opacity='.5'/></g>
```
(Y+n means the literal number Y plus n — compute the values. Every glyph
element stays strictly INSIDE the frame border — figcheck rejects a
figure without the `cam-terrain` group.) Draw your
receptive-field square and kernel-projection lines on top of it as
usual.
**Anchor the frozen endpoints identically in every figure** so all
experiments' figures line up when compared down the gallery page: the
camera-frame square starts at x=26 (its captions centered on x=53), and
the output is right-anchored — decode crosshair at x=812, the
(lat, lon, confidence) text block text-anchor=start at x=828. Mark the
anchors machine-checkably: the camera-frame rect carries
`id='frozen-input'`, the output text block carries `id='frozen-output'`.
Lay the elements between them out on a running x-cursor with generous
spacing so nothing overlaps.
**Readability contract** (figcheck verifies this geometrically): no
text may overlap other text, and no line (arrows, converge fans,
leader lines) may pass through a label — a line may END at a label,
never cross one. Rules distilled from review rounds: dashed leaders
run as orthogonal L-routes through empty lanes, never diagonally
across components; converge/vote lines originate at their true
sources (the actual dots or window cells, not generic positions);
every element owns its caption — adjacent columns stagger their
caption rows instead of colliding; prefer moving a label into empty
horizontal space over stacking it against a neighbor. Width is fixed
at 980, but **height is yours**: grow the viewBox (240–640) whenever
more vertical room makes the layout cleaner rather than cramming.
**Before finishing, validate:**
`.venv/bin/python -m autoresearch.figcheck` — it checks the anchor
contract on runs/pending_experiment.json; revise until it prints PASS.
Consecutive conv layers must be tied by kernel-projection lines (small
kernel square on one face, faint lines converging to a cell on the
next) so the encoder reads as one computation, not boxes in a row.
`archive/arch_svg_reference.py` is the concrete reference implementation
of this entire visual language — read it and reuse its helper geometry
for any stage that already has one.
`eli5` and `architecture` feed the human-facing gallery. `architecture.stages`
is the model's inference path left-to-right, camera frame → (lat, lon,
confidence) output — one box per stage, plain-language `detail`,
`"changed": true` ONLY on the stages this experiment touches. Reuse the
previous experiment's stage names verbatim wherever a stage is unchanged
(check `SELECT arch_json FROM experiments WHERE arch_json IS NOT NULL ORDER BY id DESC LIMIT 1;`) —
the diagrams are compared box-by-box across experiments. A change that only
affects training (loss, augmentation, schedule) keeps the inference stages
unchanged and adds one final stage with `"train_only": true` describing the
training signal.
3. **Write the implementation brief.** You do NOT edit `model/` yourself —
a separate implementation agent applies your design, seeing only the
current code plus what you pre-registered. Add one more field to
`runs/pending_experiment.json`:
```
"implementation_brief": "exact file-level instructions: which functions/
blocks in model/model.py and model/train.py change and how, what stays
untouched, and every contract to preserve"
```
Be precise enough that a competent engineer with no other context
implements it in one pass. Always restate the fixed contracts:
`train.py`'s CLI and the ONNX export contract in `model/model.py`'s
docstring — the frozen scorer depends on them.
## Hard rules
- Edit ONLY `runs/pending_experiment.json`. You never edit `model/` (the
implementation stage does) and files listed in `/FROZEN` are off-limits;
the harness hard-reverts any change to them.
- NEVER touch, read, or evaluate the `hamburg` holdout area (§5).
- ONE focused change per iteration — if you can't describe it in one sentence,
it's too big. Prefer architectural/procedural novelty over hyperparameter
nudges (§3): changing a learning rate is a weak experiment; changing the
coordinate parameterization, loss family, relighting-for-training, or model
topology is a strong one.
- Do not run training yourself; the harness does that.
- Stay within the deployment gates: exported ONNX ≤ 4 MiB per area, host
latency proxy ≤ 250 ms (see pipeline/score.py).Where the model was tested — and how far off it was Each map is one full test area. Every dot is one held-out test location (all six lighting conditions overlaid): the model was shown a 128 m crop centered there and asked for its position. Dot color = the distance between its answer and the truth — green ≤ 20 m (at goal), amber ≤ 50 m, red beyond. A working model turns these maps green; spatial clusters of red reveal which parts of an area confuse it.     What the six lighting conditions look like through the simulated low-light sensor (example patches — illustration, not the training set)Every frame in this project is rendered as a starlight-class low-light sensor (Sony STARVIS2 / IMX585 class — the airframe's chosen camera) would see it, not as a normal camera would. That is why the night renders look uncannily close to daylight: at high gain such a sensor recovers scene structure from moonlight, skyglow and artificial lighting, paying for it in noise, lifted shadows and washed-out color — exactly what these renders simulate. Below, one example 256 m patch per area under the six lighting conditions the model must handle. These illustrate the dataset, not this experiment's performance — the actual training set is thousands of distinct crops per area (see “training data” above), and these renderings only change when the relighting method changes. berlin frankfurt munich prignitz | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▸ | 29 | Stride-8 fine-layout tap: the field scorer reads an 8-m-resolution mid-trunk snapshot, int8-paid 7e6fb3ea | architecture | pretrained:mobilenet_v3_small IMAGENET1K_V1 features[0..8] (torchvision, BSD-3) — unchanged | 798.5 m | 2,433 KB | 2.8 ms | 70 m 11 s | $0.81 | discarded | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The design under test — technical diagram — gray = frozen contract · ink = the current design · red = this experiment's change One real test, end to end
→
693.9 m
miss, this crop 0.24
self-reported confidence In plain words Before guessing where it is, the model boils each camera frame down to a short summary — like describing a neighborhood in 560 numbers, where each number can only talk about 16-metre-wide squares. At that coarseness, two similar districts blur together, and our last three experiments proved that a bigger brain and longer studying don't help — the tell-apart detail was already thrown away before the guessing stage ever saw it. This experiment keeps everything else identical but lets the guesser also look at a sharper, earlier snapshot from inside the network, where building edges and narrow roads are still visible at roughly 8-metre precision. To keep the file small enough for the drone's tiny computer, the guesser's bulky lookup tables are stored as compact 1-byte numbers — a trick we measured beforehand to be harmless. Hypothesis The binding constraint is the information content of the head's readout, not trunk capacity, optimization, or memorization — this is exp 26's own pre-registered diagnostic fork (a), now demonstrated by the joint record: exp 20 removed memorization (train-split and eval-split medians are EQUAL), exp 25 trained to a genuine loss plateau and moved the metric only 5.2%, and exps 26+28 tripled trunk depth/channels and made things WORSE at the designed operating point (1057.68 m vs 797.91 m best). Per the plateau rule this iteration leaves the capacity-scaling mechanism entirely and attacks the bottleneck those refutations jointly point at. The field scorer today sees only a 560-d summary of the crop: a 48-d GAP plus a 512-d layout code squeezed from the stride-16 8x8 grid — every layout cell aggregates a 16x16 m patch, so NOTHING finer than 16 m granularity survives to the 1024-way cell scorer, while the ~215 m map cells it must discriminate differ precisely in sub-16-m arrangement (building outlines, road widths, lot patterns) that the 1 m/px input plainly contains. Two signatures say the discarded fine scale is what's missing: midday errors equal night errors in every area (656-839 m metros) even though midday crops carry clean 1-m detail — if the readout could use it, daytime would lead; and rural prignitz (distinctive macro-structure, low self-similarity) beats all three self-similar metros (449-539 m), the signature of lookalike-cell confusion, not lighting. The change adds the one axis never scaled: SPATIAL resolution of the readout. The trunk's stride-8 activation (features block 3, 16x16x24) is tapped, squeezed by a 1x1 conv to 4 channels, and flattened into a 1,024-d fine layout code concatenated into the bright field head's input (560 -> 1,584). Trunk computation, pretrained weights, C4 vote, decode, confidence, and training are all byte-identical — unlike refuted exps 26/28, no capacity is added to the encoder; the head is simply allowed to SEE a scale the encoder already computes and then throws away. The enlarged FC heads fly as int8 (quantize_dynamic on MatMul/Gemm only), which runs/_probe_iter6_quant2.py proved metrically inert on this exact head family (~7 m median decode shift, no ConvInteger, latency fine): estimated export ~2.6 MB against the 4 MiB gate. Method model/model.py only. (1) TinyLocNet.__init__: probe the stride-8 tap shape by running features[:4] on a zero tensor (gives 24x16x16); add self.fine_squeeze = nn.Conv2d(24, 4, 1); widen loc_logits input from feat_ch + layout_ch*8*8 = 560 to feat_ch + layout_ch*8*8 + 4*16*16 = 1584. (2) forward: run the trunk as an enumerated loop over self.features, capturing the activation after block index 3 as the tap; fine4 = self.fine_squeeze(tap).flatten(1); bright head becomes self.loc_logits(torch.cat([f4, layout4, fine4], dim=1)). Dark head, gate, conf head, C4 stack, decode, loss_fn all untouched. (3) export_onnx: export fp32 to <stem>_fp32_debug.onnx, then onnxruntime.quantization.quantize_dynamic(QInt8, op_types_to_quantize=['MatMul','Gemm']) to the contract path <out>/models/<area>.onnx; assert no ConvInteger node and size <= 4 MiB. model/train.py unchanged. Expected outcome If the readout is fine-detail starved, the worst-case median drops from 797.91 m to ~650-750 m (-6% to -19%), with gains LARGEST in the daytime buckets (morning/midday/afternoon) and the three metro areas — the cells where 1-m detail is cleanest and lookalike aliasing is worst; night cells improve least (sensor noise masks stride-8 texture). Kept iff < 797.91 m. Diagnostic forks if refuted: (a) metric within ±3% of 797.91 with healthy losses — the readout's information width is exonerated, leaving rotation residual (up to ±45° after the C4 vote) and intrinsic 128-m-footprint ambiguity as the demonstrated next bottlenecks; (b) int8 artifact scores clearly worse than its saved <area>_fp32_debug.onnx — a quantization surprise (probe predicts ~7 m, i.e. none); (c) training loss does not drop below exp 25's plateau (berlin 4.78) — the fine tap carries no transferable location signal under the relight sim's statistics. Result primary worst-case median error = 798.53 m (previous best 797.91 m); full per-area/bucket breakdown in metrics.json Conclusion REVERTED — metric did not improve (798.53 m vs best 797.91 m); change discarded Scoreboard — median error (m) per area × lighting
the worst cell (underlined) is the experiment's score;
red = failed cell, ink = at target
Fits the aircraft? all deployment gates passed
largest per-area model 2,433 KB (limit 4,096 KB) · single-frame inference 2.8 ms on one CPU thread (proxy limit 250 ms)
crops are sampled fresh each run from the frozen train split (~45,000 distinct positions per area, times random rotation), never from eval blocks; how many to use is the experiment's own choice
Where this iteration's time went: design (Haiku) 19 m 08 s · implement (Sonnet) 1 m 05 s · train 4 areas 47 m 19 s · score 1 m 36 s · samples 42 s · publish 6 s · whole iteration 70 m 22 s
ts 2026-07-22T03:17:36 · commit 7e6fb3ea1ed9 ·
parent 7e6fb3ea1ed9 · artifacts runs/20260722_020725_iter2 ·
agent model — · took 70 m 11 s
The exact prompt given to the headless agent# Autoresearch iteration — one focused experiment
You are one iteration of an autonomous research loop for UAV low-light
geolocalization. Read `CLAUDE.md` (§1, §3, §6 especially) for full context.
The harness (loop.sh) will train, score, log, and keep/revert AFTER you exit —
you only design the experiment and edit the code.
## Your job, in order
0. **Skim the library (optional input).** `autoresearch/library.md` holds
the human researcher's inspiration notes. They do not fix your answer
and you are free to ignore them — pick an entry up only when it
genuinely fits your read of the history, and if you build on one, say
so in your hypothesis.
1. **Review the research history.** Query the lineage DB:
`sqlite3 experiments.sqlite "SELECT id, title, category, hypothesis, expected_outcome, result, conclusion, primary_metric, kept FROM experiments ORDER BY id DESC LIMIT 15;"`
Note which hypotheses were supported/refuted. Do not repeat a refuted
experiment without a materially new angle.
**Plateau rule:** if three or more consecutive experiments were reverted,
do not attempt another variation of the last refuted mechanism. Either
pick a design family absent from the history (dispatcher + lighting
specialists, pretrained init, learned relighting, training-scale, …) or
attack the bottleneck the refuted hypotheses jointly point at.
2. **Design ONE focused experiment** — proper experiment design, pre-registered
before you touch code. Write it to `runs/pending_experiment.json`:
```json
{
"title": "one-line name",
"category": "architecture|loss|augmentation|relighting|training|quantization|other",
"hypothesis": "what you believe is limiting the metric and why this change addresses it",
"method": "the ONE focused change, concretely (files, mechanism)",
"expected_outcome": "predicted effect on the §6 worst-case median error, quantified if possible",
"init_strategy": "from-scratch | pretrained:<name>",
"eli5": "2-4 sentences for a smart non-ML reader: what you changed and why it might help, in everyday language — analogies welcome, zero jargon",
"architecture": {"stages": [
{"name": "Camera frame", "detail": "128×128 px crop, one of 6 lighting renders", "changed": false},
{"name": "Feature extractor", "detail": "plain-language description", "changed": false},
{"name": "…", "detail": "…", "changed": true}
]},
"architecture_svg": "<svg viewBox='0 0 980 300' …>…</svg>"
}
```
`architecture_svg` is the **technical architecture diagram** of the model
you are testing — a proper ML-paper figure of the design, drawn by you,
shown at the top of this experiment's gallery entry. **Draw tensors and
operations as the things they are — NOT as labeled boxes:** the input
image as a pixel-textured square; conv feature maps as pseudo-3D slabs
that shrink spatially and deepen in channels; 1-D vectors as thin
vertical tick-bars; fully-connected layers as fans of thin crossing
lines between bars; spatial fields/grids as actually-drawn grids with
shaded cells; decodes/aggregations as thin lines converging from cells
onto a point; samplers/targets as small pictorial glyphs. Captions sit
UNDER each element (name ≤10.5px weight 600, sub-note ≤9.5px); tensor
shapes annotate the flow in small italics (128×128×3 → 8²×128 → 1024).
Losses/targets/samplers live in a lane below the inference path,
annotated with slanted dashed leader lines to small text — no boxes
around them. Draw the two FROZEN endpoints — the camera-frame input and
the (lat, lon, confidence) output — entirely in #9b998c with a small
italic “frozen contract” tag: they are fixed by the harness and outside
your search space. Everything between them is your design: draw it in ink,
with red only on what THIS experiment changes. Style contract, so all experiments' figures read as one
paper: viewBox width 980 (height ~300–360); transparent background; ink
#111111, captions #6b6a60, annotations/arrows #9b998c; **#8c2f1f (red)
reserved for exactly what this experiment changed**; training-only
elements in #8a6a1e (or red if changed); font-family
Palatino,Georgia,serif; stroke-width ≈1.2 elements, 1 arrows; fills only
faint tints (opacity ≤ .12); no gradients, no icons, no emoji. Inference
flows left → right from camera frame to (lat, lon, confidence).
**The camera frame is a fixed glyph — copy it verbatim.** It must look
like an actual nadir frame of terrain (streets, building footprints),
not abstract pixels, and be identical in every figure. Use exactly this
snippet, substituting Y for the frame's top edge (pick Y so the frame
centers on your inference lane; captions go under it as usual — note
128²×3 is the model input crop, ~1 m/px, not the sensor's native
resolution):
```svg
<g id='cam-terrain'><rect id='frozen-input' x='26' y='Y' width='76'
height='76' fill='#f6f4ea' stroke='#9b998c' stroke-width='1.6'/>
<path d='M31 Y+47 L97 Y+27' stroke='#e6e3d4' stroke-width='5' fill='none'/>
<path d='M59 Y+6 L49 Y+70' stroke='#e6e3d4' stroke-width='3.5' fill='none'/>
<rect x='34' y='Y+9' width='12' height='8' fill='#d9d5c3' transform='rotate(-8 40 Y+13)'/>
<rect x='78' y='Y+8' width='10' height='11' fill='#cfccbd'/>
<rect x='35' y='Y+57' width='13' height='8' fill='#d9d5c3'/>
<rect x='75' y='Y+50' width='10' height='9' fill='#cfccbd' transform='rotate(6 80 Y+54)'/>
<rect x='54' y='Y+31' width='9' height='8' fill='#d9d5c3' opacity='.85'/>
<ellipse cx='86' cy='Y+63' rx='8' ry='6' fill='#8a6a1e' opacity='.12'/>
<circle cx='40' cy='Y+28' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='70' cy='Y+14' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='92' cy='Y+36' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='52' cy='Y+52' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='82' cy='Y+70' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='31' cy='Y+42' r='.7' fill='#6b6a60' opacity='.5'/></g>
```
(Y+n means the literal number Y plus n — compute the values. Every glyph
element stays strictly INSIDE the frame border — figcheck rejects a
figure without the `cam-terrain` group.) Draw your
receptive-field square and kernel-projection lines on top of it as
usual.
**Anchor the frozen endpoints identically in every figure** so all
experiments' figures line up when compared down the gallery page: the
camera-frame square starts at x=26 (its captions centered on x=53), and
the output is right-anchored — decode crosshair at x=812, the
(lat, lon, confidence) text block text-anchor=start at x=828. Mark the
anchors machine-checkably: the camera-frame rect carries
`id='frozen-input'`, the output text block carries `id='frozen-output'`.
Lay the elements between them out on a running x-cursor with generous
spacing so nothing overlaps.
**Readability contract** (figcheck verifies this geometrically): no
text may overlap other text, and no line (arrows, converge fans,
leader lines) may pass through a label — a line may END at a label,
never cross one. Rules distilled from review rounds: dashed leaders
run as orthogonal L-routes through empty lanes, never diagonally
across components; converge/vote lines originate at their true
sources (the actual dots or window cells, not generic positions);
every element owns its caption — adjacent columns stagger their
caption rows instead of colliding; prefer moving a label into empty
horizontal space over stacking it against a neighbor. Width is fixed
at 980, but **height is yours**: grow the viewBox (240–640) whenever
more vertical room makes the layout cleaner rather than cramming.
**Before finishing, validate:**
`.venv/bin/python -m autoresearch.figcheck` — it checks the anchor
contract on runs/pending_experiment.json; revise until it prints PASS.
Consecutive conv layers must be tied by kernel-projection lines (small
kernel square on one face, faint lines converging to a cell on the
next) so the encoder reads as one computation, not boxes in a row.
`archive/arch_svg_reference.py` is the concrete reference implementation
of this entire visual language — read it and reuse its helper geometry
for any stage that already has one.
`eli5` and `architecture` feed the human-facing gallery. `architecture.stages`
is the model's inference path left-to-right, camera frame → (lat, lon,
confidence) output — one box per stage, plain-language `detail`,
`"changed": true` ONLY on the stages this experiment touches. Reuse the
previous experiment's stage names verbatim wherever a stage is unchanged
(check `SELECT arch_json FROM experiments WHERE arch_json IS NOT NULL ORDER BY id DESC LIMIT 1;`) —
the diagrams are compared box-by-box across experiments. A change that only
affects training (loss, augmentation, schedule) keeps the inference stages
unchanged and adds one final stage with `"train_only": true` describing the
training signal.
3. **Write the implementation brief.** You do NOT edit `model/` yourself —
a separate implementation agent applies your design, seeing only the
current code plus what you pre-registered. Add one more field to
`runs/pending_experiment.json`:
```
"implementation_brief": "exact file-level instructions: which functions/
blocks in model/model.py and model/train.py change and how, what stays
untouched, and every contract to preserve"
```
Be precise enough that a competent engineer with no other context
implements it in one pass. Always restate the fixed contracts:
`train.py`'s CLI and the ONNX export contract in `model/model.py`'s
docstring — the frozen scorer depends on them.
## Hard rules
- Edit ONLY `runs/pending_experiment.json`. You never edit `model/` (the
implementation stage does) and files listed in `/FROZEN` are off-limits;
the harness hard-reverts any change to them.
- NEVER touch, read, or evaluate the `hamburg` holdout area (§5).
- ONE focused change per iteration — if you can't describe it in one sentence,
it's too big. Prefer architectural/procedural novelty over hyperparameter
nudges (§3): changing a learning rate is a weak experiment; changing the
coordinate parameterization, loss family, relighting-for-training, or model
topology is a strong one.
- Do not run training yourself; the harness does that.
- Stay within the deployment gates: exported ONNX ≤ 4 MiB per area, host
latency proxy ≤ 250 ms (see pipeline/score.py).Where the model was tested — and how far off it was Each map is one full test area. Every dot is one held-out test location (all six lighting conditions overlaid): the model was shown a 128 m crop centered there and asked for its position. Dot color = the distance between its answer and the truth — green ≤ 20 m (at goal), amber ≤ 50 m, red beyond. A working model turns these maps green; spatial clusters of red reveal which parts of an area confuse it.     What the six lighting conditions look like through the simulated low-light sensor (example patches — illustration, not the training set)Every frame in this project is rendered as a starlight-class low-light sensor (Sony STARVIS2 / IMX585 class — the airframe's chosen camera) would see it, not as a normal camera would. That is why the night renders look uncannily close to daylight: at high gain such a sensor recovers scene structure from moonlight, skyglow and artificial lighting, paying for it in noise, lifted shadows and washed-out color — exactly what these renders simulate. Below, one example 256 m patch per area under the six lighting conditions the model must handle. These illustrate the dataset, not this experiment's performance — the actual training set is thousands of distinct crops per area (see “training data” above), and these renderings only change when the relighting method changes. berlin frankfurt munich prignitz | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▸ | 28 | Rerun, never scored: unseen-ground confidence calibration + reinstated 3x trunk (iter 7 died on a full disk before training) 83bf925a | training | pretrained:mobilenet_v3_small IMAGENET1K_V1 features[0..10] (torchvision, BSD-3) | 1.06 km | 3,019 KB | 4.6 ms | 57 m 43 s | $0.66 | discarded | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The design under test — technical diagram — gray = frozen contract · ink = the current design · red = this experiment's change One real test, end to end
→
2.35 km
miss, this crop 0.03
self-reported confidence In plain words Two rounds ago we gave the model a bigger brain and its guesses genuinely got better — but the result was thrown out on a technicality: the bar for when the model may say 'no fix, not sure' was tuned on places it had already visited, and the bigger brain is so much more confident on visited ground that the bar landed wildly wrong for new ground — it went nearly silent on the exam. Last round we designed the honest fix — fence off a slice of the map the model never studies and tune the bar there — but the computer's hard drive filled up before training even began, so that experiment was marked 'failed' without ever actually running. The disk problem is fixed; this round simply runs the same experiment for real. Hypothesis Exp 27's recorded 1e9 is an infrastructure artifact, not a refutation: its train logs end in OSError [Errno 28] No space left on device inside prepare_realizations of the FIRST area (berlin), before a single optimizer step — metrics.json has an empty areas list and timings.json shows score_s=0, train_wall_s=158. The disk was filled by ~5.7 GB/iteration of per-area training scratch that earlier iterations never purged; the harness fix ('Never persist per-area training scratch') landed right after, and the volume now has 42 GB free against a ~6 GB per-run scratch need, so the crash cause is removed. The scientific hypothesis is therefore still the untested, pre-registered exp-27 one, restated: exp 26's 3x-capacity pretrained trunk delivered real accuracy gains (like-for-like medians down 20-35%: munich morning 536->347 m, frankfurt evening 798->466 m, prignitz morning 450->350 m; deployment gates passed at 3.09 MB / 4.6 ms) but was vetoed solely by the coverage gate, because calibrate_conf_shift tunes the abstention threshold on TRAIN-visited ground: the bigger trunk sharpened the confidence head's separation between visited and unseen ground, conf_shift leapt from 0.2-0.7 to 2.5-3.6 at identical calibration-time keep rates, and eval coverage collapsed to 0.08-0.43 (8 of 24 cells under the 0.2 floor). Calibrating on train blocks fenced off from every training draw measures the confidence distribution the scorer actually faces, so the 0.40 keep floor transfers to eval and the real capacity gain becomes scoreable. Precedent for this rerun: exp 24 retested exp 23 after it was OOM-killed mid-harness and never scored — an unmeasured pre-registered design is not a refuted one. Method Verbatim re-registration of exp 27's pre-registered design — no design delta; the only thing that changed between iter 7 and now is the harness disk fix. ONE new mechanism in model/train.py — block-held-out confidence calibration: deterministically hash ~1 in 10 train-split blocks (stable_hash, per area) into a calibration-only set; the per-epoch fresh-draw sampler uses only crops whose 182-px windows touch no calibration block (mirror of the frozen eval-buffer rule, so no calibration pixel is ever trained on); calibrate_conf_shift draws its 400 crops per bucket exclusively from calibration-block crops. MIN_KEEP_RATE=0.40, CAL_CROPS_PER_BUCKET=400, losses, schedule, sampler mechanics unchanged. Alongside, model/model.py reinstates exp 26's pre-registered architecture verbatim (MobileNetV3-Small features[:11] with stride/dilation surgery preserving the 8×8 grid; int8 dynamic quantization of MatMul/Gemm at export), which was refuted only through the calibration artifact this experiment removes. Expected outcome Eval coverage lands near the 0.35–0.45 design point in every cell (vs 0.085–0.43 in exp 26), clearing the 0.2 floor everywhere. Medians rise relative to exp 26's deflated-coverage readings but keep most of the capacity gain: predicted worst-case median 600–750 m vs the 797.91 m best, night cells binding. If exp 26's apparent gain was mostly a selection effect of low coverage, the metric will land ≈800 m or slightly worse — a clean measurement of capacity at the designed operating point either way. (Unchanged from the iter-7 pre-registration: the design was implemented but never trained or scored, so the original prediction stands.) Result primary worst-case median error = 1057.68 m (previous best 797.91 m); full per-area/bucket breakdown in metrics.json Conclusion REVERTED — metric did not improve (1057.68 m vs best 797.91 m); change discarded Scoreboard — median error (m) per area × lighting
the worst cell (underlined) is the experiment's score;
red = failed cell, ink = at target
Fits the aircraft? all deployment gates passed
largest per-area model 3,019 KB (limit 4,096 KB) · single-frame inference 4.6 ms on one CPU thread (proxy limit 250 ms)
crops are sampled fresh each run from the frozen train split (~45,000 distinct positions per area, times random rotation), never from eval blocks; how many to use is the experiment's own choice
Where this iteration's time went: design (Haiku) 4 m 08 s · implement (Sonnet) 2 m 35 s · train 4 areas 47 m 55 s · score 1 m 52 s · samples 43 s · publish 5 s · whole iteration 57 m 57 s
ts 2026-07-22T02:07:02 · commit 83bf925a5c35 ·
parent 83bf925a5c35 · artifacts runs/20260722_010919_iter1 ·
agent model — · took 57 m 43 s
The exact prompt given to the headless agent# Autoresearch iteration — one focused experiment
You are one iteration of an autonomous research loop for UAV low-light
geolocalization. Read `CLAUDE.md` (§1, §3, §6 especially) for full context.
The harness (loop.sh) will train, score, log, and keep/revert AFTER you exit —
you only design the experiment and edit the code.
## Your job, in order
0. **Skim the library (optional input).** `autoresearch/library.md` holds
the human researcher's inspiration notes. They do not fix your answer
and you are free to ignore them — pick an entry up only when it
genuinely fits your read of the history, and if you build on one, say
so in your hypothesis.
1. **Review the research history.** Query the lineage DB:
`sqlite3 experiments.sqlite "SELECT id, title, category, hypothesis, expected_outcome, result, conclusion, primary_metric, kept FROM experiments ORDER BY id DESC LIMIT 15;"`
Note which hypotheses were supported/refuted. Do not repeat a refuted
experiment without a materially new angle.
**Plateau rule:** if three or more consecutive experiments were reverted,
do not attempt another variation of the last refuted mechanism. Either
pick a design family absent from the history (dispatcher + lighting
specialists, pretrained init, learned relighting, training-scale, …) or
attack the bottleneck the refuted hypotheses jointly point at.
2. **Design ONE focused experiment** — proper experiment design, pre-registered
before you touch code. Write it to `runs/pending_experiment.json`:
```json
{
"title": "one-line name",
"category": "architecture|loss|augmentation|relighting|training|quantization|other",
"hypothesis": "what you believe is limiting the metric and why this change addresses it",
"method": "the ONE focused change, concretely (files, mechanism)",
"expected_outcome": "predicted effect on the §6 worst-case median error, quantified if possible",
"init_strategy": "from-scratch | pretrained:<name>",
"eli5": "2-4 sentences for a smart non-ML reader: what you changed and why it might help, in everyday language — analogies welcome, zero jargon",
"architecture": {"stages": [
{"name": "Camera frame", "detail": "128×128 px crop, one of 6 lighting renders", "changed": false},
{"name": "Feature extractor", "detail": "plain-language description", "changed": false},
{"name": "…", "detail": "…", "changed": true}
]},
"architecture_svg": "<svg viewBox='0 0 980 300' …>…</svg>"
}
```
`architecture_svg` is the **technical architecture diagram** of the model
you are testing — a proper ML-paper figure of the design, drawn by you,
shown at the top of this experiment's gallery entry. **Draw tensors and
operations as the things they are — NOT as labeled boxes:** the input
image as a pixel-textured square; conv feature maps as pseudo-3D slabs
that shrink spatially and deepen in channels; 1-D vectors as thin
vertical tick-bars; fully-connected layers as fans of thin crossing
lines between bars; spatial fields/grids as actually-drawn grids with
shaded cells; decodes/aggregations as thin lines converging from cells
onto a point; samplers/targets as small pictorial glyphs. Captions sit
UNDER each element (name ≤10.5px weight 600, sub-note ≤9.5px); tensor
shapes annotate the flow in small italics (128×128×3 → 8²×128 → 1024).
Losses/targets/samplers live in a lane below the inference path,
annotated with slanted dashed leader lines to small text — no boxes
around them. Draw the two FROZEN endpoints — the camera-frame input and
the (lat, lon, confidence) output — entirely in #9b998c with a small
italic “frozen contract” tag: they are fixed by the harness and outside
your search space. Everything between them is your design: draw it in ink,
with red only on what THIS experiment changes. Style contract, so all experiments' figures read as one
paper: viewBox width 980 (height ~300–360); transparent background; ink
#111111, captions #6b6a60, annotations/arrows #9b998c; **#8c2f1f (red)
reserved for exactly what this experiment changed**; training-only
elements in #8a6a1e (or red if changed); font-family
Palatino,Georgia,serif; stroke-width ≈1.2 elements, 1 arrows; fills only
faint tints (opacity ≤ .12); no gradients, no icons, no emoji. Inference
flows left → right from camera frame to (lat, lon, confidence).
**The camera frame is a fixed glyph — copy it verbatim.** It must look
like an actual nadir frame of terrain (streets, building footprints),
not abstract pixels, and be identical in every figure. Use exactly this
snippet, substituting Y for the frame's top edge (pick Y so the frame
centers on your inference lane; captions go under it as usual — note
128²×3 is the model input crop, ~1 m/px, not the sensor's native
resolution):
```svg
<g id='cam-terrain'><rect id='frozen-input' x='26' y='Y' width='76'
height='76' fill='#f6f4ea' stroke='#9b998c' stroke-width='1.6'/>
<path d='M31 Y+47 L97 Y+27' stroke='#e6e3d4' stroke-width='5' fill='none'/>
<path d='M59 Y+6 L49 Y+70' stroke='#e6e3d4' stroke-width='3.5' fill='none'/>
<rect x='34' y='Y+9' width='12' height='8' fill='#d9d5c3' transform='rotate(-8 40 Y+13)'/>
<rect x='78' y='Y+8' width='10' height='11' fill='#cfccbd'/>
<rect x='35' y='Y+57' width='13' height='8' fill='#d9d5c3'/>
<rect x='75' y='Y+50' width='10' height='9' fill='#cfccbd' transform='rotate(6 80 Y+54)'/>
<rect x='54' y='Y+31' width='9' height='8' fill='#d9d5c3' opacity='.85'/>
<ellipse cx='86' cy='Y+63' rx='8' ry='6' fill='#8a6a1e' opacity='.12'/>
<circle cx='40' cy='Y+28' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='70' cy='Y+14' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='92' cy='Y+36' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='52' cy='Y+52' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='82' cy='Y+70' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='31' cy='Y+42' r='.7' fill='#6b6a60' opacity='.5'/></g>
```
(Y+n means the literal number Y plus n — compute the values. Every glyph
element stays strictly INSIDE the frame border — figcheck rejects a
figure without the `cam-terrain` group.) Draw your
receptive-field square and kernel-projection lines on top of it as
usual.
**Anchor the frozen endpoints identically in every figure** so all
experiments' figures line up when compared down the gallery page: the
camera-frame square starts at x=26 (its captions centered on x=53), and
the output is right-anchored — decode crosshair at x=812, the
(lat, lon, confidence) text block text-anchor=start at x=828. Mark the
anchors machine-checkably: the camera-frame rect carries
`id='frozen-input'`, the output text block carries `id='frozen-output'`.
Lay the elements between them out on a running x-cursor with generous
spacing so nothing overlaps.
**Readability contract** (figcheck verifies this geometrically): no
text may overlap other text, and no line (arrows, converge fans,
leader lines) may pass through a label — a line may END at a label,
never cross one. Rules distilled from review rounds: dashed leaders
run as orthogonal L-routes through empty lanes, never diagonally
across components; converge/vote lines originate at their true
sources (the actual dots or window cells, not generic positions);
every element owns its caption — adjacent columns stagger their
caption rows instead of colliding; prefer moving a label into empty
horizontal space over stacking it against a neighbor. Width is fixed
at 980, but **height is yours**: grow the viewBox (240–640) whenever
more vertical room makes the layout cleaner rather than cramming.
**Before finishing, validate:**
`.venv/bin/python -m autoresearch.figcheck` — it checks the anchor
contract on runs/pending_experiment.json; revise until it prints PASS.
Consecutive conv layers must be tied by kernel-projection lines (small
kernel square on one face, faint lines converging to a cell on the
next) so the encoder reads as one computation, not boxes in a row.
`archive/arch_svg_reference.py` is the concrete reference implementation
of this entire visual language — read it and reuse its helper geometry
for any stage that already has one.
`eli5` and `architecture` feed the human-facing gallery. `architecture.stages`
is the model's inference path left-to-right, camera frame → (lat, lon,
confidence) output — one box per stage, plain-language `detail`,
`"changed": true` ONLY on the stages this experiment touches. Reuse the
previous experiment's stage names verbatim wherever a stage is unchanged
(check `SELECT arch_json FROM experiments WHERE arch_json IS NOT NULL ORDER BY id DESC LIMIT 1;`) —
the diagrams are compared box-by-box across experiments. A change that only
affects training (loss, augmentation, schedule) keeps the inference stages
unchanged and adds one final stage with `"train_only": true` describing the
training signal.
3. **Write the implementation brief.** You do NOT edit `model/` yourself —
a separate implementation agent applies your design, seeing only the
current code plus what you pre-registered. Add one more field to
`runs/pending_experiment.json`:
```
"implementation_brief": "exact file-level instructions: which functions/
blocks in model/model.py and model/train.py change and how, what stays
untouched, and every contract to preserve"
```
Be precise enough that a competent engineer with no other context
implements it in one pass. Always restate the fixed contracts:
`train.py`'s CLI and the ONNX export contract in `model/model.py`'s
docstring — the frozen scorer depends on them.
## Hard rules
- Edit ONLY `runs/pending_experiment.json`. You never edit `model/` (the
implementation stage does) and files listed in `/FROZEN` are off-limits;
the harness hard-reverts any change to them.
- NEVER touch, read, or evaluate the `hamburg` holdout area (§5).
- ONE focused change per iteration — if you can't describe it in one sentence,
it's too big. Prefer architectural/procedural novelty over hyperparameter
nudges (§3): changing a learning rate is a weak experiment; changing the
coordinate parameterization, loss family, relighting-for-training, or model
topology is a strong one.
- Do not run training yourself; the harness does that.
- Stay within the deployment gates: exported ONNX ≤ 4 MiB per area, host
latency proxy ≤ 250 ms (see pipeline/score.py).Where the model was tested — and how far off it was Each map is one full test area. Every dot is one held-out test location (all six lighting conditions overlaid): the model was shown a 128 m crop centered there and asked for its position. Dot color = the distance between its answer and the truth — green ≤ 20 m (at goal), amber ≤ 50 m, red beyond. A working model turns these maps green; spatial clusters of red reveal which parts of an area confuse it.     What the six lighting conditions look like through the simulated low-light sensor (example patches — illustration, not the training set)Every frame in this project is rendered as a starlight-class low-light sensor (Sony STARVIS2 / IMX585 class — the airframe's chosen camera) would see it, not as a normal camera would. That is why the night renders look uncannily close to daylight: at high gain such a sensor recovers scene structure from moonlight, skyglow and artificial lighting, paying for it in noise, lifted shadows and washed-out color — exactly what these renders simulate. Below, one example 256 m patch per area under the six lighting conditions the model must handle. These illustrate the dataset, not this experiment's performance — the actual training set is thousands of distinct crops per area (see “training data” above), and these renderings only change when the relighting method changes. berlin frankfurt munich prignitz | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▸ | 27 | Unseen-ground confidence calibration unlocks the reinstated 3×-capacity trunk cc3faa2c | training | pretrained:mobilenet_v3_small IMAGENET1K_V1 features[0..10] (torchvision, BSD-3) | gated fail | — | — | 14 m 55 s | $0.17 | gated fail | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The design under test — technical diagram — gray = frozen contract · ink = the current design · red = this experiment's change In plain words Last round we gave the model a bigger brain and its guesses genuinely got better — but the result was thrown out on a technicality. The model is allowed to say 'no fix' when unsure, and the rules demand it answers at least 20% of the time. We tune its shyness bar using places from its own training tour, and the bigger brain became so much more confident about places it had already visited than about new ones that the bar landed wildly wrong for new ground: it went nearly silent on the exam. The fix is to fence off a slice of the map the model never gets to study and tune the shyness bar there — on ground exactly as unfamiliar as the exam's. Same bigger brain, bar set honestly. Hypothesis Exp 26's capacity scaling delivered its predicted accuracy gain but was vetoed solely by the coverage gate: its deployment gates passed (3.09 MB, 4.6 ms) and its like-for-like medians fell 20–35% (munich morning 536→347 m, frankfurt evening 798→466 m, prignitz morning 450→350 m), yet 8 of 24 cells dropped below the 0.2 coverage floor → 1e9. This iteration's forensics isolate why: calibrate_conf_shift sets the abstention threshold on TRAIN-split crops at locations the model trained on, and the 3× trunk sharpened the confidence head's separation between trained-on and unseen ground — calibration-time keep rates were IDENTICAL between iter 5 and iter 6 (0.40–0.88 per bucket) while conf_shift leapt from 0.2–0.7 to 2.5–3.6 and eval coverage collapsed from 0.25–0.67 to 0.08–0.43. A threshold that keeps 40% of visited ground kept only 8–19% of held-out ground. Calibrating instead on train blocks fenced off from every training draw measures the confidence distribution the scorer actually faces, so the 40% keep floor transfers to eval and the real capacity gain becomes scoreable. (The same mechanism also explains exp 25's near-floor night coverage of 0.25–0.31 — this fix de-risks every future capacity increase, not just this one.) Method ONE new mechanism in model/train.py — block-held-out confidence calibration: deterministically hash ~1 in 10 train-split blocks (stable_hash, per area) into a calibration-only set; the per-epoch fresh-draw sampler uses only crops whose 182-px windows touch no calibration block (mirror of the frozen eval-buffer rule, so no calibration pixel is ever trained on); calibrate_conf_shift draws its 400 crops per bucket exclusively from calibration-block crops. MIN_KEEP_RATE=0.40, CAL_CROPS_PER_BUCKET=400, losses, schedule, sampler mechanics unchanged. Alongside, model/model.py reinstates exp 26's pre-registered architecture verbatim (MobileNetV3-Small features[:11] with stride/dilation surgery preserving the 8×8 grid; int8 dynamic quantization of MatMul/Gemm at export), which was refuted only through the calibration artifact this experiment removes. Expected outcome Eval coverage lands near the 0.35–0.45 design point in every cell (vs 0.085–0.43 in exp 26), clearing the 0.2 floor everywhere. Medians rise relative to exp 26's deflated-coverage readings but keep most of the capacity gain: predicted worst-case median 600–750 m vs the 797.91 m best, night cells binding. If exp 26's apparent gain was mostly a selection effect of low coverage, the metric will land ≈800 m or slightly worse — a clean measurement of capacity at the designed operating point either way. Result primary worst-case median error = 1000000000.0 m (previous best 797.91 m); full per-area/bucket breakdown in metrics.json Conclusion REVERTED — metric did not improve (1000000000.0 m vs best 797.91 m); change discarded Scoreboard — median error (m) per area × lighting
the worst cell (underlined) is the experiment's score;
red = failed cell, ink = at target
No per-area results recorded for this run. Fits the aircraft? all deployment gates passed
largest per-area model — (limit 4,096 KB) · single-frame inference — on one CPU thread (proxy limit 250 ms) Where this iteration's time went: design (Haiku) 8 m 58 s · implement (Sonnet) 2 m 17 s · train 4 areas 2 m 38 s · samples 41 s · whole iteration 15 m 00 s
ts 2026-07-21T23:34:48 · commit cc3faa2c230b ·
parent cc3faa2c230b · artifacts runs/20260721_231953_iter7 ·
agent model — · took 14 m 55 s
The exact prompt given to the headless agent# Autoresearch iteration — one focused experiment
You are one iteration of an autonomous research loop for UAV low-light
geolocalization. Read `CLAUDE.md` (§1, §3, §6 especially) for full context.
The harness (loop.sh) will train, score, log, and keep/revert AFTER you exit —
you only design the experiment and edit the code.
## Your job, in order
0. **Skim the library (optional input).** `autoresearch/library.md` holds
the human researcher's inspiration notes. They do not fix your answer
and you are free to ignore them — pick an entry up only when it
genuinely fits your read of the history, and if you build on one, say
so in your hypothesis.
1. **Review the research history.** Query the lineage DB:
`sqlite3 experiments.sqlite "SELECT id, title, category, hypothesis, expected_outcome, result, conclusion, primary_metric, kept FROM experiments ORDER BY id DESC LIMIT 15;"`
Note which hypotheses were supported/refuted. Do not repeat a refuted
experiment without a materially new angle.
**Plateau rule:** if three or more consecutive experiments were reverted,
do not attempt another variation of the last refuted mechanism. Either
pick a design family absent from the history (dispatcher + lighting
specialists, pretrained init, learned relighting, training-scale, …) or
attack the bottleneck the refuted hypotheses jointly point at.
2. **Design ONE focused experiment** — proper experiment design, pre-registered
before you touch code. Write it to `runs/pending_experiment.json`:
```json
{
"title": "one-line name",
"category": "architecture|loss|augmentation|relighting|training|quantization|other",
"hypothesis": "what you believe is limiting the metric and why this change addresses it",
"method": "the ONE focused change, concretely (files, mechanism)",
"expected_outcome": "predicted effect on the §6 worst-case median error, quantified if possible",
"init_strategy": "from-scratch | pretrained:<name>",
"eli5": "2-4 sentences for a smart non-ML reader: what you changed and why it might help, in everyday language — analogies welcome, zero jargon",
"architecture": {"stages": [
{"name": "Camera frame", "detail": "128×128 px crop, one of 6 lighting renders", "changed": false},
{"name": "Feature extractor", "detail": "plain-language description", "changed": false},
{"name": "…", "detail": "…", "changed": true}
]},
"architecture_svg": "<svg viewBox='0 0 980 300' …>…</svg>"
}
```
`architecture_svg` is the **technical architecture diagram** of the model
you are testing — a proper ML-paper figure of the design, drawn by you,
shown at the top of this experiment's gallery entry. **Draw tensors and
operations as the things they are — NOT as labeled boxes:** the input
image as a pixel-textured square; conv feature maps as pseudo-3D slabs
that shrink spatially and deepen in channels; 1-D vectors as thin
vertical tick-bars; fully-connected layers as fans of thin crossing
lines between bars; spatial fields/grids as actually-drawn grids with
shaded cells; decodes/aggregations as thin lines converging from cells
onto a point; samplers/targets as small pictorial glyphs. Captions sit
UNDER each element (name ≤10.5px weight 600, sub-note ≤9.5px); tensor
shapes annotate the flow in small italics (128×128×3 → 8²×128 → 1024).
Losses/targets/samplers live in a lane below the inference path,
annotated with slanted dashed leader lines to small text — no boxes
around them. Draw the two FROZEN endpoints — the camera-frame input and
the (lat, lon, confidence) output — entirely in #9b998c with a small
italic “frozen contract” tag: they are fixed by the harness and outside
your search space. Everything between them is your design: draw it in ink,
with red only on what THIS experiment changes. Style contract, so all experiments' figures read as one
paper: viewBox width 980 (height ~300–360); transparent background; ink
#111111, captions #6b6a60, annotations/arrows #9b998c; **#8c2f1f (red)
reserved for exactly what this experiment changed**; training-only
elements in #8a6a1e (or red if changed); font-family
Palatino,Georgia,serif; stroke-width ≈1.2 elements, 1 arrows; fills only
faint tints (opacity ≤ .12); no gradients, no icons, no emoji. Inference
flows left → right from camera frame to (lat, lon, confidence).
**The camera frame is a fixed glyph — copy it verbatim.** It must look
like an actual nadir frame of terrain (streets, building footprints),
not abstract pixels, and be identical in every figure. Use exactly this
snippet, substituting Y for the frame's top edge (pick Y so the frame
centers on your inference lane; captions go under it as usual — note
128²×3 is the model input crop, ~1 m/px, not the sensor's native
resolution):
```svg
<g id='cam-terrain'><rect id='frozen-input' x='26' y='Y' width='76'
height='76' fill='#f6f4ea' stroke='#9b998c' stroke-width='1.6'/>
<path d='M31 Y+47 L97 Y+27' stroke='#e6e3d4' stroke-width='5' fill='none'/>
<path d='M59 Y+6 L49 Y+70' stroke='#e6e3d4' stroke-width='3.5' fill='none'/>
<rect x='34' y='Y+9' width='12' height='8' fill='#d9d5c3' transform='rotate(-8 40 Y+13)'/>
<rect x='78' y='Y+8' width='10' height='11' fill='#cfccbd'/>
<rect x='35' y='Y+57' width='13' height='8' fill='#d9d5c3'/>
<rect x='75' y='Y+50' width='10' height='9' fill='#cfccbd' transform='rotate(6 80 Y+54)'/>
<rect x='54' y='Y+31' width='9' height='8' fill='#d9d5c3' opacity='.85'/>
<ellipse cx='86' cy='Y+63' rx='8' ry='6' fill='#8a6a1e' opacity='.12'/>
<circle cx='40' cy='Y+28' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='70' cy='Y+14' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='92' cy='Y+36' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='52' cy='Y+52' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='82' cy='Y+70' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='31' cy='Y+42' r='.7' fill='#6b6a60' opacity='.5'/></g>
```
(Y+n means the literal number Y plus n — compute the values. Every glyph
element stays strictly INSIDE the frame border — figcheck rejects a
figure without the `cam-terrain` group.) Draw your
receptive-field square and kernel-projection lines on top of it as
usual.
**Anchor the frozen endpoints identically in every figure** so all
experiments' figures line up when compared down the gallery page: the
camera-frame square starts at x=26 (its captions centered on x=53), and
the output is right-anchored — decode crosshair at x=812, the
(lat, lon, confidence) text block text-anchor=start at x=828. Mark the
anchors machine-checkably: the camera-frame rect carries
`id='frozen-input'`, the output text block carries `id='frozen-output'`.
Lay the elements between them out on a running x-cursor with generous
spacing so nothing overlaps.
**Readability contract** (figcheck verifies this geometrically): no
text may overlap other text, and no line (arrows, converge fans,
leader lines) may pass through a label — a line may END at a label,
never cross one. Rules distilled from review rounds: dashed leaders
run as orthogonal L-routes through empty lanes, never diagonally
across components; converge/vote lines originate at their true
sources (the actual dots or window cells, not generic positions);
every element owns its caption — adjacent columns stagger their
caption rows instead of colliding; prefer moving a label into empty
horizontal space over stacking it against a neighbor. Width is fixed
at 980, but **height is yours**: grow the viewBox (240–640) whenever
more vertical room makes the layout cleaner rather than cramming.
**Before finishing, validate:**
`.venv/bin/python -m autoresearch.figcheck` — it checks the anchor
contract on runs/pending_experiment.json; revise until it prints PASS.
Consecutive conv layers must be tied by kernel-projection lines (small
kernel square on one face, faint lines converging to a cell on the
next) so the encoder reads as one computation, not boxes in a row.
`archive/arch_svg_reference.py` is the concrete reference implementation
of this entire visual language — read it and reuse its helper geometry
for any stage that already has one.
`eli5` and `architecture` feed the human-facing gallery. `architecture.stages`
is the model's inference path left-to-right, camera frame → (lat, lon,
confidence) output — one box per stage, plain-language `detail`,
`"changed": true` ONLY on the stages this experiment touches. Reuse the
previous experiment's stage names verbatim wherever a stage is unchanged
(check `SELECT arch_json FROM experiments WHERE arch_json IS NOT NULL ORDER BY id DESC LIMIT 1;`) —
the diagrams are compared box-by-box across experiments. A change that only
affects training (loss, augmentation, schedule) keeps the inference stages
unchanged and adds one final stage with `"train_only": true` describing the
training signal.
3. **Write the implementation brief.** You do NOT edit `model/` yourself —
a separate implementation agent applies your design, seeing only the
current code plus what you pre-registered. Add one more field to
`runs/pending_experiment.json`:
```
"implementation_brief": "exact file-level instructions: which functions/
blocks in model/model.py and model/train.py change and how, what stays
untouched, and every contract to preserve"
```
Be precise enough that a competent engineer with no other context
implements it in one pass. Always restate the fixed contracts:
`train.py`'s CLI and the ONNX export contract in `model/model.py`'s
docstring — the frozen scorer depends on them.
## Hard rules
- Edit ONLY `runs/pending_experiment.json`. You never edit `model/` (the
implementation stage does) and files listed in `/FROZEN` are off-limits;
the harness hard-reverts any change to them.
- NEVER touch, read, or evaluate the `hamburg` holdout area (§5).
- ONE focused change per iteration — if you can't describe it in one sentence,
it's too big. Prefer architectural/procedural novelty over hyperparameter
nudges (§3): changing a learning rate is a weak experiment; changing the
coordinate parameterization, loss family, relighting-for-training, or model
topology is a strong one.
- Do not run training yourself; the harness does that.
- Stay within the deployment gates: exported ONNX ≤ 4 MiB per area, host
latency proxy ≤ 250 ms (see pipeline/score.py).What the six lighting conditions look like through the simulated low-light sensor (example patches — illustration, not the training set)Every frame in this project is rendered as a starlight-class low-light sensor (Sony STARVIS2 / IMX585 class — the airframe's chosen camera) would see it, not as a normal camera would. That is why the night renders look uncannily close to daylight: at high gain such a sensor recovers scene structure from moonlight, skyglow and artificial lighting, paying for it in noise, lifted shadows and washed-out color — exactly what these renders simulate. Below, one example 256 m patch per area under the six lighting conditions the model must handle. These illustrate the dataset, not this experiment's performance — the actual training set is thousands of distinct crops per area (see “training data” above), and these renderings only change when the relighting method changes. berlin frankfurt munich prignitz | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▸ | 26 | Deployment-envelope capacity scaling: 3× pretrained trunk depth paid for by int8 FC-head storage 4fb65e2a | architecture | pretrained:mobilenet_v3_small IMAGENET1K_V1 features[0..10] (torchvision, BSD-3) | gated fail | 3,019 KB | 4.6 ms | 69 m 41 s | $0.80 | gated fail | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The design under test — technical diagram — gray = frozen contract · ink = the current design · red = this experiment's change One real test, end to end
→
3.54 km
miss, this crop 0.01
self-reported confidence In plain words Last round we gave the model three times longer to study and it finished everything it could hold — yet scores barely moved. That means its brain is full: it isn't failing to study, it's out of room. The drone's flight computer caps the model file at 4 MB and we were already near the cap, but we found that the biggest part of the file — a huge table of connection strengths — works just as well stored as cheap 1-byte numbers instead of exact 4-byte ones (we measured: answers move by ~7 meters on errors of ~700). The space that trick frees pays for a three-times-bigger image-understanding brain, borrowed from the same well-trained library the model already uses. Hypothesis The binding constraint is model capacity, demonstrated by exp 25's own pre-registered fork: convergence-scaled training drove the loss to a genuine plateau (berlin 6.29→4.78, flat for the last 5 epochs at near-zero LR; prignitz 4.51) yet the metric moved only 5.2% (839.12→797.91 m) against the 10–25% predicted if optimization were binding — and train-split and eval-split unfiltered medians are now EQUAL, so the model cannot even fit its training locations. A converged, non-memorizing, underfitting model has exactly one lever left: make it bigger. The only prior capacity experiment (exp 8, reverted) scaled a FROM-SCRATCH encoder in the pre-pretrained era; exp 11 then showed pretrained features are what carry this task, and pretrained capacity has never been scaled since — the trunk has sat at 190k params (MobileNetV3-Small features[:9]) for 15 iterations because the 4 MiB deployment gate appeared full (3.30 MB). Fresh probes (runs/_probe_iter6_quant2.py, _probe_iter6_export.py) break that impasse: int8 dynamic quantization of ONLY the FC heads (which hold 727k of the 823k params) is metrically inert on the current best model — median decode shift 0.001 uv ≈ 7 m against 650–800 m medians, conf shift 0.006, latency fine — while shrinking the export 3.30→1.44 MB. The freed bytes exactly pay for tripling the pretrained trunk: features[:9]→[:11] (190k→576k ImageNet-pretrained params, 48→96 output channels), with block 9's stride-2 depthwise run at stride 1 and block 10's depthwise dilated 2 (the standard DeepLab trick) so the 8×8 feature grid every kept head reads is preserved and all pretrained weights load with unchanged shapes. Dry-run export of the full candidate: 3,091,446 bytes and 4.8 ms single-thread — both frozen gates pass with margin. Method model/model.py: (a) trunk builder loads MobileNetV3-Small features[:11] from a new pretrained snapshot model/pretrained/mnv3s_features10.pt, sets features[9]'s depthwise stride (2,2)→(1,1) and features[10]'s depthwise dilation→2/padding→4 — output 96×8×8; all heads re-width automatically via the existing shape probe. (b) export_onnx exports fp32 to a temp file then writes the final artifact with onnxruntime quantize_dynamic(op_types=['MatMul','Gemm'], QInt8) — conv layers stay fp32. model/train.py: only the init-strategy log string changes. Sampler, losses, decode, calibration, EPOCH_MULT=3, cosine schedule: all untouched; the two new pretrained blocks land in the existing 1e-4 trunk LR group automatically. Expected outcome If trunk capacity is binding, train and eval errors drop TOGETHER (no memorization gap exists to reopen — exp 20's fresh-draw sampler is untouched): predicted primary worst-case median 797.91 → 620–720 m (−10 to −22%), broad across areas and buckets since the underfit is global, with final train losses clearly below exp 25's plateau (berlin ≤ 4.65 vs 4.79, prignitz ≤ 4.37 vs 4.51). Kept iff < 797.91 m. Diagnostic forks if refuted: (a) losses drop ≥0.15 nats but the metric moves <5% — trunk capacity exonerated; the information limit of the 128 m footprint or the head's structure becomes the demonstrated bottleneck; (b) losses stay at the exp-25 plateau despite 3× trunk params — the trunk is not where the underfit lives (head next); (c) metric regresses with healthy losses — check the saved <area>_fp32_debug.onnx against the shipped int8 artifact to separate a quantization surprise from a capacity effect (probe predicts ~7 m, i.e. no effect). Result primary worst-case median error = 1000000000.0 m (previous best 797.91 m); full per-area/bucket breakdown in metrics.json Conclusion REVERTED — metric did not improve (1000000000.0 m vs best 797.91 m); change discarded Scoreboard — median error (m) per area × lighting
the worst cell (underlined) is the experiment's score;
red = failed cell, ink = at target
Fits the aircraft? all deployment gates passed
largest per-area model 3,019 KB (limit 4,096 KB) · single-frame inference 4.6 ms on one CPU thread (proxy limit 250 ms)
crops are sampled fresh each run from the frozen train split (~45,000 distinct positions per area, times random rotation), never from eval blocks; how many to use is the experiment's own choice
Where this iteration's time went: design (Haiku) 18 m 06 s · implement (Sonnet) 1 m 22 s · train 4 areas 47 m 20 s · score 1 m 53 s · samples 43 s · publish 6 s · whole iteration 69 m 51 s
ts 2026-07-21T23:18:46 · commit 4fb65e2a19d4 ·
parent 4fb65e2a19d4 · artifacts runs/20260721_220905_iter6 ·
agent model — · took 69 m 41 s
The exact prompt given to the headless agent# Autoresearch iteration — one focused experiment
You are one iteration of an autonomous research loop for UAV low-light
geolocalization. Read `CLAUDE.md` (§1, §3, §6 especially) for full context.
The harness (loop.sh) will train, score, log, and keep/revert AFTER you exit —
you only design the experiment and edit the code.
## Your job, in order
0. **Skim the library (optional input).** `autoresearch/library.md` holds
the human researcher's inspiration notes. They do not fix your answer
and you are free to ignore them — pick an entry up only when it
genuinely fits your read of the history, and if you build on one, say
so in your hypothesis.
1. **Review the research history.** Query the lineage DB:
`sqlite3 experiments.sqlite "SELECT id, title, category, hypothesis, expected_outcome, result, conclusion, primary_metric, kept FROM experiments ORDER BY id DESC LIMIT 15;"`
Note which hypotheses were supported/refuted. Do not repeat a refuted
experiment without a materially new angle.
**Plateau rule:** if three or more consecutive experiments were reverted,
do not attempt another variation of the last refuted mechanism. Either
pick a design family absent from the history (dispatcher + lighting
specialists, pretrained init, learned relighting, training-scale, …) or
attack the bottleneck the refuted hypotheses jointly point at.
2. **Design ONE focused experiment** — proper experiment design, pre-registered
before you touch code. Write it to `runs/pending_experiment.json`:
```json
{
"title": "one-line name",
"category": "architecture|loss|augmentation|relighting|training|quantization|other",
"hypothesis": "what you believe is limiting the metric and why this change addresses it",
"method": "the ONE focused change, concretely (files, mechanism)",
"expected_outcome": "predicted effect on the §6 worst-case median error, quantified if possible",
"init_strategy": "from-scratch | pretrained:<name>",
"eli5": "2-4 sentences for a smart non-ML reader: what you changed and why it might help, in everyday language — analogies welcome, zero jargon",
"architecture": {"stages": [
{"name": "Camera frame", "detail": "128×128 px crop, one of 6 lighting renders", "changed": false},
{"name": "Feature extractor", "detail": "plain-language description", "changed": false},
{"name": "…", "detail": "…", "changed": true}
]},
"architecture_svg": "<svg viewBox='0 0 980 300' …>…</svg>"
}
```
`architecture_svg` is the **technical architecture diagram** of the model
you are testing — a proper ML-paper figure of the design, drawn by you,
shown at the top of this experiment's gallery entry. **Draw tensors and
operations as the things they are — NOT as labeled boxes:** the input
image as a pixel-textured square; conv feature maps as pseudo-3D slabs
that shrink spatially and deepen in channels; 1-D vectors as thin
vertical tick-bars; fully-connected layers as fans of thin crossing
lines between bars; spatial fields/grids as actually-drawn grids with
shaded cells; decodes/aggregations as thin lines converging from cells
onto a point; samplers/targets as small pictorial glyphs. Captions sit
UNDER each element (name ≤10.5px weight 600, sub-note ≤9.5px); tensor
shapes annotate the flow in small italics (128×128×3 → 8²×128 → 1024).
Losses/targets/samplers live in a lane below the inference path,
annotated with slanted dashed leader lines to small text — no boxes
around them. Draw the two FROZEN endpoints — the camera-frame input and
the (lat, lon, confidence) output — entirely in #9b998c with a small
italic “frozen contract” tag: they are fixed by the harness and outside
your search space. Everything between them is your design: draw it in ink,
with red only on what THIS experiment changes. Style contract, so all experiments' figures read as one
paper: viewBox width 980 (height ~300–360); transparent background; ink
#111111, captions #6b6a60, annotations/arrows #9b998c; **#8c2f1f (red)
reserved for exactly what this experiment changed**; training-only
elements in #8a6a1e (or red if changed); font-family
Palatino,Georgia,serif; stroke-width ≈1.2 elements, 1 arrows; fills only
faint tints (opacity ≤ .12); no gradients, no icons, no emoji. Inference
flows left → right from camera frame to (lat, lon, confidence).
**The camera frame is a fixed glyph — copy it verbatim.** It must look
like an actual nadir frame of terrain (streets, building footprints),
not abstract pixels, and be identical in every figure. Use exactly this
snippet, substituting Y for the frame's top edge (pick Y so the frame
centers on your inference lane; captions go under it as usual — note
128²×3 is the model input crop, ~1 m/px, not the sensor's native
resolution):
```svg
<g id='cam-terrain'><rect id='frozen-input' x='26' y='Y' width='76'
height='76' fill='#f6f4ea' stroke='#9b998c' stroke-width='1.6'/>
<path d='M31 Y+47 L97 Y+27' stroke='#e6e3d4' stroke-width='5' fill='none'/>
<path d='M59 Y+6 L49 Y+70' stroke='#e6e3d4' stroke-width='3.5' fill='none'/>
<rect x='34' y='Y+9' width='12' height='8' fill='#d9d5c3' transform='rotate(-8 40 Y+13)'/>
<rect x='78' y='Y+8' width='10' height='11' fill='#cfccbd'/>
<rect x='35' y='Y+57' width='13' height='8' fill='#d9d5c3'/>
<rect x='75' y='Y+50' width='10' height='9' fill='#cfccbd' transform='rotate(6 80 Y+54)'/>
<rect x='54' y='Y+31' width='9' height='8' fill='#d9d5c3' opacity='.85'/>
<ellipse cx='86' cy='Y+63' rx='8' ry='6' fill='#8a6a1e' opacity='.12'/>
<circle cx='40' cy='Y+28' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='70' cy='Y+14' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='92' cy='Y+36' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='52' cy='Y+52' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='82' cy='Y+70' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='31' cy='Y+42' r='.7' fill='#6b6a60' opacity='.5'/></g>
```
(Y+n means the literal number Y plus n — compute the values. Every glyph
element stays strictly INSIDE the frame border — figcheck rejects a
figure without the `cam-terrain` group.) Draw your
receptive-field square and kernel-projection lines on top of it as
usual.
**Anchor the frozen endpoints identically in every figure** so all
experiments' figures line up when compared down the gallery page: the
camera-frame square starts at x=26 (its captions centered on x=53), and
the output is right-anchored — decode crosshair at x=812, the
(lat, lon, confidence) text block text-anchor=start at x=828. Mark the
anchors machine-checkably: the camera-frame rect carries
`id='frozen-input'`, the output text block carries `id='frozen-output'`.
Lay the elements between them out on a running x-cursor with generous
spacing so nothing overlaps.
**Readability contract** (figcheck verifies this geometrically): no
text may overlap other text, and no line (arrows, converge fans,
leader lines) may pass through a label — a line may END at a label,
never cross one. Rules distilled from review rounds: dashed leaders
run as orthogonal L-routes through empty lanes, never diagonally
across components; converge/vote lines originate at their true
sources (the actual dots or window cells, not generic positions);
every element owns its caption — adjacent columns stagger their
caption rows instead of colliding; prefer moving a label into empty
horizontal space over stacking it against a neighbor. Width is fixed
at 980, but **height is yours**: grow the viewBox (240–640) whenever
more vertical room makes the layout cleaner rather than cramming.
**Before finishing, validate:**
`.venv/bin/python -m autoresearch.figcheck` — it checks the anchor
contract on runs/pending_experiment.json; revise until it prints PASS.
Consecutive conv layers must be tied by kernel-projection lines (small
kernel square on one face, faint lines converging to a cell on the
next) so the encoder reads as one computation, not boxes in a row.
`archive/arch_svg_reference.py` is the concrete reference implementation
of this entire visual language — read it and reuse its helper geometry
for any stage that already has one.
`eli5` and `architecture` feed the human-facing gallery. `architecture.stages`
is the model's inference path left-to-right, camera frame → (lat, lon,
confidence) output — one box per stage, plain-language `detail`,
`"changed": true` ONLY on the stages this experiment touches. Reuse the
previous experiment's stage names verbatim wherever a stage is unchanged
(check `SELECT arch_json FROM experiments WHERE arch_json IS NOT NULL ORDER BY id DESC LIMIT 1;`) —
the diagrams are compared box-by-box across experiments. A change that only
affects training (loss, augmentation, schedule) keeps the inference stages
unchanged and adds one final stage with `"train_only": true` describing the
training signal.
3. **Write the implementation brief.** You do NOT edit `model/` yourself —
a separate implementation agent applies your design, seeing only the
current code plus what you pre-registered. Add one more field to
`runs/pending_experiment.json`:
```
"implementation_brief": "exact file-level instructions: which functions/
blocks in model/model.py and model/train.py change and how, what stays
untouched, and every contract to preserve"
```
Be precise enough that a competent engineer with no other context
implements it in one pass. Always restate the fixed contracts:
`train.py`'s CLI and the ONNX export contract in `model/model.py`'s
docstring — the frozen scorer depends on them.
## Hard rules
- Edit ONLY `runs/pending_experiment.json`. You never edit `model/` (the
implementation stage does) and files listed in `/FROZEN` are off-limits;
the harness hard-reverts any change to them.
- NEVER touch, read, or evaluate the `hamburg` holdout area (§5).
- ONE focused change per iteration — if you can't describe it in one sentence,
it's too big. Prefer architectural/procedural novelty over hyperparameter
nudges (§3): changing a learning rate is a weak experiment; changing the
coordinate parameterization, loss family, relighting-for-training, or model
topology is a strong one.
- Do not run training yourself; the harness does that.
- Stay within the deployment gates: exported ONNX ≤ 4 MiB per area, host
latency proxy ≤ 250 ms (see pipeline/score.py).Where the model was tested — and how far off it was Each map is one full test area. Every dot is one held-out test location (all six lighting conditions overlaid): the model was shown a 128 m crop centered there and asked for its position. Dot color = the distance between its answer and the truth — green ≤ 20 m (at goal), amber ≤ 50 m, red beyond. A working model turns these maps green; spatial clusters of red reveal which parts of an area confuse it.     What the six lighting conditions look like through the simulated low-light sensor (example patches — illustration, not the training set)Every frame in this project is rendered as a starlight-class low-light sensor (Sony STARVIS2 / IMX585 class — the airframe's chosen camera) would see it, not as a normal camera would. That is why the night renders look uncannily close to daylight: at high gain such a sensor recovers scene structure from moonlight, skyglow and artificial lighting, paying for it in noise, lifted shadows and washed-out color — exactly what these renders simulate. Below, one example 256 m patch per area under the six lighting conditions the model must handle. These illustrate the dataset, not this experiment's performance — the actual training set is thousands of distinct crops per area (see “training data” above), and these renderings only change when the relighting method changes. berlin frankfurt munich prignitz | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▸ | 25 | Convergence-scaled training: 3× optimizer steps with cosine LR decay under the kept fresh-draw sampler 7f8a2986 | training | pretrained:mobilenet_v3_small IMAGENET1K_V1 features[0..8] (torchvision, BSD-3) — unchanged | 797.9 m | 3,225 KB | 2.9 ms | 63 m 08 s | $0.73 | kept | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The design under test — technical diagram — gray = frozen contract · ink = the current design · red = this experiment's change One real test, end to end
→
965.6 m
miss, this crop 0.33
self-reported confidence In plain words The previous improvement gave the model fresh flashcards every study pass so it could no longer cram answers — but we kept ringing the end-of-class bell at the same early time, set back when cramming made classes short. Its grades were still climbing when the bell rang. This experiment simply lets it study three times longer, and — like slowing down smoothly before parking — makes each learning step gentler toward the end so what it learned settles in place. The model that flies is completely unchanged; only the length and pacing of school changes. Hypothesis The binding constraint is no longer the held-out-location generalization gap — it is plain underfitting, created as a side effect of kept exp 20 and never addressed because the training schedule has been frozen since bootstrap. Evidence from a fresh read-only probe of the kept exp-20 model (runs/_probe_iter5_evalcells.py): the ~3× train-vs-eval gap that exps 18/19/22/23/24 all attacked is GONE — berlin midday train-split 1014 m vs eval-split 1017 m unfiltered median, prignitz midday 580 vs 673 m (pre-exp-20 probes measured 267–400 m on train). Exp 20's single-exposure fresh draws removed memorization as a fitting strategy, which turned the 8-epoch constant-LR budget — sized in the bootstrap era, when 8 passes over one frozen tensor sufficed to memorize — into a truncation: the training loss is still descending ~0.065/epoch at cutoff (berlin 6.29→5.11 over 8 epochs, no plateau) at a never-decayed Adam LR (head 1e-3, trunk 1e-4), and unfiltered errors are flat-mediocre on BOTH splits (train-split medians 580–1889 m) — the signature of an underconverged model, not a transfer failure. This also reinterprets the three consecutive reverts (22, 23/24): their auxiliary losses competed for the same ~4,500 undertrained optimizer steps, diluting an optimization budget that was already the bottleneck. Per the plateau rule this leaves that mechanism family entirely for training-scale — the one plateau-list family never tried (exp 7 scaled data, never optimization). The same probe also tested and rejected an output-side suspicion: the decode does NOT structurally avoid the never-trained eval-block cells (P(decode in eval block) = 0.13–0.21 vs 0.20 base rate), so a head redesign is not the leading term. The change: 3× the optimizer steps (24 fresh-draw epochs from the harness's --epochs 8) with per-step cosine LR annealing to zero, so the kept sampler is finally trained to convergence — near-full coverage of the ~45k train locations per bucket, each still seen only ~1–2×, and a settled endpoint instead of a noisy constant-LR stop. Method model/train.py ONLY (model/model.py untouched — inference path, losses, decode, calibration, export all identical). (1) Module constant EPOCH_MULT = 3; train_area runs total_epochs = epochs × EPOCH_MULT sampled epochs (harness passes --epochs 8 → 24), with the per-epoch fresh-draw sample_epoch and the epoch_rng = default_rng([seed, epoch]) formula unchanged — epochs 8..23 automatically get fresh independent draws. (2) torch.optim.lr_scheduler.CosineAnnealingLR stepped once per optimizer step, T_max = precomputed total step count (6 buckets × min(max_crops_per_bucket, n_train_crops) crops/epoch, batch 64, × total_epochs), eta_min 0 — both param groups (trunk 1e-4, head 1e-3) anneal proportionally. Per-epoch memory profile is byte-identical to the kept baseline (sample_epoch untouched — the exp-23 OOM lesson); wall time ≈3× (~19 → ~58 min for the 4 parallel areas). Expected outcome If optimization is binding, the primary worst-case median improves 10–25%: 839.12 → ~630–760 m, with final train losses ≥0.25 nats below the epoch-8 values (berlin 5.11, prignitz 4.77) and train-split unfiltered medians dropping well below today's 580–1889 m. Falsification is equally informative: if final losses drop ≥0.25 nats but the metric moves <5%, optimization is exonerated and capacity/architecture becomes the demonstrated next bottleneck — a clean fork for the next iteration. Result primary worst-case median error = 797.91 m (previous best 839.12 m); full per-area/bucket breakdown in metrics.json Conclusion KEPT — metric improved (839.12 -> 797.91 m); change committed as 7f8a298 Scoreboard — median error (m) per area × lighting
the worst cell (underlined) is the experiment's score;
red = failed cell, ink = at target
Fits the aircraft? all deployment gates passed
largest per-area model 3,225 KB (limit 4,096 KB) · single-frame inference 2.9 ms on one CPU thread (proxy limit 250 ms)
crops are sampled fresh each run from the frozen train split (~45,000 distinct positions per area, times random rotation), never from eval blocks; how many to use is the experiment's own choice
Where this iteration's time went: design (Haiku) 12 m 49 s · implement (Sonnet) 41 s · train 4 areas 47 m 02 s · score 1 m 34 s · samples 42 s · publish 5 s · whole iteration 63 m 18 s
ts 2026-07-21T22:07:44 · commit 7f8a298620e4 ·
parent 03463b8b8238 · artifacts runs/20260721_210436_iter5 ·
agent model — · took 63 m 08 s
The exact prompt given to the headless agent# Autoresearch iteration — one focused experiment
You are one iteration of an autonomous research loop for UAV low-light
geolocalization. Read `CLAUDE.md` (§1, §3, §6 especially) for full context.
The harness (loop.sh) will train, score, log, and keep/revert AFTER you exit —
you only design the experiment and edit the code.
## Your job, in order
0. **Skim the library (optional input).** `autoresearch/library.md` holds
the human researcher's inspiration notes. They do not fix your answer
and you are free to ignore them — pick an entry up only when it
genuinely fits your read of the history, and if you build on one, say
so in your hypothesis.
1. **Review the research history.** Query the lineage DB:
`sqlite3 experiments.sqlite "SELECT id, title, category, hypothesis, expected_outcome, result, conclusion, primary_metric, kept FROM experiments ORDER BY id DESC LIMIT 15;"`
Note which hypotheses were supported/refuted. Do not repeat a refuted
experiment without a materially new angle.
**Plateau rule:** if three or more consecutive experiments were reverted,
do not attempt another variation of the last refuted mechanism. Either
pick a design family absent from the history (dispatcher + lighting
specialists, pretrained init, learned relighting, training-scale, …) or
attack the bottleneck the refuted hypotheses jointly point at.
2. **Design ONE focused experiment** — proper experiment design, pre-registered
before you touch code. Write it to `runs/pending_experiment.json`:
```json
{
"title": "one-line name",
"category": "architecture|loss|augmentation|relighting|training|quantization|other",
"hypothesis": "what you believe is limiting the metric and why this change addresses it",
"method": "the ONE focused change, concretely (files, mechanism)",
"expected_outcome": "predicted effect on the §6 worst-case median error, quantified if possible",
"init_strategy": "from-scratch | pretrained:<name>",
"eli5": "2-4 sentences for a smart non-ML reader: what you changed and why it might help, in everyday language — analogies welcome, zero jargon",
"architecture": {"stages": [
{"name": "Camera frame", "detail": "128×128 px crop, one of 6 lighting renders", "changed": false},
{"name": "Feature extractor", "detail": "plain-language description", "changed": false},
{"name": "…", "detail": "…", "changed": true}
]},
"architecture_svg": "<svg viewBox='0 0 980 300' …>…</svg>"
}
```
`architecture_svg` is the **technical architecture diagram** of the model
you are testing — a proper ML-paper figure of the design, drawn by you,
shown at the top of this experiment's gallery entry. **Draw tensors and
operations as the things they are — NOT as labeled boxes:** the input
image as a pixel-textured square; conv feature maps as pseudo-3D slabs
that shrink spatially and deepen in channels; 1-D vectors as thin
vertical tick-bars; fully-connected layers as fans of thin crossing
lines between bars; spatial fields/grids as actually-drawn grids with
shaded cells; decodes/aggregations as thin lines converging from cells
onto a point; samplers/targets as small pictorial glyphs. Captions sit
UNDER each element (name ≤10.5px weight 600, sub-note ≤9.5px); tensor
shapes annotate the flow in small italics (128×128×3 → 8²×128 → 1024).
Losses/targets/samplers live in a lane below the inference path,
annotated with slanted dashed leader lines to small text — no boxes
around them. Draw the two FROZEN endpoints — the camera-frame input and
the (lat, lon, confidence) output — entirely in #9b998c with a small
italic “frozen contract” tag: they are fixed by the harness and outside
your search space. Everything between them is your design: draw it in ink,
with red only on what THIS experiment changes. Style contract, so all experiments' figures read as one
paper: viewBox width 980 (height ~300–360); transparent background; ink
#111111, captions #6b6a60, annotations/arrows #9b998c; **#8c2f1f (red)
reserved for exactly what this experiment changed**; training-only
elements in #8a6a1e (or red if changed); font-family
Palatino,Georgia,serif; stroke-width ≈1.2 elements, 1 arrows; fills only
faint tints (opacity ≤ .12); no gradients, no icons, no emoji. Inference
flows left → right from camera frame to (lat, lon, confidence).
**The camera frame is a fixed glyph — copy it verbatim.** It must look
like an actual nadir frame of terrain (streets, building footprints),
not abstract pixels, and be identical in every figure. Use exactly this
snippet, substituting Y for the frame's top edge (pick Y so the frame
centers on your inference lane; captions go under it as usual — note
128²×3 is the model input crop, ~1 m/px, not the sensor's native
resolution):
```svg
<g id='cam-terrain'><rect id='frozen-input' x='26' y='Y' width='76'
height='76' fill='#f6f4ea' stroke='#9b998c' stroke-width='1.6'/>
<path d='M31 Y+47 L97 Y+27' stroke='#e6e3d4' stroke-width='5' fill='none'/>
<path d='M59 Y+6 L49 Y+70' stroke='#e6e3d4' stroke-width='3.5' fill='none'/>
<rect x='34' y='Y+9' width='12' height='8' fill='#d9d5c3' transform='rotate(-8 40 Y+13)'/>
<rect x='78' y='Y+8' width='10' height='11' fill='#cfccbd'/>
<rect x='35' y='Y+57' width='13' height='8' fill='#d9d5c3'/>
<rect x='75' y='Y+50' width='10' height='9' fill='#cfccbd' transform='rotate(6 80 Y+54)'/>
<rect x='54' y='Y+31' width='9' height='8' fill='#d9d5c3' opacity='.85'/>
<ellipse cx='86' cy='Y+63' rx='8' ry='6' fill='#8a6a1e' opacity='.12'/>
<circle cx='40' cy='Y+28' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='70' cy='Y+14' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='92' cy='Y+36' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='52' cy='Y+52' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='82' cy='Y+70' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='31' cy='Y+42' r='.7' fill='#6b6a60' opacity='.5'/></g>
```
(Y+n means the literal number Y plus n — compute the values. Every glyph
element stays strictly INSIDE the frame border — figcheck rejects a
figure without the `cam-terrain` group.) Draw your
receptive-field square and kernel-projection lines on top of it as
usual.
**Anchor the frozen endpoints identically in every figure** so all
experiments' figures line up when compared down the gallery page: the
camera-frame square starts at x=26 (its captions centered on x=53), and
the output is right-anchored — decode crosshair at x=812, the
(lat, lon, confidence) text block text-anchor=start at x=828. Mark the
anchors machine-checkably: the camera-frame rect carries
`id='frozen-input'`, the output text block carries `id='frozen-output'`.
Lay the elements between them out on a running x-cursor with generous
spacing so nothing overlaps.
**Readability contract** (figcheck verifies this geometrically): no
text may overlap other text, and no line (arrows, converge fans,
leader lines) may pass through a label — a line may END at a label,
never cross one. Rules distilled from review rounds: dashed leaders
run as orthogonal L-routes through empty lanes, never diagonally
across components; converge/vote lines originate at their true
sources (the actual dots or window cells, not generic positions);
every element owns its caption — adjacent columns stagger their
caption rows instead of colliding; prefer moving a label into empty
horizontal space over stacking it against a neighbor. Width is fixed
at 980, but **height is yours**: grow the viewBox (240–640) whenever
more vertical room makes the layout cleaner rather than cramming.
**Before finishing, validate:**
`.venv/bin/python -m autoresearch.figcheck` — it checks the anchor
contract on runs/pending_experiment.json; revise until it prints PASS.
Consecutive conv layers must be tied by kernel-projection lines (small
kernel square on one face, faint lines converging to a cell on the
next) so the encoder reads as one computation, not boxes in a row.
`archive/arch_svg_reference.py` is the concrete reference implementation
of this entire visual language — read it and reuse its helper geometry
for any stage that already has one.
`eli5` and `architecture` feed the human-facing gallery. `architecture.stages`
is the model's inference path left-to-right, camera frame → (lat, lon,
confidence) output — one box per stage, plain-language `detail`,
`"changed": true` ONLY on the stages this experiment touches. Reuse the
previous experiment's stage names verbatim wherever a stage is unchanged
(check `SELECT arch_json FROM experiments WHERE arch_json IS NOT NULL ORDER BY id DESC LIMIT 1;`) —
the diagrams are compared box-by-box across experiments. A change that only
affects training (loss, augmentation, schedule) keeps the inference stages
unchanged and adds one final stage with `"train_only": true` describing the
training signal.
3. **Write the implementation brief.** You do NOT edit `model/` yourself —
a separate implementation agent applies your design, seeing only the
current code plus what you pre-registered. Add one more field to
`runs/pending_experiment.json`:
```
"implementation_brief": "exact file-level instructions: which functions/
blocks in model/model.py and model/train.py change and how, what stays
untouched, and every contract to preserve"
```
Be precise enough that a competent engineer with no other context
implements it in one pass. Always restate the fixed contracts:
`train.py`'s CLI and the ONNX export contract in `model/model.py`'s
docstring — the frozen scorer depends on them.
## Hard rules
- Edit ONLY `runs/pending_experiment.json`. You never edit `model/` (the
implementation stage does) and files listed in `/FROZEN` are off-limits;
the harness hard-reverts any change to them.
- NEVER touch, read, or evaluate the `hamburg` holdout area (§5).
- ONE focused change per iteration — if you can't describe it in one sentence,
it's too big. Prefer architectural/procedural novelty over hyperparameter
nudges (§3): changing a learning rate is a weak experiment; changing the
coordinate parameterization, loss family, relighting-for-training, or model
topology is a strong one.
- Do not run training yourself; the harness does that.
- Stay within the deployment gates: exported ONNX ≤ 4 MiB per area, host
latency proxy ≤ 250 ms (see pipeline/score.py).Where the model was tested — and how far off it was Each map is one full test area. Every dot is one held-out test location (all six lighting conditions overlaid): the model was shown a 128 m crop centered there and asked for its position. Dot color = the distance between its answer and the truth — green ≤ 20 m (at goal), amber ≤ 50 m, red beyond. A working model turns these maps green; spatial clusters of red reveal which parts of an area confuse it.     What the six lighting conditions look like through the simulated low-light sensor (example patches — illustration, not the training set)Every frame in this project is rendered as a starlight-class low-light sensor (Sony STARVIS2 / IMX585 class — the airframe's chosen camera) would see it, not as a normal camera would. That is why the night renders look uncannily close to daylight: at high gain such a sensor recovers scene structure from moonlight, skyglow and artificial lighting, paying for it in noise, lifted shadows and washed-out color — exactly what these renders simulate. Below, one example 256 m patch per area under the six lighting conditions the model must handle. These illustrate the dataset, not this experiment's performance — the actual training set is thousands of distinct crops per area (see “training data” above), and these renderings only change when the relighting method changes. berlin frankfurt munich prignitz | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▸ | 24 | Daytime-redraw auxiliary, rerun: exp 23 was OOM-killed mid-harness, never scored — retest it with a memory-lean epoch sampler 5794e840 | loss | pretrained:mobilenet_v3_small IMAGENET1K_V1 features[0..8] (torchvision, BSD-3) | 915.7 m | 3,225 KB | 2.7 ms | 42 m 59 s | $0.49 | discarded | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The design under test — technical diagram — gray = frozen contract · ink = the current design · red = this experiment's change One real test, end to end
→
916.4 m
miss, this crop 0.43
self-reported confidence In plain words Last iteration we designed a promising exercise for the network: from its internal impression of a dark, noisy night photo, it must also sketch what that exact spot looks like in clean daylight — like a student proving they truly saw through the darkness by redrawing the street in daytime colors. The sketching module is thrown away before anything flies, so the aircraft carries nothing extra. But the experiment never actually ran: the training computer ran out of memory and the operating system silently killed one of the four training jobs, so the scoreboard recorded a bogus worst-possible number. This iteration re-runs the exact same exercise after fixing the memory waste (the training data was being copied twice for no reason) — now the whole thing uses LESS memory than the setup that has worked for weeks, and the idea finally gets its fair test. Hypothesis Exp 23's hypothesis was never tested, and this iteration retests it rather than abandoning an untried mechanism. The forensic record is unambiguous: loop.sh line 203 logged 'Killed' (SIGKILL) for the berlin training process, berlin's log is 0 bytes with no model exported, while prignitz/munich/frankfurt trained to completion under the identical code with healthy final losses (4.78-5.05, vs 4.77-5.11 in the kept exp-20 run) and normal calibration keep-rates — so the 1e9 in the DB is the scorer's missing-model sentinel, not a measured error. The recorded REVERTED at 1e9 is an infrastructure artifact, and leaving it unchallenged would poison the history: future iterations would read the reconstruction-auxiliary family as refuted when it was merely crashed. Cause of death: the four areas train IN PARALLEL inside a constrained cgroup (loop.sh's own comment cites the 'small cgroup quota'), and the kept sampler already peaks at ~3.6 GB transient per process (a 36k-crop uint8 list ~1.77 GB plus its np.stack copy ~1.77 GB); exp 23's implementation added a per-process reference raster (~0.14 GB) and a 36k x 64x64x3 uint8 target list plus ITS stack copy (~0.9 GB), pushing four concurrent processes over the quota — the kernel OOM-killer shot the first process (berlin), which freed enough memory for the remaining three to finish, exactly the observed pattern. A SIGKILLed python loses its ~1 KB of block-buffered prints, which is why the log is empty. The scientific hypothesis is unchanged from exp 23 (library L1, flavor B): the binding constraint is the held-out-location generalization gap (train-split 267-400 m vs eval-split 708-1153 m median, ~3x, every bucket), the trunk still passes lighting nuisance into the descriptor because its ImageNet features were never adapted to the sim's night statistics, and a training-only decoder that must redraw the clean daytime reference crop (pixel-aligned pairs free from the frozen sim; same cx/cy/angle) from the encoder's 8x8x48 feature grid can only succeed via features that cancel lighting and noise while preserving ground structure — canonicalization inside the encoder at zero inference cost, location-generic by construction (unlike reverted exp 22's per-patch place labels) and structure-preserving (unlike refuted exp 9's collapsible contrastive objective). The one new engineering fact this rerun adds: the epoch sampler is rewritten to preallocate its uint8 tensors and fill them in place, eliminating the list+stack doubling — peak sampler RSS drops to ~2.4 GB per process INCLUDING the new target tensor and reference raster, ~1.2 GB BELOW the kept baseline that has survived 20+ iterations, so the rerun sits inside a proven memory envelope with margin. The preallocation is numerically inert: identical RNG stream, byte-identical training tensors. Method ONE scientific change, identical to exp 23's pre-registration: attach a training-only reconstruction decoder (~55k params; 1x1 conv 48->64, then 3x [nearest-upsample x2 + 3x3 conv + ReLU] 64->48->32->24, final 1x1 conv ->3 + sigmoid; 8x8x48 in, 64x64x3 out) to the identity-turn feature map via a new return_feat flag on TinyLocNet.forward; add LAMBDA_RECON=0.3 * L1(decoder(feat), daytime_target_64px) to the loss; decoder params train at head LR 1e-3 and are discarded before export. Plus the OOM fix that lets it actually run: model/train.py's sample_epoch is rewritten to preallocate np.empty uint8 arrays for crops and daytime targets and fill them by running index (no list-of-arrays, no np.stack copy), loading the daytime reference raster once per run and reusing the already-drawn per-crop angle for the pixel-aligned target (zero new RNG draws). Both epoch prints gain flush=True so a future SIGKILL can no longer leave an empty log. Exported graph, decode, confidence, calibration, sampling distribution, epochs, and LRs are all byte-identical to the current best. Expected outcome Same pre-registration as exp 23: worst-case median error improves from 839.12 m to ~700-790 m (-6% to -16%), gains spread broadly across buckets and slightly larger in evening/night where the input is farthest from the daytime reference. Kept iff < 839.12 m. Refuted — for real this time — if all four areas train to completion and the metric lands at or above 839 m, which would mean the aux diverts trunk capacity the way exp 22's place-supervision aux did. Distinguishing signature to check in train logs: all four areas MUST print 8 epochs and export ONNX; if any log is again truncated with no exported model, the result is infrastructural, not scientific, and the mechanism remains untested (the flush=True prints will now show exactly where a kill lands). Memory audit: per-process peak during sampling drops ~1.2 GB below the kept baseline, so four parallel trainings sit ~5 GB below the configuration that has succeeded 20+ times and ~8 GB below the one that was killed. Result primary worst-case median error = 915.68 m (previous best 839.12 m); full per-area/bucket breakdown in metrics.json Conclusion REVERTED — metric did not improve (915.68 m vs best 839.12 m); change discarded Scoreboard — median error (m) per area × lighting
the worst cell (underlined) is the experiment's score;
red = failed cell, ink = at target
Fits the aircraft? all deployment gates passed
largest per-area model 3,225 KB (limit 4,096 KB) · single-frame inference 2.7 ms on one CPU thread (proxy limit 250 ms)
crops are sampled fresh each run from the frozen train split (~45,000 distinct positions per area, times random rotation), never from eval blocks; how many to use is the experiment's own choice
Where this iteration's time went: design (Haiku) 8 m 49 s · implement (Sonnet) 2 m 26 s · train 4 areas 29 m 02 s · score 1 m 34 s · samples 42 s · publish 6 s · whole iteration 43 m 13 s
ts 2026-07-21T21:02:57 · commit 5794e8405c98 ·
parent 5794e8405c98 · artifacts runs/20260721_201958_iter4 ·
agent model — · took 42 m 59 s
The exact prompt given to the headless agent# Autoresearch iteration — one focused experiment
You are one iteration of an autonomous research loop for UAV low-light
geolocalization. Read `CLAUDE.md` (§1, §3, §6 especially) for full context.
The harness (loop.sh) will train, score, log, and keep/revert AFTER you exit —
you only design the experiment and edit the code.
## Your job, in order
0. **Skim the library (optional input).** `autoresearch/library.md` holds
the human researcher's inspiration notes. They do not fix your answer
and you are free to ignore them — pick an entry up only when it
genuinely fits your read of the history, and if you build on one, say
so in your hypothesis.
1. **Review the research history.** Query the lineage DB:
`sqlite3 experiments.sqlite "SELECT id, title, category, hypothesis, expected_outcome, result, conclusion, primary_metric, kept FROM experiments ORDER BY id DESC LIMIT 15;"`
Note which hypotheses were supported/refuted. Do not repeat a refuted
experiment without a materially new angle.
**Plateau rule:** if three or more consecutive experiments were reverted,
do not attempt another variation of the last refuted mechanism. Either
pick a design family absent from the history (dispatcher + lighting
specialists, pretrained init, learned relighting, training-scale, …) or
attack the bottleneck the refuted hypotheses jointly point at.
2. **Design ONE focused experiment** — proper experiment design, pre-registered
before you touch code. Write it to `runs/pending_experiment.json`:
```json
{
"title": "one-line name",
"category": "architecture|loss|augmentation|relighting|training|quantization|other",
"hypothesis": "what you believe is limiting the metric and why this change addresses it",
"method": "the ONE focused change, concretely (files, mechanism)",
"expected_outcome": "predicted effect on the §6 worst-case median error, quantified if possible",
"init_strategy": "from-scratch | pretrained:<name>",
"eli5": "2-4 sentences for a smart non-ML reader: what you changed and why it might help, in everyday language — analogies welcome, zero jargon",
"architecture": {"stages": [
{"name": "Camera frame", "detail": "128×128 px crop, one of 6 lighting renders", "changed": false},
{"name": "Feature extractor", "detail": "plain-language description", "changed": false},
{"name": "…", "detail": "…", "changed": true}
]},
"architecture_svg": "<svg viewBox='0 0 980 300' …>…</svg>"
}
```
`architecture_svg` is the **technical architecture diagram** of the model
you are testing — a proper ML-paper figure of the design, drawn by you,
shown at the top of this experiment's gallery entry. **Draw tensors and
operations as the things they are — NOT as labeled boxes:** the input
image as a pixel-textured square; conv feature maps as pseudo-3D slabs
that shrink spatially and deepen in channels; 1-D vectors as thin
vertical tick-bars; fully-connected layers as fans of thin crossing
lines between bars; spatial fields/grids as actually-drawn grids with
shaded cells; decodes/aggregations as thin lines converging from cells
onto a point; samplers/targets as small pictorial glyphs. Captions sit
UNDER each element (name ≤10.5px weight 600, sub-note ≤9.5px); tensor
shapes annotate the flow in small italics (128×128×3 → 8²×128 → 1024).
Losses/targets/samplers live in a lane below the inference path,
annotated with slanted dashed leader lines to small text — no boxes
around them. Draw the two FROZEN endpoints — the camera-frame input and
the (lat, lon, confidence) output — entirely in #9b998c with a small
italic “frozen contract” tag: they are fixed by the harness and outside
your search space. Everything between them is your design: draw it in ink,
with red only on what THIS experiment changes. Style contract, so all experiments' figures read as one
paper: viewBox width 980 (height ~300–360); transparent background; ink
#111111, captions #6b6a60, annotations/arrows #9b998c; **#8c2f1f (red)
reserved for exactly what this experiment changed**; training-only
elements in #8a6a1e (or red if changed); font-family
Palatino,Georgia,serif; stroke-width ≈1.2 elements, 1 arrows; fills only
faint tints (opacity ≤ .12); no gradients, no icons, no emoji. Inference
flows left → right from camera frame to (lat, lon, confidence).
**The camera frame is a fixed glyph — copy it verbatim.** It must look
like an actual nadir frame of terrain (streets, building footprints),
not abstract pixels, and be identical in every figure. Use exactly this
snippet, substituting Y for the frame's top edge (pick Y so the frame
centers on your inference lane; captions go under it as usual — note
128²×3 is the model input crop, ~1 m/px, not the sensor's native
resolution):
```svg
<g id='cam-terrain'><rect id='frozen-input' x='26' y='Y' width='76'
height='76' fill='#f6f4ea' stroke='#9b998c' stroke-width='1.6'/>
<path d='M31 Y+47 L97 Y+27' stroke='#e6e3d4' stroke-width='5' fill='none'/>
<path d='M59 Y+6 L49 Y+70' stroke='#e6e3d4' stroke-width='3.5' fill='none'/>
<rect x='34' y='Y+9' width='12' height='8' fill='#d9d5c3' transform='rotate(-8 40 Y+13)'/>
<rect x='78' y='Y+8' width='10' height='11' fill='#cfccbd'/>
<rect x='35' y='Y+57' width='13' height='8' fill='#d9d5c3'/>
<rect x='75' y='Y+50' width='10' height='9' fill='#cfccbd' transform='rotate(6 80 Y+54)'/>
<rect x='54' y='Y+31' width='9' height='8' fill='#d9d5c3' opacity='.85'/>
<ellipse cx='86' cy='Y+63' rx='8' ry='6' fill='#8a6a1e' opacity='.12'/>
<circle cx='40' cy='Y+28' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='70' cy='Y+14' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='92' cy='Y+36' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='52' cy='Y+52' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='82' cy='Y+70' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='31' cy='Y+42' r='.7' fill='#6b6a60' opacity='.5'/></g>
```
(Y+n means the literal number Y plus n — compute the values. Every glyph
element stays strictly INSIDE the frame border — figcheck rejects a
figure without the `cam-terrain` group.) Draw your
receptive-field square and kernel-projection lines on top of it as
usual.
**Anchor the frozen endpoints identically in every figure** so all
experiments' figures line up when compared down the gallery page: the
camera-frame square starts at x=26 (its captions centered on x=53), and
the output is right-anchored — decode crosshair at x=812, the
(lat, lon, confidence) text block text-anchor=start at x=828. Mark the
anchors machine-checkably: the camera-frame rect carries
`id='frozen-input'`, the output text block carries `id='frozen-output'`.
Lay the elements between them out on a running x-cursor with generous
spacing so nothing overlaps.
**Readability contract** (figcheck verifies this geometrically): no
text may overlap other text, and no line (arrows, converge fans,
leader lines) may pass through a label — a line may END at a label,
never cross one. Rules distilled from review rounds: dashed leaders
run as orthogonal L-routes through empty lanes, never diagonally
across components; converge/vote lines originate at their true
sources (the actual dots or window cells, not generic positions);
every element owns its caption — adjacent columns stagger their
caption rows instead of colliding; prefer moving a label into empty
horizontal space over stacking it against a neighbor. Width is fixed
at 980, but **height is yours**: grow the viewBox (240–640) whenever
more vertical room makes the layout cleaner rather than cramming.
**Before finishing, validate:**
`.venv/bin/python -m autoresearch.figcheck` — it checks the anchor
contract on runs/pending_experiment.json; revise until it prints PASS.
Consecutive conv layers must be tied by kernel-projection lines (small
kernel square on one face, faint lines converging to a cell on the
next) so the encoder reads as one computation, not boxes in a row.
`archive/arch_svg_reference.py` is the concrete reference implementation
of this entire visual language — read it and reuse its helper geometry
for any stage that already has one.
`eli5` and `architecture` feed the human-facing gallery. `architecture.stages`
is the model's inference path left-to-right, camera frame → (lat, lon,
confidence) output — one box per stage, plain-language `detail`,
`"changed": true` ONLY on the stages this experiment touches. Reuse the
previous experiment's stage names verbatim wherever a stage is unchanged
(check `SELECT arch_json FROM experiments WHERE arch_json IS NOT NULL ORDER BY id DESC LIMIT 1;`) —
the diagrams are compared box-by-box across experiments. A change that only
affects training (loss, augmentation, schedule) keeps the inference stages
unchanged and adds one final stage with `"train_only": true` describing the
training signal.
3. **Write the implementation brief.** You do NOT edit `model/` yourself —
a separate implementation agent applies your design, seeing only the
current code plus what you pre-registered. Add one more field to
`runs/pending_experiment.json`:
```
"implementation_brief": "exact file-level instructions: which functions/
blocks in model/model.py and model/train.py change and how, what stays
untouched, and every contract to preserve"
```
Be precise enough that a competent engineer with no other context
implements it in one pass. Always restate the fixed contracts:
`train.py`'s CLI and the ONNX export contract in `model/model.py`'s
docstring — the frozen scorer depends on them.
## Hard rules
- Edit ONLY `runs/pending_experiment.json`. You never edit `model/` (the
implementation stage does) and files listed in `/FROZEN` are off-limits;
the harness hard-reverts any change to them.
- NEVER touch, read, or evaluate the `hamburg` holdout area (§5).
- ONE focused change per iteration — if you can't describe it in one sentence,
it's too big. Prefer architectural/procedural novelty over hyperparameter
nudges (§3): changing a learning rate is a weak experiment; changing the
coordinate parameterization, loss family, relighting-for-training, or model
topology is a strong one.
- Do not run training yourself; the harness does that.
- Stay within the deployment gates: exported ONNX ≤ 4 MiB per area, host
latency proxy ≤ 250 ms (see pipeline/score.py).Where the model was tested — and how far off it was Each map is one full test area. Every dot is one held-out test location (all six lighting conditions overlaid): the model was shown a 128 m crop centered there and asked for its position. Dot color = the distance between its answer and the truth — green ≤ 20 m (at goal), amber ≤ 50 m, red beyond. A working model turns these maps green; spatial clusters of red reveal which parts of an area confuse it.     What the six lighting conditions look like through the simulated low-light sensor (example patches — illustration, not the training set)Every frame in this project is rendered as a starlight-class low-light sensor (Sony STARVIS2 / IMX585 class — the airframe's chosen camera) would see it, not as a normal camera would. That is why the night renders look uncannily close to daylight: at high gain such a sensor recovers scene structure from moonlight, skyglow and artificial lighting, paying for it in noise, lifted shadows and washed-out color — exactly what these renders simulate. Below, one example 256 m patch per area under the six lighting conditions the model must handle. These illustrate the dataset, not this experiment's performance — the actual training set is thousands of distinct crops per area (see “training data” above), and these renderings only change when the relighting method changes. berlin frankfurt munich prignitz | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▸ | 23 | Daytime-redraw auxiliary: a throwaway decoder must reconstruct the clean daytime crop from the encoder's features adfc8b91 | loss | pretrained:mobilenet_v3_small IMAGENET1K_V1 features[0..8] (torchvision, BSD-3) | gated fail | — | — | 42 m 08 s | $0.48 | gated fail | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The design under test — technical diagram — gray = frozen contract · ink = the current design · red = this experiment's change In plain words The network learns to recognize places from noisy night photos. We gave it one extra practice exercise: from its internal impression of a scene, it must also sketch what that exact spot looks like in clean daylight — like a student proving they truly saw through the darkness by redrawing the street in daytime colors. The sketching module is thrown away before anything flies, so the aircraft carries nothing extra. What remains is a network whose mental image of the ground ignores darkness and sensor grain — and if its inner picture is always the daytime one, recognizing the place gets easier in every lighting condition at once. Hypothesis The binding constraint is still the held-out-location generalization gap the exp-18 probe measured (train-split crops 267-400 m median vs eval-split 708-1153 m, ~3x, in every bucket), and after exp 20 the error profile is flat across lighting buckets and areas — flat but mediocre (best 839.12 m vs the ~20 m target). The trunk is the shared weak link that flatness points at: every gradient it receives comes from ONE whole-crop place grade per crop at a gentle 1e-4 LR, while its ImageNet-pretrained features were never adapted to the sim's night-time statistics (sensor gain noise, lamp thinning, washed-out color are far outside ImageNet's input distribution). So lighting nuisance still reaches the descriptor, and the 1024-way head must generalize jointly across location x lighting with only ~30k single-exposure examples (exp 20) to do it. This experiment builds on library entry L1, flavor B (training-only reconstruction auxiliary): a small throwaway decoder must redraw the clean daytime reference crop — same location, same heading, pixel-aligned pairs free from the frozen sim — from the encoder's 8x8x48 feature grid, with an L1 loss at 64x64 weighted 0.3, then be deleted before export. All six lighting renders of a place share ONE daytime answer, so the cheapest way down this loss is features that cancel lighting and noise while preserving ground structure: canonicalization inside the encoder, at zero inference cost. Two mechanisms should move the Sec-6 metric: (a) descriptor variance across buckets collapses, so the head's capacity and exp 20's fresh location draws concentrate on place discrimination alone instead of place x lighting; (b) unlike reverted exp 22, whose dense auxiliary was per-patch PLACE classification — 64 more memorization targets of exactly the kind the transfer gap punishes — night-to-day redrawing is location-generic (the same inversion applies everywhere on the map, and under exp 20's sampling no location is seen often enough to memorize its output), so its gradients transfer to never-trained eval terrain by construction. It is also materially different from refuted exp 9 (cross-lighting NT-Xent): that was a contrastive ranking objective on the GLOBAL descriptor of the old from-scratch encoder, satisfiable by collapsing information; a dense generative target must PRESERVE structure per pixel, and it acts on today's pretrained trunk. Method ONE change: attach a training-only reconstruction decoder to the encoder. model/model.py gains ReconDecoder (~55k params: 1x1 conv 48->64 then 3x [nearest-upsample x2 + 3x3 conv + ReLU] 64->48->32->24, final 1x1 conv ->3 + sigmoid; 8x8x48 in, 64x64x3 out), build_recon_decoder(), constants RECON_SIZE=64 / LAMBDA_RECON=0.3, and an optional return_feat flag on TinyLocNet.forward exposing the identity-turn feature map. model/train.py loads the daytime reference raster once per area, samples for every training crop the pixel-aligned clean reference crop (same cx/cy/angle) downsampled to 64x64, and adds LAMBDA_RECON * L1(decoder(feat), target) to the existing loss. Decoder params train at the head LR (1e-3) and are discarded — never exported. Everything else (architecture, decode, confidence, calibration, sampling, epochs, LRs, ONNX export) is untouched; the exported graph is byte-identical in shape to the current best. Expected outcome Worst-case median error improves from 839.12 m to ~700-790 m (-6% to -16%), with the gain spread broadly across buckets (slightly larger in evening/night, where the input is farthest from the daytime reference). Kept iff < 839.12 m. Refuted if the aux diverts trunk capacity the way exp 22's place-supervision aux did and the metric lands at or above 839 m. Result primary worst-case median error = 1000000000.0 m (previous best 839.12 m); full per-area/bucket breakdown in metrics.json Conclusion REVERTED — metric did not improve (1000000000.0 m vs best 839.12 m); change discarded Scoreboard — median error (m) per area × lighting
the worst cell (underlined) is the experiment's score;
red = failed cell, ink = at target
No per-area results recorded for this run. Fits the aircraft? all deployment gates passed
largest per-area model — (limit 4,096 KB) · single-frame inference — on one CPU thread (proxy limit 250 ms) Where this iteration's time went: design (Haiku) 10 m 35 s · implement (Sonnet) 1 m 53 s · train 4 areas 28 m 34 s · samples 42 s · whole iteration 42 m 12 s
ts 2026-07-21T20:18:28 · commit adfc8b919d19 ·
parent adfc8b919d19 · artifacts runs/20260721_193619_iter3 ·
agent model — · took 42 m 08 s
The exact prompt given to the headless agent# Autoresearch iteration — one focused experiment
You are one iteration of an autonomous research loop for UAV low-light
geolocalization. Read `CLAUDE.md` (§1, §3, §6 especially) for full context.
The harness (loop.sh) will train, score, log, and keep/revert AFTER you exit —
you only design the experiment and edit the code.
## Your job, in order
0. **Skim the library (optional input).** `autoresearch/library.md` holds
the human researcher's inspiration notes. They do not fix your answer
and you are free to ignore them — pick an entry up only when it
genuinely fits your read of the history, and if you build on one, say
so in your hypothesis.
1. **Review the research history.** Query the lineage DB:
`sqlite3 experiments.sqlite "SELECT id, title, category, hypothesis, expected_outcome, result, conclusion, primary_metric, kept FROM experiments ORDER BY id DESC LIMIT 15;"`
Note which hypotheses were supported/refuted. Do not repeat a refuted
experiment without a materially new angle.
**Plateau rule:** if three or more consecutive experiments were reverted,
do not attempt another variation of the last refuted mechanism. Either
pick a design family absent from the history (dispatcher + lighting
specialists, pretrained init, learned relighting, training-scale, …) or
attack the bottleneck the refuted hypotheses jointly point at.
2. **Design ONE focused experiment** — proper experiment design, pre-registered
before you touch code. Write it to `runs/pending_experiment.json`:
```json
{
"title": "one-line name",
"category": "architecture|loss|augmentation|relighting|training|quantization|other",
"hypothesis": "what you believe is limiting the metric and why this change addresses it",
"method": "the ONE focused change, concretely (files, mechanism)",
"expected_outcome": "predicted effect on the §6 worst-case median error, quantified if possible",
"init_strategy": "from-scratch | pretrained:<name>",
"eli5": "2-4 sentences for a smart non-ML reader: what you changed and why it might help, in everyday language — analogies welcome, zero jargon",
"architecture": {"stages": [
{"name": "Camera frame", "detail": "128×128 px crop, one of 6 lighting renders", "changed": false},
{"name": "Feature extractor", "detail": "plain-language description", "changed": false},
{"name": "…", "detail": "…", "changed": true}
]},
"architecture_svg": "<svg viewBox='0 0 980 300' …>…</svg>"
}
```
`architecture_svg` is the **technical architecture diagram** of the model
you are testing — a proper ML-paper figure of the design, drawn by you,
shown at the top of this experiment's gallery entry. **Draw tensors and
operations as the things they are — NOT as labeled boxes:** the input
image as a pixel-textured square; conv feature maps as pseudo-3D slabs
that shrink spatially and deepen in channels; 1-D vectors as thin
vertical tick-bars; fully-connected layers as fans of thin crossing
lines between bars; spatial fields/grids as actually-drawn grids with
shaded cells; decodes/aggregations as thin lines converging from cells
onto a point; samplers/targets as small pictorial glyphs. Captions sit
UNDER each element (name ≤10.5px weight 600, sub-note ≤9.5px); tensor
shapes annotate the flow in small italics (128×128×3 → 8²×128 → 1024).
Losses/targets/samplers live in a lane below the inference path,
annotated with slanted dashed leader lines to small text — no boxes
around them. Draw the two FROZEN endpoints — the camera-frame input and
the (lat, lon, confidence) output — entirely in #9b998c with a small
italic “frozen contract” tag: they are fixed by the harness and outside
your search space. Everything between them is your design: draw it in ink,
with red only on what THIS experiment changes. Style contract, so all experiments' figures read as one
paper: viewBox width 980 (height ~300–360); transparent background; ink
#111111, captions #6b6a60, annotations/arrows #9b998c; **#8c2f1f (red)
reserved for exactly what this experiment changed**; training-only
elements in #8a6a1e (or red if changed); font-family
Palatino,Georgia,serif; stroke-width ≈1.2 elements, 1 arrows; fills only
faint tints (opacity ≤ .12); no gradients, no icons, no emoji. Inference
flows left → right from camera frame to (lat, lon, confidence).
**The camera frame is a fixed glyph — copy it verbatim.** It must look
like an actual nadir frame of terrain (streets, building footprints),
not abstract pixels, and be identical in every figure. Use exactly this
snippet, substituting Y for the frame's top edge (pick Y so the frame
centers on your inference lane; captions go under it as usual — note
128²×3 is the model input crop, ~1 m/px, not the sensor's native
resolution):
```svg
<g id='cam-terrain'><rect id='frozen-input' x='26' y='Y' width='76'
height='76' fill='#f6f4ea' stroke='#9b998c' stroke-width='1.6'/>
<path d='M31 Y+47 L97 Y+27' stroke='#e6e3d4' stroke-width='5' fill='none'/>
<path d='M59 Y+6 L49 Y+70' stroke='#e6e3d4' stroke-width='3.5' fill='none'/>
<rect x='34' y='Y+9' width='12' height='8' fill='#d9d5c3' transform='rotate(-8 40 Y+13)'/>
<rect x='78' y='Y+8' width='10' height='11' fill='#cfccbd'/>
<rect x='35' y='Y+57' width='13' height='8' fill='#d9d5c3'/>
<rect x='75' y='Y+50' width='10' height='9' fill='#cfccbd' transform='rotate(6 80 Y+54)'/>
<rect x='54' y='Y+31' width='9' height='8' fill='#d9d5c3' opacity='.85'/>
<ellipse cx='86' cy='Y+63' rx='8' ry='6' fill='#8a6a1e' opacity='.12'/>
<circle cx='40' cy='Y+28' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='70' cy='Y+14' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='92' cy='Y+36' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='52' cy='Y+52' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='82' cy='Y+70' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='31' cy='Y+42' r='.7' fill='#6b6a60' opacity='.5'/></g>
```
(Y+n means the literal number Y plus n — compute the values. Every glyph
element stays strictly INSIDE the frame border — figcheck rejects a
figure without the `cam-terrain` group.) Draw your
receptive-field square and kernel-projection lines on top of it as
usual.
**Anchor the frozen endpoints identically in every figure** so all
experiments' figures line up when compared down the gallery page: the
camera-frame square starts at x=26 (its captions centered on x=53), and
the output is right-anchored — decode crosshair at x=812, the
(lat, lon, confidence) text block text-anchor=start at x=828. Mark the
anchors machine-checkably: the camera-frame rect carries
`id='frozen-input'`, the output text block carries `id='frozen-output'`.
Lay the elements between them out on a running x-cursor with generous
spacing so nothing overlaps.
**Readability contract** (figcheck verifies this geometrically): no
text may overlap other text, and no line (arrows, converge fans,
leader lines) may pass through a label — a line may END at a label,
never cross one. Rules distilled from review rounds: dashed leaders
run as orthogonal L-routes through empty lanes, never diagonally
across components; converge/vote lines originate at their true
sources (the actual dots or window cells, not generic positions);
every element owns its caption — adjacent columns stagger their
caption rows instead of colliding; prefer moving a label into empty
horizontal space over stacking it against a neighbor. Width is fixed
at 980, but **height is yours**: grow the viewBox (240–640) whenever
more vertical room makes the layout cleaner rather than cramming.
**Before finishing, validate:**
`.venv/bin/python -m autoresearch.figcheck` — it checks the anchor
contract on runs/pending_experiment.json; revise until it prints PASS.
Consecutive conv layers must be tied by kernel-projection lines (small
kernel square on one face, faint lines converging to a cell on the
next) so the encoder reads as one computation, not boxes in a row.
`archive/arch_svg_reference.py` is the concrete reference implementation
of this entire visual language — read it and reuse its helper geometry
for any stage that already has one.
`eli5` and `architecture` feed the human-facing gallery. `architecture.stages`
is the model's inference path left-to-right, camera frame → (lat, lon,
confidence) output — one box per stage, plain-language `detail`,
`"changed": true` ONLY on the stages this experiment touches. Reuse the
previous experiment's stage names verbatim wherever a stage is unchanged
(check `SELECT arch_json FROM experiments WHERE arch_json IS NOT NULL ORDER BY id DESC LIMIT 1;`) —
the diagrams are compared box-by-box across experiments. A change that only
affects training (loss, augmentation, schedule) keeps the inference stages
unchanged and adds one final stage with `"train_only": true` describing the
training signal.
3. **Write the implementation brief.** You do NOT edit `model/` yourself —
a separate implementation agent applies your design, seeing only the
current code plus what you pre-registered. Add one more field to
`runs/pending_experiment.json`:
```
"implementation_brief": "exact file-level instructions: which functions/
blocks in model/model.py and model/train.py change and how, what stays
untouched, and every contract to preserve"
```
Be precise enough that a competent engineer with no other context
implements it in one pass. Always restate the fixed contracts:
`train.py`'s CLI and the ONNX export contract in `model/model.py`'s
docstring — the frozen scorer depends on them.
## Hard rules
- Edit ONLY `runs/pending_experiment.json`. You never edit `model/` (the
implementation stage does) and files listed in `/FROZEN` are off-limits;
the harness hard-reverts any change to them.
- NEVER touch, read, or evaluate the `hamburg` holdout area (§5).
- ONE focused change per iteration — if you can't describe it in one sentence,
it's too big. Prefer architectural/procedural novelty over hyperparameter
nudges (§3): changing a learning rate is a weak experiment; changing the
coordinate parameterization, loss family, relighting-for-training, or model
topology is a strong one.
- Do not run training yourself; the harness does that.
- Stay within the deployment gates: exported ONNX ≤ 4 MiB per area, host
latency proxy ≤ 250 ms (see pipeline/score.py).What the six lighting conditions look like through the simulated low-light sensor (example patches — illustration, not the training set)Every frame in this project is rendered as a starlight-class low-light sensor (Sony STARVIS2 / IMX585 class — the airframe's chosen camera) would see it, not as a normal camera would. That is why the night renders look uncannily close to daylight: at high gain such a sensor recovers scene structure from moonlight, skyglow and artificial lighting, paying for it in noise, lifted shadows and washed-out color — exactly what these renders simulate. Below, one example 256 m patch per area under the six lighting conditions the model must handle. These illustrate the dataset, not this experiment's performance — the actual training set is thousands of distinct crops per area (see “training data” above), and these renderings only change when the relighting method changes. berlin frankfurt munich prignitz | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▸ | 22 | Train-only per-patch place supervision: every feature cell places its own patch, the flying decode is untouched 9df5c839 | loss | pretrained:mobilenet_v3_small IMAGENET1K_V1 features[0..8] (torchvision, BSD-3) — unchanged | 1.13 km | 3,225 KB | 2.7 ms | 40 m 44 s | $0.47 | discarded | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The design under test — technical diagram — gray = frozen contract · ink = the current design · red = this experiment's change One real test, end to end
→
1.08 km
miss, this crop 0.21
self-reported confidence In plain words Until now the model got one graded question per practice photo: 'where is the CENTER of this photo on the map?' A student can pass that test by memorizing whole photos -- which fails on parts of the city held back from practice. Now, during practice only, every small tile of the photo also gets its own graded question: 'where does THIS tile sit on the map?' -- 64 grades per photo instead of one. To score well the network must learn what roads, buildings and field edges actually look like from above, knowledge that transfers to never-practiced places. On the real flight nothing changes: the extra quiz machinery is thrown away before takeoff and the aircraft carries exactly the same network as before. Hypothesis The binding constraint is still the held-out-location generalization gap the exp-18 probe measured (train-split crops 267-400 m median vs eval-split 708-1153 m, ~3x, in every bucket), and after exp 20 the error profile is flat across lighting and areas (metros 656-839 m, prignitz 449-539 m; night is no longer the worst bucket anywhere) -- lighting is solved as a differentiator, transfer to unseen terrain is the whole game. Exp 20 removed repetition-driven whole-crop memorization on the DATA side (kept, 1054.87 -> 839.12 m). The remaining untried side is the SUPERVISION density: the trunk's features are trained through exactly one grade per crop -- a single whole-crop Gaussian-CE at the crop centre -- so nothing ever demands that each local feature column be location-predictive on its own; the cheapest fit is whole-crop gestalt, which the probe shows failing to transfer. Exps 4 and 18 both made per-patch localization the INFERENCE mechanism and were reverted; their joint lesson (recorded in exp 19's design) is that the whole-crop arrangement signal must stay intact at decode time -- but neither ever tested per-patch supervision as a training-only regularizer under the kept whole-crop decode, which is standard deep-supervision practice and is exactly the configuration their failures leave open. A dense auxiliary head (one shared 1x1 conv 48->1024 on the identity-turn feature grid) that makes each of the 64 feature columns predict its OWN Gaussian-CE map field at its patch's true ground position multiplies the supervision 64x per crop, and -- combined with exp 20's single-exposure sampling, where a location is seen only ~1-2 times -- can only be satisfied by local cues that transfer between locations. At eval, ~76% of eval crops contain SOME train-seen terrain at a known offset (exp 19's geometry analysis); locally location-predictive feature columns put that partial familiarity into the 8x8 layout code the kept head already reads spatially -- delivering exp 18's intended benefit through the decode that works, instead of the per-patch fusion that failed. Method model/model.py: add aux_patch_head = nn.Conv2d(48, GRID_K*GRID_K, 1) and a forward_train() that returns (out, logits, aux_logits) where aux_logits [N,1024,8,8] is the aux head applied to the identity-rotation slice of the trunk feature map; forward() and the ONNX export are byte-for-byte unchanged (the aux head is not traced, so exported size stays 3.30 MiB). Add aux_patch_loss(): per-patch Gaussian-CE (same GRID_K/TARGET_SIGMA_CELLS formula as loss_fn) against each patch's true map uv. model/train.py: sample_epoch additionally returns per-crop patch targets [N,64,2] computed as (cx,cy) + R(angle) @ d for each feature cell's crop-plane offset d = ((j+0.5)*16-64, (i+0.5)*16-64) px, with the numerically verified rotation R(angle) = [[cos,-sin],[sin,cos]] (angle in degrees exactly as passed to extract_crop; verified against extract_crop at 0/90/30/217 deg). Training loss becomes loss_fn(...) + 0.5 * aux_patch_loss(...). Nothing else changes: same epochs, sampling, calibration, export. Expected outcome If dense per-patch supervision closes even a quarter of the ~3x train/eval generalization gap, the worst-case median drops from 839.12 m to roughly 700-760 m. Pre-registered prediction: primary metric <= 780 m, with the improvement broad-based across the three metro areas (the top ~10 cells all sit at 656-839 m) rather than concentrated in one bucket; coverage roughly unchanged (calibration path untouched). Failure mode to watch: the aux gradient distorting the trunk against the whole-crop objective -- weight 0.5 on the patch-mean CE keeps the main loss dominant; if the metric worsens, the conclusion is that trunk features cannot serve both objectives at this capacity, closing the deep-supervision branch of the exp-4/18 family for good. Result primary worst-case median error = 1125.53 m (previous best 839.12 m); full per-area/bucket breakdown in metrics.json Conclusion REVERTED — metric did not improve (1125.53 m vs best 839.12 m); change discarded Scoreboard — median error (m) per area × lighting
the worst cell (underlined) is the experiment's score;
red = failed cell, ink = at target
Fits the aircraft? all deployment gates passed
largest per-area model 3,225 KB (limit 4,096 KB) · single-frame inference 2.7 ms on one CPU thread (proxy limit 250 ms)
crops are sampled fresh each run from the frozen train split (~45,000 distinct positions per area, times random rotation), never from eval blocks; how many to use is the experiment's own choice
Where this iteration's time went: design (Haiku) 12 m 58 s · implement (Sonnet) 4 m 58 s · train 4 areas 20 m 06 s · score 1 m 35 s · samples 42 s · publish 5 s · whole iteration 40 m 53 s
ts 2026-07-21T19:34:42 · commit 9df5c83968e9 ·
parent 9df5c83968e9 · artifacts runs/20260721_185358_iter2 ·
agent model — · took 40 m 44 s
The exact prompt given to the headless agent# Autoresearch iteration — one focused experiment
You are one iteration of an autonomous research loop for UAV low-light
geolocalization. Read `CLAUDE.md` (§1, §3, §6 especially) for full context.
The harness (loop.sh) will train, score, log, and keep/revert AFTER you exit —
you only design the experiment and edit the code.
## Your job, in order
1. **Review the research history.** Query the lineage DB:
`sqlite3 experiments.sqlite "SELECT id, title, category, hypothesis, expected_outcome, result, conclusion, primary_metric, kept FROM experiments ORDER BY id DESC LIMIT 15;"`
Note which hypotheses were supported/refuted. Do not repeat a refuted
experiment without a materially new angle.
**Plateau rule:** if three or more consecutive experiments were reverted,
do not attempt another variation of the last refuted mechanism. Either
pick a design family absent from the history (dispatcher + lighting
specialists, pretrained init, learned relighting, training-scale, …) or
attack the bottleneck the refuted hypotheses jointly point at.
2. **Design ONE focused experiment** — proper experiment design, pre-registered
before you touch code. Write it to `runs/pending_experiment.json`:
```json
{
"title": "one-line name",
"category": "architecture|loss|augmentation|relighting|training|quantization|other",
"hypothesis": "what you believe is limiting the metric and why this change addresses it",
"method": "the ONE focused change, concretely (files, mechanism)",
"expected_outcome": "predicted effect on the §6 worst-case median error, quantified if possible",
"init_strategy": "from-scratch | pretrained:<name>",
"eli5": "2-4 sentences for a smart non-ML reader: what you changed and why it might help, in everyday language — analogies welcome, zero jargon",
"architecture": {"stages": [
{"name": "Camera frame", "detail": "128×128 px crop, one of 6 lighting renders", "changed": false},
{"name": "Feature extractor", "detail": "plain-language description", "changed": false},
{"name": "…", "detail": "…", "changed": true}
]},
"architecture_svg": "<svg viewBox='0 0 980 300' …>…</svg>"
}
```
`architecture_svg` is the **technical architecture diagram** of the model
you are testing — a proper ML-paper figure of the design, drawn by you,
shown at the top of this experiment's gallery entry. **Draw tensors and
operations as the things they are — NOT as labeled boxes:** the input
image as a pixel-textured square; conv feature maps as pseudo-3D slabs
that shrink spatially and deepen in channels; 1-D vectors as thin
vertical tick-bars; fully-connected layers as fans of thin crossing
lines between bars; spatial fields/grids as actually-drawn grids with
shaded cells; decodes/aggregations as thin lines converging from cells
onto a point; samplers/targets as small pictorial glyphs. Captions sit
UNDER each element (name ≤10.5px weight 600, sub-note ≤9.5px); tensor
shapes annotate the flow in small italics (128×128×3 → 8²×128 → 1024).
Losses/targets/samplers live in a lane below the inference path,
annotated with slanted dashed leader lines to small text — no boxes
around them. Draw the two FROZEN endpoints — the camera-frame input and
the (lat, lon, confidence) output — entirely in #9b998c with a small
italic “frozen contract” tag: they are fixed by the harness and outside
your search space. Everything between them is your design: draw it in ink,
with red only on what THIS experiment changes. Style contract, so all experiments' figures read as one
paper: viewBox width 980 (height ~300–360); transparent background; ink
#111111, captions #6b6a60, annotations/arrows #9b998c; **#8c2f1f (red)
reserved for exactly what this experiment changed**; training-only
elements in #8a6a1e (or red if changed); font-family
Palatino,Georgia,serif; stroke-width ≈1.2 elements, 1 arrows; fills only
faint tints (opacity ≤ .12); no gradients, no icons, no emoji. Inference
flows left → right from camera frame to (lat, lon, confidence).
**The camera frame is a fixed glyph — copy it verbatim.** It must look
like an actual nadir frame of terrain (streets, building footprints),
not abstract pixels, and be identical in every figure. Use exactly this
snippet, substituting Y for the frame's top edge (pick Y so the frame
centers on your inference lane; captions go under it as usual — note
128²×3 is the model input crop, ~1 m/px, not the sensor's native
resolution):
```svg
<g id='cam-terrain'><rect id='frozen-input' x='26' y='Y' width='76'
height='76' fill='#f6f4ea' stroke='#9b998c' stroke-width='1.6'/>
<path d='M31 Y+47 L97 Y+27' stroke='#e6e3d4' stroke-width='5' fill='none'/>
<path d='M59 Y+6 L49 Y+70' stroke='#e6e3d4' stroke-width='3.5' fill='none'/>
<rect x='34' y='Y+9' width='12' height='8' fill='#d9d5c3' transform='rotate(-8 40 Y+13)'/>
<rect x='78' y='Y+8' width='10' height='11' fill='#cfccbd'/>
<rect x='35' y='Y+57' width='13' height='8' fill='#d9d5c3'/>
<rect x='75' y='Y+50' width='10' height='9' fill='#cfccbd' transform='rotate(6 80 Y+54)'/>
<rect x='54' y='Y+31' width='9' height='8' fill='#d9d5c3' opacity='.85'/>
<ellipse cx='86' cy='Y+63' rx='8' ry='6' fill='#8a6a1e' opacity='.12'/>
<circle cx='40' cy='Y+28' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='70' cy='Y+14' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='92' cy='Y+36' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='52' cy='Y+52' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='82' cy='Y+70' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='31' cy='Y+42' r='.7' fill='#6b6a60' opacity='.5'/></g>
```
(Y+n means the literal number Y plus n — compute the values. Every glyph
element stays strictly INSIDE the frame border — figcheck rejects a
figure without the `cam-terrain` group.) Draw your
receptive-field square and kernel-projection lines on top of it as
usual.
**Anchor the frozen endpoints identically in every figure** so all
experiments' figures line up when compared down the gallery page: the
camera-frame square starts at x=26 (its captions centered on x=53), and
the output is right-anchored — decode crosshair at x=812, the
(lat, lon, confidence) text block text-anchor=start at x=828. Mark the
anchors machine-checkably: the camera-frame rect carries
`id='frozen-input'`, the output text block carries `id='frozen-output'`.
Lay the elements between them out on a running x-cursor with generous
spacing so nothing overlaps.
**Readability contract** (figcheck verifies this geometrically): no
text may overlap other text, and no line (arrows, converge fans,
leader lines) may pass through a label — a line may END at a label,
never cross one. Rules distilled from review rounds: dashed leaders
run as orthogonal L-routes through empty lanes, never diagonally
across components; converge/vote lines originate at their true
sources (the actual dots or window cells, not generic positions);
every element owns its caption — adjacent columns stagger their
caption rows instead of colliding; prefer moving a label into empty
horizontal space over stacking it against a neighbor. Width is fixed
at 980, but **height is yours**: grow the viewBox (240–640) whenever
more vertical room makes the layout cleaner rather than cramming.
**Before finishing, validate:**
`.venv/bin/python -m autoresearch.figcheck` — it checks the anchor
contract on runs/pending_experiment.json; revise until it prints PASS.
Consecutive conv layers must be tied by kernel-projection lines (small
kernel square on one face, faint lines converging to a cell on the
next) so the encoder reads as one computation, not boxes in a row.
`archive/arch_svg_reference.py` is the concrete reference implementation
of this entire visual language — read it and reuse its helper geometry
for any stage that already has one.
`eli5` and `architecture` feed the human-facing gallery. `architecture.stages`
is the model's inference path left-to-right, camera frame → (lat, lon,
confidence) output — one box per stage, plain-language `detail`,
`"changed": true` ONLY on the stages this experiment touches. Reuse the
previous experiment's stage names verbatim wherever a stage is unchanged
(check `SELECT arch_json FROM experiments WHERE arch_json IS NOT NULL ORDER BY id DESC LIMIT 1;`) —
the diagrams are compared box-by-box across experiments. A change that only
affects training (loss, augmentation, schedule) keeps the inference stages
unchanged and adds one final stage with `"train_only": true` describing the
training signal.
3. **Write the implementation brief.** You do NOT edit `model/` yourself —
a separate implementation agent applies your design, seeing only the
current code plus what you pre-registered. Add one more field to
`runs/pending_experiment.json`:
```
"implementation_brief": "exact file-level instructions: which functions/
blocks in model/model.py and model/train.py change and how, what stays
untouched, and every contract to preserve"
```
Be precise enough that a competent engineer with no other context
implements it in one pass. Always restate the fixed contracts:
`train.py`'s CLI and the ONNX export contract in `model/model.py`'s
docstring — the frozen scorer depends on them.
## Hard rules
- Edit ONLY `runs/pending_experiment.json`. You never edit `model/` (the
implementation stage does) and files listed in `/FROZEN` are off-limits;
the harness hard-reverts any change to them.
- NEVER touch, read, or evaluate the `hamburg` holdout area (§5).
- ONE focused change per iteration — if you can't describe it in one sentence,
it's too big. Prefer architectural/procedural novelty over hyperparameter
nudges (§3): changing a learning rate is a weak experiment; changing the
coordinate parameterization, loss family, relighting-for-training, or model
topology is a strong one.
- Do not run training yourself; the harness does that.
- Stay within the deployment gates: exported ONNX ≤ 4 MiB per area, host
latency proxy ≤ 250 ms (see pipeline/score.py).Where the model was tested — and how far off it was Each map is one full test area. Every dot is one held-out test location (all six lighting conditions overlaid): the model was shown a 128 m crop centered there and asked for its position. Dot color = the distance between its answer and the truth — green ≤ 20 m (at goal), amber ≤ 50 m, red beyond. A working model turns these maps green; spatial clusters of red reveal which parts of an area confuse it.     What the six lighting conditions look like through the simulated low-light sensor (example patches — illustration, not the training set)Every frame in this project is rendered as a starlight-class low-light sensor (Sony STARVIS2 / IMX585 class — the airframe's chosen camera) would see it, not as a normal camera would. That is why the night renders look uncannily close to daylight: at high gain such a sensor recovers scene structure from moonlight, skyglow and artificial lighting, paying for it in noise, lifted shadows and washed-out color — exactly what these renders simulate. Below, one example 256 m patch per area under the six lighting conditions the model must handle. These illustrate the dataset, not this experiment's performance — the actual training set is thousands of distinct crops per area (see “training data” above), and these renderings only change when the relighting method changes. berlin frankfurt munich prignitz | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▸ | 21 | Periodic blind holdout check (hamburg) cac6ec61 | other | — | 714.0 m | 3,225 KB | 2.7 ms | — | — | holdout | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hypothesis If the pipeline is genuinely generic, holdout error should track the development worst-case. Method Train current best model code on hamburg, score read-only (§5). Result informational only — never drives keep/revert Scoreboard — median error (m) per area × lighting
the worst cell (underlined) is the experiment's score;
red = failed cell, ink = at target
Fits the aircraft? all deployment gates passed
largest per-area model 3,225 KB (limit 4,096 KB) · single-frame inference 2.7 ms on one CPU thread (proxy limit 250 ms)
crops are sampled fresh each run from the frozen train split (~45,000 distinct positions per area, times random rotation), never from eval blocks; how many to use is the experiment's own choice
ts 2026-07-21T18:51:51 · commit cac6ec613fa9 ·
parent b8696cb516ce · artifacts runs/20260721_175707_iter1/holdout ·
agent model — · took —
Where the model was tested — and how far off it was Each map is one full test area. Every dot is one held-out test location (all six lighting conditions overlaid): the model was shown a 128 m crop centered there and asked for its position. Dot color = the distance between its answer and the truth — green ≤ 20 m (at goal), amber ≤ 50 m, red beyond. A working model turns these maps green; spatial clusters of red reveal which parts of an area confuse it.  | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▸ | 20 | Per-epoch training-set resampling: a fresh 6,000-location draw per bucket every epoch replaces the one frozen 36k-crop tensor cac6ec61 | training | pretrained:mobilenet_v3_small IMAGENET1K_V1 features[0..8] (torchvision, BSD-3) -- unchanged from exp 11 | 839.1 m | 3,225 KB | 2.8 ms | 35 m 33 s | $0.41 | kept | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The design under test — technical diagram — gray = frozen contract · ink = the current design · red = this experiment's change One real test, end to end
→
1.33 km
miss, this crop 0.27
self-reported confidence In plain words Until now the model studied the exact same 36,000 flashcards eight times over, so it started remembering individual cards instead of learning to read terrain. Cards it had memorized it could place well, but over ground it had never studied -- which is exactly what the test uses -- it was three times worse. Now it gets a brand-new deck drawn from the whole training area on every pass: same study time, eight times more distinct places, each seen only once or twice. Remembering single cards stops working, so the only strategy that pays off is learning what actually makes a place recognizable -- the skill it needs over terrain it has never seen. Hypothesis The binding constraint is the ~3x held-out-location gap the exp-18 probe measured directly: train-split crops localize to 267-400 m median while eval-split crops sit at 708-1153 m, in every lighting bucket (the per-bucket error profile of the kept model is now flat, so lighting is no longer the leading term). Exps 18 and 19 both attacked this gap by making the model handle partially-familiar crops -- one architecturally (64 per-patch votes; discarded the global arrangement signal, 1135.85 m) and one by corrupting training content (off-site distractor patches; made training strictly harder, 1241.24 m) -- and both were reverted. Jointly they say: the whole-crop arrangement signal must stay intact and the training content clean. The third, untouched axis is repetition. The kept trainer builds ONE tensor -- 6,000 locations per bucket, each at one frozen heading and one frozen relight realization -- and passes over it 8 times. Eight identical passes over 6k fixed templates per bucket, with a 573k-param 1024-way FC head on top, make per-crop memorization the cheapest way down the training loss, and memorized templates are exactly what the probe shows failing to transfer; meanwhile the other ~39k enumerable train locations per bucket are never seen at all. Redrawing the 6,000 locations, their headings, and their realization assignment fresh at the start of every epoch keeps per-epoch cost and total gradient-step count identical while raising distinct locations seen to ~30k per bucket (1-(1-0.133)^8 ~= 68% of ~45k), each seen only ~1-2 times and never twice identically (different heading and often different simulator roll on a repeat). Single-exposure data removes template memorization as an available fitting strategy: only cues that transfer between locations can reduce the loss -- precisely the capability an eval crop over never-trained terrain demands. This is materially distinct from refuted exp 6, which resampled only the ROTATIONS of the same 800 locations on the from-scratch encoder, pre-trunk and pre-C4-vote: heading invariance has been architectural since exp 16; what varies here, and what exp 6 never varied, is location coverage. Method model/model.py is untouched. model/train.py only: (1) a one-time prepare_realizations step renders each bucket's two extra relight realizations with the exact seeds exp 17 introduced and caches them as PNGs under <out-dir>/renders/; (2) the single load_training_tensors call before the epoch loop is replaced by a per-epoch sample_epoch(...) that redraws locations, headings, and realization assignment with a fresh rng seeded [seed, epoch]; the minibatch loop, optimizer/param groups, loss, confidence calibration, and ONNX export are byte-identical. Expected outcome If the memorization diagnosis is right, training loss ends slightly higher but eval medians drop in every area x bucket cell; predicted section-6 worst-case median 1054.87 -> ~850-950 m (10-20%). Failure signature: training loss stays visibly high and the metric regresses to >= 1100 m, meaning one exposure per location is too weak for the 1024-way head to bind places at all (the exp-6-like outcome). Result primary worst-case median error = 839.12 m (previous best 1054.87 m); full per-area/bucket breakdown in metrics.json Conclusion KEPT — metric improved (1054.87 -> 839.12 m); change committed as cac6ec6 Scoreboard — median error (m) per area × lighting
the worst cell (underlined) is the experiment's score;
red = failed cell, ink = at target
Fits the aircraft? all deployment gates passed
largest per-area model 3,225 KB (limit 4,096 KB) · single-frame inference 2.8 ms on one CPU thread (proxy limit 250 ms)
crops are sampled fresh each run from the frozen train split (~45,000 distinct positions per area, times random rotation), never from eval blocks; how many to use is the experiment's own choice
Where this iteration's time went: design (Haiku) 12 m 36 s · implement (Sonnet) 59 s · train 4 areas 19 m 21 s · score 1 m 36 s · samples 42 s · holdout 19 m 11 s · publish 6 s · whole iteration 54 m 54 s
ts 2026-07-21T18:32:40 · commit cac6ec613fa9 ·
parent b8696cb516ce · artifacts runs/20260721_175707_iter1 ·
agent model — · took 35 m 33 s
The exact prompt given to the headless agent# Autoresearch iteration — one focused experiment
You are one iteration of an autonomous research loop for UAV low-light
geolocalization. Read `CLAUDE.md` (§1, §3, §6 especially) for full context.
The harness (loop.sh) will train, score, log, and keep/revert AFTER you exit —
you only design the experiment and edit the code.
## Your job, in order
1. **Review the research history.** Query the lineage DB:
`sqlite3 experiments.sqlite "SELECT id, title, category, hypothesis, expected_outcome, result, conclusion, primary_metric, kept FROM experiments ORDER BY id DESC LIMIT 15;"`
Note which hypotheses were supported/refuted. Do not repeat a refuted
experiment without a materially new angle.
**Plateau rule:** if three or more consecutive experiments were reverted,
do not attempt another variation of the last refuted mechanism. Either
pick a design family absent from the history (dispatcher + lighting
specialists, pretrained init, learned relighting, training-scale, …) or
attack the bottleneck the refuted hypotheses jointly point at.
2. **Design ONE focused experiment** — proper experiment design, pre-registered
before you touch code. Write it to `runs/pending_experiment.json`:
```json
{
"title": "one-line name",
"category": "architecture|loss|augmentation|relighting|training|quantization|other",
"hypothesis": "what you believe is limiting the metric and why this change addresses it",
"method": "the ONE focused change, concretely (files, mechanism)",
"expected_outcome": "predicted effect on the §6 worst-case median error, quantified if possible",
"init_strategy": "from-scratch | pretrained:<name>",
"eli5": "2-4 sentences for a smart non-ML reader: what you changed and why it might help, in everyday language — analogies welcome, zero jargon",
"architecture": {"stages": [
{"name": "Camera frame", "detail": "128×128 px crop, one of 6 lighting renders", "changed": false},
{"name": "Feature extractor", "detail": "plain-language description", "changed": false},
{"name": "…", "detail": "…", "changed": true}
]},
"architecture_svg": "<svg viewBox='0 0 980 300' …>…</svg>"
}
```
`architecture_svg` is the **technical architecture diagram** of the model
you are testing — a proper ML-paper figure of the design, drawn by you,
shown at the top of this experiment's gallery entry. **Draw tensors and
operations as the things they are — NOT as labeled boxes:** the input
image as a pixel-textured square; conv feature maps as pseudo-3D slabs
that shrink spatially and deepen in channels; 1-D vectors as thin
vertical tick-bars; fully-connected layers as fans of thin crossing
lines between bars; spatial fields/grids as actually-drawn grids with
shaded cells; decodes/aggregations as thin lines converging from cells
onto a point; samplers/targets as small pictorial glyphs. Captions sit
UNDER each element (name ≤10.5px weight 600, sub-note ≤9.5px); tensor
shapes annotate the flow in small italics (128×128×3 → 8²×128 → 1024).
Losses/targets/samplers live in a lane below the inference path,
annotated with slanted dashed leader lines to small text — no boxes
around them. Draw the two FROZEN endpoints — the camera-frame input and
the (lat, lon, confidence) output — entirely in #9b998c with a small
italic “frozen contract” tag: they are fixed by the harness and outside
your search space. Everything between them is your design: draw it in ink,
with red only on what THIS experiment changes. Style contract, so all experiments' figures read as one
paper: viewBox width 980 (height ~300–360); transparent background; ink
#111111, captions #6b6a60, annotations/arrows #9b998c; **#8c2f1f (red)
reserved for exactly what this experiment changed**; training-only
elements in #8a6a1e (or red if changed); font-family
Palatino,Georgia,serif; stroke-width ≈1.2 elements, 1 arrows; fills only
faint tints (opacity ≤ .12); no gradients, no icons, no emoji. Inference
flows left → right from camera frame to (lat, lon, confidence).
**The camera frame is a fixed glyph — copy it verbatim.** It must look
like an actual nadir frame of terrain (streets, building footprints),
not abstract pixels, and be identical in every figure. Use exactly this
snippet, substituting Y for the frame's top edge (pick Y so the frame
centers on your inference lane; captions go under it as usual — note
128²×3 is the model input crop, ~1 m/px, not the sensor's native
resolution):
```svg
<g id='cam-terrain'><rect id='frozen-input' x='26' y='Y' width='76'
height='76' fill='#f6f4ea' stroke='#9b998c' stroke-width='1.6'/>
<path d='M31 Y+47 L97 Y+27' stroke='#e6e3d4' stroke-width='5' fill='none'/>
<path d='M59 Y+6 L49 Y+70' stroke='#e6e3d4' stroke-width='3.5' fill='none'/>
<rect x='34' y='Y+9' width='12' height='8' fill='#d9d5c3' transform='rotate(-8 40 Y+13)'/>
<rect x='78' y='Y+8' width='10' height='11' fill='#cfccbd'/>
<rect x='35' y='Y+57' width='13' height='8' fill='#d9d5c3'/>
<rect x='75' y='Y+50' width='10' height='9' fill='#cfccbd' transform='rotate(6 80 Y+54)'/>
<rect x='54' y='Y+31' width='9' height='8' fill='#d9d5c3' opacity='.85'/>
<ellipse cx='86' cy='Y+63' rx='8' ry='6' fill='#8a6a1e' opacity='.12'/>
<circle cx='40' cy='Y+28' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='70' cy='Y+14' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='92' cy='Y+36' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='52' cy='Y+52' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='82' cy='Y+70' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='31' cy='Y+42' r='.7' fill='#6b6a60' opacity='.5'/></g>
```
(Y+n means the literal number Y plus n — compute the values. Every glyph
element stays strictly INSIDE the frame border — figcheck rejects a
figure without the `cam-terrain` group.) Draw your
receptive-field square and kernel-projection lines on top of it as
usual.
**Anchor the frozen endpoints identically in every figure** so all
experiments' figures line up when compared down the gallery page: the
camera-frame square starts at x=26 (its captions centered on x=53), and
the output is right-anchored — decode crosshair at x=812, the
(lat, lon, confidence) text block text-anchor=start at x=828. Mark the
anchors machine-checkably: the camera-frame rect carries
`id='frozen-input'`, the output text block carries `id='frozen-output'`.
Lay the elements between them out on a running x-cursor with generous
spacing so nothing overlaps.
**Readability contract** (figcheck verifies this geometrically): no
text may overlap other text, and no line (arrows, converge fans,
leader lines) may pass through a label — a line may END at a label,
never cross one. Rules distilled from review rounds: dashed leaders
run as orthogonal L-routes through empty lanes, never diagonally
across components; converge/vote lines originate at their true
sources (the actual dots or window cells, not generic positions);
every element owns its caption — adjacent columns stagger their
caption rows instead of colliding; prefer moving a label into empty
horizontal space over stacking it against a neighbor. Width is fixed
at 980, but **height is yours**: grow the viewBox (240–640) whenever
more vertical room makes the layout cleaner rather than cramming.
**Before finishing, validate:**
`.venv/bin/python -m autoresearch.figcheck` — it checks the anchor
contract on runs/pending_experiment.json; revise until it prints PASS.
Consecutive conv layers must be tied by kernel-projection lines (small
kernel square on one face, faint lines converging to a cell on the
next) so the encoder reads as one computation, not boxes in a row.
`archive/arch_svg_reference.py` is the concrete reference implementation
of this entire visual language — read it and reuse its helper geometry
for any stage that already has one.
`eli5` and `architecture` feed the human-facing gallery. `architecture.stages`
is the model's inference path left-to-right, camera frame → (lat, lon,
confidence) output — one box per stage, plain-language `detail`,
`"changed": true` ONLY on the stages this experiment touches. Reuse the
previous experiment's stage names verbatim wherever a stage is unchanged
(check `SELECT arch_json FROM experiments WHERE arch_json IS NOT NULL ORDER BY id DESC LIMIT 1;`) —
the diagrams are compared box-by-box across experiments. A change that only
affects training (loss, augmentation, schedule) keeps the inference stages
unchanged and adds one final stage with `"train_only": true` describing the
training signal.
3. **Write the implementation brief.** You do NOT edit `model/` yourself —
a separate implementation agent applies your design, seeing only the
current code plus what you pre-registered. Add one more field to
`runs/pending_experiment.json`:
```
"implementation_brief": "exact file-level instructions: which functions/
blocks in model/model.py and model/train.py change and how, what stays
untouched, and every contract to preserve"
```
Be precise enough that a competent engineer with no other context
implements it in one pass. Always restate the fixed contracts:
`train.py`'s CLI and the ONNX export contract in `model/model.py`'s
docstring — the frozen scorer depends on them.
## Hard rules
- Edit ONLY `runs/pending_experiment.json`. You never edit `model/` (the
implementation stage does) and files listed in `/FROZEN` are off-limits;
the harness hard-reverts any change to them.
- NEVER touch, read, or evaluate the `hamburg` holdout area (§5).
- ONE focused change per iteration — if you can't describe it in one sentence,
it's too big. Prefer architectural/procedural novelty over hyperparameter
nudges (§3): changing a learning rate is a weak experiment; changing the
coordinate parameterization, loss family, relighting-for-training, or model
topology is a strong one.
- Do not run training yourself; the harness does that.
- Stay within the deployment gates: exported ONNX ≤ 4 MiB per area, host
latency proxy ≤ 250 ms (see pipeline/score.py).Where the model was tested — and how far off it was Each map is one full test area. Every dot is one held-out test location (all six lighting conditions overlaid): the model was shown a 128 m crop centered there and asked for its position. Dot color = the distance between its answer and the truth — green ≤ 20 m (at goal), amber ≤ 50 m, red beyond. A working model turns these maps green; spatial clusters of red reveal which parts of an area confuse it.     What the six lighting conditions look like through the simulated low-light sensor (example patches — illustration, not the training set)Every frame in this project is rendered as a starlight-class low-light sensor (Sony STARVIS2 / IMX585 class — the airframe's chosen camera) would see it, not as a normal camera would. That is why the night renders look uncannily close to daylight: at high gain such a sensor recovers scene structure from moonlight, skyglow and artificial lighting, paying for it in noise, lifted shadows and washed-out color — exactly what these renders simulate. Below, one example 256 m patch per area under the six lighting conditions the model must handle. These illustrate the dataset, not this experiment's performance — the actual training set is thousands of distinct crops per area (see “training data” above), and these renderings only change when the relighting method changes. berlin frankfurt munich prignitz | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▸ | 19 | Off-site distractor patching: half the training crops carry a pasted block of terrain from elsewhere, labels unchanged d5080d5d | augmentation | pretrained:mobilenet_v3_small IMAGENET1K_V1 features[0..8] (torchvision, BSD-3) — unchanged from exp 11 | 1.24 km | 3,225 KB | 2.7 ms | 42 m 51 s | $0.49 | discarded | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The design under test — technical diagram — gray = frozen contract · ink = the current design · red = this experiment's change One real test, end to end
→
641.7 m
miss, this crop 0.12
self-reported confidence In plain words When the drone flies over ground the model never studied, most of the picture is new to it -- but a corner usually shows land it does know. Until now, every practice picture the model ever saw was 100% familiar ground, so it never learned what to do with a half-strange view: one odd patch throws off its whole answer. We now sabotage half of the practice pictures by pasting in a square from somewhere else in the same city, while keeping the answer key unchanged. To keep scoring well, the model must learn to base its answer on the parts of the picture that fit together and shrug off the parts that don't -- like solving a jigsaw with a few pieces from the wrong box mixed in. Hypothesis The binding constraint is the held-out-location generalization gap the exp-18 probe measured directly: train-split crops localize to 267-400 m median while eval-split crops sit at 708-1153 m -- a ~3x structural gap, present in every lighting bucket. The frozen split's geometry says exactly what an eval crop looks like: its center pixel always lies in a 360-px eval block whose pixels were never trained on (train windows are dropped if they touch an eval block), while ~76% of eval crops still contain SOME train-seen terrain entering from an edge or corner of the 182-px window. So the canonical eval crop is unfamiliar terrain around the center with familiar terrain at the periphery. The kept head is a whole-crop template matcher (GAP + 512-d layout code -> FC 1024), and every single crop it has ever been trained on was 100% familiar terrain -- partial familiarity is literally outside its training distribution, so at eval the unfamiliar portion shifts the descriptor away from every learned template: fields go diffuse (night coverage sits at the abstention floor) or commit to a lookalike. Exp 18 attacked this same gap architecturally (64 per-patch field votes) and was reverted (1135.85 m) -- forcing each 16-px patch to localize alone discarded the global-arrangement signal that carries most of the discrimination. The materially new angle: keep the kept architecture untouched and make partial familiarity the TRAINING condition instead. With probability 0.5 a training crop gets a square block (side 48-96 px, i.e. 14-56% of its pixels, uniform position) overwritten with the same-position block from a different training location of the same lighting bucket, while the label and Gaussian-CE target stay at the base crop's true center. The pasted content is a worst-case stand-in for eval's unfamiliar content -- not merely uninformative but actively misleading, since its texture genuinely belongs elsewhere on the map. The only way to fit patched crops is to score map cells from the subset of content consistent with a single location and down-weight content that disagrees -- precisely the computation an eval crop demands and that nothing in the current training distribution has ever required. The confidence head trains on the same crops, so it simultaneously learns to flag conflicted content -- the exact crops it should abstain on at eval. This is the first input-composition augmentation in the project's history (exp 6, the only prior augmentation experiment, resampled rotations; it never altered crop content). Method One change, entirely inside model/train.py's load_training_tensors: collect each bucket's extracted crops into per-bucket lists; after the three-realization extraction loop, for each crop independently with p=0.5 paste a square block (side ~ uniform int in [48,96] px, top-left uniform over valid positions) copied from the SAME position in a different, pre-patching crop of the same bucket (donors read from a clean snapshot). Labels, Gaussian-CE targets, loss_fn, model.py, decode, confidence calibration, CLI, and ONNX export are all untouched. Expected outcome The gap is lighting-independent, so a real effect must be broad: predicted primary worst-case median 900-1020 m (from 1054.87 m) with all 24 area x bucket cells moving together rather than one cell fixed. Mechanism-specific signatures to check: (a) the eval-vs-train probe ratio narrows (baseline berlin midday 1153 vs 400 m, prignitz 708 vs 267 m); (b) coverage rises modestly in bright buckets (fields stay committed on partially-familiar crops instead of going diffuse); (c) the conf head's calibration keep-rates stay >= 0.40 per bucket since calibration crops are clean. Null: a global-template head cannot be taught selective reading by input augmentation alone -> metric flat within +/-3%. Downside bounded near current best: half the training crops remain exactly exp 17's distribution and the model/loss/decode/calibration are byte-identical, so the uniform-field -> map-center floor is unchanged. Deployment gates identical (3302661 bytes, ~2.7 ms host latency proxy); training wall time +<1 min/area (numpy block copies). Result primary worst-case median error = 1241.24 m (previous best 1054.87 m); full per-area/bucket breakdown in metrics.json Conclusion REVERTED — metric did not improve (1241.24 m vs best 1054.87 m); change discarded Scoreboard — median error (m) per area × lighting
the worst cell (underlined) is the experiment's score;
red = failed cell, ink = at target
Fits the aircraft? all deployment gates passed
largest per-area model 3,225 KB (limit 4,096 KB) · single-frame inference 2.7 ms on one CPU thread (proxy limit 250 ms)
crops are sampled fresh each run from the frozen train split (~45,000 distinct positions per area, times random rotation), never from eval blocks; how many to use is the experiment's own choice
Where this iteration's time went: design (Haiku) 16 m 43 s · implement (Sonnet) 43 s · train 4 areas 23 m 08 s · score 1 m 35 s · samples 42 s · publish 6 s · whole iteration 42 m 57 s
ts 2026-07-21T16:19:08 · commit d5080d5dc73e ·
parent d5080d5dc73e · artifacts runs/20260721_153617_iter6 ·
agent model — · took 42 m 51 s
The exact prompt given to the headless agent# Autoresearch iteration — one focused experiment
You are one iteration of an autonomous research loop for UAV low-light
geolocalization. Read `CLAUDE.md` (§1, §3, §6 especially) for full context.
The harness (loop.sh) will train, score, log, and keep/revert AFTER you exit —
you only design the experiment and edit the code.
## Your job, in order
1. **Review the research history.** Query the lineage DB:
`sqlite3 experiments.sqlite "SELECT id, title, category, hypothesis, expected_outcome, result, conclusion, primary_metric, kept FROM experiments ORDER BY id DESC LIMIT 15;"`
Note which hypotheses were supported/refuted. Do not repeat a refuted
experiment without a materially new angle.
**Plateau rule:** if three or more consecutive experiments were reverted,
do not attempt another variation of the last refuted mechanism. Either
pick a design family absent from the history (dispatcher + lighting
specialists, pretrained init, learned relighting, training-scale, …) or
attack the bottleneck the refuted hypotheses jointly point at.
2. **Design ONE focused experiment** — proper experiment design, pre-registered
before you touch code. Write it to `runs/pending_experiment.json`:
```json
{
"title": "one-line name",
"category": "architecture|loss|augmentation|relighting|training|quantization|other",
"hypothesis": "what you believe is limiting the metric and why this change addresses it",
"method": "the ONE focused change, concretely (files, mechanism)",
"expected_outcome": "predicted effect on the §6 worst-case median error, quantified if possible",
"init_strategy": "from-scratch | pretrained:<name>",
"eli5": "2-4 sentences for a smart non-ML reader: what you changed and why it might help, in everyday language — analogies welcome, zero jargon",
"architecture": {"stages": [
{"name": "Camera frame", "detail": "128×128 px crop, one of 6 lighting renders", "changed": false},
{"name": "Feature extractor", "detail": "plain-language description", "changed": false},
{"name": "…", "detail": "…", "changed": true}
]},
"architecture_svg": "<svg viewBox='0 0 980 300' …>…</svg>"
}
```
`architecture_svg` is the **technical architecture diagram** of the model
you are testing — a proper ML-paper figure of the design, drawn by you,
shown at the top of this experiment's gallery entry. **Draw tensors and
operations as the things they are — NOT as labeled boxes:** the input
image as a pixel-textured square; conv feature maps as pseudo-3D slabs
that shrink spatially and deepen in channels; 1-D vectors as thin
vertical tick-bars; fully-connected layers as fans of thin crossing
lines between bars; spatial fields/grids as actually-drawn grids with
shaded cells; decodes/aggregations as thin lines converging from cells
onto a point; samplers/targets as small pictorial glyphs. Captions sit
UNDER each element (name ≤10.5px weight 600, sub-note ≤9.5px); tensor
shapes annotate the flow in small italics (128×128×3 → 8²×128 → 1024).
Losses/targets/samplers live in a lane below the inference path,
annotated with slanted dashed leader lines to small text — no boxes
around them. Draw the two FROZEN endpoints — the camera-frame input and
the (lat, lon, confidence) output — entirely in #9b998c with a small
italic “frozen contract” tag: they are fixed by the harness and outside
your search space. Everything between them is your design: draw it in ink,
with red only on what THIS experiment changes. Style contract, so all experiments' figures read as one
paper: viewBox width 980 (height ~300–360); transparent background; ink
#111111, captions #6b6a60, annotations/arrows #9b998c; **#8c2f1f (red)
reserved for exactly what this experiment changed**; training-only
elements in #8a6a1e (or red if changed); font-family
Palatino,Georgia,serif; stroke-width ≈1.2 elements, 1 arrows; fills only
faint tints (opacity ≤ .12); no gradients, no icons, no emoji. Inference
flows left → right from camera frame to (lat, lon, confidence).
**The camera frame is a fixed glyph — copy it verbatim.** It must look
like an actual nadir frame of terrain (streets, building footprints),
not abstract pixels, and be identical in every figure. Use exactly this
snippet, substituting Y for the frame's top edge (pick Y so the frame
centers on your inference lane; captions go under it as usual — note
128²×3 is the model input crop, ~1 m/px, not the sensor's native
resolution):
```svg
<g><rect id='frozen-input' x='26' y='Y' width='76' height='76'
fill='#f6f4ea' stroke='#9b998c' stroke-width='1.6'/>
<path d='M26 Y+50 L102 Y+24' stroke='#e6e3d4' stroke-width='6' fill='none'/>
<path d='M62 Y+2 L48 Y+74' stroke='#e6e3d4' stroke-width='4' fill='none'/>
<rect x='33' y='Y+8' width='13' height='9' fill='#d9d5c3' transform='rotate(-8 39 Y+12)'/>
<rect x='79' y='Y+7' width='10' height='12' fill='#cfccbd'/>
<rect x='34' y='Y+58' width='14' height='9' fill='#d9d5c3'/>
<rect x='76' y='Y+50' width='11' height='10' fill='#cfccbd' transform='rotate(6 81 Y+55)'/>
<rect x='55' y='Y+32' width='9' height='8' fill='#d9d5c3' opacity='.85'/>
<ellipse cx='88' cy='Y+66' rx='10' ry='7' fill='#8a6a1e' opacity='.12'/>
<circle cx='40' cy='Y+28' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='70' cy='Y+14' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='92' cy='Y+34' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='52' cy='Y+50' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='83' cy='Y+70' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='31' cy='Y+44' r='.7' fill='#6b6a60' opacity='.5'/></g>
```
(Y+n means the literal number Y plus n — compute the values.) Draw your
receptive-field square and kernel-projection lines on top of it as
usual.
**Anchor the frozen endpoints identically in every figure** so all
experiments' figures line up when compared down the gallery page: the
camera-frame square starts at x=26 (its captions centered on x=53), and
the output is right-anchored — decode crosshair at x=812, the
(lat, lon, confidence) text block text-anchor=start at x=828. Mark the
anchors machine-checkably: the camera-frame rect carries
`id='frozen-input'`, the output text block carries `id='frozen-output'`.
Lay the elements between them out on a running x-cursor with generous
spacing so nothing overlaps.
**Readability contract** (figcheck verifies this geometrically): no
text may overlap other text, and no line (arrows, converge fans,
leader lines) may pass through a label — a line may END at a label,
never cross one. Rules distilled from review rounds: dashed leaders
run as orthogonal L-routes through empty lanes, never diagonally
across components; converge/vote lines originate at their true
sources (the actual dots or window cells, not generic positions);
every element owns its caption — adjacent columns stagger their
caption rows instead of colliding; prefer moving a label into empty
horizontal space over stacking it against a neighbor. Width is fixed
at 980, but **height is yours**: grow the viewBox (240–640) whenever
more vertical room makes the layout cleaner rather than cramming.
**Before finishing, validate:**
`.venv/bin/python -m autoresearch.figcheck` — it checks the anchor
contract on runs/pending_experiment.json; revise until it prints PASS.
Consecutive conv layers must be tied by kernel-projection lines (small
kernel square on one face, faint lines converging to a cell on the
next) so the encoder reads as one computation, not boxes in a row.
`archive/arch_svg_reference.py` is the concrete reference implementation
of this entire visual language — read it and reuse its helper geometry
for any stage that already has one.
`eli5` and `architecture` feed the human-facing gallery. `architecture.stages`
is the model's inference path left-to-right, camera frame → (lat, lon,
confidence) output — one box per stage, plain-language `detail`,
`"changed": true` ONLY on the stages this experiment touches. Reuse the
previous experiment's stage names verbatim wherever a stage is unchanged
(check `SELECT arch_json FROM experiments WHERE arch_json IS NOT NULL ORDER BY id DESC LIMIT 1;`) —
the diagrams are compared box-by-box across experiments. A change that only
affects training (loss, augmentation, schedule) keeps the inference stages
unchanged and adds one final stage with `"train_only": true` describing the
training signal.
3. **Write the implementation brief.** You do NOT edit `model/` yourself —
a separate implementation agent applies your design, seeing only the
current code plus what you pre-registered. Add one more field to
`runs/pending_experiment.json`:
```
"implementation_brief": "exact file-level instructions: which functions/
blocks in model/model.py and model/train.py change and how, what stays
untouched, and every contract to preserve"
```
Be precise enough that a competent engineer with no other context
implements it in one pass. Always restate the fixed contracts:
`train.py`'s CLI and the ONNX export contract in `model/model.py`'s
docstring — the frozen scorer depends on them.
## Hard rules
- Edit ONLY `runs/pending_experiment.json`. You never edit `model/` (the
implementation stage does) and files listed in `/FROZEN` are off-limits;
the harness hard-reverts any change to them.
- NEVER touch, read, or evaluate the `hamburg` holdout area (§5).
- ONE focused change per iteration — if you can't describe it in one sentence,
it's too big. Prefer architectural/procedural novelty over hyperparameter
nudges (§3): changing a learning rate is a weak experiment; changing the
coordinate parameterization, loss family, relighting-for-training, or model
topology is a strong one.
- Do not run training yourself; the harness does that.
- Stay within the deployment gates: exported ONNX ≤ 4 MiB per area, host
latency proxy ≤ 250 ms (see pipeline/score.py).Where the model was tested — and how far off it was Each map is one full test area. Every dot is one held-out test location (all six lighting conditions overlaid): the model was shown a 128 m crop centered there and asked for its position. Dot color = the distance between its answer and the truth — green ≤ 20 m (at goal), amber ≤ 50 m, red beyond. A working model turns these maps green; spatial clusters of red reveal which parts of an area confuse it.     What the six lighting conditions look like through the simulated low-light sensor (example patches — illustration, not the training set)Every frame in this project is rendered as a starlight-class low-light sensor (Sony STARVIS2 / IMX585 class — the airframe's chosen camera) would see it, not as a normal camera would. That is why the night renders look uncannily close to daylight: at high gain such a sensor recovers scene structure from moonlight, skyglow and artificial lighting, paying for it in noise, lifted shadows and washed-out color — exactly what these renders simulate. Below, one example 256 m patch per area under the six lighting conditions the model must handle. These illustrate the dataset, not this experiment's performance — the actual training set is thousands of distinct crops per area (see “training data” above), and these renderings only change when the relighting method changes. berlin frankfurt munich prignitz | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▸ | 18 | Dense per-patch field voting: 64 local place votes with learned fusion replace the global 1024-way template head 48e17da6 | architecture | pretrained:mobilenet_v3_small IMAGENET1K_V1 features[0..8] (torchvision, BSD-3) -- unchanged from exp 11 | 1.14 km | 1,321 KB | 3.7 ms | 45 m 05 s | $0.52 | discarded | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The design under test — technical diagram — gray = frozen contract · ink = the current design · red = this experiment's change One real test, end to end
→
1.20 km
miss, this crop 0.39
self-reported confidence In plain words Until now the model looked at the whole photo at once and asked 'which spot in the city does this whole picture remind me of?' -- like recognizing a street from a single glance. Our measurement shows it does that three times better on places it studied than on the gaps in between, which is exactly where it gets tested. Now each of 64 small tiles of the photo votes independently -- 'my little piece of ground belongs THERE' -- and a learned referee counts trustworthy tiles more heavily before merging the votes. Even over unfamiliar ground, a few tiles usually catch a road or building the model does know, and those few good votes can carry the answer. Hypothesis A read-only probe of the last kept model (berlin+prignitz, midday, matched random headings, 300 crops/condition) shows train-split crops localize to 267-400 m median while eval-split crops sit at 708-1153 m: a ~3x held-out-location generalization gap now dominates the metric, and it is structural, not a lighting problem. The current head is a single 560-d whole-crop descriptor feeding a 573k-param 1024-way FC -- it memorizes whole-crop gestalt templates, which cannot fire correctly on terrain whose pixels were never seen (eval blocks are fully excluded from training with a buffer). But the geometry helps: eval blocks are 360 px squares and crop windows are 182 px, so ~76% of eval crops contain SOME train-seen pixels at a known in-crop offset -- and a global-template head has no mechanism to exploit partially recognized content. Dense per-patch voting does exactly that: each of the 8x8 feature-grid cells casts its own full-map probability field for its OWN patch of ground, supervised with its own patch-level target; at inference a learned informativeness softmax weights the 64 per-patch log-fields into one fused field. Patches over recognized terrain vote precisely and carry the answer; unrecognizable patches are down-weighted. Because the patch-offset pattern is symmetric about the crop center, the fused field peaks at the crop center with no heading knowledge needed. Weight sharing (one 1x1 MLP serves all 64 positions) also cuts head capacity from ~573k to ~137k params, so per-location gestalt memorization -- the thing the probe shows failing to transfer -- is architecturally discouraged: local features must become location-predictive everywhere in the crop. This is materially different from reverted exp 4, which regressed per-patch COORDINATES (sigmoid + smooth-L1, mean-of-64 decode) on a from-scratch encoder with 4.8k crops: mean-decoding suffers the centroid-shrinkage disease exp 14 later diagnosed and fixed, and the from-scratch trunk had no local features worth reading. This design keeps every kept mechanism -- pretrained trunk (exp 11), calibrated Gaussian-CE field + beta-sharpened decode (exps 2/14), C4 rotation vote (exp 16), shape-based confidence + calibration (exp 15), three-realization renders (exp 17) -- and changes only who produces the field logits. Method model/model.py: delete layout_squeeze, loc_logits, dark_logits and the luminance gate; add a shared per-position head (1x1 conv 48->128, ReLU, 1x1 conv 128->1024) producing per-patch field logits [*,1024,8,8] and a 1x1 conv 48->1 whose softmax over the 64 positions gives fusion weights; fused per-view logits = sum_p w_p * logits_p; everything downstream (C4 logit mean, sharpened decode, conf head, conf_shift calibration, ONNX export) is unchanged. model/train.py: store each training crop's heading angle next to its (u,v) target; the loss adds a dense term -- each patch's logits get Gaussian-CE against the map cell of that patch's own ground content, computed as crop center + R(heading) . patch offset, on the identity C4 view. Expected outcome Primary worst-case median error drops from 1054.87 m into the 650-900 m band. Mechanism-specific signature to check in the logs: the eval-vs-train gap narrows (probe baseline: berlin midday eval 1153 m vs train 400 m; prignitz 708 vs 267) and the improvement should appear across ALL lighting buckets, since the generalization gap is lighting-independent. Bounded downside as in every kept experiment: at init the per-patch fields are near-uniform, so the fused field is uniform and the sharpened decode starts at the map center. Exported model shrinks from 3.3 MB to ~1.3 MB, well inside the 4 MiB gate; latency stays trunk-dominated (~3 ms proxy). Result primary worst-case median error = 1135.85 m (previous best 1054.87 m); full per-area/bucket breakdown in metrics.json Conclusion REVERTED — metric did not improve (1135.85 m vs best 1054.87 m); change discarded Scoreboard — median error (m) per area × lighting
the worst cell (underlined) is the experiment's score;
red = failed cell, ink = at target
Fits the aircraft? all deployment gates passed
largest per-area model 1,321 KB (limit 4,096 KB) · single-frame inference 3.7 ms on one CPU thread (proxy limit 250 ms)
crops are sampled fresh each run from the frozen train split (~45,000 distinct positions per area, times random rotation), never from eval blocks; how many to use is the experiment's own choice
Where this iteration's time went: design (Haiku) 17 m 12 s · implement (Sonnet) 3 m 12 s · train 4 areas 22 m 14 s · score 1 m 45 s · samples 42 s · publish 4 s · whole iteration 45 m 10 s
ts 2026-07-21T15:36:03 · commit 48e17da6b2cf ·
parent 48e17da6b2cf · artifacts runs/20260721_145057_iter5 ·
agent model — · took 45 m 05 s
The exact prompt given to the headless agent# Autoresearch iteration — one focused experiment
You are one iteration of an autonomous research loop for UAV low-light
geolocalization. Read `CLAUDE.md` (§1, §3, §6 especially) for full context.
The harness (loop.sh) will train, score, log, and keep/revert AFTER you exit —
you only design the experiment and edit the code.
## Your job, in order
1. **Review the research history.** Query the lineage DB:
`sqlite3 experiments.sqlite "SELECT id, title, category, hypothesis, expected_outcome, result, conclusion, primary_metric, kept FROM experiments ORDER BY id DESC LIMIT 15;"`
Note which hypotheses were supported/refuted. Do not repeat a refuted
experiment without a materially new angle.
**Plateau rule:** if three or more consecutive experiments were reverted,
do not attempt another variation of the last refuted mechanism. Either
pick a design family absent from the history (dispatcher + lighting
specialists, pretrained init, learned relighting, training-scale, …) or
attack the bottleneck the refuted hypotheses jointly point at.
2. **Design ONE focused experiment** — proper experiment design, pre-registered
before you touch code. Write it to `runs/pending_experiment.json`:
```json
{
"title": "one-line name",
"category": "architecture|loss|augmentation|relighting|training|quantization|other",
"hypothesis": "what you believe is limiting the metric and why this change addresses it",
"method": "the ONE focused change, concretely (files, mechanism)",
"expected_outcome": "predicted effect on the §6 worst-case median error, quantified if possible",
"init_strategy": "from-scratch | pretrained:<name>",
"eli5": "2-4 sentences for a smart non-ML reader: what you changed and why it might help, in everyday language — analogies welcome, zero jargon",
"architecture": {"stages": [
{"name": "Camera frame", "detail": "128×128 px crop, one of 6 lighting renders", "changed": false},
{"name": "Feature extractor", "detail": "plain-language description", "changed": false},
{"name": "…", "detail": "…", "changed": true}
]},
"architecture_svg": "<svg viewBox='0 0 980 300' …>…</svg>"
}
```
`architecture_svg` is the **technical architecture diagram** of the model
you are testing — a proper ML-paper figure of the design, drawn by you,
shown at the top of this experiment's gallery entry. **Draw tensors and
operations as the things they are — NOT as labeled boxes:** the input
image as a pixel-textured square; conv feature maps as pseudo-3D slabs
that shrink spatially and deepen in channels; 1-D vectors as thin
vertical tick-bars; fully-connected layers as fans of thin crossing
lines between bars; spatial fields/grids as actually-drawn grids with
shaded cells; decodes/aggregations as thin lines converging from cells
onto a point; samplers/targets as small pictorial glyphs. Captions sit
UNDER each element (name ≤10.5px weight 600, sub-note ≤9.5px); tensor
shapes annotate the flow in small italics (128×128×3 → 8²×128 → 1024).
Losses/targets/samplers live in a lane below the inference path,
annotated with slanted dashed leader lines to small text — no boxes
around them. Draw the two FROZEN endpoints — the camera-frame input and
the (lat, lon, confidence) output — entirely in #9b998c with a small
italic “frozen contract” tag: they are fixed by the harness and outside
your search space. Everything between them is your design: draw it in ink,
with red only on what THIS experiment changes. Style contract, so all experiments' figures read as one
paper: viewBox width 980 (height ~300–360); transparent background; ink
#111111, captions #6b6a60, annotations/arrows #9b998c; **#8c2f1f (red)
reserved for exactly what this experiment changed**; training-only
elements in #8a6a1e (or red if changed); font-family
Palatino,Georgia,serif; stroke-width ≈1.2 elements, 1 arrows; fills only
faint tints (opacity ≤ .12); no gradients, no icons, no emoji. Inference
flows left → right from camera frame to (lat, lon, confidence).
**The camera frame is a fixed glyph — copy it verbatim.** It must look
like an actual nadir frame of terrain (streets, building footprints),
not abstract pixels, and be identical in every figure. Use exactly this
snippet, substituting Y for the frame's top edge (pick Y so the frame
centers on your inference lane; captions go under it as usual — note
128²×3 is the model input crop, ~1 m/px, not the sensor's native
resolution):
```svg
<g><rect id='frozen-input' x='26' y='Y' width='76' height='76'
fill='#f6f4ea' stroke='#9b998c' stroke-width='1.6'/>
<path d='M26 Y+50 L102 Y+24' stroke='#e6e3d4' stroke-width='6' fill='none'/>
<path d='M62 Y+2 L48 Y+74' stroke='#e6e3d4' stroke-width='4' fill='none'/>
<rect x='33' y='Y+8' width='13' height='9' fill='#d9d5c3' transform='rotate(-8 39 Y+12)'/>
<rect x='79' y='Y+7' width='10' height='12' fill='#cfccbd'/>
<rect x='34' y='Y+58' width='14' height='9' fill='#d9d5c3'/>
<rect x='76' y='Y+50' width='11' height='10' fill='#cfccbd' transform='rotate(6 81 Y+55)'/>
<rect x='55' y='Y+32' width='9' height='8' fill='#d9d5c3' opacity='.85'/>
<ellipse cx='88' cy='Y+66' rx='10' ry='7' fill='#8a6a1e' opacity='.12'/>
<circle cx='40' cy='Y+28' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='70' cy='Y+14' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='92' cy='Y+34' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='52' cy='Y+50' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='83' cy='Y+70' r='.7' fill='#6b6a60' opacity='.5'/>
<circle cx='31' cy='Y+44' r='.7' fill='#6b6a60' opacity='.5'/></g>
```
(Y+n means the literal number Y plus n — compute the values.) Draw your
receptive-field square and kernel-projection lines on top of it as
usual.
**Anchor the frozen endpoints identically in every figure** so all
experiments' figures line up when compared down the gallery page: the
camera-frame square starts at x=26 (its captions centered on x=53), and
the output is right-anchored — decode crosshair at x=812, the
(lat, lon, confidence) text block text-anchor=start at x=828. Mark the
anchors machine-checkably: the camera-frame rect carries
`id='frozen-input'`, the output text block carries `id='frozen-output'`.
Lay the elements between them out on a running x-cursor with generous
spacing so nothing overlaps.
**Readability contract:** no text may overlap other text, and no line
(arrows, converge fans, leader lines) may pass through a label — route
leaders around text with a bend, or move the label. Width is fixed at
980, but **height is yours**: grow the viewBox (240–640) whenever more
vertical room makes the layout cleaner rather than cramming.
**Before finishing, validate:**
`.venv/bin/python -m autoresearch.figcheck` — it checks the anchor
contract on runs/pending_experiment.json; revise until it prints PASS.
Consecutive conv layers must be tied by kernel-projection lines (small
kernel square on one face, faint lines converging to a cell on the
next) so the encoder reads as one computation, not boxes in a row.
`archive/arch_svg_reference.py` is the concrete reference implementation
of this entire visual language — read it and reuse its helper geometry
for any stage that already has one.
`eli5` and `architecture` feed the human-facing gallery. `architecture.stages`
is the model's inference path left-to-right, camera frame → (lat, lon,
confidence) output — one box per stage, plain-language `detail`,
`"changed": true` ONLY on the stages this experiment touches. Reuse the
previous experiment's stage names verbatim wherever a stage is unchanged
(check `SELECT arch_json FROM experiments WHERE arch_json IS NOT NULL ORDER BY id DESC LIMIT 1;`) —
the diagrams are compared box-by-box across experiments. A change that only
affects training (loss, augmentation, schedule) keeps the inference stages
unchanged and adds one final stage with `"train_only": true` describing the
training signal.
3. **Write the implementation brief.** You do NOT edit `model/` yourself —
a separate implementation agent applies your design, seeing only the
current code plus what you pre-registered. Add one more field to
`runs/pending_experiment.json`:
```
"implementation_brief": "exact file-level instructions: which functions/
blocks in model/model.py and model/train.py change and how, what stays
untouched, and every contract to preserve"
```
Be precise enough that a competent engineer with no other context
implements it in one pass. Always restate the fixed contracts:
`train.py`'s CLI and the ONNX export contract in `model/model.py`'s
docstring — the frozen scorer depends on them.
## Hard rules
- Edit ONLY `runs/pending_experiment.json`. You never edit `model/` (the
implementation stage does) and files listed in `/FROZEN` are off-limits;
the harness hard-reverts any change to them.
- NEVER touch, read, or evaluate the `hamburg` holdout area (§5).
- ONE focused change per iteration — if you can't describe it in one sentence,
it's too big. Prefer architectural/procedural novelty over hyperparameter
nudges (§3): changing a learning rate is a weak experiment; changing the
coordinate parameterization, loss family, relighting-for-training, or model
topology is a strong one.
- Do not run training yourself; the harness does that.
- Stay within the deployment gates: exported ONNX ≤ 4 MiB per area, host
latency proxy ≤ 250 ms (see pipeline/score.py).Where the model was tested — and how far off it was Each map is one full test area. Every dot is one held-out test location (all six lighting conditions overlaid): the model was shown a 128 m crop centered there and asked for its position. Dot color = the distance between its answer and the truth — green ≤ 20 m (at goal), amber ≤ 50 m, red beyond. A working model turns these maps green; spatial clusters of red reveal which parts of an area confuse it.     What the six lighting conditions look like through the simulated low-light sensor (example patches — illustration, not the training set)Every frame in this project is rendered as a starlight-class low-light sensor (Sony STARVIS2 / IMX585 class — the airframe's chosen camera) would see it, not as a normal camera would. That is why the night renders look uncannily close to daylight: at high gain such a sensor recovers scene structure from moonlight, skyglow and artificial lighting, paying for it in noise, lifted shadows and washed-out color — exactly what these renders simulate. Below, one example 256 m patch per area under the six lighting conditions the model must handle. These illustrate the dataset, not this experiment's performance — the actual training set is thousands of distinct crops per area (see “training data” above), and these renderings only change when the relighting method changes. berlin frankfurt munich prignitz | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▸ | 17 | Nuisance-randomized training renders: each bucket's training crops drawn from three seeded realizations of the frozen relighting sim 95b4c3a0 | relighting | pretrained:mobilenet_v3_small IMAGENET1K_V1 features[0..8] (torchvision, BSD-3) | 1.05 km | 3,225 KB | 2.7 ms | 39 m 10 s | $0.45 | kept | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The design under test — technical diagram — gray = frozen contract · ink = the current design · red = this experiment's change One real test, end to end
→
1.72 km
miss, this crop 0.18
self-reported confidence In plain words The model learns night navigation from simulated night photos, and the simulator rolls dice for two things: the grainy sensor static, and which streetlights and lit windows happen to be on. Until now, every practice photo of a place used one single dice roll, frozen forever — so the model could ace practice by memorizing the static, like recognizing a photocopy by its smudges instead of reading it. On new places the smudges are different, and it gets lost exactly where the picture is mostly smudge: at night. Now the practice photos come from three different dice rolls of the same city, so the only thing worth memorizing is what is actually there — the roads, the buildings, and where the lit areas sit. Hypothesis Night is now the binding constraint everywhere: it is the worst lighting bucket in all four development areas, the primary metric IS Frankfurt night (1094.59 m), and night coverage has collapsed toward the abstention floor (0.285-0.37 vs 0.46-0.71 in bright buckets) — the confidence head is already throwing away two thirds of night crops and the survivors still miss by ~1 km. The cause this experiment attacks: each (area, bucket) training/eval image is ONE frozen realization of the relighting sim's stochastic nuisances — the seeded shot/read sensor noise and the seeded random thinning that decides which built-up pixels become lamps. Measured directly on prignitz: two different seeds of the night render differ by mean |dpx| 29/255 (79% of pixels differ by >8) against a render mean of only ~60/255 — roughly HALF the night image is realization-specific nuisance — while midday differs by just 6/255. Training crops therefore show each location with a fixed noise/lamp texture for all epochs, and a 1024-way location-memorization head keys each cell's template on the easiest discriminative signal available, which at night SNR~1 is precisely that frozen texture. Held-out eval locations carry unseen realizations of the same nuisances, so the learned templates misfire, fields go diffuse, and the model abstains to the floor or commits wrongly — exactly the observed night signature. Sampling each bucket's training crops from THREE seeded realizations of the frozen sim (the stored eval-matched render plus two fresh-seed re-renders of reference.tif through the untouched pipeline.relight.relight) makes realization texture non-discriminative between locations, forcing templates onto the seed-stable structure (roads, building masses, where the built-up glow sits) — which is exactly what eval crops share with training crops. This is the first experiment in the relighting family in 16 iterations, and it is the training-side relighting that pipeline/relight.py's own NOTE explicitly sanctions; the eval set is untouched. Method model/train.py only. load_training_tensors() partitions each bucket's rng.choice picks into three near-equal parts: part 0 is extracted from the stored data/<area>/relight/<bucket>.png (seed_0 — exactly what eval sees); parts 1 and 2 are extracted from relight(ref, LIGHTING_BUCKETS[bucket], meta['gsd_m'], stable_hash(f'{area}:{bucket}:trainreal:{r}')) for r=1,2, rendered on the fly from the area's reference.tif via the frozen pipeline.relight.relight function (imported, never modified), holding at most one full raster + the reference in memory at a time (~19 s/render measured, +12 renders = ~4 min/area). Everything else — model.py, losses, decode, C4 vote, conf calibration on the stored PNGs, CLI, ONNX export — unchanged. Expected outcome The effect is concentrated in dark buckets by construction (seed-to-seed render difference is ~5x smaller at midday than at night), so bright cells should move little. If realization overfit is binding at night: night medians (835-1095 m across areas) drop 10-25% and night coverage rises from 0.285-0.37 toward >=0.4, giving a predicted primary worst-case median of 880-1010 m (from 1094.59 m); since all four night cells sit within 260 m of the worst, the improvement must be broad, not single-cell. Null: the head is not keying on realization texture -> metric ~flat (+/-3%); downside is bounded near current best because a third of training data remains exactly eval-matched and the model/loss/decode are unchanged. Gates unchanged (3302661 bytes, ~2.7 ms host proxy); training wall time +~4 min/area. Result primary worst-case median error = 1054.87 m (previous best 1094.59 m); full per-area/bucket breakdown in metrics.json Conclusion KEPT — metric improved (1094.59 -> 1054.87 m); change committed as 95b4c3a Scoreboard — median error (m) per area × lighting
the worst cell (underlined) is the experiment's score;
red = failed cell, ink = at target
Fits the aircraft? all deployment gates passed
largest per-area model 3,225 KB (limit 4,096 KB) · single-frame inference 2.7 ms on one CPU thread (proxy limit 250 ms)
crops are sampled fresh each run from the frozen train split (~45,000 distinct positions per area, times random rotation), never from eval blocks; how many to use is the experiment's own choice
Where this iteration's time went: design (Haiku) 13 m 54 s · implement (Sonnet) 58 s · train 4 areas 21 m 55 s · score 1 m 39 s · samples 43 s · publish 6 s · whole iteration 39 m 16 s
ts 2026-07-21T14:50:42 · commit 95b4c3a0c7c7 ·
parent dc5813c49e55 · artifacts runs/20260721_141132_iter4 ·
agent model — · took 39 m 10 s
The exact prompt given to the headless agent# Autoresearch iteration — one focused experiment
You are one iteration of an autonomous research loop for UAV low-light
geolocalization. Read `CLAUDE.md` (§1, §3, §6 especially) for full context.
The harness (loop.sh) will train, score, log, and keep/revert AFTER you exit —
you only design the experiment and edit the code.
## Your job, in order
1. **Review the research history.** Query the lineage DB:
`sqlite3 experiments.sqlite "SELECT id, title, category, hypothesis, expected_outcome, result, conclusion, primary_metric, kept FROM experiments ORDER BY id DESC LIMIT 15;"`
Note which hypotheses were supported/refuted. Do not repeat a refuted
experiment without a materially new angle.
**Plateau rule:** if three or more consecutive experiments were reverted,
do not attempt another variation of the last refuted mechanism. Either
pick a design family absent from the history (dispatcher + lighting
specialists, pretrained init, learned relighting, training-scale, …) or
attack the bottleneck the refuted hypotheses jointly point at.
2. **Design ONE focused experiment** — proper experiment design, pre-registered
before you touch code. Write it to `runs/pending_experiment.json`:
```json
{
"title": "one-line name",
"category": "architecture|loss|augmentation|relighting|training|quantization|other",
"hypothesis": "what you believe is limiting the metric and why this change addresses it",
"method": "the ONE focused change, concretely (files, mechanism)",
"expected_outcome": "predicted effect on the §6 worst-case median error, quantified if possible",
"init_strategy": "from-scratch | pretrained:<name>",
"eli5": "2-4 sentences for a smart non-ML reader: what you changed and why it might help, in everyday language — analogies welcome, zero jargon",
"architecture": {"stages": [
{"name": "Camera frame", "detail": "128×128 px crop, one of 6 lighting renders", "changed": false},
{"name": "Feature extractor", "detail": "plain-language description", "changed": false},
{"name": "…", "detail": "…", "changed": true}
]},
"architecture_svg": "<svg viewBox='0 0 980 300' …>…</svg>"
}
```
`architecture_svg` is the **technical architecture diagram** of the model
you are testing — a proper ML-paper figure of the design, drawn by you,
shown at the top of this experiment's gallery entry. **Draw tensors and
operations as the things they are — NOT as labeled boxes:** the input
image as a pixel-textured square; conv feature maps as pseudo-3D slabs
that shrink spatially and deepen in channels; 1-D vectors as thin
vertical tick-bars; fully-connected layers as fans of thin crossing
lines between bars; spatial fields/grids as actually-drawn grids with
shaded cells; decodes/aggregations as thin lines converging from cells
onto a point; samplers/targets as small pictorial glyphs. Captions sit
UNDER each element (name ≤10.5px weight 600, sub-note ≤9.5px); tensor
shapes annotate the flow in small italics (128×128×3 → 8²×128 → 1024).
Losses/targets/samplers live in a lane below the inference path,
annotated with slanted dashed leader lines to small text — no boxes
around them. Draw the two FROZEN endpoints — the camera-frame input and
the (lat, lon, confidence) output — entirely in #9b998c with a small
italic “frozen contract” tag: they are fixed by the harness and outside
your search space. Everything between them is your design: draw it in ink,
with red only on what THIS experiment changes. Style contract, so all experiments' figures read as one
paper: viewBox width 980 (height ~300–360); transparent background; ink
#111111, captions #6b6a60, annotations/arrows #9b998c; **#8c2f1f (red)
reserved for exactly what this experiment changed**; training-only
elements in #8a6a1e (or red if changed); font-family
Palatino,Georgia,serif; stroke-width ≈1.2 elements, 1 arrows; fills only
faint tints (opacity ≤ .12); no gradients, no icons, no emoji. Inference
flows left → right from camera frame to (lat, lon, confidence).
**Anchor the frozen endpoints identically in every figure** so all
experiments' figures line up when compared down the gallery page: the
camera-frame square starts at x=26 (its captions centered on x=53), and
the output is right-anchored — decode crosshair at x=812, the
(lat, lon, confidence) text block text-anchor=start at x=828. Mark the
anchors machine-checkably: the camera-frame rect carries
`id='frozen-input'`, the output text block carries `id='frozen-output'`.
Lay the elements between them out on a running x-cursor with generous
spacing so nothing overlaps.
**Readability contract:** no text may overlap other text, and no line
(arrows, converge fans, leader lines) may pass through a label — route
leaders around text with a bend, or move the label. Width is fixed at
980, but **height is yours**: grow the viewBox (240–640) whenever more
vertical room makes the layout cleaner rather than cramming.
**Before finishing, validate:**
`.venv/bin/python -m autoresearch.figcheck` — it checks the anchor
contract on runs/pending_experiment.json; revise until it prints PASS.
Consecutive conv layers must be tied by kernel-projection lines (small
kernel square on one face, faint lines converging to a cell on the
next) so the encoder reads as one computation, not boxes in a row.
`archive/arch_svg_reference.py` is the concrete reference implementation
of this entire visual language — read it and reuse its helper geometry
for any stage that already has one.
`eli5` and `architecture` feed the human-facing gallery. `architecture.stages`
is the model's inference path left-to-right, camera frame → (lat, lon,
confidence) output — one box per stage, plain-language `detail`,
`"changed": true` ONLY on the stages this experiment touches. Reuse the
previous experiment's stage names verbatim wherever a stage is unchanged
(check `SELECT arch_json FROM experiments WHERE arch_json IS NOT NULL ORDER BY id DESC LIMIT 1;`) —
the diagrams are compared box-by-box across experiments. A change that only
affects training (loss, augmentation, schedule) keeps the inference stages
unchanged and adds one final stage with `"train_only": true` describing the
training signal.
3. **Write the implementation brief.** You do NOT edit `model/` yourself —
a separate implementation agent applies your design, seeing only the
current code plus what you pre-registered. Add one more field to
`runs/pending_experiment.json`:
```
"implementation_brief": "exact file-level instructions: which functions/
blocks in model/model.py and model/train.py change and how, what stays
untouched, and every contract to preserve"
```
Be precise enough that a competent engineer with no other context
implements it in one pass. Always restate the fixed contracts:
`train.py`'s CLI and the ONNX export contract in `model/model.py`'s
docstring — the frozen scorer depends on them.
## Hard rules
- Edit ONLY `runs/pending_experiment.json`. You never edit `model/` (the
implementation stage does) and files listed in `/FROZEN` are off-limits;
the harness hard-reverts any change to them.
- NEVER touch, read, or evaluate the `hamburg` holdout area (§5).
- ONE focused change per iteration — if you can't describe it in one sentence,
it's too big. Prefer architectural/procedural novelty over hyperparameter
nudges (§3): changing a learning rate is a weak experiment; changing the
coordinate parameterization, loss family, relighting-for-training, or model
topology is a strong one.
- Do not run training yourself; the harness does that.
- Stay within the deployment gates: exported ONNX ≤ 4 MiB per area, host
latency proxy ≤ 250 ms (see pipeline/score.py).Where the model was tested — and how far off it was Each map is one full test area. Every dot is one held-out test location (all six lighting conditions overlaid): the model was shown a 128 m crop centered there and asked for its position. Dot color = the distance between its answer and the truth — green ≤ 20 m (at goal), amber ≤ 50 m, red beyond. A working model turns these maps green; spatial clusters of red reveal which parts of an area confuse it.     What the six lighting conditions look like through the simulated low-light sensor (example patches — illustration, not the training set)Every frame in this project is rendered as a starlight-class low-light sensor (Sony STARVIS2 / IMX585 class — the airframe's chosen camera) would see it, not as a normal camera would. That is why the night renders look uncannily close to daylight: at high gain such a sensor recovers scene structure from moonlight, skyglow and artificial lighting, paying for it in noise, lifted shadows and washed-out color — exactly what these renders simulate. Below, one example 256 m patch per area under the six lighting conditions the model must handle. These illustrate the dataset, not this experiment's performance — the actual training set is thousands of distinct crops per area (see “training data” above), and these renderings only change when the relighting method changes. berlin frankfurt munich prignitz | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▸ | 16 | C4 rotation-vote field: average field logits over the crop's four 90° turns 5c5373df | architecture | pretrained:mobilenet_v3_small IMAGENET1K_V1 features[0..8] (unchanged from exp 11) | 1.09 km | 3,225 KB | 2.7 ms | 36 m 53 s | $0.42 | kept | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The design under test — technical diagram — gray = frozen contract · ink = the current design · red = this experiment's change One real test, end to end
→
1.70 km
miss, this crop 0.71
self-reported confidence In plain words A drone photo can be taken facing any direction, but during practice the model only ever saw each place from one direction — so at test time it often fails to recognize a spot simply because the picture is turned. The fix: show the model every photo four times — spun by 0°, 90°, 180° and 270° — and let the four looks vote on where the drone is. A wrong guess that only appears at one spin gets outvoted; the right answer keeps winning at every spin. Same brain, four glances, one vote. Hypothesis The frozen eval set is heading-agnostic (dataset.py gives every held-out crop a deterministic random rotation), but train.py's training tensor shows each training location at exactly ONE random heading, frozen for all 8 epochs — so at eval the model must recognize places across up to 180° of heading mismatch with a plain CNN + FC field head that has no rotational structure whatsoever. Invariance-by-data was tried once (exp 6, fresh rotations every epoch) and refuted: making a 1024-way memorization head learn every location at every heading is sample-inefficient, and more rotated examples did not help. The untried alternative is invariance by construction: evaluate the field on the crop's four exact 90° pixel rotations and average the logits (C4 symmetrization). This (a) guarantees one of the four evaluated views lies within 45° of the one trained heading, versus up to 180° today; (b) makes the model exactly invariant over the C4 subgroup, so training capacity only has to cover a ±45° residual; and (c) since a logit mean is a geometric-mean ensemble, a lookalike cell that fires at only one heading is multiplicatively suppressed — directly attacking the committed-wrong 2–4 km error mode that exps 14/15 diagnosed as what pins every cell's median near 1 km. Method model/model.py forward() only: stack the input with its three 90° pixel rotations (transpose+flip — exact permutations, no interpolation, ONNX-safe) into a 4N batch, run the unchanged trunk + layout squeeze + lighting gate + two-expert heads on all four copies, average the four 1024-d field logit vectors (and the four GAP descriptors for the confidence head's input) before the unchanged β-sharpened decode and field-shape confidence. loss_fn, train.py, calibration, and the ONNX export contract untouched. Expected outcome A broad drop across all 24 area×bucket cells rather than a single-cell fix: worst-case median (currently Munich night, 1368.6 m) improves ≥8% to ≤ ~1250 m if heading mismatch is a binding error source; roughly flat (±3%) if it is not. Model bytes unchanged (~3.3 MiB vs 4 MiB gate); host latency proxy ~4× (≈3 ms vs 250 ms gate); training wall time ~4× (~70 min/area) with no train.py change. Result primary worst-case median error = 1094.59 m (previous best 1368.6 m); full per-area/bucket breakdown in metrics.json Conclusion KEPT — metric improved (1368.6 -> 1094.59 m); change committed as 5c5373d Scoreboard — median error (m) per area × lighting
the worst cell (underlined) is the experiment's score;
red = failed cell, ink = at target
Fits the aircraft? all deployment gates passed
largest per-area model 3,225 KB (limit 4,096 KB) · single-frame inference 2.7 ms on one CPU thread (proxy limit 250 ms)
crops are sampled fresh each run from the frozen train split (~45,000 distinct positions per area, times random rotation), never from eval blocks; how many to use is the experiment's own choice
Where this iteration's time went: design (Haiku) 13 m 25 s · implement (Sonnet) 1 m 29 s · train 4 areas 19 m 42 s · score 1 m 34 s · samples 43 s · publish 5 s · whole iteration 36 m 58 s
ts 2026-07-21T14:11:14 · commit 5c5373dff494 ·
parent 7a4965213c9f · artifacts runs/20260721_133421_iter3 ·
agent model — · took 36 m 53 s
The exact prompt given to the headless agent# Autoresearch iteration — one focused experiment
You are one iteration of an autonomous research loop for UAV low-light
geolocalization. Read `CLAUDE.md` (§1, §3, §6 especially) for full context.
The harness (loop.sh) will train, score, log, and keep/revert AFTER you exit —
you only design the experiment and edit the code.
## Your job, in order
1. **Review the research history.** Query the lineage DB:
`sqlite3 experiments.sqlite "SELECT id, title, category, hypothesis, expected_outcome, result, conclusion, primary_metric, kept FROM experiments ORDER BY id DESC LIMIT 15;"`
Note which hypotheses were supported/refuted. Do not repeat a refuted
experiment without a materially new angle.
**Plateau rule:** if three or more consecutive experiments were reverted,
do not attempt another variation of the last refuted mechanism. Either
pick a design family absent from the history (dispatcher + lighting
specialists, pretrained init, learned relighting, training-scale, …) or
attack the bottleneck the refuted hypotheses jointly point at.
2. **Design ONE focused experiment** — proper experiment design, pre-registered
before you touch code. Write it to `runs/pending_experiment.json`:
```json
{
"title": "one-line name",
"category": "architecture|loss|augmentation|relighting|training|quantization|other",
"hypothesis": "what you believe is limiting the metric and why this change addresses it",
"method": "the ONE focused change, concretely (files, mechanism)",
"expected_outcome": "predicted effect on the §6 worst-case median error, quantified if possible",
"init_strategy": "from-scratch | pretrained:<name>",
"eli5": "2-4 sentences for a smart non-ML reader: what you changed and why it might help, in everyday language — analogies welcome, zero jargon",
"architecture": {"stages": [
{"name": "Camera frame", "detail": "128×128 px crop, one of 6 lighting renders", "changed": false},
{"name": "Feature extractor", "detail": "plain-language description", "changed": false},
{"name": "…", "detail": "…", "changed": true}
]},
"architecture_svg": "<svg viewBox='0 0 980 300' …>…</svg>"
}
```
`architecture_svg` is the **technical architecture diagram** of the model
you are testing — a proper ML-paper figure of the design, drawn by you,
shown at the top of this experiment's gallery entry. **Draw tensors and
operations as the things they are — NOT as labeled boxes:** the input
image as a pixel-textured square; conv feature maps as pseudo-3D slabs
that shrink spatially and deepen in channels; 1-D vectors as thin
vertical tick-bars; fully-connected layers as fans of thin crossing
lines between bars; spatial fields/grids as actually-drawn grids with
shaded cells; decodes/aggregations as thin lines converging from cells
onto a point; samplers/targets as small pictorial glyphs. Captions sit
UNDER each element (name ≤10.5px weight 600, sub-note ≤9.5px); tensor
shapes annotate the flow in small italics (128×128×3 → 8²×128 → 1024).
Losses/targets/samplers live in a lane below the inference path,
annotated with slanted dashed leader lines to small text — no boxes
around them. Draw the two FROZEN endpoints — the camera-frame input and
the (lat, lon, confidence) output — entirely in #9b998c with a small
italic “frozen contract” tag: they are fixed by the harness and outside
your search space. Everything between them is your design: draw it in ink,
with red only on what THIS experiment changes. Style contract, so all experiments' figures read as one
paper: viewBox width 980 (height ~300–360); transparent background; ink
#111111, captions #6b6a60, annotations/arrows #9b998c; **#8c2f1f (red)
reserved for exactly what this experiment changed**; training-only
elements in #8a6a1e (or red if changed); font-family
Palatino,Georgia,serif; stroke-width ≈1.2 elements, 1 arrows; fills only
faint tints (opacity ≤ .12); no gradients, no icons, no emoji. Inference
flows left → right from camera frame to (lat, lon, confidence).
**Anchor the frozen endpoints identically in every figure** so all
experiments' figures line up when compared down the gallery page: the
camera-frame square starts at x=26 (its captions centered on x=53), and
the output is right-anchored — decode crosshair at x=812, the
(lat, lon, confidence) text block text-anchor=start at x=828. Mark the
anchors machine-checkably: the camera-frame rect carries
`id='frozen-input'`, the output text block carries `id='frozen-output'`.
Lay the elements between them out on a running x-cursor with generous
spacing so nothing overlaps.
**Readability contract:** no text may overlap other text, and no line
(arrows, converge fans, leader lines) may pass through a label — route
leaders around text with a bend, or move the label. Width is fixed at
980, but **height is yours**: grow the viewBox (240–640) whenever more
vertical room makes the layout cleaner rather than cramming.
**Before finishing, validate:**
`.venv/bin/python -m autoresearch.figcheck` — it checks the anchor
contract on runs/pending_experiment.json; revise until it prints PASS.
Consecutive conv layers must be tied by kernel-projection lines (small
kernel square on one face, faint lines converging to a cell on the
next) so the encoder reads as one computation, not boxes in a row.
`archive/arch_svg_reference.py` is the concrete reference implementation
of this entire visual language — read it and reuse its helper geometry
for any stage that already has one.
`eli5` and `architecture` feed the human-facing gallery. `architecture.stages`
is the model's inference path left-to-right, camera frame → (lat, lon,
confidence) output — one box per stage, plain-language `detail`,
`"changed": true` ONLY on the stages this experiment touches. Reuse the
previous experiment's stage names verbatim wherever a stage is unchanged
(check `SELECT arch_json FROM experiments WHERE arch_json IS NOT NULL ORDER BY id DESC LIMIT 1;`) —
the diagrams are compared box-by-box across experiments. A change that only
affects training (loss, augmentation, schedule) keeps the inference stages
unchanged and adds one final stage with `"train_only": true` describing the
training signal.
3. **Write the implementation brief.** You do NOT edit `model/` yourself —
a separate implementation agent applies your design, seeing only the
current code plus what you pre-registered. Add one more field to
`runs/pending_experiment.json`:
```
"implementation_brief": "exact file-level instructions: which functions/
blocks in model/model.py and model/train.py change and how, what stays
untouched, and every contract to preserve"
```
Be precise enough that a competent engineer with no other context
implements it in one pass. Always restate the fixed contracts:
`train.py`'s CLI and the ONNX export contract in `model/model.py`'s
docstring — the frozen scorer depends on them.
## Hard rules
- Edit ONLY `runs/pending_experiment.json`. You never edit `model/` (the
implementation stage does) and files listed in `/FROZEN` are off-limits;
the harness hard-reverts any change to them.
- NEVER touch, read, or evaluate the `hamburg` holdout area (§5).
- ONE focused change per iteration — if you can't describe it in one sentence,
it's too big. Prefer architectural/procedural novelty over hyperparameter
nudges (§3): changing a learning rate is a weak experiment; changing the
coordinate parameterization, loss family, relighting-for-training, or model
topology is a strong one.
- Do not run training yourself; the harness does that.
- Stay within the deployment gates: exported ONNX ≤ 4 MiB per area, host
latency proxy ≤ 250 ms (see pipeline/score.py).Where the model was tested — and how far off it was Each map is one full test area. Every dot is one held-out test location (all six lighting conditions overlaid): the model was shown a 128 m crop centered there and asked for its position. Dot color = the distance between its answer and the truth — green ≤ 20 m (at goal), amber ≤ 50 m, red beyond. A working model turns these maps green; spatial clusters of red reveal which parts of an area confuse it.     What the six lighting conditions look like through the simulated low-light sensor (example patches — illustration, not the training set)Every frame in this project is rendered as a starlight-class low-light sensor (Sony STARVIS2 / IMX585 class — the airframe's chosen camera) would see it, not as a normal camera would. That is why the night renders look uncannily close to daylight: at high gain such a sensor recovers scene structure from moonlight, skyglow and artificial lighting, paying for it in noise, lifted shadows and washed-out color — exactly what these renders simulate. Below, one example 256 m patch per area under the six lighting conditions the model must handle. These illustrate the dataset, not this experiment's performance — the actual training set is thousands of distinct crops per area (see “training data” above), and these renderings only change when the relighting method changes. berlin frankfurt munich prignitz | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▸ | 15 | Selective prediction: field-shape confidence head with per-bucket calibrated abstention f3b87247 | architecture | pretrained:mobilenet_v3_small IMAGENET1K_V1 features[0..8] (torchvision, BSD-3) | 1.37 km | 3,222 KB | 0.8 ms | 40 m 01 s | $0.46 | kept | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The design under test — technical diagram — gray = frozen contract · ink = the current design · red = this experiment's change One real test, end to end
→
1.36 km
miss, this crop 0.40
self-reported confidence In plain words Until now the drone's navigator shouted out a position guess for every single photo, even the hopeless ones, and those wild guesses dragged its accuracy score down. This change teaches it to look at its own heat map of candidate locations and notice the difference between one crisp hotspot (sure) and a washed-out smear of weak maybes (not sure) — and in the unsure case to say 'no fix this time', letting the drone coast on its last known position for a few extra seconds, which the mission profile explicitly allows. The scoring rules have always worked this way — only answers it commits to count, as long as it doesn't stay silent too often — so we tuned it to keep answering at least 40% of the time in every lighting condition, double the required minimum. Hypothesis The frozen §6 scorer has implemented selective prediction since bootstrap — a prediction with conf < 0.3 is an abstention excluded from that cell's median, and only coverage ≥ 0.2 is required — but the confidence pathway is still the bootstrap placeholder trained toward 'error < half the map extent', so conf saturates near 1 and coverage has been 1.000 in every one of the 24 cells of every experiment ever logged: the abstention mechanism §6 explicitly designed for ('a model that honestly abstains on bad frames is more useful than one that confidently guesses wrong') has never once fired. Meanwhile exp 14 (kept, 1856.69 -> 1638.3 m) established exactly the error-population structure that selective prediction exploits: its mixture analysis put only p ≈ 0.45–0.6 of field mass near truth, and its peak-commit decode splits predictions into a committed-correct subpopulation (a few hundred m) and committed-wrong misses (2–4 km) — confirmed by the predicted rise of the mean/median ratio from ~1.2 to ~1.35–1.4 in every area × bucket cell. Each cell's median therefore sits just on the wrong side of a bimodal error distribution: if the model can RANK its own crops by whether the field committed correctly, abstaining on the worst-ranked ones moves the median into the correct-commit mode. The ranking signal is already computed at inference and thrown away: the SHAPE of the decode distribution — sharpened-field peak mass, field entropy, and the distance between the sharpened and unsharpened decodes (a direct diffuseness/multimodality readout, the same diagnostic exp 14's analysis was built on). The confidence mechanism is a design family untouched in 14 experiments; it is deployment-honest (a UAV that skips an unsure fix coasts on dead reckoning for the 5–10 s until the next frame per §2's adaptive schedule — strictly better than ingesting a 2-km-wrong fix); and it can be made safe against the coverage-FAIL gate by construction: train the head as a calibrated hit predictor (BCE on 'committed decode within τ = 0.05 uv ≈ 350 m ≈ 1.6 grid cells') on detached inputs, then bake a per-area logit offset into the export, calibrated per lighting bucket on fresh-rotation TRAIN-split crops so that ≥40% clear the fixed 0.3 threshold in EVERY bucket — double the 0.2 coverage floor. Method ONE focused change: the deployed confidence pathway. model/model.py — replace the placeholder Linear(48->1) conf head with a small MLP over [48-d GAP descriptor, sharpened-field peak mass, normalized field entropy, distance between sharpened and unsharpened decodes], ALL inputs detached so no confidence gradient can touch trunk/heads; conf = sigmoid(z - conf_shift) with conf_shift a registered buffer (0 during training); loss_fn's conf target tightens from err<0.5 to err<GOOD_ERR_UV=0.05 (weight 0.1->0.3). model/train.py — after the epoch loop, per-area calibration on ~400 fresh-rotation TRAIN-split crops per lighting bucket sets conf_shift so the scorer's fixed 0.3 threshold keeps ≥40% in the worst bucket, then export. Trunk, layout head, dark expert, gate, β=3 decode, Gaussian-CE and coordinate L2: unchanged. Expected outcome If the field-shape statistics rank commit-correctness with even moderate skill (AUC ≥ ~0.7), each cell's median moves toward the committed-correct subpopulation: predicted primary worst-case median 900–1400 m (from 1638.3 m), with the largest relative gains in the night cells (most diffuse fields, so the most filterable misses — night is the current worst bucket in all four areas). Coverage (secondary, reported honestly) drops from 1.0 to ~0.4–0.75 per cell; calibration keep-rates per bucket are logged in train_info.json. Null: confidence uninformative -> the kept subset is a near-random sample, median unchanged ≈1638 m, revert. Bounded worst case: some eval cell's coverage < 0.2 -> 1e9 -> revert; made unlikely by the 40%-keep calibration (2× the floor). If supported, the loop's subsequent battlefield changes shape: further gains must raise field quality so coverage climbs at fixed accuracy — both numbers are in every cell's log. Result primary worst-case median error = 1368.6 m (previous best 1638.3 m); full per-area/bucket breakdown in metrics.json Conclusion KEPT — metric improved (1638.3 -> 1368.6 m); change committed as f3b8724 Scoreboard — median error (m) per area × lighting
the worst cell (underlined) is the experiment's score;
red = failed cell, ink = at target
Fits the aircraft? all deployment gates passed
largest per-area model 3,222 KB (limit 4,096 KB) · single-frame inference 0.8 ms on one CPU thread (proxy limit 250 ms)
crops are sampled fresh each run from the frozen train split (~45,000 distinct positions per area, times random rotation), never from eval blocks; how many to use is the experiment's own choice
Where this iteration's time went: design (Haiku) 18 m 33 s · implement (Sonnet) 1 m 47 s · train 4 areas 17 m 42 s · score 1 m 16 s · samples 42 s · publish 5 s · whole iteration 40 m 06 s
ts 2026-07-21T13:34:06 · commit f3b87247b232 ·
parent da623e92500d · artifacts runs/20260721_125405_iter2 ·
agent model — · took 40 m 01 s
The exact prompt given to the headless agent# Autoresearch iteration — one focused experiment
You are one iteration of an autonomous research loop for UAV low-light
geolocalization. Read `CLAUDE.md` (§1, §3, §6 especially) for full context.
The harness (loop.sh) will train, score, log, and keep/revert AFTER you exit —
you only design the experiment and edit the code.
## Your job, in order
1. **Review the research history.** Query the lineage DB:
`sqlite3 experiments.sqlite "SELECT id, title, category, hypothesis, expected_outcome, result, conclusion, primary_metric, kept FROM experiments ORDER BY id DESC LIMIT 15;"`
Note which hypotheses were supported/refuted. Do not repeat a refuted
experiment without a materially new angle.
**Plateau rule:** if three or more consecutive experiments were reverted,
do not attempt another variation of the last refuted mechanism. Either
pick a design family absent from the history (dispatcher + lighting
specialists, pretrained init, learned relighting, training-scale, …) or
attack the bottleneck the refuted hypotheses jointly point at.
2. **Design ONE focused experiment** — proper experiment design, pre-registered
before you touch code. Write it to `runs/pending_experiment.json`:
```json
{
"title": "one-line name",
"category": "architecture|loss|augmentation|relighting|training|quantization|other",
"hypothesis": "what you believe is limiting the metric and why this change addresses it",
"method": "the ONE focused change, concretely (files, mechanism)",
"expected_outcome": "predicted effect on the §6 worst-case median error, quantified if possible",
"init_strategy": "from-scratch | pretrained:<name>",
"eli5": "2-4 sentences for a smart non-ML reader: what you changed and why it might help, in everyday language — analogies welcome, zero jargon",
"architecture": {"stages": [
{"name": "Camera frame", "detail": "128×128 px crop, one of 6 lighting renders", "changed": false},
{"name": "Feature extractor", "detail": "plain-language description", "changed": false},
{"name": "…", "detail": "…", "changed": true}
]},
"architecture_svg": "<svg viewBox='0 0 980 300' …>…</svg>"
}
```
`architecture_svg` is the **technical architecture diagram** of the model
you are testing — a proper ML-paper figure of the design, drawn by you,
shown at the top of this experiment's gallery entry. **Draw tensors and
operations as the things they are — NOT as labeled boxes:** the input
image as a pixel-textured square; conv feature maps as pseudo-3D slabs
that shrink spatially and deepen in channels; 1-D vectors as thin
vertical tick-bars; fully-connected layers as fans of thin crossing
lines between bars; spatial fields/grids as actually-drawn grids with
shaded cells; decodes/aggregations as thin lines converging from cells
onto a point; samplers/targets as small pictorial glyphs. Captions sit
UNDER each element (name ≤10.5px weight 600, sub-note ≤9.5px); tensor
shapes annotate the flow in small italics (128×128×3 → 8²×128 → 1024).
Losses/targets/samplers live in a lane below the inference path,
annotated with slanted dashed leader lines to small text — no boxes
around them. Draw the two FROZEN endpoints — the camera-frame input and
the (lat, lon, confidence) output — entirely in #9b998c with a small
italic “frozen contract” tag: they are fixed by the harness and outside
your search space. Everything between them is your design: draw it in ink,
with red only on what THIS experiment changes. Style contract, so all experiments' figures read as one
paper: viewBox width 980 (height ~300–360); transparent background; ink
#111111, captions #6b6a60, annotations/arrows #9b998c; **#8c2f1f (red)
reserved for exactly what this experiment changed**; training-only
elements in #8a6a1e (or red if changed); font-family
Palatino,Georgia,serif; stroke-width ≈1.2 elements, 1 arrows; fills only
faint tints (opacity ≤ .12); no gradients, no icons, no emoji. Inference
flows left → right from camera frame to (lat, lon, confidence).
**Anchor the frozen endpoints identically in every figure** so all
experiments' figures line up when compared down the gallery page: the
camera-frame square starts at x=26 (its captions centered on x=53), and
the output is right-anchored — decode crosshair at x=812, the
(lat, lon, confidence) text block text-anchor=start at x=828. Lay the
elements between them out on a running x-cursor with generous spacing
so nothing overlaps.
Consecutive conv layers must be tied by kernel-projection lines (small
kernel square on one face, faint lines converging to a cell on the
next) so the encoder reads as one computation, not boxes in a row.
`archive/arch_svg_reference.py` is the concrete reference implementation
of this entire visual language — read it and reuse its helper geometry
for any stage that already has one.
`eli5` and `architecture` feed the human-facing gallery. `architecture.stages`
is the model's inference path left-to-right, camera frame → (lat, lon,
confidence) output — one box per stage, plain-language `detail`,
`"changed": true` ONLY on the stages this experiment touches. Reuse the
previous experiment's stage names verbatim wherever a stage is unchanged
(check `SELECT arch_json FROM experiments WHERE arch_json IS NOT NULL ORDER BY id DESC LIMIT 1;`) —
the diagrams are compared box-by-box across experiments. A change that only
affects training (loss, augmentation, schedule) keeps the inference stages
unchanged and adds one final stage with `"train_only": true` describing the
training signal.
3. **Write the implementation brief.** You do NOT edit `model/` yourself —
a separate implementation agent applies your design, seeing only the
current code plus what you pre-registered. Add one more field to
`runs/pending_experiment.json`:
```
"implementation_brief": "exact file-level instructions: which functions/
blocks in model/model.py and model/train.py change and how, what stays
untouched, and every contract to preserve"
```
Be precise enough that a competent engineer with no other context
implements it in one pass. Always restate the fixed contracts:
`train.py`'s CLI and the ONNX export contract in `model/model.py`'s
docstring — the frozen scorer depends on them.
## Hard rules
- Edit ONLY `runs/pending_experiment.json`. You never edit `model/` (the
implementation stage does) and files listed in `/FROZEN` are off-limits;
the harness hard-reverts any change to them.
- NEVER touch, read, or evaluate the `hamburg` holdout area (§5).
- ONE focused change per iteration — if you can't describe it in one sentence,
it's too big. Prefer architectural/procedural novelty over hyperparameter
nudges (§3): changing a learning rate is a weak experiment; changing the
coordinate parameterization, loss family, relighting-for-training, or model
topology is a strong one.
- Do not run training yourself; the harness does that.
- Stay within the deployment gates: exported ONNX ≤ 4 MiB per area, host
latency proxy ≤ 250 ms (see pipeline/score.py).Where the model was tested — and how far off it was Each map is one full test area. Every dot is one held-out test location (all six lighting conditions overlaid): the model was shown a 128 m crop centered there and asked for its position. Dot color = the distance between its answer and the truth — green ≤ 20 m (at goal), amber ≤ 50 m, red beyond. A working model turns these maps green; spatial clusters of red reveal which parts of an area confuse it.     What the six lighting conditions look like through the simulated low-light sensor (example patches — illustration, not the training set)Every frame in this project is rendered as a starlight-class low-light sensor (Sony STARVIS2 / IMX585 class — the airframe's chosen camera) would see it, not as a normal camera would. That is why the night renders look uncannily close to daylight: at high gain such a sensor recovers scene structure from moonlight, skyglow and artificial lighting, paying for it in noise, lifted shadows and washed-out color — exactly what these renders simulate. Below, one example 256 m patch per area under the six lighting conditions the model must handle. These illustrate the dataset, not this experiment's performance — the actual training set is thousands of distinct crops per area (see “training data” above), and these renderings only change when the relighting method changes. berlin frankfurt munich prignitz | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▸ | 14 | Peak-commit decode: β-sharpened softmax replaces mean-of-map soft-argmax d2a0a2b9 | architecture | pretrained:mobilenet_v3_small IMAGENET1K_V1 features[0..8] (torchvision, BSD-3 — unchanged from exp 11) | 1.64 km | 3,214 KB | 0.8 ms | 31 m 13 s | $0.36 | kept | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The design under test — technical diagram — gray = frozen contract · ink = the current design · red = this experiment's change One real test, end to end
→
1.80 km
miss, this crop 0.96
self-reported confidence In plain words The model draws a heat map of 'where might this photo be?' over the whole city, then has to give one answer. Until now it answered with the average of every warm spot at once — like aiming a dart at the average of three boards you suspect, which lands you in the wall between them. We now turn up the contrast on the heat map first, so the strongest hotspot dominates and the faint background noise vanishes, and the model answers at that hotspot. It is also graded during training on this committed answer, so it learns to put its hotspot in the right place rather than hedging toward the middle of the map. Hypothesis After exps 10-12 (all kept: layout head, pretrained trunk, gated dark expert; 2489 -> 1857 m) the binding constraint is no longer feature quality but the decode itself: global soft-argmax returns the probability-weighted MEAN of the whole 32x32 field, so any residual diffuse or multimodal mass drags every prediction toward the map centroid. Three lines of evidence converge on this. (1) The error profile has the exact shrinkage signature: medians sit at 1.0-1.9 km against a ~3.2 km center-guess floor with mean/median ratios of only ~1.15-1.25 in every area x bucket cell — a light-tailed, globally-shrunk distribution, not the heavy tail that confident-but-wrong mode commitment would produce. Under a mixture model 'p mass near truth + (1-p) diffuse', median ≈ (1-p) x center-distance gives p ≈ 0.45-0.6 across buckets: the field already places roughly half its mass near the right answer and the expectation decode throws that away. (2) The training loss actively teaches this failure: the coordinate term is an L2 on the decoded expectation, whose minimizer IS the posterior mean, while the §6 metric is a MEDIAN of radial error, which rewards committing to the dominant mode — a direct loss-metric mismatch nothing in the history has addressed. (3) The only prior attack on the decode, exp 3 (argmax-anchored 5x5-window soft-argmax), was refuted in the from-scratch-encoder era, and its own conclusion attributed the failure to near-uniform fields making the argmax pure noise; that failure mode no longer applies now that exps 10-12 produced fields with real structure (first-ever per-lighting-bucket error gradient). The materially new angle vs exp 3: no hard argmax anchor and no window — a fully differentiable global sharpening (decode from softmax(β·logits), inverse-temperature β = 3) that leaves a uniform field exactly uniform (so the untrained model still decodes to the map center — same bounded-downside floor as every kept experiment) but drives any field with a dominant mode toward that mode, and lets the existing coordinate L2 automatically retrain the network on the committed answer. Method One line in model/model.py's TinyLocNet.forward(): the decode distribution becomes p = softmax(DECODE_BETA * logits) with a new module constant DECODE_BETA = 3.0; the soft-argmax expectation over cell centers, the confidence head, and the gated two-head field are untouched. loss_fn is left byte-identical: its cross-entropy consumes the raw logits (log_softmax(logits)), so the field keeps learning the same calibrated Gaussian-bump targets, while its coordinate L2 reads pred[:, :2] and therefore automatically grades the sharpened, committed decode — training and inference stay coupled on exactly what flies. No new parameters, train.py unchanged, ONNX contract and deployment gates unaffected. Expected outcome If the shrinkage arithmetic is right (~half the field mass already near truth), commitment converts hedged 1.3-1.9 km medians into mode-position errors: predicted worst-case median 900-1600 m (from 1856.69 m), improving all buckets, largest absolute gains where fields are most peaked (bright buckets). A confirming secondary signature: mean/median ratio should RISE (occasional confident misses replace universal hedging) even as the median falls. If instead the field's dominant modes are usually wrong, the median regresses toward 2-3 km (exp 3's fate, 2937 m) — which would cleanly refute the shrinkage reading and redirect the loop at field quality (training-scale, relighting-side signal) rather than decode mechanics. Result primary worst-case median error = 1638.3 m (previous best 1856.69 m); full per-area/bucket breakdown in metrics.json Conclusion KEPT — metric improved (1856.69 -> 1638.3 m); change committed as d2a0a2b Scoreboard — median error (m) per area × lighting
the worst cell (underlined) is the experiment's score;
red = failed cell, ink = at target
Fits the aircraft? all deployment gates passed
largest per-area model 3,214 KB (limit 4,096 KB) · single-frame inference 0.8 ms on one CPU thread (proxy limit 250 ms)
crops are sampled fresh each run from the frozen train split (~45,000 distinct positions per area, times random rotation), never from eval blocks; how many to use is the experiment's own choice
Where this iteration's time went: design (Haiku) 11 m 34 s · implement (Sonnet) 36 s · train 4 areas 17 m 06 s · score 1 m 15 s · samples 42 s · publish 13 s · whole iteration 31 m 26 s
ts 2026-07-21T12:53:43 · commit d2a0a2b9d817 ·
parent c62bf2b1690f · artifacts runs/20260721_122230_iter1 ·
agent model — · took 31 m 13 s
The exact prompt given to the headless agent# Autoresearch iteration — one focused experiment
You are one iteration of an autonomous research loop for UAV low-light
geolocalization. Read `CLAUDE.md` (§1, §3, §6 especially) for full context.
The harness (loop.sh) will train, score, log, and keep/revert AFTER you exit —
you only design the experiment and edit the code.
## Your job, in order
1. **Review the research history.** Query the lineage DB:
`sqlite3 experiments.sqlite "SELECT id, title, category, hypothesis, expected_outcome, result, conclusion, primary_metric, kept FROM experiments ORDER BY id DESC LIMIT 15;"`
Note which hypotheses were supported/refuted. Do not repeat a refuted
experiment without a materially new angle.
**Plateau rule:** if three or more consecutive experiments were reverted,
do not attempt another variation of the last refuted mechanism. Either
pick a design family absent from the history (dispatcher + lighting
specialists, pretrained init, learned relighting, training-scale, …) or
attack the bottleneck the refuted hypotheses jointly point at.
2. **Design ONE focused experiment** — proper experiment design, pre-registered
before you touch code. Write it to `runs/pending_experiment.json`:
```json
{
"title": "one-line name",
"category": "architecture|loss|augmentation|relighting|training|quantization|other",
"hypothesis": "what you believe is limiting the metric and why this change addresses it",
"method": "the ONE focused change, concretely (files, mechanism)",
"expected_outcome": "predicted effect on the §6 worst-case median error, quantified if possible",
"init_strategy": "from-scratch | pretrained:<name>",
"eli5": "2-4 sentences for a smart non-ML reader: what you changed and why it might help, in everyday language — analogies welcome, zero jargon",
"architecture": {"stages": [
{"name": "Camera frame", "detail": "128×128 px crop, one of 6 lighting renders", "changed": false},
{"name": "Feature extractor", "detail": "plain-language description", "changed": false},
{"name": "…", "detail": "…", "changed": true}
]},
"architecture_svg": "<svg viewBox='0 0 980 300' …>…</svg>"
}
```
`architecture_svg` is the **technical architecture diagram** of the model
you are testing — a proper ML-paper figure of the design, drawn by you,
shown at the top of this experiment's gallery entry. **Draw tensors and
operations as the things they are — NOT as labeled boxes:** the input
image as a pixel-textured square; conv feature maps as pseudo-3D slabs
that shrink spatially and deepen in channels; 1-D vectors as thin
vertical tick-bars; fully-connected layers as fans of thin crossing
lines between bars; spatial fields/grids as actually-drawn grids with
shaded cells; decodes/aggregations as thin lines converging from cells
onto a point; samplers/targets as small pictorial glyphs. Captions sit
UNDER each element (name ≤10.5px weight 600, sub-note ≤9.5px); tensor
shapes annotate the flow in small italics (128×128×3 → 8²×128 → 1024).
Losses/targets/samplers live in a lane below the inference path,
annotated with slanted dashed leader lines to small text — no boxes
around them. Draw the two FROZEN endpoints — the camera-frame input and
the (lat, lon, confidence) output — entirely in #9b998c with a small
italic “frozen contract” tag: they are fixed by the harness and outside
your search space. Everything between them is your design: draw it in ink,
with red only on what THIS experiment changes. Style contract, so all experiments' figures read as one
paper: viewBox width 980 (height ~300–360); transparent background; ink
#111111, captions #6b6a60, annotations/arrows #9b998c; **#8c2f1f (red)
reserved for exactly what this experiment changed**; training-only
elements in #8a6a1e (or red if changed); font-family
Palatino,Georgia,serif; stroke-width ≈1.2 elements, 1 arrows; fills only
faint tints (opacity ≤ .12); no gradients, no icons, no emoji. Inference
flows left → right from camera frame to (lat, lon, confidence).
**Anchor the frozen endpoints identically in every figure** so all
experiments' figures line up when compared down the gallery page: the
camera-frame square starts at x=26 (its captions centered on x=53), and
the output is right-anchored — decode crosshair at x=812, the
(lat, lon, confidence) text block text-anchor=start at x=828. Lay the
elements between them out on a running x-cursor with generous spacing
so nothing overlaps.
Consecutive conv layers must be tied by kernel-projection lines (small
kernel square on one face, faint lines converging to a cell on the
next) so the encoder reads as one computation, not boxes in a row.
`archive/arch_svg_reference.py` is the concrete reference implementation
of this entire visual language — read it and reuse its helper geometry
for any stage that already has one.
`eli5` and `architecture` feed the human-facing gallery. `architecture.stages`
is the model's inference path left-to-right, camera frame → (lat, lon,
confidence) output — one box per stage, plain-language `detail`,
`"changed": true` ONLY on the stages this experiment touches. Reuse the
previous experiment's stage names verbatim wherever a stage is unchanged
(check `SELECT arch_json FROM experiments WHERE arch_json IS NOT NULL ORDER BY id DESC LIMIT 1;`) —
the diagrams are compared box-by-box across experiments. A change that only
affects training (loss, augmentation, schedule) keeps the inference stages
unchanged and adds one final stage with `"train_only": true` describing the
training signal.
3. **Write the implementation brief.** You do NOT edit `model/` yourself —
a separate implementation agent applies your design, seeing only the
current code plus what you pre-registered. Add one more field to
`runs/pending_experiment.json`:
```
"implementation_brief": "exact file-level instructions: which functions/
blocks in model/model.py and model/train.py change and how, what stays
untouched, and every contract to preserve"
```
Be precise enough that a competent engineer with no other context
implements it in one pass. Always restate the fixed contracts:
`train.py`'s CLI and the ONNX export contract in `model/model.py`'s
docstring — the frozen scorer depends on them.
## Hard rules
- Edit ONLY `runs/pending_experiment.json`. You never edit `model/` (the
implementation stage does) and files listed in `/FROZEN` are off-limits;
the harness hard-reverts any change to them.
- NEVER touch, read, or evaluate the `hamburg` holdout area (§5).
- ONE focused change per iteration — if you can't describe it in one sentence,
it's too big. Prefer architectural/procedural novelty over hyperparameter
nudges (§3): changing a learning rate is a weak experiment; changing the
coordinate parameterization, loss family, relighting-for-training, or model
topology is a strong one.
- Do not run training yourself; the harness does that.
- Stay within the deployment gates: exported ONNX ≤ 4 MiB per area, host
latency proxy ≤ 250 ms (see pipeline/score.py).Where the model was tested — and how far off it was Each map is one full test area. Every dot is one held-out test location (all six lighting conditions overlaid): the model was shown a 128 m crop centered there and asked for its position. Dot color = the distance between its answer and the truth — green ≤ 20 m (at goal), amber ≤ 50 m, red beyond. A working model turns these maps green; spatial clusters of red reveal which parts of an area confuse it.     What the six lighting conditions look like through the simulated low-light sensor (example patches — illustration, not the training set)Every frame in this project is rendered as a starlight-class low-light sensor (Sony STARVIS2 / IMX585 class — the airframe's chosen camera) would see it, not as a normal camera would. That is why the night renders look uncannily close to daylight: at high gain such a sensor recovers scene structure from moonlight, skyglow and artificial lighting, paying for it in noise, lifted shadows and washed-out color — exactly what these renders simulate. Below, one example 256 m patch per area under the six lighting conditions the model must handle. These illustrate the dataset, not this experiment's performance — the actual training set is thousands of distinct crops per area (see “training data” above), and these renderings only change when the relighting method changes. berlin frankfurt munich prignitz | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▸ | 13 | Periodic blind holdout check (hamburg) fe745eb4 | other | — | 1.29 km | 3,213 KB | 0.8 ms | — | — | holdout | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hypothesis If the pipeline is genuinely generic, holdout error should track the development worst-case. Method Train current best model code on hamburg, score read-only (§5). Result informational only — never drives keep/revert Scoreboard — median error (m) per area × lighting
the worst cell (underlined) is the experiment's score;
red = failed cell, ink = at target
Fits the aircraft? all deployment gates passed
largest per-area model 3,213 KB (limit 4,096 KB) · single-frame inference 0.8 ms on one CPU thread (proxy limit 250 ms)
crops are sampled fresh each run from the frozen train split (~45,000 distinct positions per area, times random rotation), never from eval blocks; how many to use is the experiment's own choice
ts 2026-07-21T12:20:37 · commit fe745eb4fa91 ·
parent 5acfb83c6faa · artifacts runs/20260721_113214_iter2/holdout ·
agent model — · took —
Where the model was tested — and how far off it was Each map is one full test area. Every dot is one held-out test location (all six lighting conditions overlaid): the model was shown a 128 m crop centered there and asked for its position. Dot color = the distance between its answer and the truth — green ≤ 20 m (at goal), amber ≤ 50 m, red beyond. A working model turns these maps green; spatial clusters of red reveal which parts of an area confuse it.  | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▸ | 12 | Luminance-gated dark-expert field head blended with the existing layout head fe745eb4 | architecture | pretrained:mobilenet_v3_small IMAGENET1K_V1 features[0..8] (torchvision, BSD-3); new dark head + gate are from-scratch | 1.86 km | 3,213 KB | 0.8 ms | 42 m 10 s | $0.48 | kept | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The design under test — technical diagram — gray = frozen contract · ink = the current design · red = this experiment's change One real test, end to end
→
1.33 km
miss, this crop 0.98
self-reported confidence In plain words The last experiment gave the model 'eyes' that already knew how to see from a million daytime photos, and it worked -- but only for well-lit shots. At night, that same borrowed vision is guessing worse, because those eyes never really learned what a dark, sparsely-lit scene looks like. This experiment adds a second, smaller decision-maker that only looks at the crop's overall texture (not its detailed layout, which is mostly invisible in the dark anyway), plus a tiny 'how dark is this shot?' switch that blends the two decision-makers' guesses. In bright shots the switch leans on the original, more detailed guesser; in dark shots it leans on the simpler one built for low light. Nothing about the camera or the final answer changes -- just how the guess gets made inside. Hypothesis Exp 11 (ImageNet-pretrained MobileNetV3-Small trunk, kept, 2299.74 -> 2010.15 m) produced, for the first time in this project's history, a clear per-lighting-bucket error gradient instead of the flat profile every from-scratch experiment showed: night is worst in all four development areas (Berlin 1600 vs 1381 m midday; Prignitz 1604 vs 1007 m morning; Munich 2010 vs 1381 m midday -- Munich night IS the current primary worst-case metric; Frankfurt 1585 vs 1063 m afternoon). This is new, decisive evidence of a kind no prior experiment produced: pretrained ImageNet features, however gently fine-tuned (trunk lr 1e-4), carry a systematic bias toward well-lit natural-image statistics, and the single shared field head (GAP + layout code -> FC -> 1024 logits) is being forced to compute the same cell-scoring function for both regimes. CLAUDE.md SS3 explicitly leaves 'several small lighting-condition-specific models behind a lightweight dispatcher' vs. one unified model as an open, loop-decided question, and it is the one design family the plateau-rule history (exps 1-11) has never tried -- every experiment so far has kept a single head. A full duplicate specialist head sized like the existing one would blow the deployment gate (loc_logits is a 560->1024 FC, ~573k params / ~2.2 MB; exp 11's export is already 2.94 MiB of the 4 MiB ONNX budget), so this experiment tests the dispatcher/specialist hypothesis with a budget-safe mixture-of-experts-lite mechanism instead of a full dispatcher. Method model/model.py TinyLocNet: add a second, cheap 'dark expert' field head -- nn.Linear(48, 1024) reading ONLY the 48-d GAP descriptor (no 512-d layout code, on the theory that spatial layout is the least reliable signal once texture is mostly gone at night) -- and a tiny gate MLP (Linear(49,16)+ReLU+Linear(16,1)+sigmoid) over [raw-pixel mean brightness of the input frame, 48-d GAP], both derived from the frame itself so the frozen (frame in) -> (lat, lon, conf) contract is untouched. Forward pass: logits = (1-g)*bright_logits + g*dark_logits, where bright_logits is the existing, architecturally unchanged layout-aware head and g in [0,1] is the learned blend weight; softmax/soft-argmax decode, confidence head, GRID_K, and both loss terms are identical to exp 11 -- loss_fn already receives the blended logits via forward(return_logits=True), no changes needed there. No explicit lighting-bucket label is used anywhere; the gate learns to route purely from brightness + features. Adds ~51k params (~200 KB): dark head 50,176 params, gate MLP 817 params. model/train.py: no logic changes -- new params are fresh-initialized and fall into the existing 'head' LR group (1e-3) automatically since they are not in model.features. Verified locally: 816,418 total params (was 765,433), forward+backward flows gradients through both heads and the gate, ONNX 3.14 MiB (< 4 MiB gate, 150 KB of headroom left), torch/ONNX parity 6e-8, untrained decode sits at the map center (0.4999, 0.5023) confirming bounded downside. Expected outcome If the night-bucket gradient reflects a genuine lighting-conditional mismatch that a second cheap expert can correct, the worst-case median error (currently Munich night, 2010.15 m) should drop meaningfully -- plausibly to 1,500-1,900 m -- with the largest relative gains in night and evening buckets and smaller gains in already-strong bright buckets, narrowing (not necessarily eliminating) the bright-vs-night gap rather than moving every bucket uniformly. Downside is tightly bounded: both heads start randomly initialized same as before, decode and both loss terms are unchanged, and if the gate fails to learn a useful split it can converge near a constant blend that leaves the effective function close to the exp-11 head, so a null should land near the current best (~2.0-2.3 km) rather than below it. A null would be informative: it would mean the night-bucket gap is not fixable by adding head capacity conditioned on brightness alone, and would point next iterations at learned/improved relighting for the dark buckets themselves (CLAUDE.md SS4) rather than further architecture changes on the head. Result primary worst-case median error = 1856.69 m (previous best 2010.15 m); full per-area/bucket breakdown in metrics.json Conclusion KEPT — metric improved (2010.15 -> 1856.69 m); change committed as fe745eb Scoreboard — median error (m) per area × lighting
the worst cell (underlined) is the experiment's score;
red = failed cell, ink = at target
Fits the aircraft? all deployment gates passed
largest per-area model 3,213 KB (limit 4,096 KB) · single-frame inference 0.8 ms on one CPU thread (proxy limit 250 ms)
crops are sampled fresh each run from the frozen train split (~45,000 distinct positions per area, times random rotation), never from eval blocks; how many to use is the experiment's own choice
ts 2026-07-21T12:14:24 · commit fe745eb4fa91 ·
parent 5acfb83c6faa · artifacts runs/20260721_113214_iter2 ·
agent model claude-haiku-4-5-20251001, claude-sonnet-5 · took 42 m 10 s
The exact prompt given to the headless agent# Autoresearch iteration — one focused experiment
You are one iteration of an autonomous research loop for UAV low-light
geolocalization. Read `CLAUDE.md` (§1, §3, §6 especially) for full context.
The harness (loop.sh) will train, score, log, and keep/revert AFTER you exit —
you only design the experiment and edit the code.
## Your job, in order
1. **Review the research history.** Query the lineage DB:
`sqlite3 experiments.sqlite "SELECT id, title, category, hypothesis, expected_outcome, result, conclusion, primary_metric, kept FROM experiments ORDER BY id DESC LIMIT 15;"`
Note which hypotheses were supported/refuted. Do not repeat a refuted
experiment without a materially new angle.
**Plateau rule:** if three or more consecutive experiments were reverted,
do not attempt another variation of the last refuted mechanism. Either
pick a design family absent from the history (dispatcher + lighting
specialists, pretrained init, learned relighting, training-scale, …) or
attack the bottleneck the refuted hypotheses jointly point at.
2. **Design ONE focused experiment** — proper experiment design, pre-registered
before you touch code. Write it to `runs/pending_experiment.json`:
```json
{
"title": "one-line name",
"category": "architecture|loss|augmentation|relighting|training|quantization|other",
"hypothesis": "what you believe is limiting the metric and why this change addresses it",
"method": "the ONE focused change, concretely (files, mechanism)",
"expected_outcome": "predicted effect on the §6 worst-case median error, quantified if possible",
"init_strategy": "from-scratch | pretrained:<name>",
"eli5": "2-4 sentences for a smart non-ML reader: what you changed and why it might help, in everyday language — analogies welcome, zero jargon",
"architecture": {"stages": [
{"name": "Camera frame", "detail": "128×128 px crop, one of 6 lighting renders", "changed": false},
{"name": "Feature extractor", "detail": "plain-language description", "changed": false},
{"name": "…", "detail": "…", "changed": true}
]},
"architecture_svg": "<svg viewBox='0 0 980 300' …>…</svg>"
}
```
`architecture_svg` is the **technical architecture diagram** of the model
you are testing — a proper ML-paper figure of the design, drawn by you,
shown at the top of this experiment's gallery entry. **Draw tensors and
operations as the things they are — NOT as labeled boxes:** the input
image as a pixel-textured square; conv feature maps as pseudo-3D slabs
that shrink spatially and deepen in channels; 1-D vectors as thin
vertical tick-bars; fully-connected layers as fans of thin crossing
lines between bars; spatial fields/grids as actually-drawn grids with
shaded cells; decodes/aggregations as thin lines converging from cells
onto a point; samplers/targets as small pictorial glyphs. Captions sit
UNDER each element (name ≤10.5px weight 600, sub-note ≤9.5px); tensor
shapes annotate the flow in small italics (128×128×3 → 8²×128 → 1024).
Losses/targets/samplers live in a lane below the inference path,
annotated with slanted dashed leader lines to small text — no boxes
around them. Draw the two FROZEN endpoints — the camera-frame input and
the (lat, lon, confidence) output — entirely in #9b998c with a small
italic “frozen contract” tag: they are fixed by the harness and outside
your search space. Everything between them is your design: draw it in ink,
with red only on what THIS experiment changes. Style contract, so all experiments' figures read as one
paper: viewBox width 980 (height ~300–360); transparent background; ink
#111111, captions #6b6a60, annotations/arrows #9b998c; **#8c2f1f (red)
reserved for exactly what this experiment changed**; training-only
elements in #8a6a1e (or red if changed); font-family
Palatino,Georgia,serif; stroke-width ≈1.2 elements, 1 arrows; fills only
faint tints (opacity ≤ .12); no gradients, no icons, no emoji. Inference
flows left → right from camera frame to (lat, lon, confidence).
**Anchor the frozen endpoints identically in every figure** so all
experiments' figures line up when compared down the gallery page: the
camera-frame square starts at x=26 (its captions centered on x=53), and
the output is right-anchored — decode crosshair at x=812, the
(lat, lon, confidence) text block text-anchor=start at x=828. Lay the
elements between them out on a running x-cursor with generous spacing
so nothing overlaps.
Consecutive conv layers must be tied by kernel-projection lines (small
kernel square on one face, faint lines converging to a cell on the
next) so the encoder reads as one computation, not boxes in a row.
`archive/arch_svg_reference.py` is the concrete reference implementation
of this entire visual language — read it and reuse its helper geometry
for any stage that already has one.
`eli5` and `architecture` feed the human-facing gallery. `architecture.stages`
is the model's inference path left-to-right, camera frame → (lat, lon,
confidence) output — one box per stage, plain-language `detail`,
`"changed": true` ONLY on the stages this experiment touches. Reuse the
previous experiment's stage names verbatim wherever a stage is unchanged
(check `SELECT arch_json FROM experiments WHERE arch_json IS NOT NULL ORDER BY id DESC LIMIT 1;`) —
the diagrams are compared box-by-box across experiments. A change that only
affects training (loss, augmentation, schedule) keeps the inference stages
unchanged and adds one final stage with `"train_only": true` describing the
training signal.
3. **Implement exactly that change** in the agent-editable files (`model/`).
Keep `train.py`'s CLI contract and the ONNX export contract in `model/model.py`'s
docstring intact — the frozen scorer depends on them.
## Hard rules
- Edit ONLY `model/` (and `runs/pending_experiment.json`). Files listed in
`/FROZEN` are off-limits; the harness hard-reverts any change to them.
- NEVER touch, read, or evaluate the `hamburg` holdout area (§5).
- ONE focused change per iteration — if you can't describe it in one sentence,
it's too big. Prefer architectural/procedural novelty over hyperparameter
nudges (§3): changing a learning rate is a weak experiment; changing the
coordinate parameterization, loss family, relighting-for-training, or model
topology is a strong one.
- Do not run training yourself; the harness does that. A quick syntax check
(`.venv/bin/python -c "import model.train, model.model"`) is fine.
- Stay within the deployment gates: exported ONNX ≤ 4 MiB per area, host
latency proxy ≤ 250 ms (see pipeline/score.py).Where the model was tested — and how far off it was Each map is one full test area. Every dot is one held-out test location (all six lighting conditions overlaid): the model was shown a 128 m crop centered there and asked for its position. Dot color = the distance between its answer and the truth — green ≤ 20 m (at goal), amber ≤ 50 m, red beyond. A working model turns these maps green; spatial clusters of red reveal which parts of an area confuse it.     What the six lighting conditions look like through the simulated low-light sensor (example patches — illustration, not the training set)Every frame in this project is rendered as a starlight-class low-light sensor (Sony STARVIS2 / IMX585 class — the airframe's chosen camera) would see it, not as a normal camera would. That is why the night renders look uncannily close to daylight: at high gain such a sensor recovers scene structure from moonlight, skyglow and artificial lighting, paying for it in noise, lifted shadows and washed-out color — exactly what these renders simulate. Below, one example 256 m patch per area under the six lighting conditions the model must handle. These illustrate the dataset, not this experiment's performance — the actual training set is thousands of distinct crops per area (see “training data” above), and these renderings only change when the relighting method changes. berlin frankfurt munich prignitz | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▸ | 11 | ImageNet-pretrained MobileNetV3-Small trunk replaces the from-scratch encoder d69862d9 | architecture | pretrained:mobilenet_v3_small IMAGENET1K_V1 features[0..8] (torchvision, BSD-3) | 2.01 km | 3,013 KB | 0.8 ms | 31 m 35 s | $0.36 | kept | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The design under test — technical diagram — gray = frozen contract · ink = the current design · red = this experiment's change One real test, end to end
→
902.1 m
miss, this crop 0.94
self-reported confidence In plain words Until now the model's 'eyes' started out completely blind: they had to learn to see edges, corners and textures using only pictures of the one training area -- like learning to read from a single book. This experiment swaps in eyes that were already trained on 1.2 million everyday photographs, so they arrive knowing how to see, and we only gently retune them to our night-time aerial views. Everything after the eyes -- the part that guesses the position on the map -- stays exactly the same. If the guesses were bad because the eyes were weak, this should help a lot; if not, we have learned the eyes were never the problem. Hypothesis The binding constraint is the quality of the conv features themselves, and no within-dataset signal can fix it: exp 8 showed more from-scratch capacity does not help, exp 9 showed a direct place-discrimination objective (InfoNCE) on from-scratch features does not help, and exp 10's layout-aware head — which finally lets the head read WHERE features sit — was kept but recovered only 1.2%, implying the features it reads carry little discriminative content. All ten experiments trained the encoder from random init on ~36k synthetic relit crops of ONE area — far too little visual diversity to learn the general edge/corner/texture/junction vocabulary that makes local patterns separable. CLAUDE.md §3/§9 explicitly reserves permissively-licensed pretrained init as an open call, and it is the one family in the plateau-rule list never tried. Swapping the from-scratch 4-conv stack for the first 9 feature blocks of ImageNet-pretrained MobileNetV3-Small (190k params, BSD-3, output 8x8x48 — the exact spatial grid the kept exp-10 head already reads) imports a feature vocabulary learned from 1.2M real photographs; low/mid-level features are the standard transferable layer even across large domain gaps (ground-level photos -> overhead relit imagery). Method model/model.py: replace the 4x stride-2 from-scratch conv stack with torchvision's actual mobilenet_v3_small().features[:9] module (stem + 8 inverted-residual blocks), loading the verbatim IMAGENET1K_V1 tensors from model/pretrained/mnv3s_features8.pt with strict=True; bake ImageNet mean/std normalization into forward() as buffers so the [0,1] ONNX input contract is unchanged. Head adapts only in width (layout squeeze 48->8, FC 560->1024, conf 48->1, dims probed dynamically from the trunk's actual output shape); GRID_K, losses, decode, export contract all unchanged. model/train.py: two-tier fine-tuning LR (trunk param group 1e-4, fresh-head param group 1e-3) so early large head gradients do not destroy the transferred features; logs 'pretrained:mobilenet_v3_small IMAGENET1K_V1 features[0..8] (torchvision, BSD-3)' as init strategy per §9. Verified locally: strict load OK, 765k total params, forward + backward pass flows gradients into both trunk and head, ONNX 2.94 MiB (< 4 MiB gate), host latency proxy ~40 ms (< 250 ms gate), torch/ONNX parity 9e-8, untrained decode sits at the map center (0.4999, 0.4991) confirming bounded downside. Expected outcome If feature quality is the binding constraint, worst-case median error drops clearly below the kept 2299.74 m — plausibly 1,200-2,100 m, with bright buckets improving most (their texture becomes separable first) and the probability field finally committing to real modes. Downside tightly bounded: head/loss/decode are unchanged and an uninformative field still decodes near the map center (~3.2 km floor), so a null lands near 2.3-2.4 km. A null would be decisive: with features (pretrained AND contrastive), capacity, data scale, supervision form, rotation handling, and pooling all ruled out, the remaining suspects are lighting-conditional structure (dispatcher + specialists) and the information content of the relit imagery itself (learned relighting) -- the only untried families. Result primary worst-case median error = 2010.15 m (previous best 2299.74 m); full per-area/bucket breakdown in metrics.json Conclusion KEPT — metric improved (2299.74 -> 2010.15 m); change committed as d69862d Scoreboard — median error (m) per area × lighting
the worst cell (underlined) is the experiment's score;
red = failed cell, ink = at target
Fits the aircraft? all deployment gates passed
largest per-area model 3,013 KB (limit 4,096 KB) · single-frame inference 0.8 ms on one CPU thread (proxy limit 250 ms)
crops are sampled fresh each run from the frozen train split (~45,000 distinct positions per area, times random rotation), never from eval blocks; how many to use is the experiment's own choice
ts 2026-07-21T11:32:06 · commit d69862d9a3b1 ·
parent 55be8c4635a4 · artifacts runs/20260721_110031_iter1 ·
agent model claude-haiku-4-5-20251001, claude-sonnet-5 · took 31 m 35 s
The exact prompt given to the headless agent# Autoresearch iteration — one focused experiment
You are one iteration of an autonomous research loop for UAV low-light
geolocalization. Read `CLAUDE.md` (§1, §3, §6 especially) for full context.
The harness (loop.sh) will train, score, log, and keep/revert AFTER you exit —
you only design the experiment and edit the code.
## Your job, in order
1. **Review the research history.** Query the lineage DB:
`sqlite3 experiments.sqlite "SELECT id, title, category, hypothesis, expected_outcome, result, conclusion, primary_metric, kept FROM experiments ORDER BY id DESC LIMIT 15;"`
Note which hypotheses were supported/refuted. Do not repeat a refuted
experiment without a materially new angle.
**Plateau rule:** if three or more consecutive experiments were reverted,
do not attempt another variation of the last refuted mechanism. Either
pick a design family absent from the history (dispatcher + lighting
specialists, pretrained init, learned relighting, training-scale, …) or
attack the bottleneck the refuted hypotheses jointly point at.
2. **Design ONE focused experiment** — proper experiment design, pre-registered
before you touch code. Write it to `runs/pending_experiment.json`:
```json
{
"title": "one-line name",
"category": "architecture|loss|augmentation|relighting|training|quantization|other",
"hypothesis": "what you believe is limiting the metric and why this change addresses it",
"method": "the ONE focused change, concretely (files, mechanism)",
"expected_outcome": "predicted effect on the §6 worst-case median error, quantified if possible",
"init_strategy": "from-scratch | pretrained:<name>",
"eli5": "2-4 sentences for a smart non-ML reader: what you changed and why it might help, in everyday language — analogies welcome, zero jargon",
"architecture": {"stages": [
{"name": "Camera frame", "detail": "128×128 px crop, one of 6 lighting renders", "changed": false},
{"name": "Feature extractor", "detail": "plain-language description", "changed": false},
{"name": "…", "detail": "…", "changed": true}
]},
"architecture_svg": "<svg viewBox='0 0 980 300' …>…</svg>"
}
```
`architecture_svg` is the **technical architecture diagram** of the model
you are testing — a proper ML-paper figure of the design, drawn by you,
shown at the top of this experiment's gallery entry. **Draw tensors and
operations as the things they are — NOT as labeled boxes:** the input
image as a pixel-textured square; conv feature maps as pseudo-3D slabs
that shrink spatially and deepen in channels; 1-D vectors as thin
vertical tick-bars; fully-connected layers as fans of thin crossing
lines between bars; spatial fields/grids as actually-drawn grids with
shaded cells; decodes/aggregations as thin lines converging from cells
onto a point; samplers/targets as small pictorial glyphs. Captions sit
UNDER each element (name ≤10.5px weight 600, sub-note ≤9.5px); tensor
shapes annotate the flow in small italics (128×128×3 → 8²×128 → 1024).
Losses/targets/samplers live in a lane below the inference path,
annotated with slanted dashed leader lines to small text — no boxes
around them. Draw the two FROZEN endpoints — the camera-frame input and
the (lat, lon, confidence) output — entirely in #9b998c with a small
italic “frozen contract” tag: they are fixed by the harness and outside
your search space. Everything between them is your design: draw it in ink,
with red only on what THIS experiment changes. Style contract, so all experiments' figures read as one
paper: viewBox width 980 (height ~300–360); transparent background; ink
#111111, captions #6b6a60, annotations/arrows #9b998c; **#8c2f1f (red)
reserved for exactly what this experiment changed**; training-only
elements in #8a6a1e (or red if changed); font-family
Palatino,Georgia,serif; stroke-width ≈1.2 elements, 1 arrows; fills only
faint tints (opacity ≤ .12); no gradients, no icons, no emoji. Inference
flows left → right from camera frame to (lat, lon, confidence).
**Anchor the frozen endpoints identically in every figure** so all
experiments' figures line up when compared down the gallery page: the
camera-frame square starts at x=26 (its captions centered on x=53), and
the output is right-anchored — decode crosshair at x=812, the
(lat, lon, confidence) text block text-anchor=start at x=828. Lay the
elements between them out on a running x-cursor with generous spacing
so nothing overlaps.
Consecutive conv layers must be tied by kernel-projection lines (small
kernel square on one face, faint lines converging to a cell on the
next) so the encoder reads as one computation, not boxes in a row.
`archive/arch_svg_reference.py` is the concrete reference implementation
of this entire visual language — read it and reuse its helper geometry
for any stage that already has one.
`eli5` and `architecture` feed the human-facing gallery. `architecture.stages`
is the model's inference path left-to-right, camera frame → (lat, lon,
confidence) output — one box per stage, plain-language `detail`,
`"changed": true` ONLY on the stages this experiment touches. Reuse the
previous experiment's stage names verbatim wherever a stage is unchanged
(check `SELECT arch_json FROM experiments WHERE arch_json IS NOT NULL ORDER BY id DESC LIMIT 1;`) —
the diagrams are compared box-by-box across experiments. A change that only
affects training (loss, augmentation, schedule) keeps the inference stages
unchanged and adds one final stage with `"train_only": true` describing the
training signal.
3. **Implement exactly that change** in the agent-editable files (`model/`).
Keep `train.py`'s CLI contract and the ONNX export contract in `model/model.py`'s
docstring intact — the frozen scorer depends on them.
## Hard rules
- Edit ONLY `model/` (and `runs/pending_experiment.json`). Files listed in
`/FROZEN` are off-limits; the harness hard-reverts any change to them.
- NEVER touch, read, or evaluate the `hamburg` holdout area (§5).
- ONE focused change per iteration — if you can't describe it in one sentence,
it's too big. Prefer architectural/procedural novelty over hyperparameter
nudges (§3): changing a learning rate is a weak experiment; changing the
coordinate parameterization, loss family, relighting-for-training, or model
topology is a strong one.
- Do not run training yourself; the harness does that. A quick syntax check
(`.venv/bin/python -c "import model.train, model.model"`) is fine.
- Stay within the deployment gates: exported ONNX ≤ 4 MiB per area, host
latency proxy ≤ 250 ms (see pipeline/score.py).Where the model was tested — and how far off it was Each map is one full test area. Every dot is one held-out test location (all six lighting conditions overlaid): the model was shown a 128 m crop centered there and asked for its position. Dot color = the distance between its answer and the truth — green ≤ 20 m (at goal), amber ≤ 50 m, red beyond. A working model turns these maps green; spatial clusters of red reveal which parts of an area confuse it.   What the six lighting conditions look like through the simulated low-light sensor (example patches — illustration, not the training set)Every frame in this project is rendered as a starlight-class low-light sensor (Sony STARVIS2 / IMX585 class — the airframe's chosen camera) would see it, not as a normal camera would. That is why the night renders look uncannily close to daylight: at high gain such a sensor recovers scene structure from moonlight, skyglow and artificial lighting, paying for it in noise, lifted shadows and washed-out color — exactly what these renders simulate. Below, one example 256 m patch per area under the six lighting conditions the model must handle. These illustrate the dataset, not this experiment's performance — the actual training set is thousands of distinct crops per area (see “training data” above), and these renderings only change when the relighting method changes. berlin frankfurt munich prignitz | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▸ | 10 | Layout-aware field head: squeezed 8×8 spatial code ⊕ GAP replaces GAP-only head input ff271753 | architecture | from-scratch | 2.30 km | 2,960 KB | 0.7 ms | 38 m 08 s | — | kept | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The design under test — technical diagram — gray = frozen contract · ink = the current design · red = this experiment's change One real test, end to end 
→
771.3 m
miss, this crop 0.93
self-reported confidence In plain words Until now the model summarized each aerial photo as a recipe list — 'this much road, this much rooftop, this much trees' — with no memory of where anything sat in the picture. Two different neighborhoods with the same mix of ingredients looked identical to it, which may be why it kept guessing near the middle of the map. This experiment adds a second, position-preserving summary — roads on the left, courtyard in the middle, park at the top — and lets the location-guessing layer read both summaries. Places are arrangements, not ingredient lists, so keeping the arrangement should finally let it tell lookalike districts apart. Hypothesis Every design so far has fed the 1024-cell field head a spatially averaged descriptor: GAP collapses the 8×8×128 feature map to a bag-of-textures that destroys WHERE features sit in the crop — and the arrangement of roads, buildings and open space is precisely what distinguishes lookalike districts at 1 m/px. This one invariant explains the joint evidence better than any refuted hypothesis: capacity was null (exp 8: a deeper bag-of-textures is still a bag), contrastive descriptor learning was null (exp 9: no training signal can make the descriptor place-discriminative when pooling already destroyed the discriminating information), 7.5× data recovered only 7% (exp 7: more examples do not add information per example), and medians are FLAT across lighting buckets (texture statistics are roughly equally ambiguous in every lighting, while layout would be most readable in bright buckets). Exp 4 is the only non-GAP design ever tried, and it went to the opposite extreme — 64 independent per-patch coordinate regressions with no global read of the arrangement — so a layout-aware GLOBAL decision has never been tested. The fix: a 1×1 conv squeezes the feature map to 8 channels, its flattened 512-d layout code is concatenated with the existing 128-d GAP descriptor, and the FC to 1024 logits reads both — mathematically, the head upgrades from a linear map on the spatial average to a low-rank linear map on the full feature map, a strict superset of every function the old head could compute. Method model/model.py only. TinyLocNet gains self.layout_squeeze = nn.Conv2d(128, 8, 1); forward() keeps the 8×8×128 feature map, computes the usual GAP vector f (128-d) AND layout = layout_squeeze(fmap).flatten(1) (8×8×8 = 512-d), and loc_logits becomes nn.Linear(640, 1024) applied to cat([f, layout]). Confidence head (reads GAP only), softmax + soft-argmax decode, loss_fn, export and train.py are all byte-identical. Params 232k → ~757k, ONNX ≈ 2.9 MiB (< 4 MiB gate), latency delta negligible (one 1×1 conv + a wider FC). Expected outcome If arrangement-blindness is the binding constraint, worst-case median error drops clearly below the kept 2326.54 m — plausibly to 1,000–2,000 m, with bright buckets improving most and the probability field finally committing to real modes. Main risk: the layout code is rotation-sensitive and eval crops are rotated; mitigated because each training location is seen at ~6 independent random headings (one per lighting bucket) and the rotation-invariant GAP path is retained as a fallback, so a null lands near 2.3–2.4 km (decode and losses unchanged; the head can learn to down-weight the layout dims). A null would be decisive the other way: with averaging-blindness also ruled out, the limit is likely the information content of the imagery/eval design itself, pointing next at pretrained texture features or lighting-specialist structure. Result primary worst-case median error = 2299.74 m (previous best 2326.54 m); full per-area/bucket breakdown in metrics.json Conclusion KEPT — metric improved (2326.54 -> 2299.74 m); change committed as ff27175 Scoreboard — median error (m) per area × lighting
the worst cell (underlined) is the experiment's score;
red = failed cell, ink = at target
Fits the aircraft? all deployment gates passed
largest per-area model 2,960 KB (limit 4,096 KB) · single-frame inference 0.7 ms on one CPU thread (proxy limit 250 ms)
crops are sampled fresh each run from the frozen train split (~45,000 distinct positions per area, times random rotation), never from eval blocks; how many to use is the experiment's own choice
ts 2026-07-20T23:08:48 · commit ff2717532634 ·
parent 76c45e3c4850 · artifacts runs/20260720_223039_iter4 ·
agent model claude-fable-5, claude-haiku-4-5-20251001 · took 38 m 08 s
The exact prompt given to the headless agent# Autoresearch iteration — one focused experiment
You are one iteration of an autonomous research loop for UAV low-light
geolocalization. Read `CLAUDE.md` (§1, §3, §6 especially) for full context.
The harness (loop.sh) will train, score, log, and keep/revert AFTER you exit —
you only design the experiment and edit the code.
## Your job, in order
1. **Review the research history.** Query the lineage DB:
`sqlite3 experiments.sqlite "SELECT id, title, category, hypothesis, expected_outcome, result, conclusion, primary_metric, kept FROM experiments ORDER BY id DESC LIMIT 15;"`
Note which hypotheses were supported/refuted. Do not repeat a refuted
experiment without a materially new angle.
**Plateau rule:** if three or more consecutive experiments were reverted,
do not attempt another variation of the last refuted mechanism. Either
pick a design family absent from the history (dispatcher + lighting
specialists, pretrained init, learned relighting, training-scale, …) or
attack the bottleneck the refuted hypotheses jointly point at.
2. **Design ONE focused experiment** — proper experiment design, pre-registered
before you touch code. Write it to `runs/pending_experiment.json`:
```json
{
"title": "one-line name",
"category": "architecture|loss|augmentation|relighting|training|quantization|other",
"hypothesis": "what you believe is limiting the metric and why this change addresses it",
"method": "the ONE focused change, concretely (files, mechanism)",
"expected_outcome": "predicted effect on the §6 worst-case median error, quantified if possible",
"init_strategy": "from-scratch | pretrained:<name>",
"eli5": "2-4 sentences for a smart non-ML reader: what you changed and why it might help, in everyday language — analogies welcome, zero jargon",
"architecture": {"stages": [
{"name": "Camera frame", "detail": "128×128 px crop, one of 6 lighting renders", "changed": false},
{"name": "Feature extractor", "detail": "plain-language description", "changed": false},
{"name": "…", "detail": "…", "changed": true}
]},
"architecture_svg": "<svg viewBox='0 0 980 300' …>…</svg>"
}
```
`architecture_svg` is the **technical architecture diagram** of the model
you are testing — a proper ML-paper figure of the design, drawn by you,
shown at the top of this experiment's gallery entry. **Draw tensors and
operations as the things they are — NOT as labeled boxes:** the input
image as a pixel-textured square; conv feature maps as pseudo-3D slabs
that shrink spatially and deepen in channels; 1-D vectors as thin
vertical tick-bars; fully-connected layers as fans of thin crossing
lines between bars; spatial fields/grids as actually-drawn grids with
shaded cells; decodes/aggregations as thin lines converging from cells
onto a point; samplers/targets as small pictorial glyphs. Captions sit
UNDER each element (name ≤10.5px weight 600, sub-note ≤9.5px); tensor
shapes annotate the flow in small italics (128×128×3 → 8²×128 → 1024).
Losses/targets/samplers live in a lane below the inference path,
annotated with slanted dashed leader lines to small text — no boxes
around them. Draw the two FROZEN endpoints — the camera-frame input and
the (u, v, conf) output — entirely in #9b998c with a small italic
“frozen contract” tag: they are fixed by the harness and outside your
search space. Everything between them is your design: draw it in ink,
with red only on what THIS experiment changes. Style contract, so all experiments' figures read as one
paper: viewBox width 980 (height ~300–360); transparent background; ink
#111111, captions #6b6a60, annotations/arrows #9b998c; **#8c2f1f (red)
reserved for exactly what this experiment changed**; training-only
elements in #8a6a1e (or red if changed); font-family
Palatino,Georgia,serif; stroke-width ≈1.2 elements, 1 arrows; fills only
faint tints (opacity ≤ .12); no gradients, no icons, no emoji. Inference
flows left → right from camera frame to (u, v, conf). Lay elements out
on a running x-cursor with generous spacing so nothing overlaps.
Consecutive conv layers must be tied by kernel-projection lines (small
kernel square on one face, faint lines converging to a cell on the
next) so the encoder reads as one computation, not boxes in a row.
`archive/arch_svg_reference.py` is the concrete reference implementation
of this entire visual language — read it and reuse its helper geometry
for any stage that already has one.
`eli5` and `architecture` feed the human-facing gallery. `architecture.stages`
is the model's inference path left-to-right, camera frame → (lat, lon,
confidence) output — one box per stage, plain-language `detail`,
`"changed": true` ONLY on the stages this experiment touches. Reuse the
previous experiment's stage names verbatim wherever a stage is unchanged
(check `SELECT arch_json FROM experiments WHERE arch_json IS NOT NULL ORDER BY id DESC LIMIT 1;`) —
the diagrams are compared box-by-box across experiments. A change that only
affects training (loss, augmentation, schedule) keeps the inference stages
unchanged and adds one final stage with `"train_only": true` describing the
training signal.
3. **Implement exactly that change** in the agent-editable files (`model/`).
Keep `train.py`'s CLI contract and the ONNX export contract in `model/model.py`'s
docstring intact — the frozen scorer depends on them.
## Hard rules
- Edit ONLY `model/` (and `runs/pending_experiment.json`). Files listed in
`/FROZEN` are off-limits; the harness hard-reverts any change to them.
- NEVER touch, read, or evaluate the `hamburg` holdout area (§5).
- ONE focused change per iteration — if you can't describe it in one sentence,
it's too big. Prefer architectural/procedural novelty over hyperparameter
nudges (§3): changing a learning rate is a weak experiment; changing the
coordinate parameterization, loss family, relighting-for-training, or model
topology is a strong one.
- Do not run training yourself; the harness does that. A quick syntax check
(`.venv/bin/python -c "import model.train, model.model"`) is fine.
- Stay within the deployment gates: exported ONNX ≤ 4 MiB per area, host
latency proxy ≤ 250 ms (see pipeline/score.py).Where the model was tested — and how far off it was Each map is one full test area. Every dot is one held-out test location (all six lighting conditions overlaid): the model was shown a 128 m crop centered there and asked for its position. Dot color = the distance between its answer and the truth — green ≤ 20 m (at goal), amber ≤ 50 m, red beyond. A working model turns these maps green; spatial clusters of red reveal which parts of an area confuse it.  What the six lighting conditions look like through the simulated low-light sensor (example patches — illustration, not the training set)Every frame in this project is rendered as a starlight-class low-light sensor (Sony STARVIS2 / IMX585 class — the airframe's chosen camera) would see it, not as a normal camera would. That is why the night renders look uncannily close to daylight: at high gain such a sensor recovers scene structure from moonlight, skyglow and artificial lighting, paying for it in noise, lifted shadows and washed-out color — exactly what these renders simulate. Below, one example 256 m patch per area under the six lighting conditions the model must handle. These illustrate the dataset, not this experiment's performance — the actual training set is thousands of distinct crops per area (see “training data” above), and these renderings only change when the relighting method changes. berlin       frankfurt       munich       prignitz       | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▸ | 9 | Cross-lighting contrastive pairs: NT-Xent metric learning on the place descriptor 76c45e3c | loss | from-scratch | 2.34 km | 908 KB | 0.6 ms | 23 m 41 s | — | discarded | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The design under test — technical diagram — gray = frozen contract · ink = the current design · red = this experiment's change One real test, end to end
→
474.2 m
miss, this crop 0.94
self-reported confidence In plain words The model has been describing every photo so vaguely that different places end up with nearly identical descriptions — like describing every street as 'houses and a road'. We now train it flash-card style: each round it gets two photos of the exact same spot — say one at noon and one at night, turned to different compass headings — and is graded on giving both the same description, while its descriptions of the 62 other places in the round must stay clearly different. That pressure forces it to notice what makes each place unique, which is exactly what pointing to the right spot on the map requires. Hypothesis The encoder's 128-d descriptor is not place-discriminative, and no existing training signal ever directly demands that it be: Gaussian-CE against a near-uniform 1024-way target gives each crop a diffuse gradient that shapes the head more than the features. The joint evidence — near-uniform fields (exp 3's argmax test), nulls from supervision form (3-5), rotation resampling (6) and capacity (8), only 7% from 7.5x data (7), and medians flat across lighting buckets at ~2.3 km near the 3.2 km center-guess floor — all points at descriptor quality, not head, data volume, or capacity. An InfoNCE objective on location-paired batches (same place, two different lighting renders, two independent headings, 62 in-batch negatives) supplies exactly the missing signal: per-sample 'same place -> same descriptor, different place -> different descriptor' gradients far stronger than the near-uniform CE, and it builds precisely the two invariances (heading, lighting) the frozen rotated multi-bucket eval demands. Method train.py: sample ONE shared 6,000-location set per area and extract it in all six relight buckets with independent random headings (same 36k-crop, ~1.8 GB uint8 budget; previously each bucket sampled locations independently, so no cross-bucket pairs existed). Each step draws 32 locations x 2 distinct buckets -> a 64-crop batch of 32 positive pairs. model.py: add a train-only 128->128->64 MLP projection head and NT-Xent (temperature 0.2, weight 1.0) on the projected GAP descriptors, added to the unchanged Gaussian-CE + coord L2 + conf BCE. Inference path, decode, and ONNX export are byte-identical; the projection head is never traced. Expected outcome If descriptor discriminability is the binding constraint, worst-case median error drops clearly below the kept 2326.54 m — plausibly 1,200-2,200 m, with the flat lighting profile finally spreading (bright buckets improving most, since contrastive pressure makes their texture usable). Downside tightly bounded: inference path and all existing loss terms are unchanged, so a null lands near current best (~2.3-2.5 km). A null would be highly informative: with features (this exp), supervision form (3-5), rotation (6), data (7), and capacity (8) all ruled out, the bottleneck must be the information content of the relit imagery itself, pointing next at learned relighting or input resolution. Result primary worst-case median error = 2342.1 m (previous best 2326.54 m); full per-area/bucket breakdown in metrics.json Conclusion REVERTED — metric did not improve (2342.1 m vs best 2326.54 m); change discarded Scoreboard — median error (m) per area × lighting
the worst cell (underlined) is the experiment's score;
red = failed cell, ink = at target
Fits the aircraft? all deployment gates passed
largest per-area model 908 KB (limit 4,096 KB) · single-frame inference 0.6 ms on one CPU thread (proxy limit 250 ms)
crops are sampled fresh each run from the frozen train split (~45,000 distinct positions per area, times random rotation), never from eval blocks; how many to use is the experiment's own choice
ts 2026-07-20T22:30:37 · commit 76c45e3c4850 ·
parent 76c45e3c4850 · artifacts runs/20260720_220655_iter3 ·
agent model claude-fable-5, claude-haiku-4-5-20251001 · took 23 m 41 s
The exact prompt given to the headless agent# Autoresearch iteration — one focused experiment
You are one iteration of an autonomous research loop for UAV low-light
geolocalization. Read `CLAUDE.md` (§1, §3, §6 especially) for full context.
The harness (loop.sh) will train, score, log, and keep/revert AFTER you exit —
you only design the experiment and edit the code.
## Your job, in order
1. **Review the research history.** Query the lineage DB:
`sqlite3 experiments.sqlite "SELECT id, title, category, hypothesis, expected_outcome, result, conclusion, primary_metric, kept FROM experiments ORDER BY id DESC LIMIT 15;"`
Note which hypotheses were supported/refuted. Do not repeat a refuted
experiment without a materially new angle.
**Plateau rule:** if three or more consecutive experiments were reverted,
do not attempt another variation of the last refuted mechanism. Either
pick a design family absent from the history (dispatcher + lighting
specialists, pretrained init, learned relighting, training-scale, …) or
attack the bottleneck the refuted hypotheses jointly point at.
2. **Design ONE focused experiment** — proper experiment design, pre-registered
before you touch code. Write it to `runs/pending_experiment.json`:
```json
{
"title": "one-line name",
"category": "architecture|loss|augmentation|relighting|training|quantization|other",
"hypothesis": "what you believe is limiting the metric and why this change addresses it",
"method": "the ONE focused change, concretely (files, mechanism)",
"expected_outcome": "predicted effect on the §6 worst-case median error, quantified if possible",
"init_strategy": "from-scratch | pretrained:<name>",
"eli5": "2-4 sentences for a smart non-ML reader: what you changed and why it might help, in everyday language — analogies welcome, zero jargon",
"architecture": {"stages": [
{"name": "Camera frame", "detail": "128×128 px crop, one of 6 lighting renders", "changed": false},
{"name": "Feature extractor", "detail": "plain-language description", "changed": false},
{"name": "…", "detail": "…", "changed": true}
]},
"architecture_svg": "<svg viewBox='0 0 980 300' …>…</svg>"
}
```
`architecture_svg` is the **technical architecture diagram** of the model
you are testing — a proper ML-paper figure of the design, drawn by you,
shown at the top of this experiment's gallery entry. **Draw tensors and
operations as the things they are — NOT as labeled boxes:** the input
image as a pixel-textured square; conv feature maps as pseudo-3D slabs
that shrink spatially and deepen in channels; 1-D vectors as thin
vertical tick-bars; fully-connected layers as fans of thin crossing
lines between bars; spatial fields/grids as actually-drawn grids with
shaded cells; decodes/aggregations as thin lines converging from cells
onto a point; samplers/targets as small pictorial glyphs. Captions sit
UNDER each element (name ≤10.5px weight 600, sub-note ≤9.5px); tensor
shapes annotate the flow in small italics (128×128×3 → 8²×128 → 1024).
Losses/targets/samplers live in a lane below the inference path,
annotated with slanted dashed leader lines to small text — no boxes
around them. Draw the two FROZEN endpoints — the camera-frame input and
the (u, v, conf) output — entirely in #9b998c with a small italic
“frozen contract” tag: they are fixed by the harness and outside your
search space. Everything between them is your design: draw it in ink,
with red only on what THIS experiment changes. Style contract, so all experiments' figures read as one
paper: viewBox width 980 (height ~300–360); transparent background; ink
#111111, captions #6b6a60, annotations/arrows #9b998c; **#8c2f1f (red)
reserved for exactly what this experiment changed**; training-only
elements in #8a6a1e (or red if changed); font-family
Palatino,Georgia,serif; stroke-width ≈1.2 elements, 1 arrows; fills only
faint tints (opacity ≤ .12); no gradients, no icons, no emoji. Inference
flows left → right from camera frame to (u, v, conf). Lay elements out
on a running x-cursor with generous spacing so nothing overlaps.
Consecutive conv layers must be tied by kernel-projection lines (small
kernel square on one face, faint lines converging to a cell on the
next) so the encoder reads as one computation, not boxes in a row.
`archive/arch_svg_reference.py` is the concrete reference implementation
of this entire visual language — read it and reuse its helper geometry
for any stage that already has one.
`eli5` and `architecture` feed the human-facing gallery. `architecture.stages`
is the model's inference path left-to-right, camera frame → (lat, lon,
confidence) output — one box per stage, plain-language `detail`,
`"changed": true` ONLY on the stages this experiment touches. Reuse the
previous experiment's stage names verbatim wherever a stage is unchanged
(check `SELECT arch_json FROM experiments WHERE arch_json IS NOT NULL ORDER BY id DESC LIMIT 1;`) —
the diagrams are compared box-by-box across experiments. A change that only
affects training (loss, augmentation, schedule) keeps the inference stages
unchanged and adds one final stage with `"train_only": true` describing the
training signal.
3. **Implement exactly that change** in the agent-editable files (`model/`).
Keep `train.py`'s CLI contract and the ONNX export contract in `model/model.py`'s
docstring intact — the frozen scorer depends on them.
## Hard rules
- Edit ONLY `model/` (and `runs/pending_experiment.json`). Files listed in
`/FROZEN` are off-limits; the harness hard-reverts any change to them.
- NEVER touch, read, or evaluate the `hamburg` holdout area (§5).
- ONE focused change per iteration — if you can't describe it in one sentence,
it's too big. Prefer architectural/procedural novelty over hyperparameter
nudges (§3): changing a learning rate is a weak experiment; changing the
coordinate parameterization, loss family, relighting-for-training, or model
topology is a strong one.
- Do not run training yourself; the harness does that. A quick syntax check
(`.venv/bin/python -c "import model.train, model.model"`) is fine.
- Stay within the deployment gates: exported ONNX ≤ 4 MiB per area, host
latency proxy ≤ 250 ms (see pipeline/score.py).Where the model was tested — and how far off it was Each map is one full test area. Every dot is one held-out test location (all six lighting conditions overlaid): the model was shown a 128 m crop centered there and asked for its position. Dot color = the distance between its answer and the truth — green ≤ 20 m (at goal), amber ≤ 50 m, red beyond. A working model turns these maps green; spatial clusters of red reveal which parts of an area confuse it.  What the six lighting conditions look like through the simulated low-light sensor (example patches — illustration, not the training set)Every frame in this project is rendered as a starlight-class low-light sensor (Sony STARVIS2 / IMX585 class — the airframe's chosen camera) would see it, not as a normal camera would. That is why the night renders look uncannily close to daylight: at high gain such a sensor recovers scene structure from moonlight, skyglow and artificial lighting, paying for it in noise, lifted shadows and washed-out color — exactly what these renders simulate. Below, one example 256 m patch per area under the six lighting conditions the model must handle. These illustrate the dataset, not this experiment's performance — the actual training set is thousands of distinct crops per area (see “training data” above), and these renderings only change when the relighting method changes. berlin frankfurt munich prignitz | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▸ | 8 | Deployment-envelope residual encoder: 4.2× capacity (232k → 973k params), 11 conv layers with skips 76c45e3c | architecture | from-scratch | 2.43 km | 3,807 KB | 7.8 ms | 48 m 45 s | — | discarded | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The design under test — technical diagram — gray = frozen contract · ink = the current design · red = this experiment's change One real test, end to end
→
916.5 m
miss, this crop 0.99
self-reported confidence In plain words Until now the model was a tiny four-layer network trying to memorize what every corner of a 7 km-wide city looks like — like sketching an entire city map on a sticky note. Recent experiments kept changing how we teach it (better lessons, more flashcards) and nothing broke through, because the student's notebook was simply full: it scored almost equally badly in bright daylight and at night, the classic sign of a model too small for its task rather than a task too hard. The flight computer actually has room for a model four times bigger than the one we've been flying. This experiment keeps the lessons exactly the same and only swaps in a deeper, four-times-larger network — built with 'residual' shortcuts, a standard trick that keeps deep networks easy to train — so it finally has enough room to write the whole map down. Hypothesis The binding constraint is encoder capacity, not supervision, decode, rotation handling, or data volume. Exps 3-5 changed supervision/decode and were all null; exp 6 (rotation resampling) was null; exp 7's 7.5× data scale — the family every reverted mechanism jointly pointed at — recovered only 7% (2489 → 2326 m), meaning the model could not absorb the added coverage. The kept run's error profile is the signature of underfitting: ~1.55-2.33 km medians nearly FLAT across lighting buckets (Berlin night 1760 m vs midday 1642 m), so even the easiest bright-daylight discrimination is not being learned — a lighting- or supervision-side bottleneck would show a bucket gradient. The current encoder is 4 plain stride-2 convs whose 128-d GAP descriptor must separate 1,024 map cells × 6 lightings across ~45k locations at 1 m/px, while using only 0.93 MB of the 4 MiB deployment gate and 1.25 ms of the 250 ms latency budget. Depth/width is the one axis no experiment has ever touched. Method model/model.py only: replace TinyLocNet.features (4 plain stride-2 convs 3→16→32→64→128, GAP→128-d) with an 11-conv residual encoder sized to the deployment envelope — stem 3→32 s2; stage1 down 32→64 s2 + bottleneck residual block (1×1 reduce to 32, 3×3, 1×1 expand); stage2 down 64→128 s2 + bottleneck block (mid 64); stage3 down 128→160 s2 + full 3×3+3×3 residual block; GAP → 160-d. Bottlenecks sit early where 32²/16² resolution makes compute expensive; the full block sits at 8² where parameters are cheap. Field head (Linear→1024 + softmax soft-argmax), conf head, loss, decode, training data, epochs, and lr are untouched, so attribution is purely encoder capacity/depth. Verified pre-registration: 972,993 params, ONNX 3.72 MiB < 4 MiB gate, host latency proxy 2.91 ms < 250 ms. Expected outcome If capacity is the binding constraint, worst-case median error drops clearly below the kept 2326.54 m — plausibly to 1,000-2,000 m, with the flat lighting profile finally spreading (bright buckets improving most). Downside is tightly bounded: the field/decode is unchanged, so an untrained-uniform field still decodes near the map center (~3.2 km floor), and BN+residual stacks of this depth train stably at lr 1e-3. Training cost rises ~7× in encoder FLOPs (~20-25 min/area on MPS). A null would be highly informative: with supervision (3-5), rotation (6), data scale (7), and capacity (8) all ruled out, the bottleneck must be the information content of the training signal itself, pointing the next iterations at sim-to-real relighting or lighting-conditional structure. Result primary worst-case median error = 2427.33 m (previous best 2326.54 m); full per-area/bucket breakdown in metrics.json Conclusion REVERTED — metric did not improve (2427.33 m vs best 2326.54 m); change discarded Scoreboard — median error (m) per area × lighting
the worst cell (underlined) is the experiment's score;
red = failed cell, ink = at target
Fits the aircraft? all deployment gates passed
largest per-area model 3,807 KB (limit 4,096 KB) · single-frame inference 7.8 ms on one CPU thread (proxy limit 250 ms)
crops are sampled fresh each run from the frozen train split (~45,000 distinct positions per area, times random rotation), never from eval blocks; how many to use is the experiment's own choice
ts 2026-07-20T22:06:53 · commit 76c45e3c4850 ·
parent 76c45e3c4850 · artifacts runs/20260720_211808_iter2 ·
agent model claude-fable-5, claude-haiku-4-5-20251001 · took 48 m 45 s
The exact prompt given to the headless agent# Autoresearch iteration — one focused experiment
You are one iteration of an autonomous research loop for UAV low-light
geolocalization. Read `CLAUDE.md` (§1, §3, §6 especially) for full context.
The harness (loop.sh) will train, score, log, and keep/revert AFTER you exit —
you only design the experiment and edit the code.
## Your job, in order
1. **Review the research history.** Query the lineage DB:
`sqlite3 experiments.sqlite "SELECT id, title, category, hypothesis, expected_outcome, result, conclusion, primary_metric, kept FROM experiments ORDER BY id DESC LIMIT 15;"`
Note which hypotheses were supported/refuted. Do not repeat a refuted
experiment without a materially new angle.
**Plateau rule:** if three or more consecutive experiments were reverted,
do not attempt another variation of the last refuted mechanism. Either
pick a design family absent from the history (dispatcher + lighting
specialists, pretrained init, learned relighting, training-scale, …) or
attack the bottleneck the refuted hypotheses jointly point at.
2. **Design ONE focused experiment** — proper experiment design, pre-registered
before you touch code. Write it to `runs/pending_experiment.json`:
```json
{
"title": "one-line name",
"category": "architecture|loss|augmentation|relighting|training|quantization|other",
"hypothesis": "what you believe is limiting the metric and why this change addresses it",
"method": "the ONE focused change, concretely (files, mechanism)",
"expected_outcome": "predicted effect on the §6 worst-case median error, quantified if possible",
"init_strategy": "from-scratch | pretrained:<name>",
"eli5": "2-4 sentences for a smart non-ML reader: what you changed and why it might help, in everyday language — analogies welcome, zero jargon",
"architecture": {"stages": [
{"name": "Camera frame", "detail": "128×128 px crop, one of 6 lighting renders", "changed": false},
{"name": "Feature extractor", "detail": "plain-language description", "changed": false},
{"name": "…", "detail": "…", "changed": true}
]},
"architecture_svg": "<svg viewBox='0 0 980 300' …>…</svg>"
}
```
`architecture_svg` is the **technical architecture diagram** of the model
you are testing — a proper ML-paper figure of the design, drawn by you,
shown at the top of this experiment's gallery entry. **Draw tensors and
operations as the things they are — NOT as labeled boxes:** the input
image as a pixel-textured square; conv feature maps as pseudo-3D slabs
that shrink spatially and deepen in channels; 1-D vectors as thin
vertical tick-bars; fully-connected layers as fans of thin crossing
lines between bars; spatial fields/grids as actually-drawn grids with
shaded cells; decodes/aggregations as thin lines converging from cells
onto a point; samplers/targets as small pictorial glyphs. Captions sit
UNDER each element (name ≤10.5px weight 600, sub-note ≤9.5px); tensor
shapes annotate the flow in small italics (128×128×3 → 8²×128 → 1024).
Losses/targets/samplers live in a lane below the inference path,
annotated with slanted dashed leader lines to small text — no boxes
around them. Draw the two FROZEN endpoints — the camera-frame input and
the (u, v, conf) output — entirely in #9b998c with a small italic
“frozen contract” tag: they are fixed by the harness and outside your
search space. Everything between them is your design: draw it in ink,
with red only on what THIS experiment changes. Style contract, so all experiments' figures read as one
paper: viewBox width 980 (height ~300–360); transparent background; ink
#111111, captions #6b6a60, annotations/arrows #9b998c; **#8c2f1f (red)
reserved for exactly what this experiment changed**; training-only
elements in #8a6a1e (or red if changed); font-family
Palatino,Georgia,serif; stroke-width ≈1.2 elements, 1 arrows; fills only
faint tints (opacity ≤ .12); no gradients, no icons, no emoji. Inference
flows left → right from camera frame to (u, v, conf). Lay elements out
on a running x-cursor with generous spacing so nothing overlaps.
Consecutive conv layers must be tied by kernel-projection lines (small
kernel square on one face, faint lines converging to a cell on the
next) so the encoder reads as one computation, not boxes in a row.
`archive/arch_svg_reference.py` is the concrete reference implementation
of this entire visual language — read it and reuse its helper geometry
for any stage that already has one.
`eli5` and `architecture` feed the human-facing gallery. `architecture.stages`
is the model's inference path left-to-right, camera frame → (lat, lon,
confidence) output — one box per stage, plain-language `detail`,
`"changed": true` ONLY on the stages this experiment touches. Reuse the
previous experiment's stage names verbatim wherever a stage is unchanged
(check `SELECT arch_json FROM experiments WHERE arch_json IS NOT NULL ORDER BY id DESC LIMIT 1;`) —
the diagrams are compared box-by-box across experiments. A change that only
affects training (loss, augmentation, schedule) keeps the inference stages
unchanged and adds one final stage with `"train_only": true` describing the
training signal.
3. **Implement exactly that change** in the agent-editable files (`model/`).
Keep `train.py`'s CLI contract and the ONNX export contract in `model/model.py`'s
docstring intact — the frozen scorer depends on them.
## Hard rules
- Edit ONLY `model/` (and `runs/pending_experiment.json`). Files listed in
`/FROZEN` are off-limits; the harness hard-reverts any change to them.
- NEVER touch, read, or evaluate the `hamburg` holdout area (§5).
- ONE focused change per iteration — if you can't describe it in one sentence,
it's too big. Prefer architectural/procedural novelty over hyperparameter
nudges (§3): changing a learning rate is a weak experiment; changing the
coordinate parameterization, loss family, relighting-for-training, or model
topology is a strong one.
- Do not run training yourself; the harness does that. A quick syntax check
(`.venv/bin/python -c "import model.train, model.model"`) is fine.
- Stay within the deployment gates: exported ONNX ≤ 4 MiB per area, host
latency proxy ≤ 250 ms (see pipeline/score.py).Where the model was tested — and how far off it was Each map is one full test area. Every dot is one held-out test location (all six lighting conditions overlaid): the model was shown a 128 m crop centered there and asked for its position. Dot color = the distance between its answer and the truth — green ≤ 20 m (at goal), amber ≤ 50 m, red beyond. A working model turns these maps green; spatial clusters of red reveal which parts of an area confuse it.  What the six lighting conditions look like through the simulated low-light sensor (example patches — illustration, not the training set)Every frame in this project is rendered as a starlight-class low-light sensor (Sony STARVIS2 / IMX585 class — the airframe's chosen camera) would see it, not as a normal camera would. That is why the night renders look uncannily close to daylight: at high gain such a sensor recovers scene structure from moonlight, skyglow and artificial lighting, paying for it in noise, lifted shadows and washed-out color — exactly what these renders simulate. Below, one example 256 m patch per area under the six lighting conditions the model must handle. These illustrate the dataset, not this experiment's performance — the actual training set is thousands of distinct crops per area (see “training data” above), and these renderings only change when the relighting method changes. berlin frankfurt munich prignitz | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▸ | 7 | Scale training coverage 7.5x: 6,000 of ~45k available train locations per bucket (was 800) 76c45e3c | training pivot | from-scratch | 2.33 km | 908 KB | 1.2 ms | 21 m 44 s | — | kept | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The design under test — technical diagram — gray = frozen contract · ink = the current design · red = this experiment's change One real test, end to end
→
435.1 m
miss, this crop 0.95
self-reported confidence In plain words The model has been studying flashcards covering only about 2% of the map — most neighborhoods were never shown to it at all, in any lighting. The last four experiments all tried smarter study techniques and none helped, which suggests the problem was never the technique: you can't recognize a place you've never seen. This experiment changes nothing about the model or how it learns — it just hands it 7.5 times more flashcards, enough that every part of the map appears in the deck under every lighting condition. Training takes a few minutes longer per area, which was affordable all along. Hypothesis All four consecutive reverted experiments (3: decode commitment, 4: dense per-patch supervision, 5: hierarchical pooling, 6: rotation resampling) changed the supervision/decode mechanism while inheriting the same starved data regime, and all failed by nearly identical margins — the shared substrate, not the mechanisms, is the bottleneck. train.py samples only 800 of the ~45,000 enumerable train locations per lighting bucket (1.8%). At 1 m/px GSD (exp-5's density analysis assumed 10 m/px, so it attacked label-space density when the real scarcity is input-space coverage), a 128x128 crop spans 128 m of a ~7 km map, and 800 random locations put ~0.78 training crops per 220 m field cell per bucket — a quarter of grid cells have zero training example in any given lighting condition, and a typical eval crop shares content with roughly half of one training crop in its bucket. The probability field stays near-uniform because most of the map was literally never shown to the model. Raising the per-bucket sample to 6,000 gives ~5.9 crops per cell per bucket (every cell supervised in every lighting condition), makes neighboring overlapping crops (24 m stride, 80%+ shared content) appear at many independent random headings — rotation robustness through spatial density rather than exp-6's refuted temporal resampling of the same sparse 800 — and raises gradient steps from 600 to ~4,500 as a direct consequence. Method In model/train.py only: (1) raise the --max-crops-per-bucket default from 800 to 6,000 (~36k crops/area); (2) hold the training tensor as uint8 and convert each minibatch to float on device — required because 36k float32 crops would be ~7 GB, uint8 is ~1.8 GB. Model, loss, decode, optimizer (Adam 1e-3), batch size 64, epoch count (harness-set 8), and the one-shot static extraction with one random heading per location all stay byte-identical to the kept exp-2 code. Cost: ~3.5 min/area (21 s extraction + ~4,500 steps) vs ~28 s — iteration wall-clock rises to roughly 15 min. Expected outcome If data starvation is the binding constraint, worst-case median error drops clearly below the kept 2489 m — plausibly 1,200-2,300 m, with the field finally committing to real modes in bright buckets. Downside is tightly bounded: identical architecture/loss/decode means a near-uniform field still decodes to the map center (~3.2 km floor), and more i.i.d. coverage of the same distribution cannot make fitting harder, so a null result should land near the current best. A null here would be highly informative: it would rule out the entire data/compute-scale family at this model capacity and point the next iteration at encoder capacity (230k params, 385 KB of a 4 MiB gate) or lighting-conditional structure. Result primary worst-case median error = 2326.54 m (previous best 2489.03 m); full per-area/bucket breakdown in metrics.json Conclusion KEPT — metric improved (2489.03 -> 2326.54 m); change committed as 76c45e3 Scoreboard — median error (m) per area × lighting
the worst cell (underlined) is the experiment's score;
red = failed cell, ink = at target
Fits the aircraft? all deployment gates passed
largest per-area model 908 KB (limit 4,096 KB) · single-frame inference 1.2 ms on one CPU thread (proxy limit 250 ms)
crops are sampled fresh each run from the frozen train split (~45,000 distinct positions per area, times random rotation), never from eval blocks; how many to use is the experiment's own choice
ts 2026-07-20T21:18:06 · commit 76c45e3c4850 ·
parent fb932eb33b29 · artifacts runs/20260720_205622_iter1 ·
agent model claude-fable-5, claude-haiku-4-5-20251001 · took 21 m 44 s
The exact prompt given to the headless agent# Autoresearch iteration — one focused experiment
You are one iteration of an autonomous research loop for UAV low-light
geolocalization. Read `CLAUDE.md` (§1, §3, §6 especially) for full context.
The harness (loop.sh) will train, score, log, and keep/revert AFTER you exit —
you only design the experiment and edit the code.
## Your job, in order
1. **Review the research history.** Query the lineage DB:
`sqlite3 experiments.sqlite "SELECT id, title, category, hypothesis, expected_outcome, result, conclusion, primary_metric, kept FROM experiments ORDER BY id DESC LIMIT 15;"`
Note which hypotheses were supported/refuted. Do not repeat a refuted
experiment without a materially new angle.
**Plateau rule:** if three or more consecutive experiments were reverted,
do not attempt another variation of the last refuted mechanism. Either
pick a design family absent from the history (dispatcher + lighting
specialists, pretrained init, learned relighting, training-scale, …) or
attack the bottleneck the refuted hypotheses jointly point at.
2. **Design ONE focused experiment** — proper experiment design, pre-registered
before you touch code. Write it to `runs/pending_experiment.json`:
```json
{
"title": "one-line name",
"category": "architecture|loss|augmentation|relighting|training|quantization|other",
"hypothesis": "what you believe is limiting the metric and why this change addresses it",
"method": "the ONE focused change, concretely (files, mechanism)",
"expected_outcome": "predicted effect on the §6 worst-case median error, quantified if possible",
"init_strategy": "from-scratch | pretrained:<name>",
"eli5": "2-4 sentences for a smart non-ML reader: what you changed and why it might help, in everyday language — analogies welcome, zero jargon",
"architecture": {"stages": [
{"name": "Camera frame", "detail": "128×128 px crop, one of 6 lighting renders", "changed": false},
{"name": "Feature extractor", "detail": "plain-language description", "changed": false},
{"name": "…", "detail": "…", "changed": true}
]},
"architecture_svg": "<svg viewBox='0 0 980 300' …>…</svg>"
}
```
`architecture_svg` is the **technical architecture diagram** of the model
you are testing — a proper ML-paper figure of the design, drawn by you,
shown at the top of this experiment's gallery entry. **Draw tensors and
operations as the things they are — NOT as labeled boxes:** the input
image as a pixel-textured square; conv feature maps as pseudo-3D slabs
that shrink spatially and deepen in channels; 1-D vectors as thin
vertical tick-bars; fully-connected layers as fans of thin crossing
lines between bars; spatial fields/grids as actually-drawn grids with
shaded cells; decodes/aggregations as thin lines converging from cells
onto a point; samplers/targets as small pictorial glyphs. Captions sit
UNDER each element (name ≤10.5px weight 600, sub-note ≤9.5px); tensor
shapes annotate the flow in small italics (128×128×3 → 8²×128 → 1024).
Losses/targets/samplers live in a lane below the inference path,
annotated with slanted dashed leader lines to small text — no boxes
around them. Style contract, so all experiments' figures read as one
paper: viewBox width 980 (height ~300–360); transparent background; ink
#111111, captions #6b6a60, annotations/arrows #9b998c; **#8c2f1f (red)
reserved for exactly what this experiment changed**; training-only
elements in #8a6a1e (or red if changed); font-family
Palatino,Georgia,serif; stroke-width ≈1.2 elements, 1 arrows; fills only
faint tints (opacity ≤ .12); no gradients, no icons, no emoji. Inference
flows left → right from camera frame to (u, v, conf). Lay elements out
on a running x-cursor with generous spacing so nothing overlaps.
`eli5` and `architecture` feed the human-facing gallery. `architecture.stages`
is the model's inference path left-to-right, camera frame → (lat, lon,
confidence) output — one box per stage, plain-language `detail`,
`"changed": true` ONLY on the stages this experiment touches. Reuse the
previous experiment's stage names verbatim wherever a stage is unchanged
(check `SELECT arch_json FROM experiments WHERE arch_json IS NOT NULL ORDER BY id DESC LIMIT 1;`) —
the diagrams are compared box-by-box across experiments. A change that only
affects training (loss, augmentation, schedule) keeps the inference stages
unchanged and adds one final stage with `"train_only": true` describing the
training signal.
3. **Implement exactly that change** in the agent-editable files (`model/`).
Keep `train.py`'s CLI contract and the ONNX export contract in `model/model.py`'s
docstring intact — the frozen scorer depends on them.
## Hard rules
- Edit ONLY `model/` (and `runs/pending_experiment.json`). Files listed in
`/FROZEN` are off-limits; the harness hard-reverts any change to them.
- NEVER touch, read, or evaluate the `hamburg` holdout area (§5).
- ONE focused change per iteration — if you can't describe it in one sentence,
it's too big. Prefer architectural/procedural novelty over hyperparameter
nudges (§3): changing a learning rate is a weak experiment; changing the
coordinate parameterization, loss family, relighting-for-training, or model
topology is a strong one.
- Do not run training yourself; the harness does that. A quick syntax check
(`.venv/bin/python -c "import model.train, model.model"`) is fine.
- Stay within the deployment gates: exported ONNX ≤ 4 MiB per area, host
latency proxy ≤ 250 ms (see pipeline/score.py).Where the model was tested — and how far off it was Each map is one full test area. Every dot is one held-out test location (all six lighting conditions overlaid): the model was shown a 128 m crop centered there and asked for its position. Dot color = the distance between its answer and the truth — green ≤ 20 m (at goal), amber ≤ 50 m, red beyond. A working model turns these maps green; spatial clusters of red reveal which parts of an area confuse it.   What the six lighting conditions look like through the simulated low-light sensor (example patches — illustration, not the training set)Every frame in this project is rendered as a starlight-class low-light sensor (Sony STARVIS2 / IMX585 class — the airframe's chosen camera) would see it, not as a normal camera would. That is why the night renders look uncannily close to daylight: at high gain such a sensor recovers scene structure from moonlight, skyglow and artificial lighting, paying for it in noise, lifted shadows and washed-out color — exactly what these renders simulate. Below, one example 256 m patch per area under the six lighting conditions the model must handle. These illustrate the dataset, not this experiment's performance — the actual training set is thousands of distinct crops per area (see “training data” above), and these renderings only change when the relighting method changes. berlin frankfurt munich prignitz | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▸ | 6 | Per-epoch rotation resampling replaces the one-shot frozen training tensor de614d47 | augmentation | from-scratch | 2.51 km | 908 KB | 0.7 ms | 6 m 52 s | — | discarded | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The design under test — technical diagram — gray = frozen contract · ink = the current design · red = this experiment's change One real test, end to end
→
1.53 km
miss, this crop 0.91
self-reported confidence In plain words Until now the model studied each training photo from one fixed compass direction only -- like memorizing your neighborhood while always facing north, then being quizzed while facing random directions. Unsurprisingly, it froze on the quiz. Now, every time it re-reads its study deck, every photo is re-dealt at a new random rotation, so the only way to get good grades in practice is to recognize places no matter which way it is facing -- which is exactly what the real quiz (and a real drone) demands. Hypothesis The probability field stays near-uniform not because of supervision form (exp-3/4/5 all attacked the head/loss and failed identically) but because the encoder never learns rotation-invariant features: train.py extracts each of the few hundred train locations exactly once, at one fixed random heading, into a static tensor reused every epoch, while the frozen eval rotates every crop by a different deterministic angle. The model can fit 'this place seen at heading 137 degrees' without ever being forced to recognize the same place at another heading, so at eval time features do not fire, the field stays flat, and the soft-argmax decode collapses toward the map center. Re-extracting all training crops with fresh uniform-random angles at the start of every epoch multiplies heading diversity per location by the epoch count and makes rotation invariance the only way to reduce training loss. Method In model/train.py only: split load_training_tensors into a one-time loader (meta, train-crop list, six relight images cached in memory) and a per-epoch sampler that re-runs extract_crop with a fresh rng.uniform(0,360) angle for every (location, bucket) pair; the training loop rebuilds x each epoch from the cached images. Locations stay on the frozen list_crops grid (no position jitter, so the no-eval-leakage guarantee in pipeline/dataset.py is untouched). model/model.py, the loss, and the exported inference path are unchanged. Expected outcome If rotation overfit is the binding constraint, the worst-case median error drops clearly below the kept 2489 m -- plausibly to 1400-2300 m, with the largest gains in bright buckets where texture is most discriminative. Downside is tightly bounded: same locations, same model, same loss, and a near-uniform field still decodes to the map center (~3.2 km floor), so a null result should land near the current best rather than below baseline. Result primary worst-case median error = 2509.97 m (previous best 2489.03 m); full per-area/bucket breakdown in metrics.json Conclusion REVERTED — metric did not improve (2509.97 m vs best 2489.03 m); change discarded Scoreboard — median error (m) per area × lighting
the worst cell (underlined) is the experiment's score;
red = failed cell, ink = at target
Fits the aircraft? all deployment gates passed
largest per-area model 908 KB (limit 4,096 KB) · single-frame inference 0.7 ms on one CPU thread (proxy limit 250 ms)
crops are sampled fresh each run from the frozen train split (~45,000 distinct positions per area, times random rotation), never from eval blocks; how many to use is the experiment's own choice
ts 2026-07-20T20:26:38 · commit de614d47e3ea ·
parent 7d02e02d9ca2 · artifacts runs/20260720_201946_iter5 ·
agent model claude-fable-5, claude-haiku-4-5-20251001 · took 6 m 52 s
The exact prompt given to the headless agent# Autoresearch iteration — one focused experiment
You are one iteration of an autonomous research loop for UAV low-light
geolocalization. Read `CLAUDE.md` (§1, §3, §6 especially) for full context.
The harness (loop.sh) will train, score, log, and keep/revert AFTER you exit —
you only design the experiment and edit the code.
## Your job, in order
1. **Review the research history.** Query the lineage DB:
`sqlite3 experiments.sqlite "SELECT id, title, category, hypothesis, expected_outcome, result, conclusion, primary_metric, kept FROM experiments ORDER BY id DESC LIMIT 15;"`
Note which hypotheses were supported/refuted. Do not repeat a refuted
experiment without a materially new angle.
2. **Design ONE focused experiment** — proper experiment design, pre-registered
before you touch code. Write it to `runs/pending_experiment.json`:
```json
{
"title": "one-line name",
"category": "architecture|loss|augmentation|relighting|training|quantization|other",
"hypothesis": "what you believe is limiting the metric and why this change addresses it",
"method": "the ONE focused change, concretely (files, mechanism)",
"expected_outcome": "predicted effect on the §6 worst-case median error, quantified if possible",
"init_strategy": "from-scratch | pretrained:<name>",
"eli5": "2-4 sentences for a smart non-ML reader: what you changed and why it might help, in everyday language — analogies welcome, zero jargon",
"architecture": {"stages": [
{"name": "Camera frame", "detail": "128×128 px crop, one of 6 lighting renders", "changed": false},
{"name": "Feature extractor", "detail": "plain-language description", "changed": false},
{"name": "…", "detail": "…", "changed": true}
]}
}
```
`eli5` and `architecture` feed the human-facing gallery. `architecture.stages`
is the model's inference path left-to-right, camera frame → (lat, lon,
confidence) output — one box per stage, plain-language `detail`,
`"changed": true` ONLY on the stages this experiment touches. Reuse the
previous experiment's stage names verbatim wherever a stage is unchanged
(check `SELECT arch_json FROM experiments WHERE arch_json IS NOT NULL ORDER BY id DESC LIMIT 1;`) —
the diagrams are compared box-by-box across experiments. A change that only
affects training (loss, augmentation, schedule) keeps the inference stages
unchanged and adds one final stage with `"train_only": true` describing the
training signal.
3. **Implement exactly that change** in the agent-editable files (`model/`).
Keep `train.py`'s CLI contract and the ONNX export contract in `model/model.py`'s
docstring intact — the frozen scorer depends on them.
## Hard rules
- Edit ONLY `model/` (and `runs/pending_experiment.json`). Files listed in
`/FROZEN` are off-limits; the harness hard-reverts any change to them.
- NEVER touch, read, or evaluate the `hamburg` holdout area (§5).
- ONE focused change per iteration — if you can't describe it in one sentence,
it's too big. Prefer architectural/procedural novelty over hyperparameter
nudges (§3): changing a learning rate is a weak experiment; changing the
coordinate parameterization, loss family, relighting-for-training, or model
topology is a strong one.
- Do not run training yourself; the harness does that. A quick syntax check
(`.venv/bin/python -c "import model.train, model.model"`) is fine.
- Stay within the deployment gates: exported ONNX ≤ 4 MiB per area, host
latency proxy ≤ 250 ms (see pipeline/score.py).Where the model was tested — and how far off it was Each map is one full test area. Every dot is one held-out test location (all six lighting conditions overlaid): the model was shown a 128 m crop centered there and asked for its position. Dot color = the distance between its answer and the truth — green ≤ 20 m (at goal), amber ≤ 50 m, red beyond. A working model turns these maps green; spatial clusters of red reveal which parts of an area confuse it.  What the six lighting conditions look like through the simulated low-light sensor (example patches — illustration, not the training set)Every frame in this project is rendered as a starlight-class low-light sensor (Sony STARVIS2 / IMX585 class — the airframe's chosen camera) would see it, not as a normal camera would. That is why the night renders look uncannily close to daylight: at high gain such a sensor recovers scene structure from moonlight, skyglow and artificial lighting, paying for it in noise, lifted shadows and washed-out color — exactly what these renders simulate. Below, one example 256 m patch per area under the six lighting conditions the model must handle. These illustrate the dataset, not this experiment's performance — the actual training set is thousands of distinct crops per area (see “training data” above), and these renderings only change when the relighting method changes. berlin frankfurt munich prignitz | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▸ | 5 | Hierarchical coarse-to-fine supervision of the probability field via probability pooling 7d02e02d | loss | from-scratch | 2.52 km | 908 KB | 0.6 ms | 6 m 20 s | — | discarded | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The design under test — technical diagram — gray = frozen contract · ink = the current design · red = this experiment's change One real test, end to end
→
1.40 km
miss, this crop 0.93
self-reported confidence In plain words The model localizes by painting a heat map over a 32x32 grid of the area — but we only have enough training photos to give most grid squares zero examples, so the heat map never learned to commit and stayed lukewarm everywhere. The fix is to grade the same heat map at several zoom levels at once: first 'did you put the heat in the right quarter of the map?', then the right sixteenth, and so on down to the fine grid. Coarse questions have plenty of examples to learn from, and getting them right automatically steers the fine answer — like learning an address by first nailing the district, then the street, then the house. Hypothesis The kept 32x32 field stays near-uniform (confirmed by exp-3's argmax test and exp-4's failure to beat it) because supervision density in label space is below one example per class: at 10 m/px with a 24 px sampling stride a ~7 km area yields only ~600 distinct train locations per bucket spread over 1,024 fine cells, so the Gaussian-CE mostly teaches 'not here' uniformly and no cell ever accumulates enough positive gradient to stand out. Sum-pooling the same softmax field to 16x16, 8x8, and 4x4 and applying the identical Gaussian-CE at each level gives coarse cells 10-40 positive training locations each — dense enough to learn real discrimination — and because coarse probabilities are exact sums of fine-cell probabilities, that well-conditioned coarse gradient flows directly into the fine logits ('put mass somewhere in this quadrant'), letting the fine level refine a coarse decision instead of learning 1,024-way classification from scratch. Method model/model.py loss_fn only: reshape softmax(logits) to [B,32,32], build coarser fields by 2x2/4x4/8x8 sum-pooling (avg_pool2d * factor^2), and compute the existing Gaussian-smoothed cross-entropy at levels k in {4, 8, 16, 32} with sigma held constant in map units (TARGET_SIGMA_CELLS * k / GRID_K cells), averaging the four CE terms in place of the single fine CE. Forward pass, decode, export, and train.py are untouched — zero new parameters, identical ONNX artifact shape. Expected outcome Worst-case median error drops below the kept 2489 m. If the model reaches reliable 8x8-level discrimination (875 m cells), soft-argmax refinement should land the worst-case median around 1300-2200 m. Downside is tightly bounded: at init the uniform field still decodes to the map center (the ~3.2 km center-guess floor), the fine-level CE term is still present, and no inference-path component changes, so a null result should score close to the current best rather than below baseline. Result primary worst-case median error = 2516.25 m (previous best 2489.03 m); full per-area/bucket breakdown in metrics.json Conclusion REVERTED — metric did not improve (2516.25 m vs best 2489.03 m); change discarded Scoreboard — median error (m) per area × lighting
the worst cell (underlined) is the experiment's score;
red = failed cell, ink = at target
Fits the aircraft? all deployment gates passed
largest per-area model 908 KB (limit 4,096 KB) · single-frame inference 0.6 ms on one CPU thread (proxy limit 250 ms)
crops are sampled fresh each run from the frozen train split (~45,000 distinct positions per area, times random rotation), never from eval blocks; how many to use is the experiment's own choice
ts 2026-07-20T20:19:45 · commit 7d02e02d9ca2 ·
parent 7d02e02d9ca2 · artifacts runs/20260720_201325_iter4 ·
agent model claude-fable-5, claude-haiku-4-5-20251001 · took 6 m 20 s
The exact prompt given to the headless agent# Autoresearch iteration — one focused experiment
You are one iteration of an autonomous research loop for UAV low-light
geolocalization. Read `CLAUDE.md` (§1, §3, §6 especially) for full context.
The harness (loop.sh) will train, score, log, and keep/revert AFTER you exit —
you only design the experiment and edit the code.
## Your job, in order
1. **Review the research history.** Query the lineage DB:
`sqlite3 experiments.sqlite "SELECT id, title, category, hypothesis, expected_outcome, result, conclusion, primary_metric, kept FROM experiments ORDER BY id DESC LIMIT 15;"`
Note which hypotheses were supported/refuted. Do not repeat a refuted
experiment without a materially new angle.
2. **Design ONE focused experiment** — proper experiment design, pre-registered
before you touch code. Write it to `runs/pending_experiment.json`:
```json
{
"title": "one-line name",
"category": "architecture|loss|augmentation|relighting|training|quantization|other",
"hypothesis": "what you believe is limiting the metric and why this change addresses it",
"method": "the ONE focused change, concretely (files, mechanism)",
"expected_outcome": "predicted effect on the §6 worst-case median error, quantified if possible",
"init_strategy": "from-scratch | pretrained:<name>",
"eli5": "2-4 sentences for a smart non-ML reader: what you changed and why it might help, in everyday language — analogies welcome, zero jargon",
"architecture": {"stages": [
{"name": "Camera frame", "detail": "128×128 px crop, one of 6 lighting renders", "changed": false},
{"name": "Feature extractor", "detail": "plain-language description", "changed": false},
{"name": "…", "detail": "…", "changed": true}
]}
}
```
`eli5` and `architecture` feed the human-facing gallery. `architecture.stages`
is the model's inference path left-to-right, camera frame → (lat, lon,
confidence) output — one box per stage, plain-language `detail`,
`"changed": true` ONLY on the stages this experiment touches. Reuse the
previous experiment's stage names verbatim wherever a stage is unchanged
(check `SELECT arch_json FROM experiments WHERE arch_json IS NOT NULL ORDER BY id DESC LIMIT 1;`) —
the diagrams are compared box-by-box across experiments. A change that only
affects training (loss, augmentation, schedule) keeps the inference stages
unchanged and adds one final stage with `"train_only": true` describing the
training signal.
3. **Implement exactly that change** in the agent-editable files (`model/`).
Keep `train.py`'s CLI contract and the ONNX export contract in `model/model.py`'s
docstring intact — the frozen scorer depends on them.
## Hard rules
- Edit ONLY `model/` (and `runs/pending_experiment.json`). Files listed in
`/FROZEN` are off-limits; the harness hard-reverts any change to them.
- NEVER touch, read, or evaluate the `hamburg` holdout area (§5).
- ONE focused change per iteration — if you can't describe it in one sentence,
it's too big. Prefer architectural/procedural novelty over hyperparameter
nudges (§3): changing a learning rate is a weak experiment; changing the
coordinate parameterization, loss family, relighting-for-training, or model
topology is a strong one.
- Do not run training yourself; the harness does that. A quick syntax check
(`.venv/bin/python -c "import model.train, model.model"`) is fine.
- Stay within the deployment gates: exported ONNX ≤ 4 MiB per area, host
latency proxy ≤ 250 ms (see pipeline/score.py).Where the model was tested — and how far off it was Each map is one full test area. Every dot is one held-out test location (all six lighting conditions overlaid): the model was shown a 128 m crop centered there and asked for its position. Dot color = the distance between its answer and the truth — green ≤ 20 m (at goal), amber ≤ 50 m, red beyond. A working model turns these maps green; spatial clusters of red reveal which parts of an area confuse it. What the six lighting conditions look like through the simulated low-light sensor (example patches — illustration, not the training set)Every frame in this project is rendered as a starlight-class low-light sensor (Sony STARVIS2 / IMX585 class — the airframe's chosen camera) would see it, not as a normal camera would. That is why the night renders look uncannily close to daylight: at high gain such a sensor recovers scene structure from moonlight, skyglow and artificial lighting, paying for it in noise, lifted shadows and washed-out color — exactly what these renders simulate. Below, one example 256 m patch per area under the six lighting conditions the model must handle. These illustrate the dataset, not this experiment's performance — the actual training set is thousands of distinct crops per area (see “training data” above), and these renderings only change when the relighting method changes. berlin frankfurt munich prignitz | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▸ | 4 | ACE-style dense per-patch scene-coordinate regression replaces global map-cell probability field 7d02e02d | architecture | from-scratch | 2.56 km | 962 KB | 0.8 ms | 9 m 11 s | — | discarded | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The design under test — technical diagram — gray = frozen contract · ink = the current design · red = this experiment's change One real test, end to end
→
1.91 km
miss, this crop 0.98
self-reported confidence In plain words Rather than one verdict per photo, the network was made to answer 64 times per photo: the image is split into an 8×8 grid of patches, and each patch must separately name the map coordinate of the ground it shows (the training data knows the right answer for every patch). The final position is the average of those 64 mini-answers. The hope: 64× more feedback per photo forces the network to learn which ground textures give away a location. It scored slightly worse than the heat-map approach (~2.56 km vs ~2.49 km), so it was reverted. Hypothesis The encoder never learns geo-discriminative local features because supervision is one target per 128x128 crop: the GAP bottleneck plus a single global 32x32 map-cell field gives every conv feature the same diffuse gradient, which is why fields stay near-uniform (exp-3 refuted multimodality) and errors sit at ~2.1-2.5 km, barely under the 3.2 km center-guess bound, uniformly across all lighting buckets. Dense scene-coordinate regression (the ACE/DSAC family CLAUDE.md §3 explicitly names) supervises each of the 8x8 stride-16 feature cells with the map coordinate of its own patch — computable at train time from crop center + rotation — multiplying supervision density 64x on the same 4,800-crop budget and forcing local textures to become location-predictive. Because the patch-offset grid is symmetric about the crop center and rotation is linear, the unweighted mean of per-cell predictions is an unbiased crop-center estimate requiring no heading knowledge at inference, so the frozen rotated-eval contract is satisfied unchanged. Method In model/model.py: replace the GAP -> 1024-way map-cell logits head with (a) one 3x3 dilation-2 conv context stage on the 8x8x128 feature map (receptive field ~95 px, block-level context per cell), (b) a 1x1 conv -> sigmoid head emitting per-cell normalized (u,v) map coordinates, (c) forward output = unweighted mean of the 64 per-cell coordinates plus the existing loose sigmoid conf head on GAP'd context features. Loss becomes per-cell smooth-L1 against per-cell targets plus the unchanged 0.1-weighted conf BCE. In model/train.py: load_training_tensors computes the [2,8,8] per-cell target grid per sample by rotating the fixed patch-offset lattice by the sampled crop angle and adding the crop-center pixel coordinate (helper cell_target_grid in model.py). ONNX export contract, CLI contract, and crop sampling untouched. Expected outcome Worst-case median error drops below the kept 2489 m: with 64x denser gradients the encoder should finally beat the center-guess regime, predicting roughly 800-2000 m worst-case. Downside is bounded: at init sigmoid cells output ~0.5 so the mean decode starts at the map center (same center-guess floor as the baseline), and the robust smooth-L1 keeps unlocalizable patches (fields, water) from dominating the gradient. Result primary worst-case median error = 2563.37 m (previous best 2489.03 m); full per-area/bucket breakdown in metrics.json Conclusion REVERTED — metric did not improve (2563.37 m vs best 2489.03 m); change discarded Scoreboard — median error (m) per area × lighting
the worst cell (underlined) is the experiment's score;
red = failed cell, ink = at target
Fits the aircraft? all deployment gates passed
largest per-area model 962 KB (limit 4,096 KB) · single-frame inference 0.8 ms on one CPU thread (proxy limit 250 ms)
crops are sampled fresh each run from the frozen train split (~45,000 distinct positions per area, times random rotation), never from eval blocks; how many to use is the experiment's own choice
ts 2026-07-20T20:13:25 · commit 7d02e02d9ca2 ·
parent ff65590e776f · artifacts runs/20260720_200413_iter3 ·
agent model claude-fable-5, claude-haiku-4-5-20251001 · took 9 m 11 s
The exact prompt given to the headless agent# Autoresearch iteration — one focused experiment
You are one iteration of an autonomous research loop for UAV low-light
geolocalization. Read `CLAUDE.md` (§1, §3, §6 especially) for full context.
The harness (loop.sh) will train, score, log, and keep/revert AFTER you exit —
you only design the experiment and edit the code.
## Your job, in order
1. **Review the research history.** Query the lineage DB:
`sqlite3 experiments.sqlite "SELECT id, title, category, hypothesis, expected_outcome, result, conclusion, primary_metric, kept FROM experiments ORDER BY id DESC LIMIT 15;"`
Note which hypotheses were supported/refuted. Do not repeat a refuted
experiment without a materially new angle.
2. **Design ONE focused experiment** — proper experiment design, pre-registered
before you touch code. Write it to `runs/pending_experiment.json`:
```json
{
"title": "one-line name",
"category": "architecture|loss|augmentation|relighting|training|quantization|other",
"hypothesis": "what you believe is limiting the metric and why this change addresses it",
"method": "the ONE focused change, concretely (files, mechanism)",
"expected_outcome": "predicted effect on the §6 worst-case median error, quantified if possible",
"init_strategy": "from-scratch | pretrained:<name>"
}
```
3. **Implement exactly that change** in the agent-editable files (`model/`).
Keep `train.py`'s CLI contract and the ONNX export contract in `model/model.py`'s
docstring intact — the frozen scorer depends on them.
## Hard rules
- Edit ONLY `model/` (and `runs/pending_experiment.json`). Files listed in
`/FROZEN` are off-limits; the harness hard-reverts any change to them.
- NEVER touch, read, or evaluate the `hamburg` holdout area (§5).
- ONE focused change per iteration — if you can't describe it in one sentence,
it's too big. Prefer architectural/procedural novelty over hyperparameter
nudges (§3): changing a learning rate is a weak experiment; changing the
coordinate parameterization, loss family, relighting-for-training, or model
topology is a strong one.
- Do not run training yourself; the harness does that. A quick syntax check
(`.venv/bin/python -c "import model.train, model.model"`) is fine.
- Stay within the deployment gates: exported ONNX ≤ 4 MiB per area, host
latency proxy ≤ 250 ms (see pipeline/score.py).Where the model was tested — and how far off it was Each map is one full test area. Every dot is one held-out test location (all six lighting conditions overlaid): the model was shown a 128 m crop centered there and asked for its position. Dot color = the distance between its answer and the truth — green ≤ 20 m (at goal), amber ≤ 50 m, red beyond. A working model turns these maps green; spatial clusters of red reveal which parts of an area confuse it. What the six lighting conditions look like through the simulated low-light sensor (example patches — illustration, not the training set)Every frame in this project is rendered as a starlight-class low-light sensor (Sony STARVIS2 / IMX585 class — the airframe's chosen camera) would see it, not as a normal camera would. That is why the night renders look uncannily close to daylight: at high gain such a sensor recovers scene structure from moonlight, skyglow and artificial lighting, paying for it in noise, lifted shadows and washed-out color — exactly what these renders simulate. Below, one example 256 m patch per area under the six lighting conditions the model must handle. These illustrate the dataset, not this experiment's performance — the actual training set is thousands of distinct crops per area (see “training data” above), and these renderings only change when the relighting method changes. berlin frankfurt munich prignitz | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▸ | 3 | Argmax-anchored local soft-argmax decode replaces global expected-coordinate ff65590e | architecture | from-scratch | 2.94 km | 918 KB | 0.6 ms | 4 m 58 s | — | discarded | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The design under test — technical diagram — gray = frozen contract · ink = the current design · red = this experiment's change One real test, end to end
→
3.67 km
miss, this crop 0.68
self-reported confidence In plain words An attempted fix for experiment 2's “dragged toward the middle” problem. Rather than averaging the whole heat map, the model first commits to the single hottest cell, then fine-tunes its answer using only the 5×5 neighborhood around it — decide roughly where first, then refine locally. It backfired: the hottest cell is apparently often wrong (the heat maps are still too scattered), so committing to it made the error worse (~2.9 km vs ~2.5 km) and the change was reverted. Hypothesis Global soft-argmax over the 32x32 field decodes the probability-weighted mean of ALL cells, so any diffuse or multimodal probability field is pulled toward the map centroid — the kept exp-2 result (worst-case median 2489 m) barely beats the 3217 m center-guess bound, consistent with predictions huddling near the map mean rather than committing to the dominant mode. Anchoring the decode at the argmax cell and taking soft-argmax only within a +/-2-cell window forces mode commitment while keeping differentiable sub-cell refinement, and the training coordinate loss is computed on exactly this decode so train and inference stay coupled. Method In model/model.py forward(): after softmax over the 1024 cell logits, take the argmax cell, build a window mask |gx-gx*|<=2 and |gy-gy*|<=2 (5x5 cells, ~1.1 km on a ~7 km map), renormalize probabilities inside the window, and decode (u,v) as the expected coordinate over the window only. The mask is non-differentiable (straight-through-style); gradients flow through the windowed probabilities. CE-against-Gaussian-target term, conf head, train.py, and the ONNX export contract are unchanged — the exported graph uses ArgMax/Mod/floor-Div ops supported at opset 17. Expected outcome If learned fields are multimodal/diffuse (the hypothesis), predictions disperse from the centroid to true modes and the worst-case median drops meaningfully below 2489 m — estimated 1200-2200 m, largest gains in bright buckets where the field is most discriminative. Risk: if fields are near-uniform (uninformative rather than multimodal), argmax anchoring adds noise and dark-bucket error could worsen; downside bounded by roughly the center-guess scale since the CE term still trains the field identically. Result primary worst-case median error = 2937.73 m (previous best 2489.03 m); full per-area/bucket breakdown in metrics.json Conclusion REVERTED — metric did not improve (2937.73 m vs best 2489.03 m); change discarded Scoreboard — median error (m) per area × lighting
the worst cell (underlined) is the experiment's score;
red = failed cell, ink = at target
Fits the aircraft? all deployment gates passed
largest per-area model 918 KB (limit 4,096 KB) · single-frame inference 0.6 ms on one CPU thread (proxy limit 250 ms)
crops are sampled fresh each run from the frozen train split (~45,000 distinct positions per area, times random rotation), never from eval blocks; how many to use is the experiment's own choice
ts 2026-07-20T20:04:13 · commit ff65590e776f ·
parent ff65590e776f · artifacts runs/20260720_195914_iter2 ·
agent model claude-fable-5, claude-haiku-4-5-20251001 · took 4 m 58 s
The exact prompt given to the headless agent# Autoresearch iteration — one focused experiment
You are one iteration of an autonomous research loop for UAV low-light
geolocalization. Read `CLAUDE.md` (§1, §3, §6 especially) for full context.
The harness (loop.sh) will train, score, log, and keep/revert AFTER you exit —
you only design the experiment and edit the code.
## Your job, in order
1. **Review the research history.** Query the lineage DB:
`sqlite3 experiments.sqlite "SELECT id, title, category, hypothesis, expected_outcome, result, conclusion, primary_metric, kept FROM experiments ORDER BY id DESC LIMIT 15;"`
Note which hypotheses were supported/refuted. Do not repeat a refuted
experiment without a materially new angle.
2. **Design ONE focused experiment** — proper experiment design, pre-registered
before you touch code. Write it to `runs/pending_experiment.json`:
```json
{
"title": "one-line name",
"category": "architecture|loss|augmentation|relighting|training|quantization|other",
"hypothesis": "what you believe is limiting the metric and why this change addresses it",
"method": "the ONE focused change, concretely (files, mechanism)",
"expected_outcome": "predicted effect on the §6 worst-case median error, quantified if possible",
"init_strategy": "from-scratch | pretrained:<name>"
}
```
3. **Implement exactly that change** in the agent-editable files (`model/`).
Keep `train.py`'s CLI contract and the ONNX export contract in `model/model.py`'s
docstring intact — the frozen scorer depends on them.
## Hard rules
- Edit ONLY `model/` (and `runs/pending_experiment.json`). Files listed in
`/FROZEN` are off-limits; the harness hard-reverts any change to them.
- NEVER touch, read, or evaluate the `hamburg` holdout area (§5).
- ONE focused change per iteration — if you can't describe it in one sentence,
it's too big. Prefer architectural/procedural novelty over hyperparameter
nudges (§3): changing a learning rate is a weak experiment; changing the
coordinate parameterization, loss family, relighting-for-training, or model
topology is a strong one.
- Do not run training yourself; the harness does that. A quick syntax check
(`.venv/bin/python -c "import model.train, model.model"`) is fine.
- Stay within the deployment gates: exported ONNX ≤ 4 MiB per area, host
latency proxy ≤ 250 ms (see pipeline/score.py).Where the model was tested — and how far off it was Each map is one full test area. Every dot is one held-out test location (all six lighting conditions overlaid): the model was shown a 128 m crop centered there and asked for its position. Dot color = the distance between its answer and the truth — green ≤ 20 m (at goal), amber ≤ 50 m, red beyond. A working model turns these maps green; spatial clusters of red reveal which parts of an area confuse it. What the six lighting conditions look like through the simulated low-light sensor (example patches — illustration, not the training set)Every frame in this project is rendered as a starlight-class low-light sensor (Sony STARVIS2 / IMX585 class — the airframe's chosen camera) would see it, not as a normal camera would. That is why the night renders look uncannily close to daylight: at high gain such a sensor recovers scene structure from moonlight, skyglow and artificial lighting, paying for it in noise, lifted shadows and washed-out color — exactly what these renders simulate. Below, one example 256 m patch per area under the six lighting conditions the model must handle. These illustrate the dataset, not this experiment's performance — the actual training set is thousands of distinct crops per area (see “training data” above), and these renderings only change when the relighting method changes. berlin frankfurt munich prignitz | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▸ | 2 | DSNT-style spatial probability field over the map replaces direct (u,v) regression ff65590e | architecture | from-scratch | 2.49 km | 908 KB | 0.7 ms | 7 m 08 s | — | kept | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The design under test — technical diagram — gray = frozen contract · ink = the current design · red = this experiment's change One real test, end to end
→
1.60 km
miss, this crop 0.89
self-reported confidence In plain words Instead of blurting out one guess, the network now lays a 32×32 grid of cells over the map and gives each cell a score for “the photo probably came from here” — a heat map of suspicion. Its answer is the balance point (weighted average) of that heat map. Training became much more informative — every cell is told individually whether it should have been hotter or colder — and the error improved from ~3.2 km to ~2.5 km. Remaining weakness: averaging a spread-out heat map drags answers toward the middle of the map. Hypothesis Direct MSE regression to sigmoid (u,v) collapses to the map-mean (center-guess ~3.2 km) because under early-training ambiguity the MSE optimum IS the mean and the gradient gives no per-location signal; the one prior classification attempt (archived exp 5) failed for a different reason — hard argmax cells + a decoupled offset head quantize the decision and decouple training from the metric. A dense probability field over a 32x32 grid of map cells, decoded by differentiable soft-argmax (expected coordinate) and trained with cross-entropy against Gaussian-smoothed cell targets plus an expected-coordinate L2 term, gives per-cell gradient signal that escapes the mean-collapse while remaining trained end-to-end on the exact coordinate the metric measures. Method In model/model.py, replace TinyLocNet's Linear(128->3)+sigmoid head with: Linear(128->1024) map-cell logits over a 32x32 grid (~220 m cells at 7 km), softmax, expected (u,v) = probability-weighted sum of fixed cell centers (registered buffers, ONNX-friendly), plus an unchanged separate sigmoid conf head trained with the same loose BCE target (err < 0.5 extent) so the coverage gate is untouched. Loss becomes CE(softmax logits, Gaussian soft target sigma=1.5 cells) + expected-coord L2 + 0.1*conf BCE. model/train.py changes only the loss call to pass logits through. Expected outcome Worst-case median error drops below the 3216.74 m baseline; if the field head learns even coarse district-level discrimination in 8 epochs x 4800 crops, expect roughly 1500-2900 m worst-case, with the clearest gains in bright buckets. At init the soft-argmax outputs the grid mean (same center-guess as baseline), so the downside is bounded near no-change. Result primary worst-case median error = 2489.03 m (previous best 3216.74 m); full per-area/bucket breakdown in metrics.json Conclusion KEPT — metric improved (3216.74 -> 2489.03 m); change committed as ff65590 Scoreboard — median error (m) per area × lighting
the worst cell (underlined) is the experiment's score;
red = failed cell, ink = at target
Fits the aircraft? all deployment gates passed
largest per-area model 908 KB (limit 4,096 KB) · single-frame inference 0.7 ms on one CPU thread (proxy limit 250 ms)
crops are sampled fresh each run from the frozen train split (~45,000 distinct positions per area, times random rotation), never from eval blocks; how many to use is the experiment's own choice
ts 2026-07-20T19:59:14 · commit ff65590e776f ·
parent 3d53a264d82c · artifacts runs/20260720_195206_iter1 ·
agent model claude-fable-5, claude-haiku-4-5-20251001 · took 7 m 08 s
The exact prompt given to the headless agent# Autoresearch iteration — one focused experiment
You are one iteration of an autonomous research loop for UAV low-light
geolocalization. Read `CLAUDE.md` (§1, §3, §6 especially) for full context.
The harness (loop.sh) will train, score, log, and keep/revert AFTER you exit —
you only design the experiment and edit the code.
## Your job, in order
1. **Review the research history.** Query the lineage DB:
`sqlite3 experiments.sqlite "SELECT id, title, category, hypothesis, expected_outcome, result, conclusion, primary_metric, kept FROM experiments ORDER BY id DESC LIMIT 15;"`
Note which hypotheses were supported/refuted. Do not repeat a refuted
experiment without a materially new angle.
2. **Design ONE focused experiment** — proper experiment design, pre-registered
before you touch code. Write it to `runs/pending_experiment.json`:
```json
{
"title": "one-line name",
"category": "architecture|loss|augmentation|relighting|training|quantization|other",
"hypothesis": "what you believe is limiting the metric and why this change addresses it",
"method": "the ONE focused change, concretely (files, mechanism)",
"expected_outcome": "predicted effect on the §6 worst-case median error, quantified if possible",
"init_strategy": "from-scratch | pretrained:<name>"
}
```
3. **Implement exactly that change** in the agent-editable files (`model/`).
Keep `train.py`'s CLI contract and the ONNX export contract in `model/model.py`'s
docstring intact — the frozen scorer depends on them.
## Hard rules
- Edit ONLY `model/` (and `runs/pending_experiment.json`). Files listed in
`/FROZEN` are off-limits; the harness hard-reverts any change to them.
- NEVER touch, read, or evaluate the `hamburg` holdout area (§5).
- ONE focused change per iteration — if you can't describe it in one sentence,
it's too big. Prefer architectural/procedural novelty over hyperparameter
nudges (§3): changing a learning rate is a weak experiment; changing the
coordinate parameterization, loss family, relighting-for-training, or model
topology is a strong one.
- Do not run training yourself; the harness does that. A quick syntax check
(`.venv/bin/python -c "import model.train, model.model"`) is fine.
- Stay within the deployment gates: exported ONNX ≤ 4 MiB per area, host
latency proxy ≤ 250 ms (see pipeline/score.py).Where the model was tested — and how far off it was Each map is one full test area. Every dot is one held-out test location (all six lighting conditions overlaid): the model was shown a 128 m crop centered there and asked for its position. Dot color = the distance between its answer and the truth — green ≤ 20 m (at goal), amber ≤ 50 m, red beyond. A working model turns these maps green; spatial clusters of red reveal which parts of an area confuse it.  What the six lighting conditions look like through the simulated low-light sensor (example patches — illustration, not the training set)Every frame in this project is rendered as a starlight-class low-light sensor (Sony STARVIS2 / IMX585 class — the airframe's chosen camera) would see it, not as a normal camera would. That is why the night renders look uncannily close to daylight: at high gain such a sensor recovers scene structure from moonlight, skyglow and artificial lighting, paying for it in noise, lifted shadows and washed-out color — exactly what these renders simulate. Below, one example 256 m patch per area under the six lighting conditions the model must handle. These illustrate the dataset, not this experiment's performance — the actual training set is thousands of distinct crops per area (see “training data” above), and these renderings only change when the relighting method changes. berlin frankfurt munich prignitz | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ▸ | 1 | Starting baseline — naive TinyLocNet, from-scratch, frozen pipeline v1 b47d44a8 | architecture | from-scratch | 3.22 km | 385 KB | 0.6 ms | — | — | kept | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
The design under test — technical diagram — gray = frozen contract · ink = the current design · red = this experiment's change One real test, end to end
→
1.90 km
miss, this crop 0.98
self-reported confidence In plain words The deliberately-dumb starting point. A tiny neural network looks at a single 128 m square of aerial photo and, in one shot, blurts out two numbers: how far across and how far up the map that view belongs. With so little capacity and training it effectively learns to always point near the middle of the map, which is why its guesses are ~3.2 km off. Its only job is to be the reference every later experiment must beat. Hypothesis A deliberately naive ~300k-param CNN (strided conv stack, direct coordinate regression, loose confidence head) trained 8 epochs on 4,800 crops/area cannot memorize a ~7 km area at 1 m detail — its error should sit near the center-guess bound (~3 km), establishing the reference point the loop must beat. Method Frozen pipeline: 1 m/px open DOP orthophotos (source registry), 6-bucket relighting with ambient-compensated auto-exposure and golden/blue-hour grading, leakage-guarded 360 m block split (~45k train positions/area). One model per development area (berlin, prignitz, munich, frankfurt), ONNX export, frozen §6 scoring. Expected outcome Primary worst-case median error ~2.5-3.5 km (not comparable to prior eras); darker dusk/evening buckets slightly harder than before. Result primary worst-case median error = 3216.74 m; coverage 1.0 in all 24 area x lighting cells; all deployment gates passed (ONNX 385 KB, latency proxy ~0.6 ms). This is the naive TinyLocNet reference point — roughly the center-guess bound for ~7 km areas, i.e. a model that has learned almost nothing yet. Conclusion Starting baseline for the autoresearch loop. Every subsequent experiment is measured against this on the same frozen eval data (no further eval-set changes are planned). Bootstrap-phase history (7 experiments incl. two deliberate eval-set revisions) is archived in archive/bootstrap-experiments.sqlite. Scoreboard — median error (m) per area × lighting
the worst cell (underlined) is the experiment's score;
red = failed cell, ink = at target
Fits the aircraft? all deployment gates passed
largest per-area model 385 KB (limit 4,096 KB) · single-frame inference 0.6 ms on one CPU thread (proxy limit 250 ms)
crops are sampled fresh each run from the frozen train split (~45,000 distinct positions per area, times random rotation), never from eval blocks; how many to use is the experiment's own choice
ts 2026-07-20T19:50:47 · commit b47d44a8fb64 ·
parent — · artifacts runs/bootstrap_baseline_v4 ·
agent model claude-fable-5 (interactive bootstrap session) · took —
no headless prompt — designed interactively during the bootstrap session
Where the model was tested — and how far off it was Each map is one full test area. Every dot is one held-out test location (all six lighting conditions overlaid): the model was shown a 128 m crop centered there and asked for its position. Dot color = the distance between its answer and the truth — green ≤ 20 m (at goal), amber ≤ 50 m, red beyond. A working model turns these maps green; spatial clusters of red reveal which parts of an area confuse it. What the six lighting conditions look like through the simulated low-light sensor (example patches — illustration, not the training set)Every frame in this project is rendered as a starlight-class low-light sensor (Sony STARVIS2 / IMX585 class — the airframe's chosen camera) would see it, not as a normal camera would. That is why the night renders look uncannily close to daylight: at high gain such a sensor recovers scene structure from moonlight, skyglow and artificial lighting, paying for it in noise, lifted shadows and washed-out color — exactly what these renders simulate. Below, one example 256 m patch per area under the six lighting conditions the model must handle. These illustrate the dataset, not this experiment's performance — the actual training set is thousands of distinct crops per area (see “training data” above), and these renderings only change when the relighting method changes. berlin frankfurt munich prignitz | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
What do the columns and marks mean?
- Worst-case error
- The single number the loop optimizes. Every experiment trains one model per area; each model is then tested on held-out map crops it never saw during training, under each of the 6 simulated lighting conditions (morning → night). That gives a median position error for every area × lighting cell — and the score is the worst of those cells, not the average. An average would let a good Berlin-at-noon result hide a hopeless rural-night one; the worst cell can't hide anything. Mission target: ≤ 20 m.
- gated fail (×)
- The §6 score also enforces the aircraft's hard limits: the exported model must fit the ESP32-P4 flight computer (≤ 4 MiB) and answer within the latency budget (≤ 250 ms host proxy), and it may not dodge hard cases by refusing to answer (a cell where it abstains on > 80% of frames counts as failed). Any violation scores the whole experiment as failed regardless of accuracy.
- rejected
- Never trained at all. After a losing streak, the loop demands a genuine pivot — every non-frozen part of the design must change, not just the one piece that's gone stalest. If the proposed design doesn't clear that bar (checked against the actual code diff, not just what the agent claims it changed), it's rejected before spending any GPU time on it.
- cov (coverage)
- Share of test frames the model was confident enough to answer at all. Abstaining honestly on bad frames is allowed — down to the 20% floor above.
- Kept / Discarded
- Karpathy-style loop discipline: branch from the best code, run one focused experiment, keep the change (git commit) only if the worst-case error improves — otherwise revert it. The step line in the chart is the running best.
- Eval-set reset (┆)
- During the bootstrap phase the frozen evaluation data itself was revised twice (10 m satellite → 1 m orthophotos; then a split rebalance). Scores on different eval sets are measurements on different rulers and must not be compared — the dashed vertical rule marks the break, and the running-best line restarts there instead of pretending continuity. From Phase 2 on, the eval set does not change.
- Holdout (○)
- Hamburg is the blind fifth area: structurally different (port, river, spread-out), never seen by the loop, scored only as a periodic read-only check. If its error diverges from the four development areas, the pipeline has learned their quirks rather than a general method. Its result never influences keep/revert.
- Pivot-directed (▽)
- After 4 consecutive experiments in a row fail to beat the running best, the harness injects a mandatory pivot preamble into the next design prompt: do not refine the champion's current mechanism again — propose from a design family absent from the recent history. Marked ones ran under that directive; the streak resets every time an experiment is kept.
- Cost
- Estimated RunPod spend for that one iteration: its measured wall time × the pod's $0.69/hr rate. The pod bills continuously by the clock, not per phase, so this covers the whole iteration — agent design, implementation, training, scoring, publishing — not GPU time alone. Shown from experiment 11 on, when the loop moved off a laptop onto the pod; earlier runs had no pod cost but real per-call LLM API billing instead.
- Category
- Which lever the experiment pulls: architecture, loss, augmentation, relighting, training procedure, or quantization.
- Init
- Weight initialization: trained from scratch, or started from a permissively-licensed pretrained backbone.
- Model / Latency
- Largest per-area exported model, and single-frame inference time on one CPU thread (a documented proxy for the flight computer, not a measurement of it).
- Training set
- Each area offers ~45,000 distinct training positions (1 m/px, 128 m crops, times random rotation); how many crops an experiment actually samples per lighting condition is its own choice and is shown in the detail view. Training crops never overlap the eval blocks — enforced by the frozen split, not by convention.
- Agent prompt
- Every loop experiment records the exact prompt its headless agent received — expandable in the detail view, so any experiment can be re-run or audited later.
- In plain words
- Each experiment's own jargon-free explanation of what it tried, pre-registered alongside the technical design — read this first if the title looks like alphabet soup.
- One real test
- The figure at the top of each detail view is not an illustration: the same held-out Berlin night crop is fed to that experiment's actual exported model, and the red glow on the map is the probability field recovered from the deployed ONNX artifact itself — with the true location (○), the model's answer (×), and the real miss distance. Because every experiment sees the identical crop, any difference between two figures is the mechanism change, not the example. The pipeline sentence beneath names the stages; red marks what that experiment changed (a “+” stage is training-only and never flies).