“F13.2: a classifier trained on 17 cipher families' own 97-letter ciphertexts (no known words), its power on held-out texts, and K4 named against 2,000 shuffles of itself”
F13.2 (issue #85), the first F13 run. 0168 measured K4 on 71 statistics one at a time against shuffles and found nothing outside them; F7 (0018, 0027) built ciphers by family and compared single statistics. Here the statistics are combined: 17 generators (one per family or cell of the map, keys drawn at random inside each family's space, plaintext from Carter's book with no known word, half of it Sanborn-style) make 4,000 ciphertexts each; every text is measured by 0168's 71 statistics, A13's 14 tests and 70 classic cipher-identification statistics (kappa at shifts 1-60, digraphic and even-pair I.C., delta I.C., repeats); a multinomial logistic classifier is trained on 3,000 per family and its power measured on 1,000 held-out per family, written from the other end of the book. Then K4's posterior for each family is ranked against 2,000 shuffles of K4, which keep its letter counts, so a family is named only for structure beyond the counts. The search never reads the 24 known letters.
Before attacking a cipher, codebreakers often ask what kind it is: some kinds leave fingerprints, such as letters repeating at a fixed spacing. Here we build a sorting machinea classifier A program that learns to tell kinds of things apart from examples. (a classifier): we make 4,000 fake K4sthe workshop An experiment that builds thousands of pretend K4s by each suspected method and measures them, to learn which methods produce text that looks like the real one. with each of 17 kinds of cipher, measure 155 things about each, and teach the machine to tell the kinds apart. We test it on fakes it has never seen, to know how often it is right, and then show it K4the carved letters The 97 letters cut into the copper at the CIA, shown at the top of every experiment page.. Because K4's letter counts alone could fool it, K4 counts as looking like a kind only if it looks more like that kind than 2,000 shuffles of its own letters do.
Words in the margin are defined on the glossary page.
1.1
The method at work
The carved letters go in at the top; the reading comes out at the bottom.
the idea
F13.2: train a classifier on ciphertexts made by each family, measure how often it names the right one, then ask which family K4 looks like beyond its letter counts.
what happened
The run stopped before any text was made: the worker function could not be sent to the pool, because the harness loads attack.py under another module name.
the rerun
0231: the module registered under its own name, as 0229 does; the method unchanged.
2
Experiment lineage
What this experiment was built on, from the lineage page: 4 experiments before it, back to the root of its line. Its number is in bold.
keptreverted, in its family's colour×error or timeoutreverted under today's rulesbuilt on a parent in the same familybuilt on a parent in another family
3
Best candidate against the known letters
This experiment produced no candidate: the search ruled everything out before a full text was built. Its product is the bounded negative under Result and bounds below.
4
Walkthrough, by hand
Redo the method on a few letters with pencil and paper.
Take 97 letters of English and encipher them with one kind of cipher and a random key.
Measure the result: how evenly letters are spread, how often a letter recurs 1, 2, 3 ... 60 places later, repeated pairs and triples, doublets.
Do this thousands of times for each of 17 kinds, and fit weights that turn the measurements into a guess of the kind.
Check the guesses on fresh fakes: the share right is the machine's power.
Measure K4 and 2,000 shuffles of K4's letters; a kind is named only if K4 scores higher for it than nearly every shuffle.
5
Worked example, generated
The same steps run by the code on the real letters.
6
Hypothesis, for a cryptanalyst
Family, key space, the constraint under test, and the clues it reads.
Cipher-type identification (the ACA's and NCID's practice; Sinkov's statistics) asks which generator a ciphertext is typical of. F13.1 (reports/f13-solvability-map.md) lists the families English alone could decide; F13.2 asks whether K4's own statistics point at any of them, without the known letters. Classes: monoalphabetic, columnar transposition (width 2-30), substitution then transposition (F9), periodic key on any of six alphabet pairs (standard, KRYPTOS, mixed; Quagmire I-IV) in three forms at lengths 2-12, 13-26 and 27-60 (F1, F1.1, F1.2), progressive key (F1.9), ciphertext and plaintext autokey with primers 1-30 (F1.7), running key from English (F1.3), Gromark base 10 (F5), Hill 2x2 and 3x3, plain or affine (F8), trifid 27-cell cube with no 27th symbol in the output (F4), Chaocipher (F10.1), a move then the table and the table then a move (F6), and uniform random letters (the one-time pad, the null). Features: 0168's 71 (segment chi-squares, I.C. at 1-48 alphabets, repeats at widths 7, 21, 31: the width-7 doublet column and width-21 repeats among them, 04-structure.md), A13's 14 (tools/a13_structure.py), and 70 more. Tests C4(a)-(b) (a mask leaves a type signature?), C1(a) (move then table), C11a ('interval 7') and F7's fingerprints as features rather than single tests. Pre-registered: 34 cells (17 families x two feature sets: the item's named 85 and all 155); K4 names a family when its posterior for that family is above all but a share p < 0.05/34 = 0.00147 of 2,000 shuffles of K4, and that family's positive control (12 held-out plants per family, each against 1,000 of its own shuffles, the same rule) finds at least 6 of 12. A named family is a candidate, replicated with a retrained classifier and fresh shuffles before anything else.
7
Prediction
Written before the run; the result decides it.
Held-out accuracy well above chance (1/17) overall, near 100% for monoalphabetic, transposition, substitution then transposition and Hill; high for short periodic keys; but the flat families (long periodic keys, running key, autokeys, Gromark, Chaocipher, the move-and-table pairs, random letters) confused with each other near chance among themselves. K4 is put among the flat families; against its shuffles no family is named (p above 0.00147 in all 34 cells). Plants of the distinctive families are found against their own shuffles; plants of the flat families are not. Zero candidates.
8
Method
Plaintexts: 97-letter windows of Carter's Tomb of Tut-ankh-Amen (the plants' source), half clean, half Sanborn-style (full stops as X, three letters changed); training from the first 80% of the book, held-out texts and plants from the last 20%. No known word is written in.
17 generators, keys at random inside each family's space; 3,000 training and 1,000 held-out texts per family, in jobs of 100 with their own seeds.
155 features per text: 0168's all_stats (frozen), A13's battery, and kappa at shifts 1-60, digraphic and even-pair I.C., delta I.C. on both alphabets, doublets, longest repeat, distinct letters, top count, English letter log-likelihood.
Multinomial logistic regression (numpy, L2 1e-3, Adam 3,000 steps, standardized features), one on the named 85 features and one on all 155. Power: held-out confusion matrix, recall per family, AUC of each family against random letters.
K4 and 2,000 shuffles of K4 through both models; per family the share of shuffles at or above K4's posterior.
Positive control: 12 held-out plants per family, each against 1,000 of its own shuffles, the same rule.
9
Result and bounds
The harness's record: candidates, gates, loss, and the negative result this experiment leaves behind.
0candidates returned
0passed every gate
0/24known letters, best
–lowest loss
nonechampion before
The run stopped with an error: when serializing tuple item 2
How this experiment sits among the others is on the lineage page; its family's open hypotheses are on the families page.
Rendered from the record on 2026-11-05 17:14 UTC; updated after every experiment.