“F13.3: bits per letter each English judge extracts from 97 letters: the quadgram table, Kneser-Ney n-grams of order 5-8 alone and mixed with it, a word model, and the blind judge on a ladder of English-like soups”

1. Count how often each run of letters is followed by each next letter in a large amount of English.
2. Read a fresh 97-letter window one letter at a time and write down the chance the counts gave the letter that came.
3. Average minus log2 of those chances: that is the cross-entropy; 4.70 (log2 26) minus it is the evidence per letter.
4. Do the same with longer runs and with whole words, on clean book English and on text with X stops and misspellings.
5. Make fake texts that copy English's short pieces, and real words in random order, and check whether each scorer still puts real English above all of them.

Cross-entropy, letter by letter, with the quadgram table (no history before the window):

  letter      T     H     E     H     O     T     A     I     R     E     S     C
  -log2 P  3.47  1.71  0.59  5.50  2.12  4.54  3.95  6.64  4.38  3.54  3.39  5.32

  mean over the 31 letters: 2.71 bits; evidence 1.99 bits per letter (log2 26 = 4.70)

Shannon (1951) puts English at 0.6-1.3 bits per letter given long context; n-gram models reach about 2.3-2.5 with large corpora at order 5-6; the harness's quadgram table gives 2.92 (0136). Each bit per letter is 97 bits of evidence over a ciphertext-only reading, which moves F13.1's cells: F1.1 at key length 26 (202 bits), Chaocipher (177), F1.2 at 13 (225), plaintext autokey primers past 36 (F1, F1.7) all sit between the quadgram and Shannon bounds. The measurement is 0136's (cross-entropy on held-out English windows scored with no history before the window, as a search's reading is scored), extended to Sanborn-style text (judge bench classes xstops, sub3, k123: C19, C17, A6) because his plaintext is not Carter. The judge (kryptos/judge.py; reports/judge-bench.md) is not a probability model, so its evidence is bounded from below by the ladder: a scorer that separates English from soups matching English to order k extracts more than an order-k model. Pre-registered: a model is 'stronger' when its cross-entropy on held-out clean Carter is lower than the quadgram table's by at least 0.10 bits per letter, judged on the out-of-domain arm (trained without Carter); the Carter-trained models are reported as the in-domain ceiling, the case of a message in Carter's register; a scorer 'passes a rung' when at least 90% of clean and of X-stop windows score above every negative of that rung (the bench's rule).

Written after the dry run, for the out-of-domain arm: Kneser-Ney on the manuals alone is no better than the quadgram table on Carter; mixed with it, it gains 0.1-0.3 bits per letter (evidence about 1.9-2.05 bits per letter, 185-200 bits in 97): stronger by the rule. The in-domain mix reaches about 2.3 bits per letter. The word model alone is weaker than the table on 97-letter windows (cut words at both ends). Sanborn-style text costs 0.1-0.2 bits per letter on every model. Ladder: the quadgram scorer fails the 4-gram soup rung (as on the bench), the mixed n-gram passes it but fails the 6-gram soup and the word salad; the judge passes the 4-gram and 6-gram rungs and fails the word salad. So the best measured model moves F1.1 at 26 and Chaocipher into reach, not F1.2 at 13. Zero candidates.

1. Corpus: Carter (first 70% train, 70-80% validation, last 20% test), and the Army/Friedman/NSA/Lewis texts, lines kept only when 70% of their words are in the dictionary (cipher groups dropped).
2. Models: the quadgram table as an interpolated Kneser-Ney 4-gram over its own counts (Q); interpolated Kneser-Ney orders 5-8 on the corpus (KN5-KN8); Q mixed with each, weight on the validation windows (MIXn); a word model (W).
3. Cross-entropy on 97-letter windows with no prior history: test clean, X stops (full stops as X), three letters changed, K1-K3 plaintexts (overlapping windows); bits per letter = log2 26 - H; median and least predictable tenth of windows.
4. Ladder: 30 clean and 30 X-stop test windows against 40 soups sampled from Q (order 4), 40 from KN6 (order 6) and 40 word salads; each scorer (Q, the best MIX, W, the judge) by the bench's rule.
5. The judge is called once, from the main process, on 180 texts (cap 3,000).

clean Carter: Q 1.753 bits/letter; out of domain MIX8m 1.945 (K1+K2 1.935); in domain MIX8 2.382; stronger by the rule: ['KN5', 'MIX5', 'KN6', 'MIX6', 'KN7', 'MIX7', 'KN8', 'MIX8', 'MIX6m', 'MIX8m', 'W']; judge ladder: soup4 (from Q) pass, soup6 (from KN6) fail, word salad fail

What was searched, as the harness recorded it.

item

F13.3

search reads known letters

no

models

Q

the quadgram table (kryptos/data, 4.2e9 counts) as interpolated Kneser-Ney order 4

KN5

interpolated Kneser-Ney order 5, local corpus

KN6

interpolated Kneser-Ney order 6, local corpus

KN7

interpolated Kneser-Ney order 7, local corpus

KN8

interpolated Kneser-Ney order 8, local corpus

MIX5

Q and KN5 mixed, weight on Q 0.3 (validation)

MIX6

Q and KN6 mixed, weight on Q 0.3 (validation)

MIX7

Q and KN7 mixed, weight on Q 0.3 (validation)

MIX8

Q and KN8 mixed, weight on Q 0.3 (validation)

KN6m

interpolated Kneser-Ney order 6, manuals only (no Carter)

KN8m

interpolated Kneser-Ney order 8, manuals only (no Carter)

MIX6m

Q and KN6m mixed, weight on Q 0.55

MIX8m

Q and KN8m mixed, weight on Q 0.55

W

word unigrams (corpus counts, dictionary words 0.5) or single letters, summed over segmentations

corpus letters train

1,429,659

manual lines kept

26,017

test windows

clean

529

xstops

534

k123

34

k12

17

sub3

529

validation windows

268

cross entropy

Q

clean

H mean

2.947

bits per letter

1.753

H median

2.89

H p90 least predictable

3.383

bits in 97

170

windows

529

xstops

H mean

3.049

bits per letter

1.651

H median

2.987

H p90 least predictable

3.531

bits in 97

160.2

windows

534

k123

H mean

2.962

bits per letter

1.739

H median

2.952

H p90 least predictable

3.163

bits in 97

168.6

windows

34

k12

H mean

2.965

bits per letter

1.735

H median

2.943

H p90 least predictable

3.172

bits in 97

168.3

windows

17

sub3

H mean

3.29

bits per letter

1.411

H median

3.243

H p90 least predictable

3.708

bits in 97

136.8

windows

529

KN5

clean

H mean

2.64

bits per letter

2.06

H median

2.604

H p90 least predictable

3.099

bits in 97

199.9

windows

529

xstops

H mean

2.753

bits per letter

1.948

H median

2.728

H p90 least predictable

3.241

bits in 97

188.9

windows

534

k123

H mean

2.813

bits per letter

1.888

H median

2.732

H p90 least predictable

3.51

bits in 97

183.1

windows

34

k12

H mean

2.903

bits per letter

1.798

H median

2.813

H p90 least predictable

3.397

bits in 97

174.4

windows

17

sub3

H mean

3.146

bits per letter

1.555

H median

3.125

H p90 least predictable

3.569

bits in 97

150.8

windows

529

MIX5

clean

H mean

2.546

bits per letter

2.155

H median

2.517

H p90 least predictable

2.897

bits in 97

209

windows

529

xstops

H mean

2.651

bits per letter

2.05

H median

2.622

H p90 least predictable

3.036

bits in 97

198.8

windows

534

k123

H mean

2.686

bits per letter

2.014

H median

2.596

H p90 least predictable

3.162

bits in 97

195.4

windows

34

k12

H mean

2.733

bits per letter

1.968

H median

2.655

H p90 least predictable

3.084

bits in 97

190.9

windows

17

sub3

H mean

2.967

bits per letter

1.733

H median

2.953

H p90 least predictable

3.297

bits in 97

168.1

windows

529

KN6

clean

H mean

2.501

bits per letter

2.2

H median

2.458

H p90 least predictable

2.972

bits in 97

213.4

windows

529

xstops

H mean

2.616

bits per letter

2.084

H median

2.576

H p90 least predictable

3.122

bits in 97

202.2

windows

534

k123

H mean

2.643

bits per letter

2.057

H median

2.733

H p90 least predictable

3.469

bits in 97

199.6

windows

34

k12

H mean

2.977

bits per letter

1.724

H median

2.916

H p90 least predictable

3.429

bits in 97

167.2

windows

17

sub3

H mean

3.039

bits per letter

1.662

H median

3.011

H p90 least predictable

3.459

bits in 97

161.2

windows

529

MIX6

clean

H mean

2.398

bits per letter

2.302

H median

2.377

H p90 least predictable

2.755

bits in 97

223.3

windows

529

xstops

H mean

2.507

bits per letter

2.193

H median

2.475

H p90 least predictable

2.924

bits in 97

212.8

windows

534

k123

H mean

2.51

bits per letter

2.19

H median

2.541

H p90 least predictable

3.084

bits in 97

212.4

windows

34

k12

H mean

2.719

bits per letter

1.982

H median

2.68

H p90 least predictable

3.042

bits in 97

192.2

windows

17

sub3

H mean

2.847

bits per letter

1.853

H median

2.826

H p90 least predictable

3.183

bits in 97

179.7

windows

529

KN7

clean

H mean

2.446

bits per letter

2.254

H median

2.439

H p90 least predictable

2.928

bits in 97

218.7

windows

529

xstops

H mean

2.563

bits per letter

2.137

H median

2.528

H p90 least predictable

3.046

bits in 97

207.3

windows

534

k123

H mean

2.512

bits per letter

2.188

H median

2.763

H p90 least predictable

3.457

bits in 97

212.3

windows

34

k12

H mean

3.001

bits per letter

1.7

H median

2.964

H p90 least predictable

3.42

bits in 97

164.9

windows

17

sub3

H mean

2.994

bits per letter

1.707

H median

2.993

H p90 least predictable

3.404

bits in 97

165.6

windows

529

MIX7

clean

H mean

2.342

bits per letter

2.358

H median

2.329

H p90 least predictable

2.693

bits in 97

228.8

windows

529

xstops

H mean

2.452

bits per letter

2.248

H median

2.42

H p90 least predictable

2.856

bits in 97

218.1

windows

534

k123

H mean

2.401

bits per letter

2.299

H median

2.57

H p90 least predictable

3.074

bits in 97

223

windows

34

k12

H mean

2.72

bits per letter

1.98

H median

2.681

H p90 least predictable

3.049

bits in 97

192.1

windows

17

sub3

H mean

2.803

bits per letter

1.897

H median

2.794

H p90 least predictable

3.124

bits in 97

184

windows

529

KN8

clean

H mean

2.421

bits per letter

2.28

H median

2.411

H p90 least predictable

2.902

bits in 97

221.1

windows

529

xstops

H mean

2.536

bits per letter

2.165

H median

2.511

H p90 least predictable

3.026

bits in 97

210

windows

534

k123

H mean

2.43

bits per letter

2.271

H median

2.741

H p90 least predictable

3.382

bits in 97

220.2

windows

34

k12

H mean

2.978

bits per letter

1.722

H median

2.939

H p90 least predictable

3.363

bits in 97

167.1

windows

17

sub3

H mean

2.97

bits per letter

1.731

H median

2.972

H p90 least predictable

3.385

bits in 97

167.9

windows

529

MIX8

clean

H mean

2.318

bits per letter

2.382

H median

2.311

H p90 least predictable

2.682

bits in 97

231.1

windows

529

xstops

H mean

2.427

bits per letter

2.273

H median

2.404

H p90 least predictable

2.836

bits in 97

220.5

windows

534

k123

H mean

2.333

bits per letter

2.367

H median

2.55

H p90 least predictable

3.038

bits in 97

229.6

windows

34

k12

H mean

2.702

bits per letter

1.998

H median

2.644

H p90 least predictable

3.022

bits in 97

193.8

windows

17

sub3

H mean

2.784

bits per letter

1.916

H median

2.776

H p90 least predictable

3.113

bits in 97

185.8

windows

529

KN6m

clean

H mean

3.218

bits per letter

1.482

H median

3.161

H p90 least predictable

3.995

bits in 97

143.8

windows

529

xstops

H mean

3.307

bits per letter

1.393

H median

3.247

H p90 least predictable

4.079

bits in 97

135.1

windows

534

k123

H mean

3.283

bits per letter

1.418

H median

3.263

H p90 least predictable

3.734

bits in 97

137.5

windows

34

k12

H mean

3.192

bits per letter

1.508

H median

3.227

H p90 least predictable

3.704

bits in 97

146.3

windows

17

sub3

H mean

3.627

bits per letter

1.073

H median

3.576

H p90 least predictable

4.303

bits in 97

104.1

windows

529

MIX6m

clean

H mean

2.768

bits per letter

1.932

H median

2.694

H p90 least predictable

3.296

bits in 97

187.4

windows

529

xstops

H mean

2.862

bits per letter

1.839

H median

2.787

H p90 least predictable

3.404

bits in 97

178.4

windows

534

k123

H mean

2.805

bits per letter

1.895

H median

2.783

H p90 least predictable

3.101

bits in 97

183.8

windows

34

k12

H mean

2.765

bits per letter

1.936

H median

2.763

H p90 least predictable

3.038

bits in 97

187.8

windows

17

sub3

H mean

3.125

bits per letter

1.576

H median

3.074

H p90 least predictable

3.608

bits in 97

152.8

windows

529

KN8m

clean

H mean

3.174

bits per letter

1.526

H median

3.108

H p90 least predictable

3.886

bits in 97

148

windows

529

xstops

H mean

3.261

bits per letter

1.439

H median

3.213

H p90 least predictable

3.938

bits in 97

139.6

windows

534

k123

H mean

3.283

bits per letter

1.417

H median

3.3

H p90 least predictable

3.657

bits in 97

137.5

windows

34

k12

H mean

3.206

bits per letter

1.495

H median

3.236

H p90 least predictable

3.526

bits in 97

145

windows

17

sub3

H mean

3.569

bits per letter

1.131

H median

3.524

H p90 least predictable

4.179

bits in 97

109.7

windows

529

MIX8m

clean

H mean

2.755

bits per letter

1.945

H median

2.682

H p90 least predictable

3.264

bits in 97

188.7

windows

529

xstops

H mean

2.849

bits per letter

1.852

H median

2.78

H p90 least predictable

3.405

bits in 97

179.6

windows

534

k123

H mean

2.809

bits per letter

1.891

H median

2.798

H p90 least predictable

3.104

bits in 97

183.4

windows

34

k12

H mean

2.766

bits per letter

1.935

H median

2.766

H p90 least predictable

3.008

bits in 97

187.7

windows

17

sub3

H mean

3.113

bits per letter

1.587

H median

3.053

H p90 least predictable

3.594

bits in 97

154

windows

529

W

clean

H mean

2.62

bits per letter

2.081

H median

2.625

H p90 least predictable

2.942

bits in 97

201.8

windows

529

xstops

H mean

2.702

bits per letter

1.998

H median

2.685

H p90 least predictable

3.054

bits in 97

193.8

windows

534

k123

H mean

2.768

bits per letter

1.932

H median

2.749

H p90 least predictable

3.042

bits in 97

187.4

windows

34

k12

H mean

2.799

bits per letter

1.902

H median

2.813

H p90 least predictable

2.937

bits in 97

184.5

windows

17

sub3

H mean

3.136

bits per letter

1.564

H median

3.131

H p90 least predictable

3.474

bits in 97

151.8

windows

529

best model clean

MIX8

gain over Q bits per letter clean

Q

0

KN5

0.307

MIX5

0.401

KN6

0.446

MIX6

0.549

KN7

0.501

MIX7

0.605

KN8

0.526

MIX8

0.629

KN6m

-0.271

MIX6m

0.179

KN8m

-0.227

MIX8m

0.192

W

0.327

rule stronger

cross-entropy on clean held-out Carter at least 0.10 bits per letter under Q

stronger models

KN5, MIX5, KN6, MIX6, KN7, MIX7, KN8, MIX8, MIX6m, MIX8m, W

best out of domain

MIX8m

out of domain stronger

yes

ladder

Q

soup4 (from Q)

threshold

-246.83

found

clean

1

xstops

1

passes

no

soup6 (from KN6)

threshold

-231.68

found

clean

0

xstops

1

passes

no

word salad

threshold

-255.74

found

clean

8

xstops

2

passes

no

MIX8

soup4 (from Q)

threshold

-286.73

found

clean

29

xstops

30

passes

yes

soup6 (from KN6)

threshold

-165.42

found

clean

0

xstops

0

passes

no

word salad

threshold

-217.78

found

clean

15

xstops

10

passes

no

W

soup4 (from Q)

threshold

-343.88

found

clean

29

xstops

30

passes

yes

soup6 (from KN6)

threshold

-205.51

found

clean

0

xstops

0

passes

no

word salad

threshold

-171.52

found

clean

0

xstops

0

passes

no

judge

soup4 (from Q)

threshold

72

found

clean

29

xstops

29

passes

yes

soup6 (from KN6)

threshold

90

found

clean

18

xstops

17

passes

no

word salad

threshold

88

found

clean

25

xstops

24

passes

no

ladder rule

a rung passes when at least 90% of clean and of X-stop windows (30 each) score above every one of its 40 negatives

judge calls

180

judge none

0

f13 1 cells

F1.1 one free alphabet + key L = 26

key bits

202.3

Q clean

no

best out of domain clean

no

best out of domain k12

no

best clean

yes

best xstops

yes

F1.2 two free alphabets + key L = 13

key bits

224.9

Q clean

no

best out of domain clean

no

best out of domain k12

no

best clean

yes

best xstops

no

F1.7 autokey on one free alphabet, lag 30

key bits

221.1

Q clean

no

best out of domain clean

no

best out of domain k12

no

best clean

yes

best xstops

no

F10 Chaocipher

key bits

176.8

Q clean

no

best out of domain clean

yes

best out of domain k12

yes

best clean

yes

best xstops

yes

F6.2 two free alphabets after a width-10 columnar, L = 10

key bits

232.6

Q clean

no

best out of domain clean

no

best out of domain k12

no

best clean

no

best xstops

no

F1 running key from unknown English

key bits

282.8

Q clean

no

best out of domain clean

no

best out of domain k12

no

best clean

no

best xstops

no

reference 0136 quadgram H carter book

2.916

exhaustive

no

compute seconds

152