Bookmarks

Kimi's CEO🇨🇳 says every AI lab has the wrong obsession

Zhilin Yang, on why K3 beat the frontier: "Every lab, like Claude, thinks the model matters most. That's wrong. It's how you organize the people building it that wins."

Then Moonshot proved the philosophy. K3 sold out every plan on purpose, cutting their own revenue instead of throttling existing users. When Anthropic hit that wall in April, they cut users' usage 50% at peak.

His one big idea: long context is the AI era's RAM. The 128K-to-gigabytes jump, compressed into 2 years instead of 40.

The real moat: "your biggest advantage, perhaps your only advantage, is your organization."

Model, or the team behind it: which actually wins?

Transformer by hand ✍️ ~ 6 steps walkthrough below

Open the hood of a transformer and the parts list is overwhelming: embeddings, positional encoding, attention weighting, self-attention, cross-attention, multi-head attention, layer norm, skip connections, softmax, linear, Nx, shifted right, query, key, value, masking.

Which of those actually make the car run?

Two of them. Attention weighting and the feed-forward network. Everything else is an enhancement to make it run faster and longer, which is how we got from a car to a truck, and to the word "large" in large language model.

So I drew and calculated those two parts entirely by hand.

Goal: push five features through one transformer block, filling in every cell yourself.

1. Given

Five positions of input features, arriving from the previous block.

2. Attention matrix

Let us feed all five features to a query-key module (QK) and read back an attention weight matrix, A. The details of that module are a post of their own.

3. Attention weighting

We multiply the input features by A to get the attention weighted features, Z. Still five positions. The effect is to combine features across positions, horizontally: X1 becomes X1 + X2, X2 becomes X2 + X3, and so on.

4. First layer

Let us feed all five weighted features into the first layer of the FFN. Multiply by the weights and biases. This time the combining happens across feature dimensions, vertically, and each feature grows from 3 numbers to 4.

Note that every position goes through the same weight matrix. That is what "position-wise" means.

5. ReLU

We cross out the negatives. They become zeros.

6. Second layer

Let us bring it back down: 4 dimensions to 3. The output feeds the next block, which has a completely separate set of parameters, and the whole thing runs again.

You have just calculated a transformer block by hand. ✍️

The takeaway: the two parts are doing two different jobs, and neither one alone is enough. Attention mixes across positions, so a feature can see its neighbours. The FFN mixes across feature dimensions, so each position can think about itself. Horizontal, then vertical. Then that pattern repeats N times, each block with its own separate set of weights. That is the Nx from the list up top, and that is what makes the transformer run.

💾 Save this post!

#AIbyHand #Transformers #DeepLearning

Ilya Sutskever explains the 1% human training loop, and why fully autonomous AI is the wrong target

"The thing you really want is for the human teachers that tell, that teach the AI, for them to collaborate with an AI."

"You might want to think of it as being in a world where the human teachers do 1% of the world and the work and the AI do 99% of the work."

"You don't want it to be 100% AI"

"But you do want it to be a human-machine collaboration which teaches the next machine."

Most teams are trying to remove the human from the loop. Ilya's design keeps a tiny human control surface and uses it to teach the next generation.

The leverage is not zero humans. It is 1% human judgment steering 99% machine work.

• Ilya Sutskever (@ilyasut), AI researcher, on @dwarkesh_sp

Graph Convolutional Network by hand ✍️ ~ 12 steps walkthrough below

Graph Convolutional Networks (GCNs), introduced by Thomas Kipf and Max Welling in 2017, are the tool for data shaped like a graph: social networks, recommendations, biological networks, drug discovery, molecular chemistry.

I drew and calculated a simple GCN entirely by hand.

Goal: run a two-layer GCN, then a small classifier, on a five-node graph, filling in every cell yourself.

1. Given

A graph of five nodes, A to E, with edges between some of them.

2. Adjacency matrix (neighbors)

Put a 1 wherever two nodes share an edge, in both directions.

3. Adjacency matrix (self)

Add 1s down the diagonal, one self-loop per node. That is just adding the identity matrix.

4. Messages

Multiply each node's embedding by the weights and biases, then ReLU. Negatives become 0.

5. Pooling

Multiply the messages by the adjacency matrix. Each node gathers the messages of its neighbours and itself.

6. Visualize

Node A pools [3,0,1] + [1,0,0] = [4,0,1].

7. Second GCN layer

Messages again: weights, biases, ReLU.

8. Pooling again

Pool over each node and its neighbours, once more.

9. Visualize

Node C pools [1,2,4] + [1,3,5] + [0,0,1] = [2,5,10].

10. Fully connected layer

Weights, biases, ReLU. This time there are no neighbours to pool, just the node itself.

11. Linear layer

One more: weights and biases.

12. Sigmoid

Squash each score to a probability (≥ 3 → 1, 0 → 0.5, ≤ -3 → 0). That is the classification for each node.

You have just classified every node in the graph by hand. ✍️

The outputs:
A: 0 (very unlikely)
B: 1 (very likely)
C: 1 (very likely)
D: 1 (very likely)
E: 0.5 (neutral)

The takeaway: a GCN layer is two parts. The top part pools each node with its neighbours through the adjacency matrix. The bottom part is an MLP that transforms each node on its own. A transformer layer has the same two parts, with an attention matrix where the adjacency matrix was. Both matrices do one job, mixing across positions: attention over tokens, adjacency over nodes.

In my class I call the GCN the transformer's little cousin: a bit more stubborn, because its attention is fixed by the graph rather than computed from Q, K, and V. Draw the two side by side and the resemblance is hard to miss.

💾 Save this post!

#AIbyHand #GraphNeuralNetworks #DeepLearning

The circular diagram presents the C major scale, its seven diatonic triads (I–vii°), and the corresponding relative modes in a single, integrated view. At its center lies C, from which the primary triads radiate outward:

CM (I), Dm (ii), Em (iii), FM (IV), GM (V), Am (vi), and Bdim (vii°),

each formed by stacking every other note of the C–D–E–F–G–A–B scale. Encircling these are the seven modes:

Ionian, Dorian, Phrygian, Lydian, Mixolydian, Aeolian, and Locrian

aligned with their respective parent roots and annotated with their characteristic scale degrees and interval patterns.

This unified layout enables musicians to quickly visualize harmonic relationships, making it a practical reference for composition, improvisation, modulation, and transposition across keys.