Transformer by hand ✍️ ~ 6 steps walkthrough below
Studying the Transformer architecture is like opening up the engine hood of your car. So many unknown parts: embeddings, positional encoding, attention weighting, self-attention, cross-attention, multi-head attention, layer norm, skip connections, softmax, linear, Nx, shifted right, query, key, value, masking.
But which of those actually make the car run?
I think at the core, there are two most essential components:
Attention weighting and the feed-forward network.
Everything else is an enhancement to make it run faster and longer, which is how we got from a car to a truck, and to the word "large" in large language model.
So I drew and calculated those two parts entirely by hand.
Goal: push five features through one transformer block, filling in every cell yourself.
1. Given
Five positions of input features, arriving from the previous block.
2. Attention matrix
Let us feed all five features to a query-key module (QK) and read back an attention weight matrix, A. The details of that module are a post of their own.
3. Attention weighting
We multiply the input features by A to get the attention weighted features, Z. Still five positions. The effect is to combine features across positions, horizontally: X1 becomes X1 + X2, X2 becomes X2 + X3, and so on.
4. First layer
Let us feed all five weighted features into the first layer of the FFN. Multiply by the weights and biases. This time the combining happens across feature dimensions, vertically, and each feature grows from 3 numbers to 4.
Note that every position goes through the same weight matrix. That is what "position-wise" means.
5. ReLU
We cross out the negatives. They become zeros.
6. Second layer
Let us bring it back down: 4 dimensions to 3. The output feeds the next block, which has a completely separate set of parameters, and the whole thing runs again.
You have just calculated a transformer block by hand. ✍️
The takeaway: the two parts are doing two different jobs, and neither one alone is enough. Attention mixes across positions, so a feature can see its neighbours. The FFN mixes across feature dimensions, so each position can think about itself. Horizontal, then vertical. Then that pattern repeats N times, each block with its own separate set of weights. That is the Nx from the list up top, and that is what makes the transformer run.
💾 Save this post!
#AIbyHand #Transformers #DeepLearning