At the intersection of advanced mathematics and machine learning, the transformers formula serves as the architectural backbone for virtually every modern large language model. This intricate set of equations, primarily rooted in the self-attention mechanism, dictates how a model weighs the importance of different words in a sequence. Unlike older recurrent architectures, this formula allows for parallel processing and a nuanced understanding of context, enabling the stunning capabilities seen in chatbots and translation tools today.
The Core Equation: Multi-Head Attention
The central transformers formula is the Multi-Head Attention mechanism, which calculates a weighted sum of values based on the compatibility of queries and keys. The process begins with the creation of three distinct vectors for each word: the Query (Q), the Key (K), and the Value (V). These vectors are derived by multiplying the input embeddings by learned weight matrices, effectively projecting the data into different subspaces where semantic relationships can be more easily captured.
Dot-Product and Scaling
For each query vector, the formula computes a dot product with every key vector in the sequence. This operation produces a score that represents the relevance or compatibility between the two words. To prevent the dot products from growing too large in dimensionality—which would push the softmax function into regions with tiny gradients—the result is divided by the square root of the dimension of the key vectors. This scaling step is a critical detail that stabilizes the training of deep networks.
Calculate Query, Key, and Value vectors.
Compute similarity scores via dot product.
Scale the scores to maintain gradient stability.
Softmax Normalization
Following the scaling, the scores are passed through a softmax function. This mathematical operation converts the raw scores into a probability distribution, where the values range between 0 and 1 and sum to one. The resulting weights represent the confidence the model has in attending to each specific word when generating a representation for the current target word. Words with higher scores exert a stronger influence on the final output. Weighted Sum and Output Projection Once the attention weights are determined, they are applied to the value vectors (V). The formula performs a weighted sum, multiplying each value vector by its corresponding softmax score and then adding them together. The result is a new vector that contains information from the entire sequence, heavily influenced by the most relevant words. Finally, this aggregated vector is multiplied by a final weight matrix to produce the attention output, which is then passed to the next layer of the network.
Weighted Sum and Output Projection
Multi-Head Mechanism: Diversity of Representation
To capture different types of relationships—such as syntactic structure or factual association—the model employs multiple attention heads. Instead of performing a single attention function, the input is projected into multiple sets of Q, K, and V matrices. Each head learns to attend to information from different representation subspaces. The outputs of all heads are then concatenated and linearly transformed, providing the model with a richer and more diverse understanding of the context than a single head could achieve.
Positional Encoding: Injecting Order
Since the transformers formula lacks the inherent sequential processing of RNNs, it requires a special mechanism to incorporate the order of words. This is achieved through Positional Encoding. Since there is no recurrence, the position information is added to the input embeddings directly. These encodings use sine and cosine functions of different frequencies to allow the model to learn relative positions, ensuring that the sequence order is a fundamental part of the mathematical calculation.
