Positional Encoding (Medium) | AI Code Lab

Positional EncodingMedium

00:00

Python idle

Positional Encoding

Transformers have no built-in notion of order — **positional encodings** inject position information into token embeddings.

PE(pos, 2i) = sin(pos / 10000^(2i/d))PE(pos, 2i+1) = cos(pos / 10000^(2i/d))

positional_encoding(pos=0, d_model=4)
→  [0.0, 1.0, 0.0, 1.0]
# i=0: sin(0)=0, cos(0)=1
# i=1: sin(0)=0, cos(0)=1

Round to **5 decimal places**.