All labs · 👁️ Vision AI lab
How vision AI works: transformers, YOLO, ReID & tracking
Look inside the models behind self-driving cars and smart cameras. Watch attention light up inside a transformer, step through how YOLO finds objects in one pass, compare YOLO versions from v1 to YOLO26, see how a ReID model recognises the same person on another camera – and how all of it joins up into a tracker. Then code each piece yourself in Python.
A transformer reads all the words at once and lets every word look at the others to decide what matters – that's attention. GPT, ChatGPT, and vision models like ViT, TransReID and YOLOv12's attention blocks are all built from it. Follow one sentence through the machine:
- ① Embeddingwords → vectors
- ② Q · K · Vquery, key, value
- ③ Attentionwho looks at whom
- ④ MLPthink about it
- ⑤ Next wordprobabilities
① Tokens become vectors (hover a word)
Each token gets a token embedding (what the word means) + a position embedding (where it is). Colours show the numbers: blue = positive, red = negative. Here 8 numbers per word; GPT-2 uses 768.
②③ Attention: who looks at whom
Row = the word that is asking (its query). Column = the word being looked at (its key). Darker = more attention. Each row adds up to 100%.
④ MLP: each word thinks on its own
After attention mixes information between words, the MLP works on each word alone: it expands the vector 4× (GPT-2: 768 → 3,072), applies GELU, and shrinks it back. A residual connection adds the input back, and layer norm keeps numbers stable. One attention + MLP = one block; GPT-2 small stacks 12.
⑤ What comes next?
The last word's vector is turned into a score (logit) for every word in the vocabulary (GPT-2: 50,257), then softmax makes them probabilities. Low temperature = safe, high = adventurous. In this tiny demo the scores come from a word-pair table so the guesses make sense; a real model learns them.
The real thing: GPT-2 small
- Vector size
- 768
- Attention heads
- 12 per block (64 numbers each)
- Blocks
- 12
- MLP
- 768 → 3,072 → 768, GELU
- Vocabulary
- 50,257 tokens
Want the full-size version? The Transformer Explainer runs real GPT-2 in your browser. And in vision, a ViT does the same with image patches instead of words – see TransReID in tab 4.
YOLO = "You Only Look Once": one pass through one network turns the whole image into boxes and labels. Step through what happens inside:
📐 IoU playground
Drag the boxes (or their bottom-right corner). IoU = overlap ÷ union. NMS deletes a box when its IoU with a better box is above the threshold; training counts a prediction as correct when IoU ≥ 0.5.
Every modern detector has three parts: a backbone that turns pixels into features, a neck that mixes features from different sizes, and a head that turns features into boxes. Pick a version to see what changed – each one fixed a weakness of the one before.
The neck: FPN vs PAN
Nano models: accuracy vs speed
COCO AP (higher = better) and latency on a T4 GPU with TensorRT FP16 (lower = faster), as reported in each paper. YOLOv8-N's time includes NMS.
Re-identification (ReID): is the person on camera 2 the same one we saw on camera 1? A ReID model turns each person crop into a list of numbers (an embedding). Same person → vectors point the same way (high cosine similarity); different people → far apart.
🔍 Find this person (click anyone to make them the query)
🗺️ The embedding space
Colour = person, shape = camera. Before training, pictures from the same camera cluster (lighting, background). Training with an ID loss (which person is it?) and a triplet loss (pull the same person closer than anyone else) moves them until each person is one cluster. Dashed: one triplet – anchor, positive (pull), negative (push).
OSNet: look at every scale at once
Some people differ in a tiny detail (a logo, shoes), others in the big picture (whole outfit). OSNet's block has 4 streams with growing receptive fields (3×3 → 9×9, made of stacked "Lite 3×3" convs) and a unified aggregation gate that decides, per picture, how much of each stream to use. Only 2.2M parameters, a 512-number embedding, Market-1501 rank-1 94.8%.
TransReID: a transformer looks at patches
Tracking = detection on every frame + matching each new box to the right person from the last frame, so everyone keeps the same ID. Watch what happens when two people cross and someone walks behind the pillar.
This frame's matching
Each cell is a cost (lower = better match) between a track from the last frame and a new detection. The Hungarian algorithm picks the combination with the lowest total. Unmatched detections start new tracks; unmatched tracks wait a while (behind the pillar) before being dropped.
What each ingredient adds
- IoU only: match boxes that overlap. Breaks when people cross (boxes overlap both) or vanish for a few frames.
- + Motion (Kalman filter): predict where each person moves next (dashed box) and match against the prediction.
- + ReID: compare appearance embeddings too, so a person who reappears or crosses someone keeps their ID. FairMOT: cost = 0.98 × appearance + 0.02 × motion.
FairMOT's ablation on MOT17: IoU only 648 ID switches → ReID only 435 → IoU + ReID 342 → IoU + ReID + Kalman 299.
Two ways to build the whole system
SDE: separate detection & embedding (e.g. YOLO + OSNet, DeepSORT)
Simple and accurate, but the ReID network runs once per person – a crowded street gets slow.
JDE / FairMOT: one network does both
JDE (YOLOv3-style Darknet-53 + FPN) adds an embedding to every anchor: 22 FPS, MOT16 MOTA 64.4. FairMOT found anchors were unfair to ReID, so it predicts object centres (heatmap + size + offset) on a high-resolution stride-4 map with a 128-number ReID embedding at each centre: MOT17 MOTA 73.7 at ~26 FPS.