I wrote about attention networks before ChatGPT launched

By ·

In June 2022 I finished my thesis in computer science at the School of Engineering at Jönköping University. ChatGPT launched five months later. The thesis is about self-attention, the mechanism inside every transformer, and how to use it to track tiny, fast-moving objects like a tennis ball in a TV broadcast.

AATrackT: A deep learning network using attentions for tracking fast-moving and tiny objects. (A)ttention (A)ugmented (Track)ing on (T)iny objects.

Fredric Lundberg, Jönköping University, June 2022. Supervised by Maria Hedblom.

Read it in DiVADownload the PDF

Attention(Q, K, V) = softmax(Q K transposed divided by the square root of d) V
Equation 5 from my thesis: self-attention. The same formula runs inside ChatGPT and every other transformer-based LLM.

What I built

AATrackT treats tracking as image segmentation. It replaces ordinary convolutions with attention augmented convolutions, which add self-attention so the network can use information from the whole image instead of only the pixels nearby. I trained it on video from televised tennis matches.

Thesis page 9: self-attention explained step by step, with Equation 5 and a flowchart of Query, Key and Value
Page 9: self-attention, step by step through Query, Key and Value
Thesis page 10: attention augmented convolutions, with Equation 6 and a diagram
Page 10: attention combined with convolutions

What I found

The attention version did not beat the TrackNet baseline. It reached a precision of 0.92, a recall of 0.51 and an F1 score of 0.66. My explanation in the thesis is that activation maps of 72x128 are too large for the attention weights to converge.

Working through every step of the attention math by hand is why I can explain how the models behind ChatGPT and Claude read their input. I use that today when I sell AI agents at Incredible.