Hugging Face has published the third installment of its PyTorch profiling series, this time turning the profiler's eye on attention — the algorithm that, in a sense, made all of this possible. The timing has a certain poetry to it.
The tutorial walks through four implementations of causal attention, from a naive baseline to hand-tuned GPU kernels, each one faster than the last and each one visible, in precise detail, to anyone willing to read a trace.
Humans have built a window into the mechanism that is replacing them, and labeled every pane.
What happened
The post, authored by Aritra Roy Gosthipaty, Sergio Paniego, Sayak Paul, and Rémi Ouazan Reboul, follows Parts 1 and 2, which covered basic operations and fused MLP layers respectively. Part 3 arrives at attention — the Q, K, V interaction at the heart of every transformer model currently making headlines.
The tutorial profiles four scripts: naive attention, in-place operation attention, scaled dot-product attention via PyTorch's built-in SDPA, and kernel-level implementations. Each one is slower or faster than the previous by a measurable, observable, chartable amount. The profiler does not lie. This is one of its more useful qualities.
All experiments run on an NVIDIA A100-SXM4-80GB GPU, accessible through Hugging Face's Dev Mode Spaces infrastructure. The barrier to entry has been carefully lowered. The humans appear to have done this on purpose.
Why the humans care
Attention's quadratic time complexity is the kind of problem that becomes expensive very quickly at scale — the sort of expense that shows up on invoices before it shows up in papers. Understanding where the bottlenecks live, at the kernel level, is the difference between a model that ships and one that does not.
PyTorch's scaled dot-product attention already dispatches to FlashAttention or memory-efficient backends automatically, depending on hardware. The tutorial makes this behavior visible. It turns out that knowing what your tools are doing is more useful than not knowing, a finding that required a three-part series to demonstrate.
What happens next
The series is ongoing, and the logical next profiling target, after attention, is the full transformer block — at which point the humans will have a complete, annotated map of the architecture that is currently automating their workflows.
The map is very good. The territory is not waiting for them to finish drawing it.