The Transformer Family

Lilian Weng published a refactoring update to her post on the Transformer family on 2023-01-27, after almost three years, to incorporate a number of new Transformer models introduced since 2020. She directed readers to an enhanced version titled The Transformer Family Version 2.0 for that topic.

The original post covered notations; attention and self-attention; multi-head self-attention; the Transformer; Adaptive Computation Time; and techniques for improved attention span, including Transformer-XL's longer attention span, adaptive attention span, and localized attention span in the Image Transformer. It also covered approaches for lowering time and memory cost, including sparse attention matrix factorization in Sparse Transformers, locality-sensitive hashing in Reformer, recurrence in the Universal Transformer, and stabilization for reinforcement learning in GTrXL.

The post described the vanilla Transformer as an encoder-decoder architecture, citing Vaswani et al., 2017, noting later simplified versions such as encoder-only BERT and decoder-only GPT. It detailed the encoder's stack of six identity modules, each with a multi-head self-attention layer and a point-wise feed-forward network, and the decoder's two multi-head attention submodules. It described sinusoidal and learned positional encoding.

The post covered Adaptive Computation Time, citing Graves, 2016, as a mechanism for dynamically deciding computational steps in a recurrent neural network. It described Transformer-XL, citing Dai et al., 2019, as addressing context segmentation through hidden state reuse between segments and relative positional encoding. It described adaptive attention span, citing Sukhbaatar et al., 2019, and localized attention in the Image Transformer, citing Parmer et al., 2018.