This section summarizes the mechanisms that drive the development of large language models, which ultimately pave the way for applications like ChatGPT. It is based on the literature and lecture below.
Transformer
The Transformer architecture, such as that used in OpenAI’s Generative Pre-trained Transformer (GPT) and Google’s Bidirectional Encoder with Transformer (BERT), changed the landscape of natural language processing.
The seminal paper Attention Is All You Need announces the Transformer.
Transformers build on the self-attention mechanism. Attention is a mechanism for helping compute the embedding for a token by selectively attending to and integrating information from surrounding tokens. In this sense, we say that a word "attends to" certain neighboring words more than others.
"Attention" in machine learning is inspired by the biological concept of attention. Humans don’t process all the environmental information at once, but concentrate on relevant data for the current task. This selective focus applies to various senses like vision, hearing, smell, etc. Likewise, "attention" is employed in different areas of machine learning, including natural language processing and computer vision.
Transformers have several advantages over prior architectures, such as recurrent neural networks. First, they allow the models to learn the contextual connections of words by looking at other words in the input sequence to get a better understanding of a certain word and keeping the context of a word far across the input data.
Besides, transformers allow the model to attend to different parts of an input sequence in parallel. They therefore enable more efficient use of hardware advancements such as GPUs. Such computational efficiency often leads to improved accuracy because of the ability to build larger models with more robust training.
The third advantage of transformers is their general-purpose nature. They are not strongly tied to a particular task or domain and can be fine-tuned for a large number of downstream natural language processing tasks and applications. These include sentiment analysis, text classification, translation, summarization, natural language inference, semantic textual similarity, word sense disambiguation, and question answering.
The major components of a transformer are a set of encoders and decoders. Encoders encode input into a representation, which is then decoded by the decoders. The main difference between the encoder and decoder is that the former is bidirectional, whereas the latter is unidirectional. GPT models use only the decoder portion of the architecture and thereby inheriting its unidirectional nature; BERT uses the encoder with its bidirectional characteristics.
Mechanisms before transformers include Recurrent Neural Networks (RNN) and Long-Short Term Memory (LSTM). RNN processes data sequentially and is not very efficient in handling long sequences. Neither RNN nor LSTM can be trained in parallel.
Self-supervision
Self-supervised learning produces prediction problems from unlabeled data without the need for human annotators. Self-supervised tasks are therefore more scalable. Additionally, they are designed to force the model to predict parts of the inputs, making them richer and potentially more useful than models trained on a more limited label space.
Large-scale pre-training
Pre-training establishes the basis of the abilities of LLMs. By pre-training on large-scale corpora, LLMs acquire essential language understanding and generation skills.
A pre-trained model can be fine-tuned to a particular domain, dataset, or task using a relatively small size of relevant data from the new domain or language. While the pre-training stage focuses on a large quantity of data, the fine-tuning stage prioritizes quality over quantity.
The ability of LLMs taking the knowledge learnt from one task and applying it to another task is enabled by transfer learning. Pre-training is the dominant approach to transfer learning in deep learning.
Early work of pre-training started with static word representations.
word2vec
GloVe
ELMo was the first model proposed to capture contextual word presentations. In contextual embedding, each word has a different vector that expresses different meanings depending on the surrounding words.
BERT sets the “pre-training and fine-tuning” learning paradigm.
Pre-training is learning knowledge about language and the world from vast amounts of data, and the model compresses this knowledge into its parameters. The result of this stage is a base model. Pre-training is a very computationally involved process.
Fine-tuning is the process of taking a fully pre-trained model and further adapting some or all of its parameters to new data. The result of this stage is an assistant model. Fine-tuning is less computationally expensive.
Finally, the task-agnostic generative pre-training model. GPT models differentiate from BERT in that GPTs are autoregressive decoders, which are designed to predict the next word given the previous words from left to right.
GPT-3
GPT
In contrast, BERT and its descendants (e.g., RoBERTa) are bidirectional encoders, which compute contextualized representations of the input tokens from both left and right directions. This distinction makes BERT more appropriate for interpretative tasks like sentiment analysis or natural language understanding, where understanding the full context of a sentence is essential. They are not used for generation.
Reinforcement Learning from Human Feedback
To align LLMs with human values, Reinforcement Learning from Human Feedback (RLHF) has been proposed to fine-tune LLMs with the collected human feedback data to better capture human preferences such as BLEU or ROUGE.
Also see the Methods section on OpenAI's blog.
The initial GPT models exhibited three abilities.
The GPT-3.5 models enhanced the code generation and understanding abilities. But it traded the in-context learning ability for alignment with humans through RLHF.
Despite the advances and influence of LLMs, the underlying principles of LLMs remain insufficiently understood. One fundamental question is why LLMs exhibit certain emergent abilities, such as in-context learning.
The emergent ability of LLMs are formally defined as “An ability is emergent if it is not present in smaller models but is present in larger models”.
Emergence means that the behavior of a system is implicitly induced rather than explicitly constructed.It is both the source of scientific excitement and anxiety about unanticipated consequences.
Three typical emergent abilities for LLMs are in-context learning, instruction following, and step-by-step reasoning.
In-context learning refers to the ability that the language model can be adapted to a downstream task simply by providing it with a prompt (a natural language description of the task). It is an emergent property that was neither specifically trained for nor anticipated to arise.
A growing reading list.