“Large language models generate text by predicting likely word sequences from patterns learned during training rather than inherently retrieving verified facts from a database.”

We argue that to develop a holistic understanding of these systems, we must consider the problem that they were trained to solve: next-word prediction over Internet text. … Nonetheless, we should also remember a simpler fact: Language models are...language models! That is, they are statistical next-word prediction systems. This fact has some important consequences: For instance, as we have shown, language models have greater difficulty with infrequent tasks than frequent ones, even when comparing two tasks that seem equally complex to a human; and they have greater difficulty on examples with lowprobability answers than high-probability ones, even when the task is deterministic.

In contrast, LLMs are probabilistic models that excel at next word prediction based on the given context, rather than storing explicit information in defined locations.

Next-token prediction with the logarithmic loss (Shannon, 1951) is a cornerstone of autoregressive sequence modeling—particularly language model pre-training (Vaswani et al., 2017; Radford et al., 2019). It estimates a distribution over sequences (a1, . . . , aH) by jointly fitting a sequence of conditional models πb(ah | a1:h−1) to maximize log-likelihood.

A goal of statistical language modeling is to learn the joint probability function of sequences of words in a language. … The model learns simultaneously (1) a distributed representation for each word along with (2) the probability function for word sequences, expressed in terms of these representations.

Next-token prediction (NTP) over large text corpora has become the go-to paradigm to train large language models. … The process, elegantly simple, uses a large training corpus to minimize, for each context $\bm{z}_{<t}\in\mathcal{V}^{t-1}$ of $t-1$ preceding tokens, the cross-entropy (CE) loss between the model’s predicted conditional probability distribution over potential next tokens from a vocabulary $\mathcal{V}$ and the one-hot encoded actual next token $z_{t}\in\mathcal{V}$ .

Large pre-trained language models have been shown to store factual knowledge in their parameters, and achieve state-of-the-art results when fine-tuned on down stream NLP tasks. … We explore a general-purpose fine-tuning recipe for retrieval-augmented generation (RAG) — models which combine pre-trained parametric and non-parametric memory for language generation.

AR language modeling seeks to estimate the probability distribution of a text corpus with an au toregressive model [7, 27, 28]. Specifically, given a text sequence x = (x1, · · · , xT ), AR language modeling factorizes the likelihood into a forward product p(x) = Q T t=1 p(xt | x<t)

Once trained, the LLM is deployed and used for text generation (inference); this is discussed in Part 3. … Having discussed pre-training (Part 1) and alignment training (Part 2), we now discuss generation, where the LLM uses its learned parameters to produce new text by sampling one token at a time from the probability distribution it has learned to predict. … The modern LLM is best understood not as a static repository of facts, but as a dynamic system. While its parameters encapsulate the statistical structure of human knowledge, its behavior is a product of deliberate alignment constraints and inference-time conditioning.

A typical application of RAG is illustrated in Figure 2. Here, a user poses a question to ChatGPT about a recent, widely discussed news. Given ChatGPT’s reliance on pre-training data, it initially lacks the capacity to provide updates on recent developments. RAG bridges this information gap by sourcing and incorporating knowledge from external databases.

Large pre-trained language models have been shown to store factual knowledge in their parameters, and achieve state-of-the-art results when fine-tuned on downstream NLP tasks. … Pre-trained neural language models have been shown to learn a substantial amount of in-depth knowledge from data [47]. They can do so without any access to an external memory, as a parameterized implicit knowledge base [51, 52].

Despite being powerful, LLMs lack knowledge that is under-represented in their training data, and are prone to hallucinations, es pecially in open-domain settings (OpenAI, 2023). … Retrieval-augmented LLMs, therefore, have raised widespread attention as LLM outputs can be poten tially grounded on external knowledge.

These models are typically trained on very large datasets and store a substantial amount of world or domain knowledge implicitly in their parameters. However, they are also prone to hallucination and cannot represent the full long tail of knowledge from the training corpus.

Petroni et al. (2019) demonstrate that large pre trained LMs such as BERT (Devlin et al., 2019) memorize the significant amount of world knowl edge in their parameters (parametric knowledge), and Roberts et al. (2020) show that fine-tuned T5 without any reference documents (closed book QA) can achieve competitive performance on open-domain QA.

If the model learns “A equals B” it cannot infer “B equals A” unless it is also in the training data.

Retrieval-augmented generation grounds the outputs of large language models in external corpora, mitigating hallucination (Lewis et al., 2021).

Unlike mathematical Floyd (2007) and logical reasoning Pan et al. (2023), factual reasoning heavily relies on the factual knowledge encoded within LLMs, acquired through extensive pretraining on vast corpora, rather than on user-inputted premises.

This concept of building a model to predict the next most likely word given the context of a sequence of words first originated in work by Shannon [89].

A language model (LM) estimates the joint probability of a given text corpus (x1,…,xT) by factorizing it with a chain rule P(x1,…,xT)=∏t=1TP(xt|x1,…,xt−1) and sequentially modeling each conditional term in the product.

- Causal language modeling: the model has to predict the next token in the sentence (so the labels are the same as the inputs shifted to the right). … - Masked language modeling: the model has to predict some tokens that are masked in the input. It still has access to the whole sentence, so it can use the tokens before and after the tokens masked to predict their value.

Instead of learning to predict only the next token at each position, MTP adds auxiliary prediction heads that predict tokens 2, 3, or more positions ahead.

Recently, large language models (LLMs) have showcased remarkable abilities in solving various tasks, which are capable of encoding extensive volumes of world knowledge within their param eters (Brown et al., 2020a; Ouyang et al., 2022; Zhao et al., 2023).

The `NextTokenLM` embeds some input tokens, contextualizes them, then predicts the next word, computing a loss against known target.

Despite their impressive capabilities, these models often produce text that seems coher ent and plausible but factually incorrect, a problem commonly known as hallucination (Maynez et al., 2020; Zhou et al., 2020; Liu et al., 2021; Ji et al., 2023; Su et al., 2024).

Conceptually, the retrieval-augmented decoder-only LLM can be viewed as applying the sparse attention over its long context window, where the sparsity pattern is not predefined as Child et al. (2019) but determined by the standalone retriever.

Autoregressive models, such as GPTs, are trained to guess how a sequence continues; for example, whether the word sequence "I like to eat" is more likely to be followed by the word "bread" or the word "rocks". … The model's predictions are based on the properties of sequences within its training dataset.

To address the issues of hallucinations Xu et al. (2024b); Liu et al. (2024a) due to a lack of corresponding knowledge or containing outdated knowledge, retrieval-augmented generation (RAG) Gao et al. (2023); Fan et al. (2024) has been proposed for retrieving relevant chunks from a pool of candidate documents to assist LLM generation.

Following the autoregressive language model training as described in Section 9.3.3, Fig. 11.9.6 illustrates GPT pretraining with a Transformer encoder, where the target sequence is the input sequence shifted by one token. … Note that the attention pattern in the Transformer decoder enforces that each token can only attend to its past tokens (future tokens cannot be attended to because they have not yet been chosen).

Following the autoregressive language model training as described in Section 9.3.3, Fig. 11.9.6 illustrates GPT pretraining with a Transformer encoder, where the target sequence is the input sequence shifted by one token. … Note that the attention pattern in the Transformer decoder enforces that each token can only attend to its past tokens (future tokens cannot be attended to because they have not yet been chosen).

Multi-Token Prediction (MTP) extends the prediction scope to several future tokens at each position. An MTP objective adds extra prediction targets, which can improve data efficiency. It may also encourage representations that anticipate later tokens.

Unlike LLMs that rely on static training data, RAG pulls relevant text from databases, uploaded documents, or web sources. … IBM states that "in the generative phase, the LLM draws from the augmented prompt and its internal representation of its training data to synthesize" an answer.

targets (List[str]): list of the target types that the language model should predict. Can be one of "self", "future", and "past". Defaults to "future".

Next-token prediction trains a large language model by asking it to assign a high probability to the token that actually follows each position in a text sequence. … Next-token prediction teaches a base model to approximate patterns in its training text. It does not directly provide labels for factual correctness, helpfulness, or instruction following.