title	Natural Language Processing
aliases	NLP Language Models Text Mining
tags	nlp deep-learning transformers language-models

Natural Language Processing

Statistical models and structured prediction

• Maximum-entropy language models - Combines observed feature statistics in a probabilistic model and estimates parameters by maximum likelihood. Develops feature selection and estimation procedures for language-processing tasks. Berger et al. 1996.
• N-gram smoothing comparisons - Compares smoothing methods across corpora, training sizes, and n-gram orders using held-out cross-entropy. Shows that model settings and data scale affect which estimator works best. Chen and Goodman 1996.
• Conditional random fields - Models label sequences conditional on observations, allowing dependent labels and rich input features. Avoids the label-bias limitation of locally normalized maximum-entropy Markov models. Lafferty et al. 2001.
• Neural tagging from learned features - Uses a shared neural approach for part-of-speech tagging, chunking, entity recognition, and semantic roles. Learns representations from largely unlabeled text instead of engineering separate input features for every task. Collobert et al. 2011.
• Naive Bayes and SVM baselines - Studies unigram/bigram features and NB/SVM variants for sentiment and topic classification. Uses NB log-count ratios as SVM features and shows why a precisely specified baseline matters. Wang and Manning 2012.
• Neural dependency parsing - Uses learned dense features to choose transitions in a greedy dependency parser. Replaces large sparse feature-template sets with compact representations of parsing configurations. Chen and Manning 2014.

Tokenization and word representations

• word2vec - Learns word vectors through continuous bag-of-words and skip-gram prediction objectives. Produces a reusable vector per vocabulary word; the representation does not change with each sentence context. Mikolov et al. 2013.
• GloVe - Fits a weighted log-bilinear model to global word co-occurrence statistics. Combines corpus-level counts with vector representations rather than learning only from individual context windows. Pennington et al. 2014.
• Subword translation with BPE - Represents rare and unseen words as sequences of subword units. Adapts byte-pair encoding to vocabulary construction for translation, reducing dependence on whole-word dictionaries. Sennrich et al. 2016.
• fastText subword vectors - Represents words by sums of character n-gram vectors within a skip-gram model. Can construct vectors for unseen words from their subword components. Bojanowski et al. 2017.
• fastText classification - Uses a simple supervised text-classification architecture with word and n-gram features. Provides a computationally inexpensive baseline distinct from the fastText method for unsupervised word vectors. Joulin et al. 2017.
• SentencePiece - Trains subword tokenization directly from raw sentences without requiring prior word segmentation. Provides tokenization and detokenization for languages with different writing conventions. Kudo and Richardson 2018.

Contextual pretraining

• ELMo - Builds context-dependent word representations from internal states of a pretrained bidirectional language model. Allows downstream tasks to combine signals from different layers. Peters et al. 2018.
• BERT - Pretrains bidirectional Transformer representations with masked-token prediction and an original next-sentence objective. Fine-tunes the encoder for tasks such as classification and extractive question answering. Devlin et al. 2019.
  • RoBERTa - Reexamines BERT training choices, data size, and optimization. Demonstrates that training conditions can explain apparent differences between pretraining methods. Liu et al. 2019.
• XLNet - Trains autoregressive predictions across permutations of the factorization order to learn bidirectional context. Integrates Transformer-XL mechanisms without using BERT's masked-token objective. Yang et al. 2019.
• ELECTRA - Trains an encoder to distinguish original tokens from plausible generator replacements. Uses a discrimination target across input positions rather than predicting only the identities of masked tokens. Clark et al. 2020.
• ModernBERT - Updates bidirectional encoder pretraining with rotary positions and alternating local/global attention; evaluated for classification and both single-vector and late-interaction retrieval. Warner et al. 2025.

Sequence generation and translation

• Sequence-to-sequence LSTMs - Encodes an input sequence into a fixed-dimensional vector and decodes a target sequence with another LSTM. Establishes a jointly trained neural architecture for sequence transduction. Sutskever et al. 2014.
• Neural attention for translation - Learns soft alignments so each generated word can depend on relevant source positions. Relieves the fixed-vector bottleneck of early encoder-decoder translation models. Bahdanau et al. 2015.
• Transformer - Uses attention-based encoder and decoder layers for sequence transduction, dispensing with recurrence and convolution. Makes interactions between sequence positions explicit through attention. Vaswani et al. 2017.
• T5 - Expresses classification, question answering, translation, and other tasks in a common text-to-text format. Compares pretraining objectives, architectures, datasets, and transfer procedures within that framework. Raffel et al. 2020.
• BART - Pretrains an encoder-decoder by reconstructing text corrupted with noise, including span masking and sentence reordering. Connects bidirectional encoding with left-to-right generation. Lewis et al. 2020.
• Vision-language generation - Flamingo and LLaVA condition language generation on images; see Multi-View Learning for architectures and visual instruction tuning.

Multilingual learning

• XLM-R - Pretrains a masked language model across 100 languages. Studies cross-lingual transfer and the tradeoff between sharing representations and dividing finite model capacity among languages. Conneau et al. 2020.
• mT5 - Extends text-to-text pretraining to multilingual data. Studies cross-lingual task transfer and accidental translation, where generated answers switch away from the requested language. Xue et al. 2021.

Scaling and in-context learning

• Language-model scaling laws - Fits empirical relationships among language-model loss, parameter count, data size, and training compute. Its compute-allocation recommendations depend on the investigated training regime. Kaplan et al. 2020.
  • Compute-optimal training - Studies how to allocate a fixed training budget between model size and token count. Finds a different allocation from earlier scaling work; the optimum depends on the objective and cost assumptions. Hoffmann et al. 2022.
• GPT-3 and few-shot prompting - Studies task demonstrations supplied in the prompt without task-specific gradient updates. Distinguishes in-context conditioning from fine-tuning. Brown et al. 2020.
• Compute-data scaling - Models the diminishing value of repeated or paraphrased tokens. On Dolma-3, a fit on 14M/30M models predicts 60M-600M losses with log-loss RMSE 0.079. Only learning rate and decay are tuned; larger-model advice is extrapolation. Qin et al. 2026.
• Data filtering and compute - On 670M-10B-token Common Crawl pools, sufficiently trained dense models outperform tested filtered subsets in language-modeling loss. Filtering still helps at low compute; full-crawl predictions extrapolate beyond the experiments. Mohri et al. 2026.

Instruction and preference learning

• Instruction tuning / FLAN - Fine-tunes on multiple tasks expressed as natural-language instructions and evaluates unseen task types. Studies how instruction format, task diversity, and model scale affect transfer. Wei et al. 2022.
• InstructGPT - Combines supervised demonstrations with preference rankings and reinforcement learning. Studies following instructions on a specified prompt distribution; preference training does not eliminate factual errors. Ouyang et al. 2022.
• Constitutional AI - Uses written principles to guide model critiques, revised responses, and AI-generated preference feedback. Combines supervised and reinforcement learning while retaining human choices about the principles. Bai et al. 2022.
• Direct Preference Optimization - Reparameterizes a reward-based preference objective as direct policy training with a classification loss. Uses preference pairs without a separate reward-model fitting stage or online sampling during fine-tuning. Rafailov et al. 2023.

Sentence representations and retrieval

• Sentence-BERT - Uses siamese and triplet training to produce sentence embeddings comparable by cosine similarity. Enables precomputing candidate representations instead of encoding every sentence pair jointly. Reimers and Gurevych 2019.
• Dense Passage Retrieval - Learns separate question and passage encoders for dense-vector retrieval. Trains from question-passage examples to select contexts for open-domain question answering. Karpukhin et al. 2020.
• ColBERT - Encodes query and document tokens separately, then scores fine-grained matches through late interaction. Retains multiple token representations while allowing document encoding before query time. Khattab and Zaharia 2020.
• Retrieval-Augmented Generation - Combines a pretrained sequence generator with a learned retriever and an external passage index. Compares retrieval shared across a response with retrieval varying by token; retrieved context does not guarantee factuality. Lewis et al. 2020.
• SimCSE - Learns sentence embeddings contrastively using dropout views, or supervised entailment positives and contradiction negatives. Studies how these objectives shape embedding alignment and uniformity. Gao et al. 2021.
• SPLADE - Learns sparse lexical representations with explicit sparsity regularization and saturated term weights. Supports retrieval with inverted indexes and an effectiveness-sparsity tradeoff. Formal et al. 2021.
• RAGTruth - Annotates unsupported or contradictory spans in nearly 18,000 retrieval-augmented responses. Demonstrates that providing retrieved evidence does not ensure faithful generation; retrieval quality and support for generated claims need separate evaluation. Niu et al. 2024.
• Late chunking - Encodes a longer text before dividing token representations into chunks for pooling. Produces one vector per chunk informed by surrounding context, unlike late-interaction retrieval's multiple stored token vectors. Günther et al. 2024.
• Direct corpus interaction - Lets an agent search raw files with terminal tools. On 100 BrowseComp-Plus queries with GPT-5.4 nano, accuracy rises from 45% with embedding retrieval to 73%, while agent cost roughly doubles and gold-document recall falls. Li et al. 2026.

Long-context models

• Transformer-XL - Combines segment-level recurrence with a positional encoding scheme to extend context across segments. Reuses prior representations while addressing fragmentation from fixed-length training windows. Dai et al. 2019.
• Longformer - Combines local sliding-window attention with task-specific global attention. Reduces attention cost for long documents when window size and the number of global positions remain bounded. Beltagy et al. 2020.
• Lost in the Middle - Finds position-dependent failures in long-context question answering and key-value retrieval. In the studied models, relevant information near the middle can be harder to use than information near either end. Liu et al. 2024.

Decoding and reasoning procedures

• Nucleus sampling - Samples from a dynamically selected high-probability portion of the next-token distribution. Studies how decoding changes repetition and diversity even when the underlying language model is unchanged. Holtzman et al. 2020.
• PICARD constrained decoding - Rejects tokens that violate incremental parsing constraints during generation. Improves validity and text-to-SQL performance on Spider and CoSQL; a syntactically valid query can still answer the wrong question. Scholak et al. 2021.
• Chain-of-thought prompting - Supplies worked reasoning examples in the prompt before asking for a solution. Studies arithmetic, commonsense, and symbolic tasks; generated explanations are not themselves a correctness check. Wei et al. 2022.
  • Self-consistency - Samples multiple reasoning paths and aggregates their answers instead of choosing one greedy path. Improves the studied reasoning benchmarks at the cost of additional generation. Wang et al. 2023.
• ReAct - Interleaves generated reasoning with actions that query external information or interact with an environment. Studies how observations can update task plans instead of relying only on an uninterrupted text completion. Yao et al. 2023.
• Speculative decoding - A smaller model drafts tokens that the target model verifies in parallel. A sampling correction preserves the target distribution; speed depends on draft cost and acceptance. Leviathan et al. 2023.
• Process supervision - Trains reward models from human judgments of individual solution steps rather than only final answers; introduces PRM800K for studying process supervision in mathematical reasoning. Lightman et al. 2023.

Optimizing language-model programs

• DSPy - Represents LM pipelines as typed modules and bootstraps demonstrations from successful execution traces. A compiler selects module parameters against a supplied metric; the program structure and evaluator remain design choices. Khattab et al. 2024.
• TextGrad - Propagates LLM-written critiques through a computation graph to revise prompts, code or other variables. Its textual gradients are feedback, not numerical derivatives. Yuksekgonul et al. 2025.

Text-generation metrics

• BLEU - Compares generated translations with references using modified n-gram precision and a brevity penalty. A corpus-level overlap metric; it does not directly establish factual or semantic equivalence. Papineni et al. 2002.
  • SacreBLEU and reproducibility - Shows that tokenization and normalization choices can make BLEU scores incomparable. Provides standardized reference processing and reporting to make evaluations easier to reproduce. Post 2018.
• ROUGE - Evaluates summaries using overlap with reference summaries, including n-grams and sequence-based measures. Different ROUGE variants capture different forms of overlap. Lin 2004.
• chrF - Uses character n-gram precision and recall for translation evaluation. Avoids dependence on a particular word tokenizer and exposes a parameter controlling the recall-precision balance. Popović 2015.
• BERTScore - Matches candidate and reference tokens using contextual embedding similarity rather than exact lexical equality. Measures representation-level agreement; it is not a factuality verifier. Zhang et al. 2020.

Benchmarks and behavioral evaluation

• SQuAD - Tests reading comprehension with questions whose answers are spans of the supplied Wikipedia passage. Measures extractive question answering rather than unrestricted open-domain knowledge. Rajpurkar et al. 2016.
• MultiNLI - Provides entailment, contradiction, and neutral sentence pairs across genres of written and spoken English. Includes evaluation of transfer across genres. Williams et al. 2018.
• GLUE - Combines language-understanding tasks with a diagnostic suite for linguistic behavior. Encourages shared representations across tasks rather than optimizing a single dataset. Wang et al. 2019.
  • SuperGLUE - Introduces a harder collection of language-understanding tasks following progress on GLUE. Supplies a toolkit and shared evaluation; aggregate scores still combine distinct abilities. Wang et al. 2019.
• CheckList - Organizes behavioral tests by linguistic capability and test type. Uses generated cases to reveal failures that aggregate held-out accuracy can conceal. Ribeiro et al. 2020.
• MMLU - Tests knowledge and problem solving across 57 academic and professional subjects. Supports per-subject analysis; performance on these questions is not a general measure of real-world competence. Hendrycks et al. 2021.
• BIG-bench - Collects diverse tasks for examining language-model capabilities and scaling. Studies differences across tasks, model sizes, and evaluation metrics instead of treating aggregate improvement as uniform progress. Srivastava et al. 2022.
• TruthfulQA - Tests whether generated answers reproduce common misconceptions. Targets a particular source of falsehood; passing the benchmark does not establish truthfulness on arbitrary questions. Lin et al. 2022.
• HELM - Evaluates language models across shared scenarios and multiple criteria, including accuracy, calibration, robustness, bias, toxicity, and efficiency. Makes coverage gaps and metric tradeoffs explicit. Liang et al. 2023.
• Memory consolidation failures - Repeatedly distilling ground-truth solutions makes previously solved ARC-AGI tasks fail for GPT-5.4. Raw-episode stores remain competitive with consolidators in WebShop, ALFWorld and AppWorld; compression can discard useful evidence. Zhang et al. 2026.
• Raw-history memory - On MemoryAgentBench with GPT-4o-mini, tested additions of fact extraction, summary augmentation or graph expansion to dense retrieval with reranking lower accuracy. Raw-RAG trails two structured systems on PersonaMem-v2; these are offline evaluations. Bao et al. 2026; code.
• Thinking and refusal - In four moderate-size open reasoning models, tested safety defenses often trade less harmful compliance for more benign refusals. Thinking-prefix and defense interventions concern refusal behavior, not every form of safety or frontier-scale reasoning. Ri et al. 2026.

Software

• Transformers - Provides a library of pretrained Transformer architectures and task interfaces. The paper describes reusable implementations for NLP research and applications, rather than a single model release. Wolf et al. 2020.
• Stanza - Provides a multilingual NLP pipeline for tokenization, morphology, syntax, and named entities. Packages neural processing components for text analysis across languages. Qi et al. 2020.

Related topics

• Neural Networks
• Transfer Learning
• Generative Models
• Computer Vision
• Multi-View Learning