title	Transfer Learning
aliases	Domain Adaptation Fine-tuning Foundation Models
tags	transfer-learning deep-learning pre-training

Transfer Learning

• Transfer learning survey - Organizes transfer by differences between source and target domains, tasks, and available labels; covers instance, feature, parameter, and relational knowledge transfer. Pan and Yang 2010.

Pretraining and task adaptation

• ULMFiT - Pretrains a language model, adapts it to target text, then tunes a classifier. Layer-specific learning rates and gradual unfreezing control how pretrained weights change. Howard and Ruder 2018.
• BERT - Masked-token and next-sentence prediction pretrain a bidirectional encoder; task-specific output layers and encoder fine-tuning transfer it to language understanding tasks. Devlin et al. 2019.
• T5 - Casts NLP tasks as text-to-text prediction; compares pretraining objectives, architectures, data, and transfer strategies within a shared framework. Raffel et al. 2020.
• Domain- and task-adaptive pretraining - Continues language-model pretraining on unlabeled domain text or task-specific text before supervised fine-tuning; studies both separately and together. Gururangan et al. 2020.
• Language-defined classification - CLIP uses descriptions of candidate classes for zero-shot prediction without target-dataset fine-tuning; see Multi-View Learning.

Frozen features and fine-tuning

• DeCAF - Reuses activations from a supervised object-recognition CNN as fixed features for scene recognition, domain adaptation, and fine-grained classification. Donahue et al. 2014.
• How transferable are features? - Layer-transfer experiments separate task specialization from optimization difficulties caused by splitting co-adapted neurons; transferability depends on layer and task distance. Yosinski et al. 2014.
• L2-SP - Regularizes fine-tuning toward pretrained parameter values rather than zero, explicitly encouraging retention of the source solution. Li et al. 2018.
• Linear probing then fine-tuning - Fits a linear head before updating the whole model. Studies how fine-tuning can distort useful pretrained features and hurt out-of-distribution accuracy despite improving in-distribution accuracy. Kumar et al. 2022.

Parameter-efficient adaptation

• Bottleneck adapters - Inserts small trainable modules into a pretrained Transformer. The original recipe also tunes layer normalization and the task head; most backbone parameters stay fixed. Houlsby et al. 2019.
• Prefix tuning - Freezes the language model and learns continuous prefix activations across its layers, providing task-specific conditioning for generation. Li and Liang 2021.
• Prompt tuning - Learns soft prompt embeddings prepended to the input while freezing the language model; the trainable prompt operates at the input rather than every layer. Lester et al. 2021.
• LoRA - Freezes pretrained weights and learns low-rank additive updates to selected matrices. Reduces trainable parameters and task-specific storage; the base weights need not be low-rank. Hu et al. 2022.
  • QLoRA - Trains LoRA adapters through a frozen 4-bit base model. NF4 storage, double quantization, and paged optimizers reduce training memory; computation uses higher precision. Dettmers et al. 2023.
  • DoRA - Separates pretrained weights into magnitude and direction; learns magnitude changes and uses LoRA for directional updates. Liu et al. 2024.
  • LoRA Learns Less and Forgets Less - In studied code and math adaptation settings, standard low-rank LoRA learns less than full fine-tuning but retains more performance outside the target domain. Biderman et al. 2024.
• IA3 - Learns vectors that rescale attention keys, values, and intermediate feed-forward activations while freezing the backbone; introduced with the T-Few adaptation recipe. Liu et al. 2022.
• LoRA tuning evidence - With learning-rate sweeps, LoRA, PiSSA, MiLoRA, Init[AB] and DoRA reach similar peak scores on tested 0.6B-13B LLM tasks but favor different rates. Rank and batch-size sweeps cover selected settings; the study does not establish equivalence on all tasks. Lee et al. 2026.

Domain adaptation

• Covariate-shift weighting - Reweights source examples toward the target input distribution. Assumes unchanged label probabilities given inputs and source support covering the target; estimates weights discriminatively. Bickel et al. 2009.
• DANN - Uses labeled source and unlabeled target data. A gradient-reversal layer trains features to support source classification while confusing a source-versus-target discriminator. Ganin et al. 2016.
• Deep CORAL - Aligns source and target feature covariances during neural-network training, extending linear CORAL to learned nonlinear representations. Sun and Saenko 2016.
• SHOT - Adapts from a pretrained source model and unlabeled target data without revisiting source data. Freezes the classifier and updates features through information maximization and pseudo-labeling. Liang et al. 2020.

Test-time adaptation

• Test-Time Training - Jointly trains prediction and a self-supervised auxiliary task, then updates shared features on unlabeled test inputs before prediction. Includes per-sample and online variants. Sun et al. 2020.
• Tent - Minimizes prediction entropy on unlabeled test batches by updating normalization statistics and channel-wise affine parameters; requires no source data or changes to source training. Wang et al. 2021.
• CoTTA - Adapts to changing target streams using averaged teacher predictions and stochastic restoration of source weights to reduce error accumulation and forgetting. Wang et al. 2022.

Generalization to unseen domains

• Invariant Risk Minimization - Seeks a representation admitting the same optimal classifier across training environments. Counterexamples show this objective can still fail on unseen shifts. Arjovsky et al. 2019, Rosenfeld et al. 2021.
• Group DRO - Minimizes worst-group training loss using predefined groups. Experiments show regularization or early stopping matters for worst-group generalization in overparameterized networks. Sagawa et al. 2020.

Few-shot and in-context adaptation

• Matching Networks - Learns to predict a query label from a small labeled support set; adapts to new classes through the support examples without fine-tuning the network. Vinyals et al. 2016.
• Prototypical Networks - Learns an embedding space and represents each class by its mean support embedding; classifies queries by distances to these prototypes. Snell et al. 2017.
• MAML - Meta-learns initial parameters across tasks so a few gradient updates on a new task's examples produce an adapted predictor. Finn et al. 2017.
• In-context learning in GPT-3 - Specifies tasks and demonstrations in the input text, with no gradient updates at deployment. Few-shot conditioning changes the context; fine-tuning changes parameters. Brown et al. 2020.

Learning with limited or unreliable labels

• Positive-unlabeled learning - Learns from known positives and an unlabeled mixture of both classes. If labeled positives are selected uniformly within their class, label probability is proportional to class probability; selective labeling breaks this correction. Elkan and Noto 2008.
• BALD active learning - Chooses labels expected to reduce posterior uncertainty about the model, using disagreement among predictive distributions. Houlsby et al. derive it for Gaussian-process classification; querying labels differs from passive few-shot adaptation. Houlsby et al. 2011.
• Mean Teacher - Averages model weights to produce teacher targets for consistency training on perturbed inputs. This uses unlabeled observations during fitting; its benefit depends on the architecture and the relation between labeled and unlabeled data. Tarvainen and Valpola 2017.
• Programmatic weak supervision - Snorkel combines noisy labeling functions into probabilistic training labels by modeling their accuracies and dependencies, then trains a discriminative model. Development and test sets remain labeled. Ratner et al. 2017.
• Semi-supervised evaluation - Oliver et al. find that out-of-class unlabeled examples can degrade tested methods and that strong supervised baselines are often underreported. Match backbones and label-selection budgets; more unlabeled data need not improve performance. Oliver et al. 2018.
• FixMatch - Uses confident predictions on weakly augmented unlabeled images as targets for strongly augmented versions. Combines pseudo-labeling with consistency; the confidence threshold and augmentation policy determine which unlabeled evidence enters training. Sohn et al. 2020.
• Confident Learning - Uses out-of-sample probabilities to estimate the joint distribution of observed and latent labels and rank suspected errors. Assumes class-conditional label noise; exact recovery also needs constraints on prediction errors. Northcutt et al. 2021; code.

Continual adaptation and model merging

• Learning without Forgetting - Trains on new-task data while distilling the previous model's old-task outputs on those same inputs. Retains old-task supervision without storing its original training examples. Li and Hoiem 2016.
• Elastic weight consolidation - Penalizes changes to parameters important for previous tasks using a diagonal Fisher-weighted quadratic penalty; a regularization approach to reducing catastrophic forgetting. Kirkpatrick et al. 2017.
• Dark Experience Replay - Stores past examples and model logits, then matches those logits during later training. Combines replay with distillation when task boundaries blur; its comparisons require accounting for memory and replay computation. Buzzega et al. 2020.
• Task arithmetic - Defines task vectors as fine-tuned weights minus a shared pretrained initialization. Adding, subtracting or combining these vectors edits model behavior without another gradient-training run. Ilharco et al. 2023.

Evidence and evaluation

• Domain-adaptation theory - Relates target error to source error, domain divergence, and the existence of a hypothesis that works in both domains. Distribution alignment alone is insufficient. Ben-David et al. 2009.
• Negative transfer - Defines harmful transfer relative to the same learning algorithm without source data; analyzes source-target mismatch and target-label availability, and proposes filtering unrelated source examples. Wang et al. 2019.
• DomainBed study - Compares domain generalization under controlled architectures and model selection. Carefully tuned empirical risk minimization is competitive on the studied benchmarks. Gulrajani and Lopez-Paz 2021.
• WILDS - Benchmarks naturally occurring distribution shifts across hospitals, locations, time, and other groups, with prescribed splits and evaluation metrics. Koh et al. 2021.

Software

• PEFT - Hugging Face library for adapting pretrained models through a subset of parameters; integrates with Transformers, Diffusers, and Accelerate. docs.
• TorchVision models - Pretrained vision weights with associated preprocessing transforms. Select a specific weight version for reproducible transfer experiments. docs.
• DomainBed - Domain-generalization algorithms, datasets, training scripts, and model-selection procedures in a shared experimental framework. code.
• WILDS package - Dataset loading, predefined splits, group metadata, evaluation functions, and baseline code for distribution-shift benchmarks. code.

Related topics

• Feature extraction
• Neural networks