Feature Extraction and Representation Learning

🔒 Full text at this link requires payment or institutional access. Unmarked links may be unchecked.

Surveys

• Representation Learning: A Review and New Perspectives - Probabilistic models, autoencoders, manifold learning. Bengio et al. 2013.
• A Survey of Dimensionality Reduction Techniques - Categorizes dimensionality-reduction techniques and explains their mathematical foundations. Sorzano et al. 2014.
• Feature Selection and Feature Extraction in Pattern Analysis - Reviews the theory, applications, and numerical implementations of feature selection and extraction methods. Ghojogh et al. 2019.

Fixed and domain-informed features

Text and image descriptors

• Text vectorization - Sparse word/character n-grams, TF-IDF, and feature hashing. docs.
• Image descriptors - SIFT describes local keypoints; HOG summarizes local gradient orientations. Lowe 2004, Dalal and Triggs 2005.

Signal and time-series transforms

• Wavelet scattering - Cascades of wavelet transforms, nonlinearities, and averaging. Kymatio implements scattering for 1D, 2D, and 3D inputs. Andreux et al. 2020; code.
• Time-series statistics - tsfresh combines a large feature bank with relevance filtering; catch22 provides 22 selected time-series characteristics. Christ et al. 2018, Lubba et al. 2019; code, code.
• Convolutional transforms - ROCKET, MiniROCKET, and MultiROCKET produce convolution-response statistics for linear classification; extraction can expand dimensionality. Dempster et al. 2019, Dempster et al. 2020, Tan et al. 2021; code.

Linear and statistical representations

Component analysis and factorization

• PCA (principal component analysis) - Orthogonal projection maximizing retained variance of centered data. Pearson 1901; docs.
  • Dual PCA - Computes PCA through the sample Gram matrix. Ghojogh and Crowley 2019.
  • Incremental PCA - Batchwise PCA updates. docs, code.
• Truncated SVD / latent semantic analysis - Low-rank approximation without PCA's centering step, allowing sparse document-term inputs. Deerwester et al. 1990; docs.
• ICA (independent component analysis) - Linear separation of statistically independent sources. Hyvärinen and Oja 2000, Hyvärinen et al. 2001; docs.
• Factor analysis - A linear latent-factor model with feature-specific Gaussian noise. docs.
• NMF (non-negative matrix factorization) - Additive factorization of non-negative data. Lee and Seung 1999 🔒; docs.
• Robust PCA / principal component pursuit - Separates low-rank structure from sparse corruption (X = L + S), with recovery guarantees under rank, incoherence, and corruption assumptions. Candes et al. 2011.
• Topic LDA (latent Dirichlet allocation) - Document representations as topic mixtures. Blei et al. 2003, Hoffman et al. 2010.

Sparse representations

• Sparse PCA - Sparse component loadings, selecting which input variables contribute to each component. Zou et al. 2006; docs.
• Dictionary learning / sparse coding - Sparse sample codes over learned, possibly overcomplete dictionaries; sparsity applies to the codes rather than PCA loadings. Mairal et al. 2010; docs.

Supervised and multiview projections

• Fisher LDA (linear discriminant analysis) - Supervised projection maximizing between-class relative to within-class scatter. Fisher 1936, Rao 1948, Martinez and Kak 2001; docs.
• Supervised PCA - Response-based predictor screening followed by PCA. Bair et al. 2006.
• CCA / PLS - Canonical correlation analysis maximizes correlation between projected views; partial least squares extracts cross-covariance-based components, including for regression. docs.

Kernel methods and random projections

• Random projections - Gaussian or sparse random maps approximately preserve Euclidean distances under Johnson-Lindenstrauss conditions. Achlioptas 2003; docs.
• Kernel PCA - Component analysis in an implicit kernel feature space. Schölkopf et al. 1998; docs.
  • Kernel PCA extensions - Kernel PCA for novelty detection and robust kernel PCA. Hoffmann 2007, Nguyen and De la Torre 2008.
• Kernel Fisher discriminant analysis - Supervised class separation in a kernel feature space. Mika et al. 1999.
• Random Fourier features - Sampled Fourier maps approximate shift-invariant kernels. Unlike distance-preserving random projections, these construct nonlinear features and may expand dimension. Rahimi and Recht 2007.
• Nyström approximation - Sampled data landmarks approximate the kernel matrix; an alternative to Fourier sampling. Williams and Seeger 2000; docs.

Manifold learning and visualization

Distance and spectral methods

• MDS (multidimensional scaling) - Fits coordinates to dissimilarities or their ordering. Kruskal 1964, Little et al. 2018, Little et al. 2022; docs.
• Sammon mapping - Weighted distance reconstruction emphasizing smaller input distances. Sammon 1969.
• SOM (self-organizing maps) - Updates grid-organized prototypes and their neighbors. Kohonen 1982 🔒.
• Isomap - MDS on graph-approximated geodesic distances. Tenenbaum et al. 2000; docs.
• LLE (locally linear embedding) - Fits coordinates using fixed local linear reconstruction weights. Roweis and Saul 2000; docs.
  • HLLE (Hessian eigenmapping) - Manifold coordinates from local Hessian estimates. Supported by scikit-learn's LLE interface. Donoho and Grimes 2003.
• Laplacian eigenmaps / spectral embedding - Graph-local similarity represented by Laplacian eigenvectors. Belkin and Niyogi 2003.
• Maximum variance unfolding / semidefinite embedding - Maximizes spread while preserving local distances through semidefinite optimization. Weinberger et al. 2005.
• Diffusion maps - Spectral coordinates reflecting random-walk connectivity across scales. Coifman and Lafon 2006.
  • PHATE - Embeds distances between transformed diffusion probabilities to visualize continuous and branching structure. Moon et al. 2019 🔒; code.

Neighborhood objectives and learned mappings

• t-SNE - Local neighborhood probability matching. van der Maaten and Hinton 2008, van der Maaten 2014; docs, docs.
  • Extensions - Tree-SNE for hierarchical clustering; LEt-SNE for hyperspectral imagery. Robinson and Pierce-Hoffman 2020, Shukla et al. 2019; code.
• LargeVis - Approximate neighborhood-graph construction and probabilistic layout for large collections. Tang et al. 2016.
• UMAP - Embeds a fuzzy neighborhood graph. McInnes et al. 2018; code, docs.
  • Parametric UMAP - Trains a neural encoder with the UMAP objective. Sainburg et al. 2021; docs.
  • densMAP - Adds relative local-density preservation to UMAP. Narayan et al. 2021 🔒; docs.
  • MAPLE - Adds self-supervised manifold modeling to UMAP using maximum manifold capacity representations. Huang et al. 2026, Huang et al. 2026.
• TriMap - Relative similarity preservation through triplets. Amid and Warmuth 2019; code.
• PaCMAP - Combines neighbor, mid-near, and distant pairs to preserve structure at different scales. Wang et al. 2021; code.
  • LocalMAP - Adjusts the neighborhood graph during optimization to improve local cluster separation. Implemented in the PaCMAP repository. Wang et al. 2025.
• ParamRepulsor - Hard-negative mining and stronger repulsion address loss of local detail in parametric embeddings. Huang et al. 2024; code.
• New observations - Ordinary UMAP and openTSNE place new points relative to a fitted embedding. Parametric UMAP instead evaluates a learned encoder; new-point support alone does not imply a neural mapping. docs, docs.

Learned representations

Reconstruction and masking

• Autoencoders - Encoder-decoder learning through reconstruction. Hinton and Salakhutdinov 2006.
  • Denoising - Clean-input reconstruction from corrupted observations. Vincent et al. 2010.
  • Variational - Probabilistic latent-variable modeling with an approximate posterior. Kingma and Welling 2014.
  • Masked - Reconstructs hidden image patches from visible ones. He et al. 2022; code.

Contrastive learning

• SimCLR - Contrasts augmented image views using a nonlinear projection head; downstream features come from the encoder before that head. Chen et al. 2020; code.
• Contrastive Predictive Coding - Predicts future latent representations from autoregressive context using a contrastive loss and negative samples; learns sequential features without reconstructing raw observations. van den Oord et al. 2018.
• Alignment and uniformity - Relates contrastive learning to positive-pair alignment and uniform spread on the unit sphere. The analysis is asymptotic; normalization alone does not guarantee uniform learned features. Wang and Isola 2020.

Self-distillation and redundancy reduction

• BYOL - An online network predicts a moving target network's representation without explicit negative pairs. Grill et al. 2020.
• DINO - Self-distillation without labels using a momentum teacher; the precursor to DINOv2/v3. Caron et al. 2021.
• Barlow Twins - Drives cross-correlation between two augmented views toward identity, combining agreement with redundancy reduction. This objective also underlies TLDR. Zbontar et al. 2021.
• VICReg - Separates view agreement, per-feature variance control, and covariance regularization. Unlike Barlow Twins' cross-correlation objective, it explicitly constrains each branch's variance. Bardes et al. 2022; code.

Prediction in representation space

• I-JEPA - Predicts representations of masked image regions from visible context, rather than reconstructing pixels as in MAE. Assran et al. 2023.
• LeJEPA - Combines a joint-embedding predictive objective with sketched isotropic Gaussian regularization of embeddings. Balestriero and LeCun 2025.
• Spectral Ghost - Develops a spectral account of representation sufficiency connecting component analysis and self-supervised objectives. Dai et al. 2026.

Identifiability and disentanglement

• Limits of unsupervised disentanglement - Locatello et al. show that recovering independent factors requires inductive assumptions. Across seven datasets, stronger disentanglement does not consistently reduce downstream sample needs; factor recovery is a distinct goal. Locatello et al. 2019.
• Identifiable VAE / nonlinear ICA - Conditions a factorized latent prior on an observed auxiliary variable. Identifiability holds up to specified transformations under variation and regularity conditions; an ordinary VAE with an unconditional prior lacks this guarantee. Khemakhem et al. 2020.

Extraction by modality

Images and video

• DINOv2 - Scales self-supervised learning to produce reusable image-level and patch-level features. Oquab et al. 2023; code.
• DINOv3 - Gram anchoring addresses degradation of dense features during extended self-supervised training. Siméoni et al. 2025; code.
• V-JEPA 2.1 - Dense prediction over visible and masked tokens, plus intermediate-layer self-supervision, extends JEPA features to spatially structured image/video representations. Mur-Labadia et al. 2026; code.
• FINO - Adapts visual representations to scientific domains using discrete or continuous metadata alongside self-supervised learning. Gardès et al. 2026.

Text and retrieval

Sentence and document vectors

• Sentence-BERT - Siamese/triplet training makes BERT sentence vectors usable with cosine similarity, avoiding joint query-document encoding. Reimers and Gurevych 2019; code.
• E5 - Contrastive pretraining on weakly supervised text pairs; evaluated both without labeled-data fine-tuning and after supervised fine-tuning. Wang et al. 2022; code.
• Qwen3 Embedding - Adapts language-model backbones for instruction-aware multilingual text embeddings. Zhang et al. 2025; code.

Document context and chunk pooling

• Late chunking - Encodes a long context before mean-pooling individual spans; stores one context-informed vector per chunk. Conventional chunking encodes each chunk separately. Günther et al. 2024.
• pplx-embed-context - Combines diffusion-pretrained bidirectional encoders, contrastive learning, and late chunking for document-context-aware passage vectors. Eslami et al. 2026.

Sparse and multivector scoring

• ColBERT - Independently encodes queries and passages into token vectors; sums each query token's maximum similarity to a passage token. Late interaction retains multiple vectors, whereas late chunking pools each chunk. Khattab and Zaharia 2020; code.
• BGE-M3 - Jointly supports dense vectors, sparse lexical weights, and token multivectors through self-knowledge distillation. Chen et al. 2024; code.
• Multivector expressivity - Proves a worst-case representation-size separation between Chamfer similarity and single-vector inner products; this is not an empirical retrieval ranking. Jayaram 2026.

Audio and time series

• TS2Vec - Contrastive learning of contextual timestamp features, pooled for subsequences or whole series. Yue et al. 2022; code.
• MOMENT - Masked time-series pretraining supports classification, forecasting, imputation, and anomaly detection. Goswami et al. 2024; code.
• MantisV2 - Synthetic-data pretraining combined with layer selection and token aggregation for frozen time-series features. Feofanov et al. 2026; code.
• BEATs - Iterates acoustic tokenization and masked audio-token prediction to learn general audio representations. Chen et al. 2023; code.
• wav2vec 2.0 - Masks latent speech representations and predicts jointly learned quantized targets with a contrastive loss. Unlabeled audio pretraining precedes supervised speech-recognition fine-tuning. Baevski et al. 2020.

Cross-modal representations

• CLIP - Contrastive image-text alignment using natural-language supervision. Radford et al. 2021; code.
• SigLIP 2 - Combines sigmoid image-text learning with self-distillation and masked prediction for multilingual alignment and dense visual features. Tschannen et al. 2025; code.
• Qwen3-VL-Embedding - Shared vectors for text, images, document images, and video. The accompanying reranker scores pairs rather than producing independently indexable vectors. Li et al. 2026; code.
• CLAP - Contrastively trains audio and text encoders on paired recordings and descriptions. Their shared space supports audio retrieval and zero-shot classification through text prompts. Elizalde et al. 2022.

Representation compression

• Matryoshka representation learning - Trains nested vector prefixes at multiple dimensions. Compression is built into representation training; arbitrary truncation of ordinary embeddings is not equivalent. Kusupati et al. 2022; code.
• TLDR - Learns compression of existing features from neighbor pairs using the Barlow Twins redundancy-reduction objective. PCA instead maximizes retained variance. Kalantidis et al. 2022; code.
• Compression evidence - TLDR authors report +4.1 mAP points over whitened PCA at 128 dimensions on ROxford, averaging Medium/Hard protocols, with GeM-AP features and GLD-v2 fitting data. Kalantidis et al. 2022.

Evaluation and comparative evidence

Benchmarks and geometry

• MTEB / MMTEB - Text-embedding evaluation across tasks, domains, and languages. Muennighoff et al. 2022, Enevoldsen et al. 2025; code.
• Neighborhood preservation - Trustworthiness penalizes introduced neighbors; continuity penalizes lost neighbors. These measure geometry rather than held-out task performance. Venna and Kaski 2006; docs.
• Intrinsic dimension - Global and local estimators characterize data geometry; their estimates are not task-optimized embedding dimensions. Bac et al. 2021; docs.
• Local versus global structure - Huang et al. (2022) report stronger neighborhood preservation for t-SNE/art-SNE, but stronger distance-rank preservation for PCA/TriMap/PaCMAP/ForceAtlas2 on most tested datasets. Huang et al. 2022.
• Stability - Huang et al. (2022) test parameter/preprocessing sensitivity and report five-run variation. Its local/global rankings concern 2D layouts, including transcriptomic data, rather than general predictive features. Huang et al. 2022.

Extraction choices

• Projection head - SimCLR's ImageNet linear evaluation favors encoder features before the contrastive projection head over its output. The loss-space representation is not necessarily the best downstream feature. Chen et al. 2020.
• Context - Late-chunking authors report mean nDCG@10 (0-100) rising from 52.2 to 54.0 across three encoders and four BEIR datasets, using 256-token chunks. Individual settings include ties and regressions. Günther et al. 2024.
• Layer selection - MantisV2's UCR ablations favor intermediate frozen features after more pretraining updates; the 100K-sample, 100-epoch setting favors the final layer. Fine-tuning favors all layers. Feofanov et al. 2026.

Software

• Python (reduction and dimension estimation) - scikit-learn, umap-learn, openTSNE, PaCMAP/LocalMAP, PHATE, scikit-dimension. docs, docs, docs, code, code, docs.
• Python (extraction) - Sentence Transformers, timm, Kymatio, tsfresh; evaluation: MTEB. docs, docs, docs, docs, code.
• R - dimRed, dyndimred, intrinsicDimension, Rdimtools; discriminant analysis: MASS, sda. You 2020; code, code, code, code, code, code.
• JavaScript (extraction) - Transformers.js runs compatible pretrained models through ONNX Runtime with WASM/WebGPU. docs.
• JavaScript (visualization) - tsne, umap-js, dimred; interactive tool: StatSim Vis. docs, docs, docs, docs.
• C++ - Tapkee. code.

Connections

• Sparse autoencoders for interpretability - Overcomplete sparse features from internal model activations, evaluated for reconstruction and interpretability rather than compactness alone. Gao et al. 2024.

Related topics

• Feature selection
• Neural networks
• Multiview learning
• Interpretability