In industrial AI and data science engineering, algorithm selection defines the upper bound of model performance, compute cost, and deployment feasibility.
This cheat sheet eliminates outdated academic stepping stones (such as AdaBoost, SARSA, Vanilla RNN, or genetic algorithms), focusing exclusively on the 27 essential algorithms widely used in modern production AI systems.
🔗 Related Cheat Sheets: Claude Code CLI Cheat Sheet · Python Cheat Sheet · 4 Leading Always-on Agents Comparison Cheat Sheet
Modern AI Core Algorithms Master Index
| # | Algorithm (Abbr.) | Primary Task | Mathematical Foundation & Loss Mechanism | Critical Hyperparameters | Typical Industrial Applications |
|---|---|---|---|---|---|
| 01 | Linear Regression | Continuous Value Fitting | Minimize Mean Squared Error: $\mathcal{L}_{\text{MSE}} = \frac{1}{2m}\sum(y-\hat{y})^2$ | Regularization $\alpha$ (L1 Lasso / L2 Ridge) | Real-estate valuation, power consumption forecasting, linear trend analysis |
| 02 | Logistic Regression | Binary Classification | Sigmoid activation + Binary Cross-Entropy log loss | Inverse regularization $C$, class weights (class_weight) |
Ad CTR prediction, churn forecasting, credit risk scoring |
| 03 | Decision Tree (CART) | Classification / Regression | Greedy binary recursive splitting (Gini Impurity / Variance Reduction) | max_depth, min_samples_split, min_samples_leaf |
Explainable credit approvals, medical triage baselines |
| 04 | Support Vector Machine (SVM) | Classification / Regression | Maximum margin hyperplane + Kernel Trick | Penalty $C$, Gaussian kernel bandwidth $\gamma$ | High-dimensional small-sample classification, bioinformatics gene markers |
| 05 | K-Nearest Neighbors (KNN) | Instance-based Classification | Distance metrics (Euclidean / Manhattan) + majority vote | Neighbors $k$, distance weighting metric | Recommendation system cold starts, local anomaly detection |
| 06 | Naive Bayes | Text / Discrete Classification | Conditional independence assumption + Maximum A Posteriori (MAP) | Laplace smoothing coefficient $\alpha$ | Spam filtering, short-text classification, sentiment screening |
| 07 | Random Forest | Classification / Regression | Bagging row bootstrapping + random column subspace sampling | Tree count n_estimators, max_features |
Feature importance ranking, industrial predictive maintenance |
| 08 | Gradient Boosted Trees (XGBoost/LightGBM) | Tabular SOTA Modeling | Additive residual fitting along negative gradients + 2nd-order Taylor expansion | Learning rate eta, tree depth, subsample, colsample_bytree |
Kaggle tabular competitions, search ranking, fraud pricing models |
| 09 | K-Means Clustering | Unsupervised Partitioning | Iterative centroid reassignment minimizing within-cluster sum of squares (WCSS) | Number of clusters $K$, initialization init='k-means++' |
Customer persona segmentation, image color quantization |
| 10 | Hierarchical Clustering | Tree-based Clustering | Agglomerative (bottom-up) or divisive hierarchical dendrogram | Linkage criterion (ward, complete), distance threshold |
Phylogenetic tree reconstruction, organizational taxonomy clustering |
| 11 | DBSCAN | Density-based Clustering | Core point density-reachability within $\epsilon$-neighborhood; robust to noise | Neighborhood radius $\epsilon$, minimum density min_samples |
Geospatial GPS trajectory hotspot mining, network intrusion clustering |
| 12 | Principal Component Analysis (PCA) | Linear Dimensionality Reduction | Orthogonal eigendecomposition of covariance matrix maximizing projection variance | Target components $k$ or explained variance ratio | Feature noise reduction, multicollinearity elimination, data compression |
| 13 | t-SNE / UMAP | Manifold Learning & Visualization | High-dimensional distribution matching (KL divergence) / Riemannian fuzzy simplicial sets | perplexity (t-SNE) / n_neighbors, min_dist (UMAP) |
High-dimensional embedding 2D/3D visualization, single-cell RNA analysis |
| 14 | HNSW (Hierarchical Navigable Small World) | Vector ANN Search | Multi-layer probabilistic skip-list graph traversal with logarithmic complexity | Graph degree M, search width efConstruction, efSearch |
Foundation for LLM RAG pipelines, Milvus/Qdrant/Pinecone vector databases |
| 15 | Stochastic Gradient Descent + AdamW | Deep Learning Optimization | Decoupled weight decay + momentum damping + adaptive learning rates | Learning rate $\eta$, weight decay $\lambda$, betas $(\beta_1, \beta_2)$ | Universal default optimizer for Transformer and neural network training |
| 16 | Multilayer Perceptron (MLP) | Nonlinear Mapping | Dense matrix multiplications $W \cdot x + b$ with activation functions (GELU, SwiGLU) | Hidden layer sizes, dropout rate | Recommendation ranking layers (DeepFM), multimodal projection necks |
| 17 | Convolutional Neural Networks (ResNet/ConvNeXt) | Spatial Computer Vision | Local receptive fields + weight sharing + residual skip connections | Kernel size, layer strides, residual blocks | Image classification, object detection backbones, medical imaging |
| 18 | Recurrent Architectures (LSTM / GRU) | Sequential Modeling | Additive gating pathways (input/forget/output gates) preventing gradient vanishing | Hidden dimension size, recurrent dropout | Sensor telemetry forecasting, audio signal modeling, algorithmic trading |
| 19 | Graph Neural Networks (GCN / GAT) | Relational Graph Learning | Spectral graph Laplacian filtering / Spatial attention-weighted message passing | Graph layers, aggregation operator, attention heads | Financial fraud ring detection, drug discovery molecule graphs |
| 20 | Transformer | Sequence Foundation Modeling | Scaled Dot-Product Multi-Head Attention + Positional Encodings (RoPE) | Attention heads, layer count $L$, model dimension $d_{\text{model}}$ | GPT-4, LLaMA-3, Claude, Vision Transformers |
| 21 | Masked Autoencoder (MAE) | Self-Supervised Vision Pretraining | High-ratio random masking (75%) + asymmetric autoencoder reconstruction | Masking ratio, encoder/decoder depth ratio | Self-supervised visual pretraining for robotics, satellite and vision backbones |
| 22 | Proximal Policy Optimization (PPO) | RLHF Policy Optimization | Clipped surrogate objective constraining policy updates: $\text{clip}(r_t, 1-\epsilon, 1+\epsilon)$ | Clipping parameter $\epsilon$, generalized advantage estimation $\lambda$ | Classic RLHF alignment pipeline in ChatGPT / InstructGPT |
| 23 | Direct Preference Optimization (DPO) | Offline LLM Preference Alignment | Closed-form implicit reward mapping directly over human preference pairs $(y_w, y_l)$ | Reference policy temperature $\beta$ | The modern standard for open-source LLM post-training alignment |
| 24 | Variational Autoencoder (VAE) | Latent Generative Modeling | Variational inference maximizing ELBO + reparameterization trick $z = \mu + \sigma \odot \epsilon$ | Latent dimension $d_z$, KL divergence weight $\beta$ | Latent space image compression inside Stable Diffusion / Flux / Sora |
| 25 | Generative Adversarial Networks (GAN) | Adversarial Generation | Minimax zero-sum game between Generator and Discriminator | Gradient penalty weight $\lambda$, discriminator update ratio | Real-time facial upscaling, super-resolution, neural voice synthesis |
| 26 | Diffusion Models (DDPM / Flow Matching) | High-Fidelity Generation | Stepwise forward Gaussian noise addition + reverse neural denoising / vector fields | Timesteps $T$, noise schedule, classifier-free guidance scale (CFG) | Midjourney, Stable Diffusion, Flux, Sora text-to-image and text-to-video |
| 27 | Monte Carlo Tree Search (MCTS) | Strategic Planning & Reasoning | Four-stage tree rollout (Selection ➔ Expansion ➔ Simulation ➔ Backpropagation) | Exploration constant $C_{\text{puct}}$, search simulations | AlphaGo, AlphaZero, deep reasoning chains in OpenAI o1 / o3 systems |
Core Algorithm Selection Frameworks
1. Tabular Learning: Tree Ensemble Showdown
| Model | Core Splitting Mechanism | Missing Value Handling | Compute Complexity | Production Verdict |
|---|---|---|---|---|
| Random Forest | Parallel independent bagging trees | Imputation required | $O(M \cdot K \cdot N \log N)$ | Robust out-of-the-box baseline; virtually impossible to overfit |
| XGBoost | Serial residual boosting with exact 2nd-order gradients | Native direction learning | Histogram-accelerated | Top-tier predictive precision; handles complex interactions |
| LightGBM | Leaf-wise splitting + GOSS + EFB bundling | Native histogram binning | Fast, lowest memory | King of large-scale tabular datasets; industry standard |
2. Vector ANN Search: Approximate Nearest Neighbors
| Algorithm | Index Construction Mechanism | Query Latency | Memory Footprint | Production Verdict |
|---|---|---|---|---|
| Flat L2 | Exhaustive linear scan ($O(N)$) | Milliseconds to seconds | 1x (Raw vectors) | Exact ground truth baseline for evaluation only |
| IVF-PQ | Inverted file clustering + Product Quantization | Sub-millisecond ($O(\sqrt{N})$) | 0.05x - 0.2x (Compressed) | High compression for multi-billion scale datasets |
| HNSW | Multi-layer probabilistic skip-list graph | Sub-millisecond ($O(\log N)$) | 1.5x - 2.5x (Graph overhead) | Gold standard for RAG applications and vector databases |
3. LLM Post-Training: Alignment Paradigms
| Alignment Method | Training Pipeline Complexity | Required Active Models in VRAM | Training Stability | Production Recommendation |
|---|---|---|---|---|
| RLHF (PPO) | Complex (Rollouts, Value network, Reward model) | 4 Models (Actor, Critic, Reward, Reference) | Fragile, hyperparameter sensitive | High ceiling for massive enterprise clusters |
| DPO | Simple (Direct supervised loss on pairwise data) | 2 Models (Actor + frozen Reference) | Highly stable, standard cross-entropy | Industry standard for open-source model alignment |
| KTO | Simple (Binary thumbs-up/down signals) | 2 Models | Stable | Ideal when pairwise preference data is scarce |
Practical Engineering Selection Guidelines
- Tabular & Tabular Time-Series: Always start with LightGBM / XGBoost. Deep learning rarely outperforms gradient-boosted trees on tabular data.
- Semantic Search & RAG: Pair modern dense embedding models with HNSW vector indexing. Combine with hybrid BM25 lexical search and a cross-encoder reranker for best retrieval accuracy.
- LLM Alignment: Default to DPO (Direct Preference Optimization). It eliminates the fragile instability and massive VRAM overhead of training separate reward and value networks.