Machine Learning Fundamentals

This cheat sheet covers essential concepts in machine learning, including probability, statistics, linear regression, classification, and neural networks. It provides a concise overview of key terms, formulas, and techniques used in the field.

Core Principles

Machine learning models learn patterns from data.
Probability and statistics are foundational to understanding ML.
Linear regression models relationships between variables.
Classification models categorize data points.
Neural networks, especially deep ones, are powerful function approximators.
Data preprocessing and feature engineering are crucial steps.
Model evaluation metrics are essential for assessing performance.
Hyperparameter tuning is key to optimizing models.

Action Steps

Collect and preprocess data.
Split data into training, validation, and test sets.
Choose an appropriate model architecture.
Define a loss function.
Train the model using optimization algorithms (e.g., Gradient Descent).
Tune hyperparameters using validation data.
Evaluate the final model on the test set.
Iterate and refine the model as needed.

Formulas

$P(A|B) = \frac{P(B|A)P(A)}{P(B)}$ (Bayes' Rule)
$E[X] = \sum_{x} x P(x)$ (Expected Value for discrete r.v.)
$E[X] = \int_{-\infty}^{\infty} x f(x) dx$ (Expected Value for continuous r.v.)
$var(X) = E[(X - \mu)^2]$ (Variance)
$H(X) = -\sum_{i=1}^{n} p_i \log_2 p_i$ (Entropy)
$Accuracy = \frac{TP + TN}{P + N}$
$Precision = \frac{TP}{TP + FP}$
$Recall = \frac{TP}{P}$
$F1-Score = 2 \frac{Precision \times Recall}{Precision + Recall}$
$FPR = \frac{FP}{N}$
$L(y, \hat{y}) = \frac{1}{N} \sum_{n=1}^{N} (y_n - \hat{y}_n)^2$ (Mean Squared Error)
$RSS(\omega) = \sum_{n=1}^{N} (y_n - \omega^T x_n)^2$
$w^* = (X^T X)^{-1} X^T y$ (Least Squares Solution)
$p(y|x; W) = \frac{1}{1 + e^{-W x}}$ (Sigmoid function for logistic regression)
$\mathcal{L}(\theta) = \underset{\theta}{\arg\min} \sum_{n=1}^{N} \mathcal{L}(y_n, f(x_n; \theta))$ (Loss minimization)

Key Terms

Supervised Learning: Learning from labeled data where the model learns a mapping from inputs to outputs.
Unsupervised Learning: Learning from unlabeled data to discover patterns or structures.
Regression: Predicting a continuous outcome variable.
Classification: Assigning data points to predefined categories.
Feature Vector: A numerical representation of input data.
Loss Function: Measures the error between predicted and actual values.
Gradient Descent: An iterative optimization algorithm to find the minimum of a function.
Neural Network: A model inspired by the structure of the brain, composed of interconnected nodes (neurons).
Activation Function: Introduces non-linearity into neural networks.
Backpropagation: Algorithm for training neural networks by propagating errors backward.
Convolutional Neural Network (CNN): A type of neural network particularly effective for image processing.
Pooling Layer: Reduces the spatial dimensions of feature maps in CNNs.
k-Nearest Neighbors (k-NN): A non-parametric classification/regression algorithm based on nearest neighbors.
Curse of Dimensionality: Phenomenon where models perform poorly in high-dimensional spaces.
Language Modeling: Predicting the next word in a sequence.
Word Embeddings: Representations of words as dense vectors in a continuous space.
Recurrent Neural Network (RNN): Neural networks designed to handle sequential data.
Seq2Seq: Sequence-to-sequence models used for tasks like machine translation.
Attention Mechanism: Allows models to focus on relevant parts of the input sequence.
Transformer: A neural network architecture relying heavily on attention mechanisms.