Find papers

ITE 493

Machine Learning

TU BITM · Semester 8 · BITM curriculum and programme regulation

Requirement
elective
Credits
3
Study Guide
6 units available

Chapter notes

Authoritative study guide across 6 syllabus units

Open a unit to read its study notes.

Unit 1: Introduction· 3 LHs· ~4 marks

Suggest correction

Part 1: Core Concepts (2 Marks)

  • Machine Learning (Tom Mitchell’s Definition): A computer program is said to learn from experience EE with respect to some class of tasks TT and performance measure PP, if its performance at tasks in TT, as measured by PP, improves with experience EE.
  • Learning Paradigms:
    • Supervised Learning: Learning a mapping function y=f(x)y = f(x) from labeled training pairs (xi,yi)(x_i, y_i) to predict continuous outcomes (Regression) or discrete categories (Classification).
    • Unsupervised Learning: Discovering latent patterns, clusters, or distributions within unlabeled datasets {xi}\{x_i\} without predefined ground-truth targets.
    • Reinforcement Learning: An agent interacting with an environment learning an optimal policy π(a∣s)\pi(a|s) via trial-and-error to maximize cumulative discounted rewards over time.
  • Inductive Bias: The set of fundamental assumptions and preferences an algorithm makes beyond the observable training data in order to generalize to unseen instances (e.g., Occam’s razor preference for simpler decision trees).
  • Bias-Variance Tradeoff: A fundamental statistical tension where total expected prediction error decomposes into:
    Total Error=Bias2+Variance+Irreducible Noise\text{Total Error} = \text{Bias}^2 + \text{Variance} + \text{Irreducible Noise}
    • High Bias (Underfitting): Model is too simplistic, missing underlying trends.
    • High Variance (Overfitting): Model memorizes training noise, failing to generalize to validation data.

Part 2: Structural Frameworks (10 Marks)

1. The Machine Learning Engineering Lifecycle Pipeline

DATA COLLECTION ----> DATA PREPROCESSING ----> FEATURE ENGINEERING
Raw Logs, APIs, Imputation, Scaling Encoding, Selection
Databases, Sensors Outlier Handling Dimensionality Red.
MLOps & MONITORING <---- MODEL EVALUATION <---- MODEL TRAINING
Drift Detection, Cross-Validation, Gradient Descent,
Continuous CI/CD ROC-AUC, F1-Score Hyperparameter Tune

2. K-Fold Cross-Validation Framework (K=5K = 5)

  1. Dataset D partitioned into 5 equal folds
  2. Iteration 1: [ Test (F1) ] [ Train (F2) ] [ Train (F3) ] [ Train (F4) ] [ Train (F5) ] -> Score 1
  3. Iteration 2: [ Train (F1) ] [ Test (F2) ] [ Train (F3) ] [ Train (F4) ] [ Train (F5) ] -> Score 2
  4. Iteration 3: [ Train (F1) ] [ Train (F2) ] [ Test (F3) ] [ Train (F4) ] [ Train (F5) ] -> Score 3
  5. Iteration 4: [ Train (F1) ] [ Train (F2) ] [ Train (F3) ] [ Test (F4) ] [ Train (F5) ] -> Score 4
  6. Iteration 5: [ Train (F1) ] [ Train (F2) ] [ Train (F3) ] [ Train (F4) ] [ Test (F5) ] -> Score 5
  7. Overall CV Performance: E_CV = (1/K) * sum(Score_i)

Part 3: Distinction Tables

Basis of Comparison Parametric Machine Learning Models Non-Parametric Machine Learning Models
Model Structure Fixed, predetermined number of parameters regardless of data size Number of parameters grows flexibly with the training dataset size
Data Assumptions Makes strong distributional assumptions (e.g., linearity, normality) Makes very few or no rigid assumptions about data distribution
Training Speed Fast; requires optimizing fixed parameter vector Often slow; scales with O(N)O(N) or O(N2)O(N^2) dataset points
Memory Footprint Low; only stores learned weights θ\theta High; frequently needs entire training dataset in memory (e.g., KNN)
Risk Profile High bias if assumptions fail; low variance High variance and prone to overfitting if not regularized
Typical Examples Linear Regression, Logistic Regression, Linear SVM K-Nearest Neighbors (KNN), Decision Trees, RBF-SVM

Part 4: Worked Problems & Exam Traps

Problem Archetype: Decomposing Prediction Error and Combatting Overfitting

Question: A predictive model demonstrates 98.5%98.5\% accuracy on training data but drops to 67.2%67.2\% accuracy on the test set. Diagnose the model failure mode, explain the bias-variance state, and formulate four concrete engineering interventions to remediate this issue. Solution:

  1. Diagnosis:
    • High performance discrepancy indicates severe Overfitting characterized by Low Bias and High Variance.
    • The model has memorized idiosyncrasies, noise, and outliers of the training set rather than learning generalizable underlying patterns.
  2. Remediation Interventions:
    • Regularization: Introduce L1L_1 (Lasso) or L2L_2 (Ridge) penalties to shrink weights:
      J(θ)=MSE+λ∑θj2J(\theta) = \text{MSE} + \lambda \sum \theta_j^2
    • Data Augmentation / Acquisition: Collect additional training samples to penalize spurious correlations.
    • Dimensionality Reduction: Apply PCA or SelectKBest to prune irrelevant features.
    • Early Stopping / Pruning: Halt iterative training when validation loss begins to diverge, or prune decision tree depth (e.g., max_depth = 5).

TU Exam Trap: Students often confuse data leakage during preprocessing. If you normalize data (e.g., StandardScaler) using the entire dataset before splitting into train/test sets, test distribution information leaks into the training pipeline! You must fit scalers strictly on the training set only (scaler.fit_transform(X_train)), and then transform test data using those exact learned parameters (scaler.transform(X_test)).