Data Warehousing and Data Mining

Pass path: Learn the KDD process steps (Selection, Preprocessing, Transformation, Data Mining, Interpretation/Evaluation) and major distance metrics (Euclidean, Manhattan, Cosine, Jaccard). Top path: Master data preprocessing pipelines (handling missing values, binning, min-max vs z-score normalization) and dimensionality reduction via Principal Component Analysis (PCA).

Part 1: Core Concepts (2 Marks)

• ★ Knowledge Discovery in Databases (KDD): The iterative process of discovering valid, novel, potentially useful, and ultimately understandable patterns and knowledge from massive volumes of structured or unstructured data.
• ★ Euclidean vs Manhattan Distance: For vectors $\mathbf{x} = (x_1, \dots, x_n)$ and $\mathbf{y} = (y_1, \dots, y_n)$: Euclidean distance is $d(\mathbf{x}, \mathbf{y}) = \sqrt{\sum_{i=1}^n (x_i - y_i)^2}$, while Manhattan distance (L1 norm) is $d(\mathbf{x}, \mathbf{y}) = \sum_{i=1}^n |x_i - y_i|$.
• ★ Min-Max Normalization: A linear data transformation mapping values from an initial range $[\min_A, \max_A]$ to a target range $[new\_\min_A, new\_\max_A]$ via $v' = \frac{v - \min_A}{\max_A - \min_A} (new\_\max_A - new\_\min_A) + new\_\min_A$.

Part 2: Structural Frameworks (10 Marks)

1. The KDD Process Lifecycle and Comprehensive Data Preprocessing Pipeline

• Introduction: Define Data Mining as the core computational step within the broader multidisciplinary KDD framework (combining statistics, machine learning, and database systems).
• The Seven Stages of KDD:
  1. Data Cleaning: Removing noise, handling inconsistent records, and imputing missing values.
  2. Data Integration: Combining heterogeneous data sources (multiple relational databases, flat files, data cubes) while resolving schema collisions.
  3. Data Selection: Retrieving data subsets relevant to the analysis task from the operational data store.
  4. Data Transformation: Consolidating data into formats suitable for mining (normalization, aggregation, feature construction).
  5. Data Mining: Essential algorithmic phase extracting hidden data patterns (classification, clustering, association rules).
  6. Pattern Evaluation: Identifying truly interesting patterns representing actionable knowledge using statistical thresholds.
  7. Knowledge Presentation: Visualizing discovered insights to business stakeholders (charts, dashboards, decision trees).
• Data Preprocessing Techniques:
  • Data Cleaning: Methods for missing values (Ignore tuple, manual fill, global constant, attribute mean/median, most probable value via decision tree imputation).
  • Data Transformation:
    • Min-Max Normalization: Preserves exact relationships among data values; maps to $[0, 1]$.
    • Z-Score Normalization (Standardization): $v' = \frac{v - \mu}{\sigma}$, ideal when outliers exist or min/max are unknown.
    • Decimal Scaling: Normalizes by moving decimal points based on $\max(|v|)$.
  • Data Reduction:
    • Dimensionality Reduction: PCA (orthogonal projection maximizing variance), Wavelet transforms, attribute subset selection.
    • Numerosity Reduction: Parametric models (regression, log-linear) and non-parametric models (histograms, clustering, sampling).
• Conclusion: Unprocessed raw data is typically dirty, incomplete, and high-dimensional; preprocessing consumes up to 80% of total project effort but directly dictates downstream algorithmic accuracy.

Part 3: Distinction Tables

Part 4: Worked Problems & Exam Traps

• Problem 1 (Easy - Distance Calculation): Given two data points in 2D space: $P_1 = (2, 3)$ and $P_2 = (6, 6)$, calculate:
  1. Euclidean Distance
  2. Manhattan Distance
  • Solution:
    1. $\text{Euclidean Distance} = \sqrt{(6 - 2)^2 + (6 - 3)^2} = \sqrt{4^2 + 3^2} = \sqrt{16 + 9} = \sqrt{25} = 5.0$
    2. $\text{Manhattan Distance} = |6 - 2| + |6 - 3| = 4 + 3 = 7.0$
• Problem 2 (Exam-Level Normalization): Given the age attribute values: $[13, 15, 16, 16, 19, 20, 20, 21, 22, 22, 25, 25, 25, 30, 33, 35, 35, 40, 45, 70]$. Normalize the value $v = 35$:
  1. Using Min-Max Normalization onto the range $[0, 1]$.
  2. Using Z-Score Normalization where the mean $\mu = 26.85$ and standard deviation $\sigma = 12.8$.
  • Solution:
    1. Min-Max: $\min = 13$, $\max = 70$.

       Swipe to see the full formula↔

       $$v' = \frac{35 - 13}{70 - 13} \times (1 - 0) + 0 = \frac{22}{57} \approx 0.386$$
    2. Z-Score:

       Swipe to see the full formula↔

       $$v' = \frac{35 - 26.85}{12.8} = \frac{8.15}{12.8} \approx 0.637$$
• Problem 3 (Twist / Case Scenario): Why is Cosine Similarity preferred over Euclidean Distance when measuring the similarity between two text document vectors in information retrieval?
  • Solution:
    1. Euclidean distance measures the absolute geometric distance between vectors in multidimensional term-frequency space; it is heavily distorted by document length.
    2. A short article about cricket and a 100-page book about cricket share identical topic distributions, but Euclidean distance between them is massive because the book has 100x higher word counts.
    3. Cosine similarity measures the cosine of the angle between the two vectors (direction, not magnitude): $\cos(\theta) = \frac{\mathbf{A} \cdot \mathbf{B}}{\|\mathbf{A}\| \|\mathbf{B}\|}$, normalizing out document length completely.
  • Students lose marks here because they describe data mining as the entire KDD process (data mining is only one specific algorithmic phase of KDD), or they fail to square the differences in Euclidean distance.