Business Data Analytics

Descriptive vs. Prescriptive Analytics:

• Descriptive Analytics (“What happened?”): Summarizes historical raw data into understandable business reports, dashboards, and aggregations (e.g. monthly sales revenue, total website visitors).
• Prescriptive Analytics (“What should we do?”): Leverages advanced optimization algorithms, simulation models, and business rules to recommend the single best actionable course of action to capitalize on future opportunities or mitigate risk (e.g. dynamic airline ticket pricing, automated supply chain route optimization).

Feature Scaling Formulas:

• Min-Max Normalization (Rescaling to $[0, 1]$):

  Swipe to see the full formula↔

  $$x_{\text{norm}} = \frac{x - x_{\min}}{x_{\max} - x_{\min}}$$
• Z-Score Standardization (Zero mean, unit variance):

  Swipe to see the full formula↔

  $$z = \frac{x - \mu}{\sigma}$$
  Where $\mu$ is the feature mean and $\sigma$ is the standard deviation.

Precision vs. Recall:

• Precision: Proportion of positive identifications that were actually correct.

  Swipe to see the full formula↔

  $$\text{Precision} = \frac{TP}{TP + FP}$$
• Recall (Sensitivity): Proportion of actual positives that were identified correctly.

  Swipe to see the full formula↔

  $$\text{Recall} = \frac{TP}{TP + FN}$$
  Where $TP = \text{True Positives}$, $FP = \text{False Positives}$, $FN = \text{False Negatives}$.

Association Rule Lift:

A Lift of 2.5 indicates that customers who purchase item $A$ are 2.5 times more likely to purchase item $B$ than a random customer, demonstrating a strong, positive associative relationship between items $A$ and $B$.

RFM Model:

A behavioral customer segmentation framework based on three quantitative transaction metrics:

1. Recency ($R$): How recently did the customer make a purchase?
2. Frequency ($F$): How often do they purchase over a specified period?
3. Monetary Value ($M$): How much total money have they spent? Used to identify high-value VIP customers, churning loyalists, and low-engagement shoppers.

IQR Outlier Detection Method:

The Interquartile Range measures the statistical spread of the middle 50% of sorted data:

• $\text{Lower Bound} = Q_1 - 1.5 \times \text{IQR}$
• $\text{Upper Bound} = Q_3 + 1.5 \times \text{IQR}$ Any data point falling outside $[\text{Lower Bound}, \text{Upper Bound}]$ is flagged as an outlier.

Numerical Calculation ($n = 10$ sorted values):

• Lower half: $[50, 60, 70, 80, 90] \implies \mathbf{Q_1 = 70}$
• Upper half: $[100, 110, 120, 130, 450] \implies \mathbf{Q_3 = 120}$
• $\mathbf{\text{IQR}} = 120 - 70 = \mathbf{50}$

Bounds:

• $\text{Lower Bound} = 70 - (1.5 \times 50) = 70 - 75 = \mathbf{-5}$
• $\text{Upper Bound} = 120 + (1.5 \times 50) = 120 + 75 = \mathbf{195}$

Identification:

• The purchase value 450 exceeds the upper bound of 195 ($450 > 195$).
• Conclusion: 450 is a statistically verified extreme outlier.

Classification Metrics Calculation:

(a) Accuracy:

(b) Precision:

(c) Recall (Sensitivity):

(d) Specificity:

(e) F1-Score:

K-Means Clustering Algorithm:

An iterative, partition-based unsupervised learning algorithm that divides $n$ observations into $k$ distinct, non-overlapping clusters:

1. Initialize $k$ random cluster centroids.
2. Assignment Step: Assign each data point to its nearest centroid using Euclidean distance: $d(x, c) = \sqrt{\sum (x_i - c_i)^2}$.
3. Update Step: Recalculate each centroid as the arithmetic mean of all data points assigned to that cluster.
4. Repeat steps 2 and 3 until centroids converge (no reassignments).

Determining Optimal $k$:

• Elbow Method: Plots Within-Cluster Sum of Squares (WCSS / Inertia) against increasing values of $k$. As $k$ increases, WCSS decreases. The optimal $k$ is located at the ‘elbow point’ where the rate of decrease abruptly flattens.
• Silhouette Analysis: Measures how similar an object is to its own cluster compared to neighboring clusters (ranging from -1 to +1). The $k$ value with the highest average silhouette score represents optimal cluster cohesion and separation.

Four Components of a Time Series:

1. Trend ($T$): Long-term upward or downward movement in the series over several years (e.g. secular growth of smartphone transactions).
2. Seasonality ($S$): Repetitive, periodic fluctuations that recur at fixed calendar intervals (e.g. Dashain festival retail spikes every autumn).
3. Cyclical Fluctuations ($C$): Medium-to-long term swings corresponding to broader macroeconomic business cycles (expansion, recession).
4. Irregular / Noise ($I$): Unpredictable, random shocks caused by natural disasters, strikes, or political events.

Additive vs. Multiplicative Decomposition:

• Additive Model: $Y_t = T_t + S_t + C_t + I_t$. Used when the magnitude of seasonal variations remains constant regardless of the baseline level of the trend.
• Multiplicative Model: $Y_t = T_t \times S_t \times C_t \times I_t$. Used when seasonal swings increase proportionally in magnitude as the underlying trend grows.

Role of Apache Spark in Big Data Analytics:

Apache Spark is an open-source, distributed multi-language engine for executing data engineering, machine learning pipelines, and SQL queries on large-scale datasets.

Advantages over Traditional Hadoop MapReduce:

1. In-Memory Processing: Spark holds intermediate processing states and iterations in distributed RAM across cluster nodes, whereas MapReduce strictly writes intermediate results to physical disk (HDFS) after every map/reduce phase. Spark executes up to 100x faster for iterative machine learning algorithms.
2. Resilient Distributed Datasets (RDDs): Fault-tolerant collections of operational elements partitioned across cluster nodes. If a worker node crashes, Spark reconstructs lost partitions using the RDD lineage graph (DAG) without needing full data checkpoint reloads.
3. Unified API Stack: Integrates Spark SQL, Spark Streaming, MLlib (machine learning), and GraphX within a single cohesive programming model.

Ethical Principles & GDPR Compliance:

1. Consent & Purpose Limitation: Explicit, opt-in user consent must be obtained before tracking consumer behaviors. Collected data must strictly be used for the declared legitimate business purpose.
2. Data Minimization & Anonymization: Only store attributes essential for the analytical model. Strip Personally Identifiable Information (PII) using hashing or k-anonymity before training machine learning models.
3. Mitigating Algorithmic Bias: Training data frequently mirrors historical human prejudices (e.g. loan approval models rejecting marginalized zip codes). Data science teams must conduct fairness audits, disparate impact assessments, and remove proxy variables that reinforce discrimination.

(a) End-to-End Analytics Workflow:

1. Data Ingestion: Ingest raw logs from CDR servers, billing databases, customer service CRM tickets, and digital recharge portals into a cloud data lake.
2. Data Cleaning & Wrangling: Handle missing values (e.g. median imputation for monthly recharge), drop duplicate records, and remove test accounts.
3. Feature Engineering: Aggregate transactional behaviors over 30/60/90-day rolling windows; encode categorical data via One-Hot encoding; scale continuous features via standardization.
4. Model Training & Cross-Validation: Split data into 70% Train, 15% Validation, and 15% Test. Handle class imbalance (25% churn vs 75% non-churn) using SMOTE or class-weight penalization.
5. Model Evaluation: Evaluate using Precision-Recall Area Under Curve (PR-AUC) and F1-Score rather than raw accuracy.

---

(b) Five Predictive Features:

1. Recharge Frequency Drop ($F_{\Delta}$): Ratio of recharges in the last 30 days compared to the trailing 90-day average.
2. Customer Service Grievance Volume: Number of unresolved network complaints lodged in the past 14 days.
3. Data Consumption Trend: Percentage decline in daily 4G data volume (MB) over the last month.
4. Tenure (Months): Duration since SIM card registration.
5. Competitor Cross-Calling Percentage: Proportion of outbound calls made to competing mobile networks.

---

(c) Logistic Regression vs. Random Forest:

• Logistic Regression: Linear, highly interpretable; produces well-calibrated probabilities and explicit feature odds ratios ($e^\beta$). Struggles to capture complex non-linear feature interactions.
• Random Forest: Ensemble of decision trees; naturally captures non-linear relationships and feature interactions without requiring manual polynomial transformations. Less prone to overfitting than single trees. Provides explicit Gini feature importance.

---

(d) Translating Churn Probabilities to Business Action:

• Set classification threshold based on cost-benefit matrix: cost of retention offer (e.g. free data pack worth NPR 100) vs loss of customer lifetime value ($CLV = \text{NPR } 3,500$).
• Segment high-risk churners ($P(\text{churn}) > 0.70$) into automated CRM workflows: send personalized SMS bonus offers, automated discount vouchers on digital recharge apps, and prioritize VIP customer care outreach.

Step 1: Minimum Support Threshold

Total transactions $N = 5$.

---

Step 2: Apriori Frequent Itemset Generation

Candidate 1-Itemsets ($C_1$):

• {Milk}: 4 (80%) $\ge 2$ (Frequent $L_1$)
• {Bread}: 4 (80%) $\ge 2$ (Frequent $L_1$)
• {Butter}: 4 (80%) $\ge 2$ (Frequent $L_1$)
• {Eggs}: 2 (40%) $\ge 2$ (Frequent $L_1$)

Candidate 2-Itemsets ($C_2$):

• {Milk, Bread}: T1, T3, T5 $\implies \text{Count } 3 \ge 2$ (Frequent $L_2$)
• {Milk, Butter}: T1, T4, T5 $\implies \text{Count } 3 \ge 2$ (Frequent $L_2$)
• {Milk, Eggs}: T4, T5 $\implies \text{Count } 2 \ge 2$ (Frequent $L_2$)
• {Bread, Butter}: T1, T2, T5 $\implies \text{Count } 3 \ge 2$ (Frequent $L_2$)
• {Bread, Eggs}: T5 $\implies \text{Count } 1 < 2$ (Pruned)
• {Butter, Eggs}: T4, T5 $\implies \text{Count } 2 \ge 2$ (Frequent $L_2$)

Candidate 3-Itemsets ($C_3$):

• {Milk, Bread, Butter}: T1, T5 $\implies \text{Count } 2 \ge 2$ (Frequent $L_3$)
• {Milk, Butter, Eggs}: T4, T5 $\implies \text{Count } 2 \ge 2$ (Frequent $L_3$)

---

Step 3: Analysis of Rule $\text{Bread, Butter} \rightarrow \text{Milk}$

• Support count of {Bread, Butter, Milk} = 2
• Support count of {Bread, Butter} = 3
• Support count of {Milk} = 4

Metrics:

Since Confidence ($66.67\%$) exceeds Minimum Confidence ($60\%$), this is a valid strong association rule.

---

Step 4: Managerial Interpretation & Retail Merchandising Decision:

• Lift Analysis: The Lift value is $0.833 < 1.0$.
• A Lift less than 1 indicates that purchasing Bread and Butter actually has a negative associative correlation with Milk compared to the baseline purchase rate of Milk (Milk is bought by 80% of all shoppers, but only by 66.7% of Bread+Butter buyers).
• Retail Decision: Supermarket managers should NOT place Milk immediately next to Bread and Butter. Since Milk is a universal destination staple, it should be placed at the far back of the supermarket so that shoppers walk through aisles of high-margin impulse items to reach it.