7006SCN_ES_123456789_1

Online Retail II - PySpark Customer Analytics & Segmentation

A comprehensive Big Data Analytics project using PySpark on the Online Retail II dataset. The project demonstrates end-to-end data processing, machine learning, model explainability, and business intelligence through interactive Tableau dashboards.

📋 Project Overview

This project analyzes over 541,000 transactions from a UK-based online retailer (2009–2010) to perform customer segmentation, identify high-value customers, and generate actionable business insights using distributed computing with Apache Spark.

🎯 Objectives

Perform large-scale data ingestion and preprocessing using PySpark
Build RFM-based customer segmentation
Train and compare 4 Machine Learning models
Evaluate model performance, stability, and explainability (SHAP)
Create interactive Tableau dashboards for business stakeholders
Analyze scalability and cost implications

🛠️ Tech Stack

Apache Spark 3.5.3 (PySpark)
Python + Pandas + Matplotlib + Seaborn
UCI Machine Learning Repository Dataset
Tableau (for Business Intelligence Dashboards)
SHAP (Model Explainability)

📁 Project Structure

📊 Key Features

Data Quality Analysis with missing value handling
RFM Analysis for customer segmentation
Four ML Models: Logistic Regression, Random Forest, GBT, Linear SVC
Model Explainability using SHAP
Interactive Tableau Dashboards (4 dashboards)
Scalability & Cost Analysis

🚀 How to Run

Open any notebook in Google Colab
Run the installation cells
Upload online_retail_II.xlsx to Colab or use the provided path
Execute notebooks sequentially (Task 1 → Task 5)

📈 Business Insights

UK dominates revenue (85%+)
High-Value customers drive majority of revenue
Strong seasonality in November–December
Monetary value is the strongest predictor of customer lifetime value

📌 Dashboards

Dashboard 1: Data Quality & Pipeline Monitoring
Dashboard 2: Model Performance & Feature Importance
Dashboard 3: Business Insights
Dashboard 4: Scalability & Cost Analysis

🧠 Key Learnings

Efficient distributed data processing with PySpark
Importance of proper caching and repartitioning
Tree-based models outperform linear models on RFM data
SHAP provides valuable interpretability for business decisions

Made with ❤️ using PySpark & Tableau

jithdevj/7006SCN_ES_123456789_1

About

Resources

Stars

Watchers

Forks

Releases

Languages