7006SCN — Machine Learning and Big Data

Student: Rajnayan Bhagat | SID: 17042226 | Module: 7006SCN

Project

This project builds an end-to-end Big Data machine learning pipeline in PySpark to classify Higgs boson signal events from background particle collision events using the UCI HIGGS dataset in a Google Colab runtime.

Dataset

Property	Value
Name	HIGGS Dataset
Source	UCI Machine Learning Repository — https://archive.ics.uci.edu/dataset/280/higgs
Rows	11 million
Features	28 (21 low-level kinematic properties measured directly by particle detectors and 7 high-level features derived by physicists)
License	CC BY 4.0
Not Kaggle	The dataset was downloaded directly from the UCI Machine Learning Repository (archive.ics.uci.edu), an open academic data source maintained by the University of California, Irvine — completely independent of Kaggle
Not teaching	Not used in any 7006SCN lab session
Tableau Public URL	https://public.tableau.com/app/profile/rajnayan.bhagat/vizzes

Task Progress

Task	Notebook	Commits	Comments
1	Task_1.ipynb	3	Loads the HIGGS dataset into a PySpark DataFrame, inspects the schema, verifies file size and justifies the dataset against the 5 V's of Big Data. Includes ethical and licensing considerations.
2	Task_2.ipynb	3	Performs missing value checks, constructs a 5-stage PySpark Pipeline: VectorAssembler → StandardScaler → VectorSlicer → PolynomialExpansion → VectorAssembler. Produces two feature vectors — `features_28_scaled` for tree-based models and `features_63_LR` for Logistic Regression. Dataset sampled to 5% and split 80/20 into training and test sets.
3	Task_3.ipynb	12	Trains four binary classification models — Logistic Regression, Decision Tree, Random Forest and Gradient Boosted Trees (GBT). Hyperparameter tuning performed on a 5% tuning sample using CrossValidator (LR, DT) and TrainValidationSplit (RF, GBT). Best parameters used to refit each model on the full training set. Results summarised in a metrics table including AUC-ROC, Accuracy, F1, Precision, Recall and Training Time.
4	Task_4.ipynb	3	Accesses Spark UI via ngrok tunnel and REST API to evidence job completion, stage distribution and executor resource usage. Documents all caching and persisting decisions using StorageLevel.MEMORY_AND_DISK. Discusses repartitioning strategy and resource configuration on Colab's free tier.
5	Task_5.ipynb	5	Compares all four models using a full metrics table, confusion matrices, ROC curves and Precision-Recall curves. Perturbation analysis conducted by injecting 5% Gaussian noise into feature vectors to measure prediction agreement rate stability. SHAP explainability applied via a sklearn surrogate Random Forest trained on GBT predictions to identify the most influential features.
6	Task_6.ipynb	3	Exports pipeline monitoring data, model metrics, feature importances, GBT prediction confidence scores and scalability data to CSV for Tableau. Four dashboards built and published on Tableau Public Dashboard 1: Data Quality & Pipeline Monitoring Dashboard 2: Model Performance & Feature Importance Dashboard 3: Business Insights Dashboard 4: Scalability

Branch Structure

Each task has a dedicated branch that was merged into main after completion.

Branch	Topic
`main`	Stable, submission-ready code
`Task1`	Problem & Dataset
`Task2`	Data Engineering
`Task3`	ML Model Portfolio
`Task4`	Distributed Computing
`Task5`	Model Evaluation & Stability
`Task6`	Tableau Storytelling

Repository Structure

7006SCN_ES_123456789/
│
├── notebooks/ ← main branch
│ ├── Task1.ipynb           # Problem & Dataset
│ ├── Task2.ipynb           # Data Engineering
│ ├── Task3.ipynb           # ML Model Portfolio
│ ├── Task4.ipynb           # Distributed Computing
│ ├── Task5.ipynb           # Model Evaluation
│ └── Task6.ipynb           # Tableau Storytelling
│ └── outputs/                # Tableau-ready CSVs, SHAP plots (generated by notebooks)
│             ├── file_scalability.csv                 # Scalability
│             ├── file_class_distribution.csv     # Business Insights
│             ├── file_gbt_predictions.csv        # Business Insights
│             ├── file_exported_metrics.csv     # Metrics and feature importances
│             ├── file_feature_importance.csv # Metrics and feature importances
│             └── file_pipeline_monitoring.csv # Data Quality & Pipeline Monitoring
└── README.md ← this file

Running Order

Run notebooks in task order — each task depends on outputs from the previous:

1. Download the dataset

https://archive.ics.uci.edu/dataset/280/higgs Extract the gz file and upload to Google Drive.

2. Colab runtime Setup

Install Spark on your Colab runtime and connect it to the Google Drive where the extracted dataset is uploaded using code in Task_1.ipynb. Set the higgs_path variable to the directory on your Google Drive where the extracted dataset resides.

3. Run in order (each saves outputs/ for the next task)

notebooks/Task_1.ipynb notebooks/Task_2.ipynb notebooks/Task_3.ipynb notebooks/Task_4.ipynb notebooks/Task_5.ipynb notebooks/Task_6.ipynb

References

Baldi, P., Sadowski, P. and Whiteson, D. (2014) Searching for exotic particles in high-energy physics with deep learning, Nature Communications. Available at: https://doi.org/10.1038/ncomms5308
UCI Machine Learning Repository (2014) HIGGS Dataset. Available at: https://archive.ics.uci.edu/dataset/280/higgs
Apache Software Foundation (2024) PySpark Documentation. Available at: https://spark.apache.org/docs/latest/api/python/

AI-Use Declaration

Claude (Anthropic) was consulted to clarify PySpark functions, debug errors and refine report phrasing. All code was independently written, tested and executed in Google Colab. All AI-generated suggestions were critically reviewed and verified against actual notebook outputs before inclusion.

bhagatr/7006SCN_RB_17042226

About

Resources

Stars

Watchers

Forks

Releases

Languages