7006SCN_Shruthi_16893728

HIGGS Dataset Classification using PySpark and Machine Learning

Project Overview

This project presents a large-scale machine learning solution for binary classification using the HIGGS dataset. The objective is to distinguish between Higgs boson signal events and background events generated from particle collision experiments. The dataset contains approximately 11 million records and 29 attributes, making it suitable for Big Data analytics and distributed machine learning.

Apache Spark and PySpark were used for data engineering, preprocessing, feature transformation, model training, and evaluation. A scalable machine learning pipeline was developed using VectorAssembler, StandardScaler, VectorSlicer, and Polynomial Expansion techniques. Four machine learning algorithms were implemented and compared: Logistic Regression, Decision Tree, Linear SVM, and Gradient-Boosted Trees.

Model performance was evaluated using Accuracy, Precision, Recall, F1-Score, and AUC-ROC metrics. Gradient-Boosted Trees achieved the best overall predictive performance. Additional analysis included distributed computing optimisation, Spark UI monitoring, stability testing, SHAP-based explainability, and Tableau dashboards for data storytelling and business insights.

This project demonstrates how Big Data technologies can be combined with machine learning to efficiently process and analyse large datasets in a distributed environment. :contentReference[oaicite:0]{index=0}

7006SCN2526MAYJUL/7006SCN_Shruthi_16893728

About

Resources

Stars

Watchers

Forks

Releases