Machine Learning and Big Data (7006SCN)

Coursework Title

Applied Core Assessment – Machine Learning and Big Data

Student Information

**Student Name: Harshavardhan Bandarapu

**Student ID: 17143062

Module: 7006SCN – Machine Learning and Big Data

Project Overview

This project applies distributed machine learning techniques using PySpark to solve a real-world product demand classification problem. The analysis is performed on the Geographic Product Demand dataset containing over 10 million records. The complete machine learning workflow includes data ingestion, preprocessing, feature engineering, model development, distributed computing optimisation, model evaluation, and Tableau visualisation.

Dataset

Dataset: Geographic Product Demand

Source: https://huggingface.co/datasets/abhinavsandhu/geographic-product-demand

Dataset Characteristics

Approximately 10 million records
11 attributes
Large-scale structured dataset
Suitable for distributed machine learning
Entire pipeline implemented using PySpark

Repository Structure

README.md

/notebooks
│── Task1.ipynb
│── Task2.ipynb
│── Task3.ipynb
│── Task4.ipynb
│── Task5.ipynb

/Tableau
│── Task6.twbx

Task Summary

Task 1 – Problem & Dataset

Dataset loading using Spark
Schema inspection
Big Data justification (5 Vs)
Ethical and licensing considerations

Task 2 – Data Engineering

Data preprocessing
Missing value verification
Duplicate detection
Feature engineering
PySpark Pipeline construction

Task 3 – Machine Learning Models

The following classification models are implemented:

Logistic Regression
Decision Tree
Random Forest
Linear Support Vector Classifier

Hyperparameter tuning is performed using CrossValidator and ParamGridBuilder.

Task 4 – Distributed Computing

Data repartitioning
Cache optimisation
Persist optimisation
Spark UI analysis
Resource configuration

Task 5 – Model Evaluation

Accuracy
Precision
Recall
F1-score
AUC-ROC
Confusion matrices
ROC curves
Precision-Recall curves
Stability analysis
Feature importance analysis

Task 6 – Tableau Storytelling

The Tableau workbook contains four dashboards:

Data Quality & Pipeline Monitoring
Model Performance & Feature Importance
Business Insights
Scalability & Cost Analysis

Software and Libraries

Python 3
Apache Spark
PySpark MLlib
Pandas
NumPy
Matplotlib
Scikit-learn
Tableau Public

Reproducibility

The notebooks are designed to be executed sequentially:

Task1.ipynb
Task2.ipynb
Task3.ipynb
Task4.ipynb
Task5.ipynb

The Tableau workbook visualises the processed outputs generated during the machine learning workflow.

Author

Harshavardhan Bandarapu

Coventry University

7006SCN – Machine Learning and Big Data

7006SCN2526MAYJUL/7006SCN_HB_17143062

About

Resources

Stars

Watchers

Forks

Releases

Languages