Project Overview

This project applies Big Data analytics and Machine Learning techniques to optimise smart waste management in Dhaka city. The objective is to predict whether a waste bin requires collection using operational, environmental and geographical information collected from smart waste bins.

PySpark was used to implement a complete end-to-end Big Data pipeline including data ingestion, preprocessing, feature engineering, model training, distributed computing optimisation, model evaluation and Tableau storytelling.

The project demonstrates scalable machine learning techniques suitable for urban waste management systems.

Dataset Information

Dataset: Dhaka Smart Waste 10M Ultra

Source: Mendeley Data

Link: https://data.mendeley.com/datasets/ctt5kwppwt/2/files/2aa95e7a-af16-4933-9199-0756a4b2faba

Dataset Characteristics

10 million rows
More than 20 features
Structured tabular dataset
Suitable for classification and clustering tasks
Synthetic dataset with no personal information

Repository Structure

notebooks/

├── Task1.ipynb
├── Task2.ipynb
├── Task3.ipynb
├── Task4.ipynb
├── Task5.ipynb
├── Task6.ipynb

README.md

Technologies Used

Python
PySpark
Pandas
NumPy
Matplotlib
Seaborn
SHAP
Tableau Public
GitHub

Machine Learning Models

Supervised Models

Logistic Regression
Decision Tree
Random Forest
Gradient Boosted Trees (GBT)

Unsupervised Model

K-Means Clustering

Project Workflow

Task 1

Problem definition
Dataset exploration
Big Data 5Vs analysis
Ethical and licensing considerations

Task 2

Data preprocessing
Missing value handling
Feature engineering
PySpark pipeline construction

Task 3

Machine learning model development
Hyperparameter tuning
Model comparison

Task 4

Distributed computing optimisation
Repartitioning
Caching and persisting
Spark UI analysis

Task 5

Model evaluation
Confusion matrices
ROC curves
Precision-Recall curves
Stability analysis
SHAP explainability

Task 6

Tableau storytelling dashboards

Tableau Dashboards

Data Quality & Pipeline Monitoring
Model Performance & Feature Importance
Business Insights
Scalability & Cost Analysis

Ethical Considerations

The dataset was obtained from Mendeley Data and used solely for academic purposes. Since the dataset is synthetic and contains no personally identifiable information, privacy risks are minimal. Results were interpreted responsibly to avoid misleading waste management decisions.

7006SCN2526MAYJUL/7006SCN_manisha_16939613

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Contributors 1

Languages