Climate Change Twitter Big Data Analysis using Apache Spark and Machine Learning

Project Overview

This project presents a complete Big Data Analytics and Machine Learning pipeline for analysing the Climate Change Twitter Dataset using Apache Spark (PySpark). The workflow covers data loading, preprocessing, feature engineering, exploratory data analysis (EDA), machine learning model development, hyperparameter tuning, distributed computing, explainable AI (LIME), scalability analysis, and Tableau dashboard preparation.

Objectives

Process large-scale Twitter data with Apache Spark.
Clean and engineer features for ML.
Perform exploratory data analysis.
Train and compare multiple classification models.
Tune models using cross-validation.
Demonstrate Spark distributed computing.
Evaluate scalability using strong and weak scaling.
Explain predictions using LIME.
Export Tableau-ready CSV files.

Dataset

Dataset: The Climate Change Twitter Dataset

Source: https://data.mendeley.com/datasets/mw8yd7z9wc/2

Technologies

Python
Apache Spark (PySpark)
Spark MLlib
Pandas
NumPy
Matplotlib
Seaborn
Scikit-learn
LIME
Tableau

Project Tasks

Task 1 -- Big Data Loading

Configure Spark Session
Load dataset
Inspect schema
Display sample data
Verify partitions
Check dataset size

Task 2 -- Data Engineering

Remove invalid records
Create binary label
Generate time-based features
Handle missing values
Encode categorical variables
Scale numerical features

Exploratory Data Analysis

Class distribution
Topic vs aggressiveness
Temperature distribution
Correlation heatmap

Task 3 -- Machine Learning

Implemented models: - Logistic Regression - Decision Tree - Random Forest - Gradient Boosted Trees

Evaluation metrics: - Accuracy - Precision - Recall - ROC-AUC - Confusion Matrix - Classification Report - ROC Curves

Feature importance is extracted from Random Forest and Gradient Boosting.

Hyperparameter Tuning

Cross-validation is performed for all models using parameter grids.

Task 4 -- Distributed Computing

Spark resource configuration
Partition analysis
Repartitioning
Cache vs non-cache comparison
Spark UI monitoring

Task 5 -- Explainable AI

LIME is used to explain individual Random Forest predictions and visualize feature contributions.

Scalability Analysis

Strong Scaling
Weak Scaling
Cost vs Performance

Tableau Outputs

Generated CSV files include: - tableau_data_quality.csv - tableau_business_insights.csv - tableau_scalability_cost.csv

Workflow

Dataset
   ↓
Data Cleaning
   ↓
Feature Engineering
   ↓
Encoding & Scaling
   ↓
Train/Test Split
   ↓
Machine Learning Models
   ↓
Hyperparameter Tuning
   ↓
Performance Evaluation
   ↓
Explainable AI (LIME)
   ↓
Distributed Computing
   ↓
Scalability Analysis
   ↓
Tableau Dashboard

Project Structure

ClimateChange-BigData/
│
├── Climate Change Twitter Dataset.csv
├── main.ipynb
├── README.md
├── tableau_data_quality.csv
├── tableau_business_insights.csv
├── tableau_scalability_cost.csv
└── figures/

Results

Spark efficiently processes large-scale Twitter data.
Ensemble models outperform baseline classifiers.
Hyperparameter tuning improves predictive performance.
Caching reduces execution time.
LIME improves model interpretability.
Tableau dashboards support interactive analysis.

Future Work

SHAP explainability
BERT-based NLP models
Spark Streaming
MLflow deployment
Cloud deployment (AWS/Databricks)

koreb/7006SCN_Bhoomi

About

Resources

Stars

Watchers

Forks

Releases

Languages