Climate TRACE Emissions Clustering - Big Data Analytics Project

Research Question

Can emissions profiles cluster sources into distinct environmental impact categories?

Business Framing

Climate policy targeting and emissions monitoring. By clustering emission sources based on their profiles (capacity, emissions intensity, sector, gas type), policymakers can identify distinct environmental impact categories and design targeted interventions.

Dataset

Source: Climate TRACE (datania/climate-trace)

188.8 million rows, 16 columns, 1.58 GB (Parquet format)
Emissions data from Spain (ESP), 2015-2025, monthly granularity
All columns stored as strings (requires type casting)
Key columns: source_id, source_name, iso3_country, sector, subsector, source_type, capacity, capacity_factor, activity, gas (CO2, CH4, N2O, etc.), emissions_factor, emissions_quantity, start_time, end_time

Algorithms

Algorithm	Library	Environment	Purpose
KMeans	PySpark MLlib	CPU	Centroid-based baseline
BisectingKMeans	PySpark MLlib	CPU	Hierarchical divisive
GaussianMixture	PySpark MLlib	CPU	Probabilistic soft clustering
DBSCAN	cuML (RAPIDS)	GPU (A100)	Density-based, noise handling
HDBSCAN	cuML (RAPIDS)	GPU (A100)	Hierarchical density-based
MiniBatchKMeans	scikit-learn	CPU	Scalable CPU baseline

Custom Transformer

EmissionsIntensityCalc normalizes emissions by capacity and computes:

emissions_per_unit_capacity: emissions_quantity / capacity (safe division)
intensity_ratio: emissions_factor * capacity_factor
sector_share: emissions_quantity / total_emissions_in_sector

Project Structure

9th_copy/
|-- notebooks/
|   |-- 1_data_ingestion.ipynb       # Load, sample, validate, cast, save
|   |-- 2_feature_engineering.ipynb   # Encode, derive features, PCA, scale
|   |-- 3_model_training.ipynb        # Train all 6 algorithms, CrossValidator
|   |-- 4_evaluation.ipynb            # Metrics, bootstrap CI, Tableau export
|-- scripts/
|   |-- setup_environment.sh          # Environment setup for Colab Pro
|   |-- run_pipeline.py               # Pipeline orchestrator
|   |-- performance_profiler.py       # CPU/GPU performance profiling
|-- config/
|   |-- spark_config.yaml             # Spark and model configuration
|   |-- tableau_config.json           # Tableau dashboard configuration
|-- data/
|   |-- schemas/                      # Data schema definitions
|   |-- samples/                      # Sample data files
|-- tests/
|   |-- test_pipeline.py              # Unit tests for all pipeline stages
|-- tableau/
|   |-- README_tableau.md             # Tableau dashboard guide
|-- .gitignore
|-- environment.yml                   # Conda environment specification
|-- Dockerfile                        # Container definition
|-- README.md                         # This file

Execution Environment

Platform: Google Colab Pro with NVIDIA A100 GPU
PySpark: 3.5.8
cuML/cuDF: 26.02
Python: 3.10

Sampling Strategy

The dataset has 188.8M rows, which exceeds memory constraints. The sampling strategy:

Spark operations: 2M rows (streamed from HuggingFace)
GPU operations: 500K rows (subsampled from Spark data)
Visualization: 20K rows (for plot clarity)

Quick Start

Google Colab

Upload notebooks to Google Colab
Select A100 GPU runtime
Run notebooks in order (1 through 4)

Local / Docker

docker build -t climate-clustering .
docker run --gpus all -p 8888:8888 climate-clustering

Pipeline Orchestration

python scripts/run_pipeline.py --stages all
python scripts/run_pipeline.py --stages ingest,features

Running Tests

python -m pytest tests/test_pipeline.py -v

Key Hyperparameters

KMeans k range: [4, 12]
DBSCAN eps values: [0.4, 0.7, 1.1]
HDBSCAN min_cluster_size: [80, 150, 300]
PCA components: 10
CrossValidator folds: 3
Bootstrap iterations: 50

Output Artifacts

Partitioned Parquet (by sector)
Trained MLlib models (KMeans, BisectingKMeans, GMM)
Tableau CSV exports (6 files)
Visualization PNGs (5 charts)
Performance profiling report
Training results JSON

Version Information

Component	Version
PySpark	3.5.8
cuML	26.02
cuDF	26.02
scikit-learn	>=1.3
Python	3.10

endlah/7006SCN-Code-

About

Resources

Stars

Watchers

Forks

Releases

Languages