Fraud Detection Using Distributed PySpark ML

Overview

This project implements large-scale financial fraud detection using distributed machine learning with PySpark MLlib. The dataset exceeds 1GB and contains millions of transactional records with severe class imbalance (approximately 0.13% fraud rate).

The objective is to evaluate the trade-off between predictive performance and computational scalability in a Big Data environment.

The system includes:

Distributed data engineering using Spark
Financial feature engineering
Multiple machine learning models
Imbalance-aware evaluation
Scalability analysis
Model serialization for reproducibility

Dataset

Source:
CiferAI Fraud Detection Dataset
https://huggingface.co/datasets/CiferAI/Cifer-Fraud-Detection-Dataset-AF

Characteristics

Millions of transactions
Structured tabular format
Severe class imbalance
Financially sensitive transactional attributes

The dataset is stored in Parquet format for optimized distributed processing.

Feature Engineering

Engineered financial consistency features:

origin_balance_diff
dest_balance_diff
amount_to_origin_ratio
amount_to_dest_ratio

Categorical transaction types were encoded using:

StringIndexer
OneHotEncoder

Feature vectors were assembled using Spark’s VectorAssembler.

Models Implemented

Distributed (PySpark MLlib)

Logistic Regression (weighted)
Random Forest
Tuned Random Forest
Gradient Boosted Trees

Single-node Baseline

Scikit-learn Logistic Regression

Class weights were applied to address extreme imbalance.

Evaluation Metrics

Models were evaluated using:

Accuracy
Precision
Recall
F1 Score
ROC-AUC
PR-AUC

Due to severe class imbalance, PR-AUC and Recall were prioritised over Accuracy.

Threshold sensitivity analysis was performed for ensemble models.

Scalability Analysis

Training time comparison:

Model	Training Time (seconds)
Logistic Regression	~16
Gradient Boosted Trees	~334
Random Forest	~540

Ensemble models improved Recall but required substantially higher computational cost.

Model Serialization

Saved models are located in:

spark_logistic_regression/
spark_random_forest/
spark_gbt/
sklearn_logistic_regression.pkl

Spark models are saved using native MLlib format.
The scikit-learn model is serialized using pickle.

Key Insights

Extreme class imbalance is the dominant performance constraint.
Engineered balance-based features outperform raw transaction categories.
Ensemble models provide incremental improvement over linear baselines.
Distributed ML enables scalability but does not eliminate imbalance challenges.
Cost-performance trade-offs are critical in operational fraud systems.

bandarum/Fraud_detection_ML_Big_Data_CW

About

Resources

Stars

Watchers

Forks

Releases

Languages