NigerianFraudClustering

Big Data clustering analysis on the Nigerian Financial Transactions and Fraud Detection Dataset (5M rows, 45 columns, 1.88GB).

Research Question

Can transaction behavioral profiles identify fraud typologies?

Business Context

Banking fraud detection and risk profiling. By clustering transaction behavioral features, we identify distinct fraud typologies that can inform risk scoring and alert systems.

Dataset

Source: HuggingFace - Nigerian Financial Transactions

5 million transactions
45 columns including amount, transaction type, location, fraud labels
Key risk scores: spending_deviation_score, velocity_score, geo_anomaly_score, channel_risk_score, merchant_fraud_rate

Algorithms

Algorithm	Framework	Type
KMeans	PySpark MLlib	Centroid-based
BisectingKMeans	PySpark MLlib	Hierarchical centroid
GaussianMixture	PySpark MLlib	Model-based
HDBSCAN	cuML (GPU)	Density-based
AgglomerativeClustering	cuML (GPU)	Hierarchical
DBSCAN	scikit-learn (CPU)	Density-based

Pipeline

Data Ingestion - Load from HuggingFace, validate schema, save as partitioned Parquet
Feature Engineering - Cleaning, encoding, custom features, StandardScaler, t-SNE
Model Training - All 6 algorithms with hyperparameter tuning and GPU/CPU comparison
Evaluation - Silhouette, Calinski-Harabasz, Davies-Bouldin, bootstrap CI, stability (ARI)

Project Structure

2nd_copy/
  notebooks/
    1_data_ingestion.ipynb
    2_feature_engineering.ipynb
    3_model_training.ipynb
    4_evaluation.ipynb
  scripts/
    setup_environment.sh
    run_pipeline.py
    performance_profiler.py
  config/
    spark_config.yaml
    tableau_config.json
  data/
    schemas/
    samples/
  tests/
    test_pipeline.py
  tableau/
    README_tableau.md
  .gitignore
  environment.yml
  Dockerfile
  README.md

Quick Start (Google Colab Pro)

Upload notebooks to Colab
Select A100 GPU runtime
Run notebooks in order (1 through 4)
Each notebook installs its own dependencies in the first cell

Environment

PySpark 3.5.8
cuML 26.02 / cuDF 26.02 (RAPIDS)
scikit-learn >= 1.4.0
Python 3.10+

Tableau Dashboards

Four CSVs are exported for Tableau visualization:

cluster_profiles.csv - Row-level assignments with t-SNE coordinates
cluster_metrics.csv - Algorithm comparison metrics
cluster_sizes.csv - Cluster distributions and fraud rates
stability_results.csv - ARI stability scores

See tableau/README_tableau.md for dashboard setup instructions.

Testing

pytest tests/test_pipeline.py -v

Coding Style

Variable naming: abbreviated snake_case (txn_df, txn_amt, km_mdl)
Comments: inline end-of-line style
Code organization: helper functions at top, main logic below

Pinnamv/7006SCN-Code

About

Resources

Stars

Watchers

Forks

Releases

Languages