Big Data Analytics with Apache Spark using the HIGGS Dataset

This repository contains a series of six practical assignments demonstrating the implementation of Apache Spark (PySpark) for large-scale data processing, feature engineering, distributed computing, machine learning, and data visualization using the HIGGS Dataset.

The project showcases the complete big data analytics workflow—from data loading and preprocessing to predictive analysis and visualization preparation.

Project Objectives

Learn Apache Spark fundamentals
Process large datasets using distributed computing
Perform feature engineering with PySpark
Analyze Spark execution performance
Evaluate machine learning prediction results
Prepare datasets for Tableau visualization

Repository Structure

.
├── notebooks/
│   ├── Task1.ipynb
│   ├── Task2.ipynb
│   ├── Task3.ipynb
│   ├── Task4.ipynb
│   ├── Task5.ipynb
│   └── Task6.ipynb
└── README.md

Task Overview

Task 1 – Spark Setup & Dataset Loading

Description

This notebook demonstrates how to configure Apache Spark in Google Colab and load the HIGGS dataset into a Spark DataFrame.

Topics Covered

Installing PySpark
Creating Spark Session
Loading HIGGS Dataset
Reading CSV files
Viewing Dataset Schema
Displaying Sample Records

Task 2 – Data Ingestion & Partition Analysis

Description

This notebook focuses on efficient data loading from Google Drive and understanding Spark's distributed partitioning.

Topics Covered

Mounting Google Drive
Reading compressed CSV files
Assigning feature names
Schema inspection
Checking number of partitions
Understanding distributed storage

Task 3 – Feature Engineering

Description

This notebook performs feature engineering using Spark SQL functions to generate additional variables for machine learning.

Topics Covered

Column Renaming
Feature Transformation
Creating Derived Features
SQL Expressions
Data Preparation

Task 4 – Distributed Computing Performance

Description

This notebook demonstrates Spark's distributed processing capabilities and evaluates execution performance.

Topics Covered

Distributed Processing
Data Caching
Performance Optimization
Execution Time Analysis
Spark Persistence

Task 5 – Machine Learning Prediction Analysis

Description

This notebook evaluates prediction outputs generated from multiple machine learning algorithms.

Models Evaluated

Logistic Regression
Decision Tree
Random Forest

Topics Covered

Reading Prediction Files
Loading Parquet Files
Comparing Model Outputs
Prediction Analysis

Task 6 – Tableau Data Preparation

Description

This notebook prepares processed prediction datasets for visualization in Tableau.

Topics Covered

Reading Prediction Files
Data Cleaning
Dataset Export
Visualization Preparation

Dataset Information

Dataset: HIGGS Dataset

The HIGGS dataset is a benchmark dataset used in particle physics to distinguish between signal processes that produce Higgs bosons and background processes.

Dataset Characteristics

Over 11 million observations
28 numerical input features
1 binary target variable
Large-scale dataset suitable for distributed computing

Technologies Used

Python
Apache Spark (PySpark)
Spark SQL
Spark MLlib
Google Colab
Pandas
Parquet
Tableau

Project Workflow

HIGGS Dataset
       │
       ▼
Task 1
Spark Setup & Data Loading
       │
       ▼
Task 2
Data Ingestion & Partitioning
       │
       ▼
Task 3
Feature Engineering
       │
       ▼
Task 4
Distributed Computing Analysis
       │
       ▼
Task 5
Machine Learning Prediction Evaluation
       │
       ▼
Task 6
Visualization Dataset Preparation
       │
       ▼
Tableau Dashboard

Installation

Clone this repository:

git clone https://github.com/username/your-repository.git

Navigate into the project folder:

cd your-repository

Install the required package:

pip install pyspark

Running the Project

Execute the notebooks in the following order:

Task1.ipynb
Task2.ipynb
Task3.ipynb
Task4.ipynb
Task5.ipynb
Task6.ipynb

Expected Outputs

The project generates:

Spark DataFrames
Engineered Features
Distributed Computing Metrics
Machine Learning Predictions
Parquet Output Files
Tableau-ready Datasets

tirumalast/7006SCN_Initials_16688319

About

Resources

Stars

Watchers

Forks

Releases

Languages