Offensive Language Detection in Social Media

This project implements a machine learning pipeline for detecting offensive language in social media using the OLID dataset. The pipeline includes data exploration, cleaning, feature extraction (TF-IDF, custom features, and GloVe embeddings), and model training with SMOTE.

Key Files

cleaningPreprocessing.py: Functions to clean and preprocess tweet text.

data_loading_exploration.py: Loads the dataset and generates basic EDA figures.

feature_extraction.py: Extracts TF-IDF features.

custom_features.py: Custom feature extraction (punctuation counts, user mentions, sentiment).

enhanced_model_improvements.py: Builds and trains the enhanced model (TF-IDF + custom features + GloVe).

main.py: Main script to run the complete pipeline.

test_evaluation.py: Evaluates the saved model on the test set.

OLIDv1.0/ (folder): Contains the OLID dataset files.

Requirements Python 3.9 Libraries: pandas, numpy, scikit-learn, seaborn, matplotlib, joblib, vaderSentiment

Due to file size limitations, the glove.6B.100d.txt file (over 300MB) is not included in this repository. Please download it from the GloVe website and place it in the project directory before running the code. (https://nlp.stanford.edu/projects/glove/)

How to Run:

Train and save the model: python3 main.py

Evaluate the saved model: python3 test_evaluation.py

kuzmict/cw1_nlp_10210411

About

Resources

Stars

Watchers

Forks

Releases

Languages