Hate Speech Detection in Videos and Memes

This repository contains code for detecting hate speech in videos and memes using multimodal architectures. The project includes Simple Embedding Fusion (aka Simple Fusion) and MO-Hate architectures.

Table of Contents

• Dataset
• Installation
• Preprocessing
• Running Simple Fusion Experiments
• Running MO-Hate Architecture Experiments
• Testing on Specific Videos/Memes

Dataset

Download

The datasets used in this project are HateMM and Hateful Memes. You can download it from the following link:

• HateMM Dataset
• Hateful Memes Dataset on Kaggle or Hugging Face

Installation

To install the necessary packages, run the following command:

pip install -r requirements.txt

Preprocessing

Before running the experiments, you need to preprocess the dataset to extract features. The preprocessing scripts are located in the Preprocessing/ directory. Make sure to specify the correct path to the downloaded datasets in the below code files.

1. Video Frames and Audio Transcript:

   python Preprocessing/frameExtract.py

   and then

   python Preprocessing/WhisperTranscript.py

2. Audio Features:

   python Preprocessing/AudioMFCC_Features.py

   or

   python Preprocessing/CLAP_and_Wav2Vec2_features.py

3. Text Features:

   python Preprocessing/BERT_HXP_Embeddings.py

4. Image Features:

   python Preprocessing/CLIP_image_features.py

   or

   python Preprocessing/DINOv2_image_features.py

   or

   python Preprocessing/ViT_Memes_Features.py

5. Video Features:

   python Preprocessing/ViT_VideoFrame_Features.py

Running Simple Embedding Fusion Experiments

The Simple Fusion architecture experiments are implemented in the Simple Fusion/ directory. To run the simple fusion experiments, follow these steps:

1. Train the Simple Fusion Model on HateMM:

   python Simple-Fusion/HateMM_Fusion.py

2. Train the Simple Fusion Model on Hateful Memes:

   python Simple-Fusion/HateMemesFusion.py

Running MO-Hate Architecture Experiments

The MO-Hate architecture experiments are implemented in the MO-Hate/ directory. To run these experiments, follow these steps:

1. Train the MO-Hate Model on HateMM:

   python MO-Hate/main.py

2. Train the MO-Hate Model on Hateful Memes:

   python MO-Hate/memes.py

Testing on Specific Videos/Memes

For testing either on random or specific images from the test set, follow these steps:

Load the trained model and run the file: python test_videos.py or python test_memes.py

Note: Some parts of the preprocessing and training codes for Simple Embedding Fusion and MO-Hate have been taken from their respective GitHub repositories. Please refer to the following links for more details:

• HateMM
• MO-Sarcation