Twitter Celebrity Matcher

The objective of this project is to match celebrity users with their respective tweets by making use of Semantic Textual Similarity.

This app finds similar Twitter users based on their tweets. It works in two ways - 1. Get a list of the most similar celebrity Twitter accounts based on a predefined Twitter celebrity list (915\ Twitter celebrity accounts). 2. Find similarities between two Twitter users based on their tweets.

View this project on kaggle - kaggle.com/code/ahmedshahriarsakib/twitter-celebrity-matcher-sbert-tweepy

:computer: Streamlit App | :ballotboxwithcheck: How it works? | :hammerandwrench: Project Structure | :floppydisk: Dataset |:camera: Screenshots | 🔎 Findings | :arrow_down: Built With --- | --- | --- | --- |--- | --- | ---|

Setup 🔧

Add the following to your .env file to work with tweepy: ``` ACCESSKEY=YOURACCESSKEY ACCESSSECRET=YOURACCESSSECRET CONSUMERKEY=YOURCONSUMERKEY CONSUMERSECRET=YOURCONSUMERSECRET

```

Virtual Environment Setup

Set up a virtual environment (preferably python 3.8+, features such as walrus operator was used) and run: $ pip install -r requirements.txt Check main.py file to run the streamlit app or API.

Docker Setup

Build - docker build -t twitter-celebrity-matcher .

Run - docker run -p 8501:8501 twitter-celebrity-matcher

If you have CUDA enabled GPU, you can set up pytorch (for SBERT) with pip -

$ pip install --no-cache-dir --force-reinstall torch==1.11.0+cu113 torchvision==0.12.0+cu113 torchaudio===0.11.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html

Static type testing - $ mypy main.py --ignore-missing-imports

Streamlit App💻

The app file is located at app/app.py.

To run the app, run the following command after installing necessary dependencies from requirements.txt -

$ streamlit run main.py will start the app at default port 8501.

App Walkthrough

Project Structure

TwitterCelebrityMatcher ├── api │ ├── __init__.py │ ├── api.py ├── app │ ├── __init__.py │ ├── app.py | ├── appdata.py │ ├── apphome.py │ ├── appinit.py │ ├── utils.py ├── celebrity-listing │ ├── Top-1000-Celebrity-Twitter-Accounts.csv ├── core │ ├── __init__.py │ ├── dataprep.py │ ├── matcher.py │ ├── scraper.py │ ├── utils.py ├── twitter-celebrity-embed-data │ ├── twitter-celebrity-embed-data.csv └── .env | config.py | main.py - api - fastApi script - app - streamlit app scripts - app.py - main app script - appdata.py - dataclass contains configuration variables - apphome.py - core functionalities - appinit.py - helper script to initialize the core objects (scraper and matcher) - utils.py - utility functions (e.g. social sharing html script) - core - contains 3 helper scripts - dataprep.py - scripts for all preprocessing tasks ( tweet cleaning, embedding generation etc.) - matcher.py - core functionalities to match twitter users - matcher.py - twitter scraper helper script - utils.py - utility functions - config.py - all configuration variables - main.py - run 3 apps - console scripts, streamlit app or API

utitlities directory contains a standalone tweepy scraper scripts - tweepy_scraper.py and emoticon proprocessing scripts (original from ekphrasis module)

How it works?

The celebrity tweets were collected using tweepy. After preprocessing the tweets, the tweets were embedded using sentence-transformers pretrained multilingual model - paraphrase-multilingual-MiniLM-L12-v2. The similarity score between tweets embeddings is calculated using cosine similarity.

This app - 1. Takes a Twitter username 2. Scrapes the tweets if it's unavailable 3. Generate embeddings of the given Twitter account's tweets 4. Calculate the mean embedding of the tweets (limitation ~3200 tweets) 5. Finds the cosine similarity between - Given user's tweets embeddings and celebrity tweets embeddings - Given two user's tweets embeddings 6. Returns the most similar celebrity Twitter accounts based on similarity score or just score between two users

Dataset

List of Celebrity Twitter Accounts

Open Source GitHub Gist - Top-1000-Celebrity-Twitter-Accounts.csv

NB: - There are some duplicates in the dataset. (986 after filtering) - There are some unofficial Celebrity accounts (ex - twitter.com/sonunigam) with a very small amount of tweets. We can filter those users based on their tweet count. Here is a good research paper on this topic - 25 Tweets to Know You: A New Model to Predict Personality with Social Media

I uploaded the relevant datasets on kaggle. Feel free to use the datasets -

1. Top 1000 Twitter Celebrity Accounts - Kaggle
2. Top 1000 Twitter Celebrity Tweets And Embeddings - Kaggle
3. Emoticon Dictionary - Kaggle

Generated Embeddings of Celebrity Tweets

The final embeddings of all celebrity tweets - - Twitter Celebrity Embed Data

Data Preprocessing

The tweets were collected and saved as bytestring to preserve the text encoding (some tweets may contain complicated symbol/emojis etc.). So first the tweets are decoded as string and then some preprocessing techniques are applied.

Current pipeline : - Removed hashtags, urls, and mentions - Replaced emoticons with their textual representation (ekphrasis) - Replaced emoji with their textual representation (demoji package)

Model

• sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2

Word embedding dimension : 384

Download SBERT pretrained models directly from here

Screenshots

Match Top Celebrities

Match 1v1 Celebrities

User Interface

Result Section

Findings!🔎

• Most of the celebrities are from music, film or sports industry. The similarity results in these categories are very impressive.
  

Music

Sports

Regional Film Industry

• The multilingual model does a great job finding region/culture specific attributes. Below is an example of bollywood celebrities

Author

• The number of authors in this celebrity list is comparatively lower than other dominating professions (actors, singers, footballers etc.). So in this example the first few results (first two from the results -> both are above 0.94 score ) were good but then the score drops. The drop rate is higher that the results from the dominating celebrity professions where the top score is around 0.96+ and first 10 or 20 from the list have very close score. This will vary but with only 5 users the performance of the pretrained model is great in this case

Multiple Roles - Business Magnet/ Philanthropist

• From the screenshot below, it's very clear that there are few people on the list who are very active in multiple categories.
  • The closest of them all (for Bill Gates) is Richard Branson although the score is comparatively lower (0.82) than what we see in other categories. Considering how close their roles are (both are billionaires, investor, tech) this is a very positive result.
  • Leonardo DiCaprio is the second placeholder because he falls into Philanthropist category, same goes for Tony Robbins
  • Other famous billionaires such as - anand mahindra and Ratan N. Tata makes it to the top 10
  • Most of the top people are either founders of large organizations, activists, philanthropists or authors. (Bill Gates published few books, so this could be another factor)

NB: Due to space limitation, the screenshots above displays only top 10 results, try the app to view more top results which are very similar to the query

FYI

Test Workstation (RAM 16gb 2400Hz, AMD 2600x, 1050ti (768 CUDA cores))

• With CUDA enabled GPU the app runs ~5x faster than CPU.
• With the current scraping scripts, it takes ~14-16 seconds to download tweets
• Takes ~2h 5m in total to preprocess and generate embedding for all users' (915 total) tweets

Built With

demoji==1.1.0 ekphrasis==0.5.1 jupyter==1.0.0 python-dotenv==0.19.2 schedule==1.1.0 sentence-transformers==2.2.0 torch==1.11.0+cu113 torchaudio==0.11.0+cu113 torchvision==0.12.0+cu113 streamlit==1.7.0 tweepy==4.7.0

Reference

• Identifying Insomnia From Social Media Posts: Psycholinguistic Analyses of User Tweets
• An Effective BERT-Based Pipeline for Twitter Sentiment Analysis: A Case Study in Italian
• SBERT Multilingual Models
• Twitter Sentiment Analysis with Deep Learning using BERT and Hugging Face
• Twitter Sentiment Analysis with Twint and Textblob