https://github.com/animesh/analyzing-twitter-trends-on-covid-19-vaccinations

A quantitative study comprising Twitter discussions and thematic analysis for COVID-19 vaccines

https://github.com/animesh/analyzing-twitter-trends-on-covid-19-vaccinations

Repository

A quantitative study comprising Twitter discussions and thematic analysis for COVID-19 vaccines

https://github.com/animesh/Analyzing-Twitter-Trends-On-COVID-19-Vaccinations/blob/main/

# Analyzing-Twitter-Trends-On-COVID-19-Vaccinations
A quantitative study comprising Twitter discussions and thematic analysis for COVID-19 vaccines 

Paper published at: https://infodemiology.jmir.org/2022/1/e33909

## TABLE OF CONTENTS

* [Background](#background)
* [Objective](#objective)
* [Tools and Packages](#tools)
* [Data Collection](#data-collection)
* [Data Pre-Processing](#data-preprocessing)
* [Data Modeling](#data-modeling)
* [Data Visualization](#data-visualization)
* [Results](#results)
* [Conclusion](#conclusion)
* [References](#references)
* [Challenges and Future Work](#challenges-and-futurework)



## BACKGROUND 
The COVID-19 pandemic has killed 3.2 million people and infected 150 million around the world as of April 30, 2021. Growing human rights concerns, vaccine movements, and skepticism towards the vaccines, its effects and efficacy have resulted in a multitude of conversations on social media and the process of vaccination becoming a complicated task. No major studies have been conducted to analyze peoples perception of COVID-19 vaccines on social media for the year 2021



## OBJECTIVE 
 To extract information from tweets (between January and April, 2021) related to COVID vaccine where opinions are highly unstructured, heterogeneous and are either positive or negative, or neutral and identify driving factors for the change in sentiments 
 
 To explore conversations and abstract "topics" that occur in the collected tweets using topic modeling and text analytics backed by breakthrough events in the timeline 
 
 To visualize the trends in sentiments of Twitter users and popularity associated with the discovered topics 
 

 

## TOOLS


  

    
Task
    
Technique
 
    
Tools/Packages Used
  
  

    
Data Collection
    
Tweet extraction from Twitter 
 
    
snscrape
  
  

    
Data Pre-processing
    
Removed punctuation, stopwords, URLs, emojis, lemmatization
 
    
re, nltk,CountVectorizer, pandas, numpy
  
  

    
Data Modeling
    
Unsupervised LDA
 
    
pyLDAvis.sklearn, LatentDirichletAllocation, sklearn 
  
  

    
Text Analytics
    
Topic Modeling, Sentiment analysis
 
    
vaderSentiment, corextopic
  
  

    
Data Visualization
    
Multi-attribute plots
 
    
matplotlib, seaborn, Tableau, wordcloud
  
  

    
Environments & Platforms
    
 
 
    
MS Excel, Google Colab, Jupyter Notebook, Twitter
  





## DATA-COLLECTION 


  

    
Method
    
Notes
 
  
  

    
Tweepy
    
3200 tweets; no historical data
 
  
  

    
GetOldTweets3
    
Twitter has removed the endpoint the GetOldTweets3 uses 
 
  
  

    
TWINT
    
Twitter throws a more strict device + IP-ban after a certain amount of queries
 
  
  

    
snscrape
    
Scrapped 100K tweets - 96,641 English tweets
 
  
  

    
Octoparse (software)
    
Very time consuming with the event loop
 
  


 Data Collection: Identifying COVID-19 Vaccines Content 

 Package used: snscrape
Language: English
Keywords: covid vaccine
Timeframe: January 1, 2021 to March 31, 2021
Number of tweets collected = 2.74 million
January - 884,011 tweets | February - 800,008 tweets | March - 1,127,854 tweets
No null values identified

 Data Coverage: 
 With covid vaccine as the search terms, we believe that our set of keywords provides reasonable coverage and is representative of tweets communicating about COVID-19 vaccines 

Individual tweets = 2.1 million 

Organizational tweets = 0.59 million 




## DATA-PREPROCESSING

 Data Cleaning  

 Removed punctuation using remove_punct function with library re 
 
 Removed URLs and emojis in Tokenization using library re 
 
 Removed stopwords using nltk 
 
 Lemmatization of Tweets using nltk.WordNetLemmatizer() 
 
 

 Individual vs Organizational Tweets  

 Created a Bag-of-Words with ~175 keywords to filter on Display Names 
 
 Removed 22% of the data 
 
 2,109,427 tweets remain after removing organizational accounts 
 
 Assigned week numbers (1 to 12) to the dataset 
 



## DATA-MODELING

 Unsupervised LDA 
To understand the abstract topics hidden in the tweets unsupervised LDA technique was implemented using the library 'pyLDAvis'. We discovered 18 different topics considering the cluster size and no overlapping amongst the clusters
 Sentiment Analysis 
Sentiment analysis is a supervised machine learning problem with different types of analysis. We considered a fine-grained sentiment classification with five levels of sentiments - overly positive, positive, neutral, negative, and overly negative. We used VADER (Valence Aware Dictionary for Sentiment Reasoning) as a rule-based model to examine the impact of COVID-19 vaccine on the attitude of Twitter users during the pandemic. 
 CorEx 
Correlation Explanation (CorEx) provides a flexible framework for learning topics that are maximally informative about a corpus of text. Through anchor words, we seeded and guided the topic model towards topics of substantive interest, which allowed us to interact with and refine topics in a way that is not possible with traditional topic models. Normalized Topic Correlation (NTC) represents the correlations within an individual document explained by a particular topic. 



## DATA-VISUALIZATION 

 Unsupervised LDA 



 Trends in Sentiment Analysis 
 


 Distribution of Sentiments 
 


 Vaccine Conversation Trends 
 


 Popular Topics 




## RESULTS 
 Discovered 13 unique topics from the tweets across 12 weeks from Jan21 to Mar21 
February had the lowest number of tweets (594,050) as compared to January (695,890) and March (819,487) about COVID vaccinations
Positive sentiment contributed the most in overall sentiments of Twitter users (732,395), followed by neutral (579,493) and negative (525,866) sentiments
People were discussing the most about topics like Vaccination status, Travel, Pandemic, Politics, Vaccine Approval
Topics that remained underrepresented were People Against Vaccine, Political and COVID leaders, Who Got Vaccinated



## CONCLUSION 
This study focused on demonstrating the conversations around COVID-19 vaccines on Twitter using a dataset created with tweets from individuals leveraging Machine Learning and Text Analytics approach. We performed exploratory data analysis using Unsupervised LDA to identify initial implicit topics. The dataset was further analysed for positive and negative sentiments. We further performed topic modeling for a deeper understanding of topics and their popularity across time. 



## REFERENCES 
 Praveen SV, Ittamalla R, Deepak G. Analyzing the attitude of Indian citizens towards COVID-19 vaccine - A text analytics study. Diabetes Metab Syndr. 2021 Mar-Apr;15(2):595-599. doi: 10.1016/j.dsx.2021.02.031. Epub 2021 Feb 27. PMID: 33714134; PMCID: PMC7910132 
 DeVerna, M., Pierri, F., Truong, B., Bollenbacher, J., Axelrod, D., Loynes, N., . . . Bryden, J. (2021, April 20). CoVaxxy: A collection of ENGLISH-LANGUAGE Twitter posts About COVID-19 Vaccines 
 K. Hazel Kwon, J. Hunter Priniski & Monica Chadha (2018): Disentangling User Samples: A Supervised Machine Learning Approach to Proxy-population Mismatch in Twitter Research, Communication Methods and Measures, DOI: 10.1080/19312458.2018.1430755 
 Scraping Tweets with snscrape - https://betterprogramming.pub/how-to-scrape-tweets-with-snscrape-90124ed006af 
 Vader Sentiment Analysis - 
https://github.com/cjhutto/vaderSentiment 
 Unsupervised LDA - 
https://www.kaggle.com/keitazoumana/topic-modeling-with-lda 



## CHALLENGES-AND-FUTUREWORK 
 Challenges : Identifying package for tweet scraping and recognizing limitations on extraction, large execution times and runtime errors due to memory limitation for parts of data modeling 
 Future Work 
 Low impact insights from VADER Sentiment Analysis opens up a scope for deep dive into topics independently like People For/Against vaccines 
 Explore conversations and sentiments in organizational tweets 
 Number of active COVID cases, recoveries and deaths for the three months 
 






This project was made in collaboration with Harsh Shah and Vivek Kumar, do check out some of the amazing projects they've worked on.

Owner

GitHub Events