https://github.com/ahmedshahriar/pulsepoint-data-analytics
EDA, data processing, cleaning and extensive geospatial analysis on a selenium based web scraped dataset
Science Score: 13.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
○.zenodo.json file
-
○DOI references
-
○Academic publication links
-
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (10.9%) to scientific vocabulary
Keywords
Repository
EDA, data processing, cleaning and extensive geospatial analysis on a selenium based web scraped dataset
Basic Info
Statistics
- Stars: 5
- Watchers: 2
- Forks: 1
- Open Issues: 0
- Releases: 0
Topics
Metadata Files
README.md
PulsePoint Data Analytics
Summary
In this project I performed data cleaning & preprocessing, feature extraction, extensive geospatial and time series analysis of the Pulsepoint Emergency Data as well as apply some clustering and dimensionality reduction techniques.
Please check Project Wiki for an in depth overview
Dataset
PulsePoint offers a web client at web.pulsepoint.org that allows users to view the same data that appears in PulsePoint Respond with a browser.
The dataset was collected via web scraping using python libraries (selenium, postgreSQL). The logs were collected from 2021-05-02 to 2021-12-31.
NB: As per the copyright policy held by https://web.pulsepoint.org see more - https://www.pulsepoint.org/eula "The PulsePoint app and its data are available to users for their own personal use and cannot be redistributed", Hence I am not intending to publish the dataset
With this data, it is possible to assess the impact of local emergencies over the period as well as the regions where they were concentrated.
The project answers questions which could be inferred from this dataset, such as:
Location-based: - What are the major incidents in terms of numbers? - What are the major states and cities in terms of the number of incidents? - Which regions have the highest number of incidents?
Time-based: - Which dates have a higher number of incidents occurred? - Which days of the week have a higher number of incidents occurred? - Which time of the day has a higher number of incidents occurred?
Analysis: - Find geolocations of the major locations - Visualize major locations taking the number of incidents that occurred at those regions into account - Cluster cities based on the time, duration, and the number of incidents that occurred at those places
This project also serves as my assignment for the course -
Also, you can view this notebook on kaggle -
Preview -
Built With
tqdm==4.62.3
wordcloud==1.5.0
geopy==1.17.0
plotly==4.4.1
folium==0.8.3
geopandas==0.10.2
pdpipe==0.0.70
yellowbrick==1.3.post1
scikit-learn==0.22.2.post1
Owner
- Name: Ahmed Shahriar Sakib
- Login: ahmedshahriar
- Kind: user
- Location: Ontario, Canada
- Company: @criticalml-uw
- Website: https://ahmedshahriar.com
- Twitter: ahmed__shahriar
- Repositories: 5
- Profile: https://github.com/ahmedshahriar
Software Engineer, an expert in web scraping & automation, data analytics, and machine learning. Kaggle Master.
GitHub Events
Total
- Watch event: 1
- Fork event: 1
Last Year
- Watch event: 1
- Fork event: 1
Issues and Pull Requests
Last synced: about 1 year ago
All Time
- Total issues: 0
- Total pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Total issue authors: 0
- Total pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 0
- Pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Issue authors: 0
- Pull request authors: 0
- Average comments per issue: 0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0