htmldate
htmldate: A Python package to extract publication dates from web pages - Published in JOSS (2020)
LISC
LISC: A Python Package for Scientific Literature Collection and Analysis - Published in JOSS (2019)
trafilatura
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
ia-webscraping
An AWS workflow for collecting webpages from the Internet Archive
https://github.com/datamade/car-scraper
💲Make spreadsheets out of Chicago Association of REALTORS® reports
pixarfilms
:movie_camera: R data package to explore Pixar films, the people, and reception data
https://github.com/cimentadaj/dataharvesting
Material for the course 'Data Harvesting' for the masters in computational social science - UC3M
https://github.com/ahmedshahriar/bd-medicine-scraper
Scrapy-Django PostgreSQL integrated API with Proxy IP configuration that scrapes all medicine data (meds, prices, generics, companies, indications) from Bangladesh (30k+ pages)
MDPIexploreR
A simple package to explore MDPI´s articles by journal. A series of functions help to obtain lists of papers, obtain data from them (turnaround times, special issues and articles types) and create summary graphs.
https://github.com/adamouization/formula1-crash-stats
:car: :collision: :car: Flask web application parsing F1 DNF stats from Wikipedia and visualising them in charts and tables.
https://github.com/erictleung/sports-popularity-in-usa
:basketball: Analysis of sports popularity in the USA
scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
letsrept
letsRept is an R package to sample reptile species information from The Reptile Database website
https://github.com/ahmedshahriar/bd-ponno
Scrapy Mongodb Djongo integrated API that scrapes popular e-commerce sites (10+) from Bangladesh
https://github.com/ahmedshahriar/twittercelebritymatcher
Match celebrity users with their respective tweets by making use of Semantic Textual Similarity on over 900+ celebrity users' 2.5 million+ scraped tweets utilizing SBERT, streamlit, tweepy and FastAPI
https://github.com/ahmedshahriar/depression-tweets-scraper
A Scraper that scrapes '#depression' tweets daily powered by GitHub action and snscrape (stopped at June 30,2023)
music-analysis
Analysis of influence of music using multiple methods, i.e. clustering musical elements, graph centrality of samplings, predicting popularity from lyrics (2018)
semantic-outlier-removal
Code and data for SORE (ACL 2025), a semantic boilerplate remover.
pyhdb
pyHDB - Ferramenta de auxílio metodológico para pesquisas na interface da Hemeroteca Digital Brasileira da Biblioteca Nacional. Desenvolvida por Eric Brasil (IHLM-UNILAB) como parte de pesquisa acadêmica da área de História Digital.
ferramentas_scielo_v2
Esse repositório é parte dos projetos desenvolvidos pelos membros do LABHDUFBA e tem como objetivo oferecer ferramentas de raspagem, organização e análise de artigos ciêntíficos publicados na plataforma Scielo.br.
legiscrawler
An automation webcrawler based on Selenium library for retrieving parliamentary questions on The Website of Taiwan Legislative Yuan (https://lis.ly.gov.tw/). 🕸️ 🕸️ 爬立法委員問政專輯的爬蟲小幫手 🛠️🧰
https://github.com/chiragagg5k/finscraper
scraping and processing sec edgar financial data and training fastercnn table detection model with it
https://github.com/csinva/clinical-rule-survey
Analyzing clinical decision instruments through the lens of data and large language models.
universitatespodcastdata
An R package for downloading, extracting, and analyzing interview transcripts from the Universitates podcast series. It provides tools for data processing, searching, and visualization