https://github.com/caimeng2/uniscraper
A universal scraper that grabs text from multiple types of webpages.
Science Score: 23.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
○.zenodo.json file
-
○DOI references
-
○Academic publication links
-
✓Committers with academic emails
6 of 10 committers (60.0%) from academic institutions -
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (7.1%) to scientific vocabulary
Keywords
Keywords from Contributors
Repository
A universal scraper that grabs text from multiple types of webpages.
Basic Info
Statistics
- Stars: 6
- Watchers: 1
- Forks: 31
- Open Issues: 5
- Releases: 0
Topics
Metadata Files
README.md
UniScraper
Description
Uniscraper is a universal scraper that collects text from multiple types of webpages. Currently it supports html (including dynamic webpages that use javascript), online pdfs, word documents, presentation slides, and spreadsheets.
Installation instructions
Clone the git repo:
git clone https://github.com/caimeng2/UniScraper.git
Set up a conda environment by running the following command:
conda env create --prefix ./envs --file environment.yml
conda activate ./envs
Dependency
bs4 webdriver_manager pandas selenium requests python-docx python-pptx pdfminer
Example usage
Please run example.ipynb to see example usage.
Owner
- Name: Meng Cai
- Login: caimeng2
- Kind: user
- Repositories: 3
- Profile: https://github.com/caimeng2
GitHub Events
Total
- Watch event: 1
- Pull request event: 1
- Fork event: 1
Last Year
- Watch event: 1
- Pull request event: 1
- Fork event: 1
Committers
Last synced: 7 months ago
Top Committers
| Name | Commits | |
|---|---|---|
| Cai, Meng | m****1@g****m | 27 |
| Jack | j****u@g****m | 3 |
| Dirk Colbry | c****i@m****u | 2 |
| schmi710 | 5****0 | 1 |
| Baozbao | 7****0 | 1 |
| trachtsa | t****a@m****u | 1 |
| singhjy1 | s****1@m****u | 1 |
| krupczaka | k****k@m****u | 1 |
| Yash Manne | m****s@m****u | 1 |
| Ishaan Pathak | p****s@m****u | 1 |
Committer Domains (Top 20 + Academic)
Issues and Pull Requests
Last synced: 6 months ago
All Time
- Total issues: 11
- Total pull requests: 12
- Average time to close issues: 7 months
- Average time to close pull requests: 5 months
- Total issue authors: 11
- Total pull request authors: 10
- Average comments per issue: 0.64
- Average comments per pull request: 0.0
- Merged pull requests: 10
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 0
- Pull requests: 2
- Average time to close issues: N/A
- Average time to close pull requests: about 9 hours
- Issue authors: 0
- Pull request authors: 1
- Average comments per issue: 0
- Average comments per pull request: 0.0
- Merged pull requests: 2
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
- alyaqou1 (1)
- bowerm37 (1)
- haasjack (1)
- skowalski34 (1)
- schmi710 (1)
- morri726 (1)
- chenqili2020 (1)
- fjevtic46 (1)
- singhjy1 (1)
- krupczaka (1)
Pull Request Authors
- colbrydi (2)
- butzujac (2)
- pathakis (1)
- yashmanne (1)
- trachtsa (1)
- krupczaka (1)
- schmi710 (1)
- morri726 (1)
- chenqili2020 (1)
- fjevtic46 (1)
Top Labels
Issue Labels
Pull Request Labels
Dependencies
- bs4
- jupyter
- nltk
- pandas
- pdoc3
- pip
- pylint
- pytest
- requests