Data

Tools

Indexes

Applications

Experiments

Open Source Science

articledownloader

Uses publisher APIs to programmatically retrieve scientific journal articles for text mining.

https://github.com/olivettigroup/article-downloader

Science Score: 72.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ✓
    CITATION.cff file
    Found CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ○
    DOI references
  • ✓
    Academic publication links
    Links to: zenodo.org
  • ✓
    Committers with academic emails
    3 of 8 committers (37.5%) from academic institutions
  • ✓
    Institutional organization owner
    Organization olivettigroup has institutional domain (olivetti.mit.edu)
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (15.3%) to scientific vocabulary

Keywords

api-wrapper python
Last synced: 6 months ago · JSON representation ·

Repository

Uses publisher APIs to programmatically retrieve scientific journal articles for text mining.

Basic Info
Statistics
  • Stars: 130
  • Watchers: 16
  • Forks: 35
  • Open Issues: 3
  • Releases: 4
Topics
api-wrapper python
Created over 11 years ago · Last pushed about 2 years ago
Metadata Files
Readme License Citation

README.md

article-downloader

Circle CI Documentation Status DOI

Uses publisher-approved APIs to programmatically retrieve large amounts of scientific journal articles for text mining. Exposes a top-level ArticleDownloader class which provides methods for retrieving lists of DOIs (== unique article IDs) from text search queries, downloading HTML and PDF articles given DOIs, and programmatically sweeping through search parameters for large scale downloading.

Important Note: This package is only intended to be used for publisher-approved text-mining activities! The code in this repository only provides an interface to existing publisher APIs and web routes; you need your own set of API keys / permissions to download articles from any source that isn't open-access.

Full API Documentation

You can read the documentation for this repository here.

Installation

Use pip install articledownloader. If you don't have pip installed, you could also download the ZIP containing all the files in this repo and manually import the ArticleDownloader class into your own Python code.

Usage

Use the ArticleDownloader class to download articles. You'll need an API key, and please respect each publisher's terms of use.

It's usually best to add your API key to your environment variables with something like export API_KEY=xxxxx.

You can find DOIs using a CSV where the first column corresponds to search queries, and these queries will be used to find articles and retrieve their DOIs.

Examples

Downloading a single PDF article

```python from articledownloader.articledownloader import ArticleDownloader downloader = ArticleDownloader(elsapikey='yourelsevierAPIkey') myfile = open('my_path/something.pdf', 'w') # Need to use 'wb' on Windows

downloader.getpdffromdoi('mydoi', my_file, 'crossref') ```

Downloading a single HTML article

```python from articledownloader.articledownloader import ArticleDownloader downloader = ArticleDownloader(elsapikey='yourelsevierAPIkey') myfile = open('my_path/something.html', 'w')

downloader.gethtmlfromdoi('mydoi', my_file, 'elsevier') ```

Getting metadata

```python from articledownloader.articledownloader import ArticleDownloader downloader = ArticleDownloader(elsapikey='yourelsevierAPI_key')

Get 500 DOIs from articles published after the year 2000 from a single journal

downloader.getdoisfromjournalissn('journalissn', rows=500, pubafter=2000)

Get the title for a single article (only works with CrossRef for now)

downloader.gettitlefromdoi('mydoi', 'crossref')

Get the abstract for a single article (only works with Elsevier for now)

downloader.getabstractfromdoi('mydoi', 'elsevier') ```

Using search queries to find DOIs

CSV file:

search query 001,
search query 002,
search query 003,
.
.
.

Python: ```python from articledownloader.articledownloader import ArticleDownloader downloader = ArticleDownloader('yourAPIkey')

grab up to 5 articles per search

queries = downloader.loadqueriesfromcsv(open('pathtocsvfile', 'r'))

dois = [] for query in queries: dois.append(downloader.getdoisfrom_search(query))

for i, doi in enumerate(dois): myfile = open(str(i) + '.pdf', 'w') downloader.getpdffromdoi(doi, myfile, 'crossref') #or 'elsevier' myfile.close() ```

Owner

  • Name: olivettigroup
  • Login: olivettigroup
  • Kind: organization
  • Location: MIT

Citation (CITATION.cff) 

# This CITATION.cff file was generated with cffinit.
# Visit https://bit.ly/cffinit to generate yours today!

cff-version: 1.2.0
title: Article Downloader
message: >-
  If you use this software, please cite it using the
  metadata from this file.
type: software
authors:
  - given-names: 'Kim'
    family-names: Edward
    email: elsao@mit.edu
    affiliation: MIT
identifiers:
  - type: url
    value: 'https://github.com/olivettigroup/article-downloader'
repository-code: 'https://github.com/olivettigroup/article-downloader'
abstract: The Codebase for Article Downloader
keywords:
  - 'NLP, Materials Science, Scientific Data'
version: '1.0'

GitHub Events

Total
  • Watch event: 13
  • Fork event: 1
Last Year
  • Watch event: 13
  • Fork event: 1

Committers

Last synced: almost 3 years ago

All Time
  • Total Commits: 145
  • Total Committers: 8
  • Avg Commits per committer: 18.125
  • Development Distribution Score (DDS): 0.131
Top Committers
Name Email Commits
Edward Kim e****n@g****m 126
Edward Kim e****k@m****u 7
Zach Jensen z****n@m****u 5
Vicky Gong v****g@m****u 2
Edward Kim e****m@c****o 2
Katrin Leinweber 9****r@u****m 1
Christian Knittl-Frank l****l@g****m 1
zjensen262 3****2@u****m 1
Committer Domains (Top 20 + Academic)
mit.edu: 3 citrine.io: 1

Issues and Pull Requests

Last synced: 6 months ago

All Time
  • Total issues: 9
  • Total pull requests: 2
  • Average time to close issues: 3 months
  • Average time to close pull requests: 6 days
  • Total issue authors: 8
  • Total pull request authors: 2
  • Average comments per issue: 3.89
  • Average comments per pull request: 0.5
  • Merged pull requests: 2
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 0
  • Pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 0
  • Pull request authors: 0
  • Average comments per issue: 0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • spbatchu (2)
  • duhaime (1)
  • akamxkd (1)
  • DICPZhou (1)
  • legg0028 (1)
  • Xianjun-Yang (1)
  • sgbaird (1)
  • aman31kmr (1)
Pull Request Authors
  • katrinleinweber (1)
  • lcnittl (1)
Top Labels
Issue Labels
Pull Request Labels

Packages

  • Total packages: 1
  • Total downloads:
    • pypi 177 last-month
  • Total dependent packages: 0
  • Total dependent repositories: 4
  • Total versions: 72
  • Total maintainers: 2
pypi.org: articledownloader

A module for downloading scientific journal articles

  • Versions: 72
  • Dependent Packages: 0
  • Dependent Repositories: 4
  • Downloads: 177 Last month
Rankings
Forks count: 7.1%
Stargazers count: 7.3%
Dependent repos count: 7.5%
Average: 9.2%
Dependent packages count: 10.1%
Downloads: 14.0%
Maintainers (2)
eddotman zjensen
Last synced: 6 months ago

Dependencies

requirements.txt pypi
  • autologging *
  • htmlparser *
  • requests *
setup.py pypi