Data

Tools

Indexes

Applications

Experiments

Open Source Science

twarc

A command line tool (and Python library) for archiving Twitter JSON

https://github.com/docnow/twarc

Science Score: 20.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ○
    codemeta.json file
  • ○
    .zenodo.json file
  • ○
    DOI references
  • ✓
    Academic publication links
    Links to: zenodo.org
  • ✓
    Committers with academic emails
    18 of 71 committers (25.4%) from academic institutions
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (14.3%) to scientific vocabulary

Keywords from Contributors

distributed open-science rdf pypi parallel data-mining genome fuzzing geojson interactive
Last synced: 10 months ago · JSON representation

Repository

A command line tool (and Python library) for archiving Twitter JSON

Basic Info
Statistics
  • Stars: 1,383
  • Watchers: 33
  • Forks: 256
  • Open Issues: 49
  • Releases: 131
Created over 13 years ago · Last pushed over 2 years ago
Metadata Files
Readme License

README.md

twarc

DOI Build Status Standard Premium v2

twarc is a command line tool and Python library for collecting and archiving Twitter JSON data via the Twitter API. It has separate commands (twarc and twarc2) for working with the older v1.1 API and the newer v2 API and Academic Access (respectively).

twarc has been developed with generous support from the Mellon Foundation.

Contributing

New features are welcome and encouraged for twarc. However, to keep the core twarc library and command line tool sustainable we will look at new functionality with the following principles in mind:

  1. Purpose: twarc is for collection and archiving of Twitter data via the Twitter API.
  2. Sustainability: keeping the surface area of twarc and it's dependencies small enough to ensure high quality.
  3. Utility: what is exposed by twarc should be applicable to different people, projects and domains, and not specific use cases.
  4. API consistency: as much as sensible we aim to make twarc consistent with the Twitter API, and also aim to make twarc consistent with itself - so commands in core twarc should work similarly to each other, and twarc functionality should align towards the Twitter API.

For features and approaches that fall outside of this, twarc enables external packages to hook into the twarc2 command line tool via click-plugins. This means that if you want to propose new functionality, you can create your own package without coordinating with core twarc.

Documentation

The documentation is managed at ReadTheDocs. If you would like to improve the documentation you can edit the Markdown files in docs or add new ones. Then send a pull request and we can add it.

To view your documentation locally you should be able to:

pip install -r requirements-mkdocs.txt
pip install -e .
mkdocs serve
open http://127.0.0.1:8000/

If you prefer you can create a page on the wiki to workshop the documentation, and then when/if you think it's ready to be merged with the documentation create an issue. Please feel free to create whatever documentation is useful in the wiki area.

Code

If you are interested in adding functionality to twarc or fixing something that's broken here are the steps to setting up your development environment:

git clone https://github.com/docnow/twarc
cd twarc
pip install -r requirements.txt

Create a .env file that included Twitter App keys to use during testing:

BEARER_TOKEN=CHANGEME
CONSUMER_KEY=CHANGEME
CONSUMER_SECRET=CHANGEME
ACCESS_TOKEN=CHANGEME
ACCESS_TOKEN_SECRET=CHANGEME

Now run the tests:

python setup.py test

Add your code and some new tests, and send a pull request!

Owner

  • Name: Documenting the Now
  • Login: DocNow
  • Kind: organization
  • Email: info@docnow.io

tools supporting the ethical collection, use, and preservation of social media

GitHub Events

Total
  • Issues event: 1
  • Watch event: 25
  • Fork event: 3
Last Year
  • Issues event: 1
  • Watch event: 25
  • Fork event: 3

Committers

Last synced: about 1 year ago

All Time
  • Total Commits: 1,486
  • Total Committers: 71
  • Avg Commits per committer: 20.93
  • Development Distribution Score (DDS): 0.476
Past Year
  • Commits: 0
  • Committers: 0
  • Avg Commits per committer: 0.0
  • Development Distribution Score (DDS): 0.0
Top Committers
Name Email Commits
Ed Summers e****s@p****m 778
Igor Brigadir i****r@g****m 257
Sam Hames s****s@q****u 104
hugovk h****k 80
Peter Binkley p****y@u****a 37
Justin Littman j****n@g****u 23
Francis Kayiwa k****a@p****u 20
tinafigueroa 3****a 20
nruest r****n@g****m 11
Daniel Chudnov d****d@u****u 9
Walmir w****n@g****m 9
David Thiel d****l@s****u 7
Elizabeth Alpert e****t@q****u 6
cosmicoptima p****l@g****m 6
Ryan Chartier r****i@u****a 6
Alex Leith a****h@t****g 6
hauselin h****n@g****m 5
Frederik Elwert f****t@r****e 4
kngoledge k****e@g****m 4
Daniyal d****n@g****m 4
Melanie Walsh m****h@w****u 4
Andreas Segerberg a****g@g****m 4
ianmilligan1 i****n@u****a 4
R. Miles McCain d****e@s****l 4
Mirko Lenz i****o@m****e 4
Andy Chosak c****k 4
Alice Miller a****r@q****u 4
Leonard Lausen l****d@l****l 3
haruna w****w@y****p 3
nicholas w****s@g****m 3
and 41 more...
Committer Domains (Top 20 + Academic)
qut.edu.au: 4 ualberta.ca: 2 stanford.edu: 2 ncsu.edu: 2 pobox.com: 1 gwu.edu: 1 princeton.edu: 1 umich.edu: 1 tidetech.org: 1 rub.de: 1 wustl.edu: 1 uwaterloo.ca: 1 sendmiles.email: 1 mirko-lenz.de: 1 lausen.nl: 1 yahoo.co.jp: 1 iosa.it: 1 gitter.im: 1 uchicago.edu: 1 shu.edu: 1 ucsb.edu: 1

Issues and Pull Requests

Last synced: 11 months ago

All Time
  • Total issues: 76
  • Total pull requests: 30
  • Average time to close issues: 21 days
  • Average time to close pull requests: 20 days
  • Total issue authors: 40
  • Total pull request authors: 12
  • Average comments per issue: 4.5
  • Average comments per pull request: 1.6
  • Merged pull requests: 24
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 1
  • Pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 1
  • Pull request authors: 0
  • Average comments per issue: 0.0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • igorbrigadir (21)
  • edsu (5)
  • myrainbowandsky (4)
  • santoshbs (3)
  • SamHames (3)
  • numeroteca (3)
  • HarryHe11 (2)
  • frogger (2)
  • miguelwon (2)
  • lutemis (1)
  • tadavoila (1)
  • axhiao (1)
  • fishfree (1)
  • hauselin (1)
  • aborruso (1)
Pull Request Authors
  • igorbrigadir (10)
  • SamHames (9)
  • edsu (2)
  • hauselin (1)
  • mirkolenz (1)
  • ntorba (1)
  • eferm (1)
  • boyd-nguyen (1)
  • Quiet27 (1)
  • tirkarthi (1)
  • betsybookwyrm (1)
  • CharleoY (1)
Top Labels
Issue Labels
enhancement (15) good first issue (15) bug (10) cli (7) v2 (5) documentation (3) plugins (2)
Pull Request Labels
enhancement (1) good first issue (1)

Packages

  • Total packages: 5
  • Total downloads:
    • homebrew 18 last-month
    • pypi 3,870 last-month
  • Total dependent packages: 4
    (may contain duplicates)
  • Total dependent repositories: 97
    (may contain duplicates)
  • Total versions: 497
  • Total maintainers: 2
pypi.org: twarc

Archive tweets from the command line

  • Versions: 166
  • Dependent Packages: 4
  • Dependent Repositories: 97
  • Downloads: 3,870 Last month
Rankings
Dependent repos count: 1.5%
Stargazers count: 1.8%
Dependent packages count: 2.4%
Average: 2.6%
Forks count: 3.3%
Downloads: 4.3%
Maintainers (2)
esummers igorbrigadir
Last synced: 11 months ago
proxy.golang.org: github.com/DocNow/twarc
  • Versions: 160
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Dependent packages count: 5.4%
Average: 5.6%
Dependent repos count: 5.8%
Last synced: 11 months ago
proxy.golang.org: github.com/docnow/twarc
  • Versions: 160
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Dependent packages count: 6.5%
Average: 6.7%
Dependent repos count: 6.9%
Last synced: 11 months ago
conda-forge.org: twarc
  • Versions: 1
  • Dependent Packages: 0
  • Dependent Repositories: 0
Rankings
Forks count: 10.2%
Stargazers count: 10.3%
Average: 26.4%
Dependent repos count: 34.0%
Dependent packages count: 51.2%
Last synced: 11 months ago
formulae.brew.sh: twarc

Command-line tool and Python library for archiving Twitter JSON

  • Versions: 10
  • Dependent Packages: 0
  • Dependent Repositories: 0
  • Downloads: 18 Last month
Rankings
Forks count: 9.9%
Stargazers count: 14.4%
Dependent packages count: 19.0%
Average: 29.8%
Dependent repos count: 50.7%
Downloads: 55.2%
Last synced: 11 months ago

Dependencies

requirements-mkdocs.txt pypi
  • click >=7,<9
  • click-config-file >=0.6
  • click-plugins >=1
  • humanize >=3.9
  • mkdocs >=1.2
  • mkdocs-click >=0.4
  • mkdocs-material >=7.2
  • mkdocstrings >=0.15
  • python-dateutil >=2.8
  • requests_oauthlib >=1.3
  • tqdm >=4.62
requirements.txt pypi
  • click >=7,<9
  • click-config-file >=0.6
  • click-plugins >=1
  • humanize >=3.9
  • python-dateutil >=2.8
  • requests_oauthlib >=1.3
  • tqdm >=4.62
.github/workflows/main.yml actions
  • actions/checkout v2 composite
  • actions/setup-python v2 composite
  • pypa/gh-action-pypi-publish release/v1 composite
  • rtCamp/action-slack-notify v2 composite
setup.py pypi