Recent Releases of twarc
twarc - v2.14.0
What's Changed
Small fix to add missing fields for blue verified in both v1.1 and v2.
- Add
verified_typeuser field https://github.com/DocNow/twarc/pull/682 - Add
include_ext_is_blue_verifiedhttps://github.com/DocNow/twarc/pull/690 - Updates to tutorial doc https://github.com/DocNow/twarc/commit/ae7a161d66e936042d9a8724d9b788c10dd656b4
Full Changelog: https://github.com/DocNow/twarc/compare/v2.13.0...v2.14.0
- Python
Published by igorbrigadir over 3 years ago
twarc - v2.13.0
What's Changed
- bump python version to >=3.6 by @igorbrigadir in https://github.com/DocNow/twarc/pull/660
- Updates to tutorial doc by @boyd-nguyen in https://github.com/DocNow/twarc/pull/665
- Twarc Tutorial by @SamHames in https://github.com/DocNow/twarc/pull/558
- Add twarc demo video by @Quiet27 in https://github.com/DocNow/twarc/pull/677
- Add missing variants field to media by @igorbrigadir in https://github.com/DocNow/twarc/pull/679
New Contributors
- @boyd-nguyen made their first contribution in https://github.com/DocNow/twarc/pull/665
- @Quiet27 made their first contribution in https://github.com/DocNow/twarc/pull/677
Full Changelog: https://github.com/DocNow/twarc/compare/v2.12.0...v2.13.0
- Python
Published by edsu over 3 years ago
twarc - Release v2.11.2
This is a small bug fix release for an issue with progress bars being used incorrectly when reading data from stdin: #652.
- Python
Published by SamHames almost 4 years ago
twarc - v2.11.1
What's Changed
- Add sort_order parameter for search api by @mirkolenz in https://github.com/DocNow/twarc/pull/645
- Append matching rules from stream when flattening by @igorbrigadir in https://github.com/DocNow/twarc/pull/646
- Fix bug where --max-results could not be set with --no-context-annota… by @SamHames in https://github.com/DocNow/twarc/pull/648
New Contributors
- @mirkolenz made their first contribution in https://github.com/DocNow/twarc/pull/645
Full Changelog: https://github.com/DocNow/twarc/compare/v2.10.4...v2.11.1
- Python
Published by edsu almost 4 years ago
twarc - v2.11.0
What's Changed
- Add sort_order parameter for search api by @mirkolenz in https://github.com/DocNow/twarc/pull/645
- Append matching rules from stream when flattening by @igorbrigadir in https://github.com/DocNow/twarc/pull/646
New Contributors
- @mirkolenz made their first contribution in https://github.com/DocNow/twarc/pull/645
Full Changelog: https://github.com/DocNow/twarc/compare/v2.10.4...v2.11.0
- Python
Published by edsu about 4 years ago
twarc - v2.10.4
This release contains two bug fixes:
- A fix to the
ensure_flattenedfunction that handles valid API responses that contain errors, but no data #627 - A fix to the v1.1
user_lookupfunction that raises a useful error when a string is passed, preventing the lookup of single character usernames. Thanks to @hauselin.
- Python
Published by SamHames about 4 years ago
twarc - v2.10.2
What's Changed
- Set daemon attribute instead of using setDaemon method that was deprecated in Python 3.10. by @tirkarthi in https://github.com/DocNow/twarc/pull/617
- Fix method name typo introduced by refactor by @SamHames in https://github.com/DocNow/twarc/pull/621
New Contributors
- @tirkarthi made their first contribution in https://github.com/DocNow/twarc/pull/617
Full Changelog: https://github.com/DocNow/twarc/compare/v2.10.1...v2.10.2
- Python
Published by edsu over 4 years ago
twarc - v2.9.4
This release is functionally identical to v2.9.3, which contained a bugfix for an issue with the streaming API raising an exception and stopping early.
Due to a mistake in the release process v2.9.3 wasn't deployed to PyPI. Rather than edit history to re-release that version, this new release is being made instead.
- Python
Published by SamHames over 4 years ago
twarc - v2.9.0
What's Changed
- More badges in the readme by @igorbrigadir in https://github.com/DocNow/twarc/pull/577
- Support likes and retweets endpoints. by @SamHames in https://github.com/DocNow/twarc/pull/588
Full Changelog: https://github.com/DocNow/twarc/compare/v2.8.3...v2.9.0
- Python
Published by SamHames over 4 years ago
twarc -
This release includes some improvements to how the mandatory the mandatory one second sleep between requests to the search api is handled with some of the twarc2 commands. See #575 for details.
- Python
Published by edsu over 4 years ago
twarc -
v2.8.1 includes a small update to the twarc search --help message that links
to Twitter's Building Queries for Search Tweets to help users figure out what's
possible.
https://developer.twitter.com/en/docs/twitter-api/tweets/search/integrate/build-a-query
- Python
Published by edsu over 4 years ago
twarc -
v2.8.0 adds some new controls for shaping the data that is returned from the Twitter API. The default behavior is for twarc to retrieve the fullest representation of a tweet by requesting all tweet, user, media, place and poll fields as well as all available expansions. This is generally good practice with twarc because it means that downstream processing of the collected data can rely on have all this data at its disposal. However there may be cases where you want to customize the data that comes back. This is not recommended practice but it could be useful in some contexts.
The following options allow you to fine tune the types of data that are requested when using the following sub-commands: search, searches, tweet, sample, hydrate, users, mentions, timeline, timelines, conversation, conversations, and stream. The options include:
--expansions TEXT Comma separated list of expansions to
retrieve. Default is all available.
--tweet-fields TEXT Comma separated list of tweet fields to
retrieve. Default is all available.
--user-fields TEXT Comma separated list of user fields to
retrieve. Default is all available.
--media-fields TEXT Comma separated list of media fields to
retrieve. Default is all available.
--place-fields TEXT Comma separated list of place fields to
retrieve. Default is all available.
--poll-fields TEXT Comma separated list of poll fields to
retrieve. Default is all available.
These correspond to the API Fields and Expansions.
There is also --minimal-fields which requests just a minimal subset of data, and --no-context-annotations that does not include context-annotations, which allows more tweets to be fetched at one time (500 instead of 100). This also applies to the sub-commands: search, searches, tweet, sample, hydrate, users, mentions, timeline, timelines, conversation, conversations, stream.
--minimal-fields By default twarc gets all available data.
This option requests the minimal retrievable
amount of data - only IDs and object
references are retrieved. Setting this makes
--max-results 500 the default. NOTE: This
argument is mutually exclusive with
arguments: [--counts-only, --poll-fields,
--media-fields, --expansions, --no-context-
annotations, --place-fields, --user-fields,
--tweet-fields].
- Python
Published by edsu over 4 years ago
twarc -
A bugfix release to apply black formatting rules.
- Python
Published by edsu almost 5 years ago
twarc -
A bugfix release to apply black formatting rules.
- Python
Published by edsu almost 5 years ago
twarc -
- Add start-time/since-id parameters in the timeline CLI command to the timelines CLI command.
- Ensure that sample command only writes JSON on stdout.
- Python
Published by edsu almost 5 years ago
twarc -
v2.7.0 adds a new places command to search for places and their identifiers, which can be used in search and stream queries. Even though it's still on the 1.1 endpoint the 1.1/geo/search.json API endpoint makes these place identifiers available when searching by the name, geo coordinates, or ip address.
``` Usage: twarc2 places [OPTIONS] VALUE [OUTFILE]
Search for places by place name, geo coordinates or ip address.
Options: --type [name|geo|ip] How to search for places (defaults to name) --granularity [neighborhood|city|admin|country] What type of places to search for (defaults to neighborhood) --max-results INTEGER Maximum results to return --json Output raw JSON response --help Show this message and exit. ```
There is a corresponding method twarc.client2.Twarc2.geo() method which you
can use to do the lookup yourself from Python.
- Python
Published by edsu almost 5 years ago
twarc - v2.5.0
This release includes new functionality for working with Twitter's new Batch Compliance API which allow you to upload large datasets of Tweet or user IDs to retrieve their compliance status in order to determine what data requires action in order to bring your datasets into compliance.
``` Usage: twarc2 compliance-job [OPTIONS] COMMAND [ARGS]...
Create, retrieve and list batch compliance jobs for Tweets and Users.
Options: --help Show this message and exit.
Commands: create Create a new compliance job and upload tweet IDs. download Download the compliance job with the specified ID. get Returns status and download information about the job ID. list Returns a list of compliance jobs by job type and status. ```
- Python
Published by edsu almost 5 years ago
twarc - v2.4.2
This release ensures that the timeline, timelines, conversation and conversations commands default to a --start-time of 2006-03-21 (the first day of tweets) when being instructed to use the /tweets/search/all endpoint behind the scenese. For example when doing:
twarc2 timeline --use-search jack
or:
twarc2 conversation --archive 21
Previously it was defaulting to the last 30 days (which is an unfortunate default set by the /tweets/search/all endpoint). Many thanks to Darren Halpin and @SamHames for identifying and fixing the issue!
- Python
Published by edsu almost 5 years ago
twarc - v2.3.8
This bugfix release reuses twarc.decorators2.catchrequestexceptions in the
context of streaming responses (twarc2 sample and twarc2 stream) that use the response.iter_lines method. Hopefully this will address #505 but it will require testing by people who continue seeing
the error in the wild.
- Python
Published by edsu about 5 years ago
twarc - v2.3.7
v2.3.7 includes new functionality that adds progress bars for twarc2 commands like search, hydrate, timeline and more. These visual indications of how much data has been collected and how much there is to go are extremely useful in data collection jobs. Progress bars display by default when you instruct twarc to write output to a file (#490).
Additionally there is new code to catch 503 Twitter API errors that have recently been occurring much more regularly (#499). Apparently a big reason for these errors was the load that requesting 500 tweets from the search/all endpoint while also asking for context annotations. Twitter recently announced they were no longer making context annotations available for requests asking for more than 100 tweets. Since it's one of twarc's design principles to maximize the representation of tweets the search command has been adjusted to default to 100 now instead of 500, at least for the time being (#504).
- Python
Published by edsu about 5 years ago
twarc - v2.3.0
This release includes a new command for retrieving data from the new counts API.
``` Usage: twarc2 counts [OPTIONS] QUERY [OUTFILE]
Return counts of tweets matching a query.
Options: --since-id INTEGER Count tweets sent after tweet id --until-id INTEGER Count tweets sent prior to tweet id --start-time [%Y-%m-%d|%Y-%m-%dT%H:%M:%S] Count tweets created after UTC time (ISO 8601/RFC 3339), e.g. 2021-01-01T12:31:04 --end-time [%Y-%m-%d|%Y-%m-%dT%H:%M:%S] Count tweets sent before UTC time (ISO 8601/RFC 3339) --archive Count using the full archive (requires Academic Research track) --granularity [day|hour|minute] Aggregation level for counts. Can be one of: day, hour, minute. Default is hour. --limit INTEGER Maximum number of days of results to save (minimum is 30 days) --text Output the counts as human readable text --csv Output counts as CSV --help Show this message and exit. ```
- Python
Published by edsu about 5 years ago
twarc - v2.1.4
The log will now include a message about where the config file has been loaded from. In addition --verbose can be used to cause more information to be logged, such as the keys that are being used to access the API. Most of the time this is probably not a good idea since it makes your keys available in a log, but it can be useful in situations where debugging error responses from the API.
- Python
Published by edsu about 5 years ago
twarc - v2.1.2
This release adds two new twarc2 subcommands:
conversations
conversations will read a file of tweets and look for any conversations they are a part of and will download the full conversation thread for them.
twarc2 conversations tweets.jsonl > conversations.jsonl
You can also give it a file of conversation_ids to download instead:
twarc2 conversations ids.txt > conversations.jsonl
timelines
Similarly timelines will read in a file of tweet ids and will download the user timeline for any user who authored the tweets.
twarc2 timelines tweets.jsonl > timelines.jsonl
You can also give it a file of user ids or usernames:
twarc2 timelines users.txt > timelines.jsonl
This functionality was first developed in the twarc-timelines plugin which has been renamed to twarc-timeline-archive because it does some extra things like writing timelines to separate directories and being able to be run on a schedule without redownloading previously downloaded data.
- Python
Published by edsu about 5 years ago
twarc - v2.1.0
v2.1.0 removes the --flatten option from many commands in the hopes of encouraging users to mostly use the original data as retrieved from the Twitter API. The subcommand twarc2 flatten remains mostly for use in data processing pipelines that expect line oriented json where each object is a tweet:
twarc2 search blacklivesmatter | twarc2 flatten | jq .text
The twarc.expansions.flatten() function has been updated to always return a list of tweets, and twarc.expansions.ensure_flattened() can be used to make sure data has been flattened already when processing tweet data. Since it is designed for use in twarc plugins and other pieces of software that need to work with tweets it is also available for import from twarc:
from twarc import ensure_flattened
In addition this release also includes twarc conversation for retrieving tweets from a a particular conversation thread.
- Python
Published by edsu about 5 years ago
twarc - v2.0.13
This bugfix release irons out some wrinkles that have been discovered during usage:
- Fix for handling search responses with missing
datastanza but that contain a next token for another page of results. #464 - Stream diagnostics now to go stderr to not interfere with JSON being written to stdout. #456
- twarc can be run from the command line as a module now: e.g.
python -m twarc2 search barackobama#455 - twarc2 command help text indicates times are UTC
- Python
Published by edsu about 5 years ago
twarc - v2.0.6
When users are searching the full archive the default --start-date is set to the Twitter Epoch (2006-03-21). Without this it will default to the month prior to whatever the --end-date is, which can lead to confusion when users think they are searching everything. See #434 for context.
- Python
Published by edsu about 5 years ago
twarc - v2.0.0
v2.0.0 is a big effort by many to support the new Twitter v2 API. When you install twarc now you will continue to have the twarc command line as you did before, but there will be a new command twarc2 for interacting with the v2 API. Consult the documentation to see how it works!
twarc2 --help
It is worth noting that there is now documentation on ReadTheDocs and there is a new mechanism for extending the twarc2 command with plugins.
- Python
Published by edsu over 5 years ago
twarc - v1.12.0
This release adds initial support for the Gnip Full Archive premium endpoint if you are fortunate enough to have access to that. Many thanks to @epicfaace for adding the functionality and testing it.
- Python
Published by edsu over 5 years ago
twarc - v1.10.1
v1.10.1 includes option to manually enter user access token and secret during twarc configure. This is in response to several reports of users who are not given the PIN when authenticating with their app at Twitter. User access token and secret can be generated manually at https://developer.twitter.com/en/apps
- Python
Published by edsu about 6 years ago
twarc - v1.10.0
v1.10.0 changes the behavior of the command line client so that it can also fetch retweets from a file of tweet_ids.
twarc retweets ids.txt > retweets.jsonl
This is in addition to the previous behavior where it accepted a tweet id to fetch the retweets for:
twarc retweets 20 > retweets.jsonl
You can also comma separate the tweet ids on the command line if you want:
twarc retweets 20,21 > retweets.jsonl
The internal interface to the Twarc.reteets has changed to now accept an iterator of tweet ids.
```python from twarc import Twarc
twitter = Twarc()
for tweet in twitter.retweets([20, 21]): print(tweet['id_str']) # etc ``` If you have been using the retweets method in your code you will want to adjust it to pass in a list of ids rather than the bare ids.
- Python
Published by edsu about 6 years ago
twarc - v1.9.0
Premium Search API
v1.9.0 adds new functionality that allows you to use the Twitter Premium Search API endpoints. To use the Premium Search you will need to visit the Twitter Developer Dashboard and set up an environment that is attached to one of your apps. Then you should be able to use the label for your environment in your twarc search command.
For example to use the docnowdev environment to search the 30 day endpoint you can:
twarc search blacklivesmatter --30day docnowdev > tweets.jsonl
or to search the full archive endpoint:
twarc search blacklivesmatter --fullarchive docnowdev > tweets.jsonl
Warning: Depending on your query this could quickly use up your budget! So you will likely also want to use --from_date and/or --to_date to limit the time range that you are searching. You can also use --limit to limit the total number of tweets that are retrieved.
twarc search blacklivesmatter --30day docnowdev --to_date 2013-08-01 > tweets.jsonl
If your app is only authorized for the sandbox you must use the --sandbox parameter which will alter the maximum number of tweets you will can retrieve in a request down to 100.
This functionality is also made available through the new Twarc.premium_search method.
Twitter Labs
v1.9.0 also includes some initial support for the Twitter Labs endpoints. At the moment only the sample stream is supported, but we anticipate adding more as they are requested.
twarc --app_auth labs_v1_sample > sample.jsonl
- Python
Published by edsu about 6 years ago
twarc - v1.8.4
This is a bugfix release for the CSV conversion which was not extracting tweet text from enclosed tweet objects in retweets which hold the fullest representation of the tweet text.
See #329 for context, and thanks to @harshbhatt for the bug report!
- Python
Published by edsu about 6 years ago
twarc - v1.8.2
The twarc command catches SIGTERM so that users can ctrl-c to stop the process. But previously it did so quietly, so as not to clutter up the console with a stack trace. But being quiet can be confusing when users put twarc into the background, and then log out, which can cause SIGTERM being sent to the process.
To aid in diagnosing twarc will now log when it has received a SIGTERM message before it stops. Thanks to @mielverkerken for identifying the issue, and helping diagnose it.
- Python
Published by edsu over 6 years ago
twarc - v1.8.0
This release includes a new command line option --app_auth that lets you control whether App Auth is used for connecting to the Twitter API instead of User Auth (the default).
This can be super useful in cases when you know the rate limits are better for App Auth (search) but not so useful in cases where it isn't (hydration).
For example when using User Auth you are limited to issuing 180 request per 15 minutes, and with App Auth you can issue 450 requests per 15 minutes. If you are searching for a large number of tweets switching to App Auth means you can retrieve 4,320,000 tweets in 24 hours, as compared with 1,728,000.
- Python
Published by edsu over 6 years ago
twarc - v1.7.1
v1.7.1 adds the ability to filter tweets on language. You can use from the command line, with one or more --lang arguments. So for example this will filter tweets that mention Paris or Madrid that are written in French or Spanish.
twarc filter paris,madrid --lang es --lang fr
You can also use programmatically:
python
for tweet in t.filter('paris,madrid', lang=['es', 'sp']):
print(tweet['full_text'])
Thanks for the idea @tugrulz and for your patience!
- Python
Published by edsu about 7 years ago
twarc - v1.4.3
v1.4.3 release includes three new features:
Español
Thanks to Tina Figueroa @tinafigueroa the Twarc documentation has been translated into Spanish!! Hopefully this will open up twarc and DocNow to new communities of users worldwide. Thanks also to @kayiwa for the assist.
Configure
Unless you tell it to do otherwise twarc's configuration is found at $HOME:.twarc and it contains a set of profiles that contain that API keys to use when talking to the Twitter API. Previously you could only run twarc configure to add the initial profile, and subsequent profiles needed to be added manually with a text editor.
With this release you can repeatedly call configure if you want to add new profiles. In addition it will only prompt you for the Twitter application's consumer key and consumer secret and will give you a URL to visit in your browser to grant user access to the application.
This is useful in situations where you want to easily reuse the same application but with different user accounts. It's also handy in instructional settings where the instructor may want to create an application for class and give the keys to students who will not be required to link their Twitter account to their cell phone. In those situations it's a good idea to delete the application after the instruction has completed.
Here's what it looks like on the console:
``` Twarc needs to know a few things before it can talk to Twitter on your behalf.
Please enter your Twitter application credentials from apps.twitter.com:
consumer key: VYhiNthiswontworksL7yjho8Vg consumer secret: S8tanL6o7A4hNneitherwillthisqPCjSR2Z6NfZrAHfhWA0
Please log into Twitter and visit this URL in your browser: https://api.twitter.com/oauth/authorize?oauth_token=mlZQowblahblahblahMHkpAAABYumdhl4
After you have authorized the application please enter the displayed PIN: 7814642
The credentials for edsu have been saved to your configuration file at /Users/edsu/.twarc
✨ ✨ ✨ Happy twarcing! ✨ ✨ ✨ ```
Protected
It appears that Twitter's search and streaming APIs do not return protected tweets even if the authenticating user is entitled to see them. This is a good thing, but rather than relying on this undocumented behavior v1.4.3 will inspect all tweet and user objects and filter out protected ones unless twarc was specifically asked to not do that. You can use the --protected to control this. By default it is set to false, which filters out protected tweets.
- Python
Published by edsu about 8 years ago
twarc - v1.4.0
v1.4.0 adds a new output format csv-excel that modifies some of the columns to prevent newlines from being present in quoted strings. While it's valid to have newlines within quoted column values, they cause some problems for Excel. Thanks to @justinlittman for diagnosing and resolving the issue.
- Python
Published by edsu over 8 years ago
twarc - v1.3.5
This bugfix release fixes an install problem on Windows that was causing the twarc.py command line to fail to execute. It also includes a fix for not often used subcommands like places that were failing because they weren't encoding JSON properly.
- Python
Published by edsu over 8 years ago
twarc - v1.3.1
This release brings CSV output to twarc.
twarc search blacklivesmatter --format csv > blacklivematter.csv
If you prefer you can output directly to a named file:
twarc search blacklivesmatter --format csv --outupt blacklivesmatter.csv
You can also split the output across numbered files if you want. For example this will rotate files every 10,000 rows:
twarc search blacklivesmatter --format csv --split 10000 --output blacklivesmatter.csv
Which will write files like:
blacklivesmatter-001.csv
blacklivesmatter-002.csv
blacklivesmatter-003.csv
...
- Python
Published by edsu over 8 years ago