Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/lromul/gramtion

Twitter bot for generating photo descriptions (alt text)

https://github.com/lromul/gramtion

Science Score: 13.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ○
    .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ○
    Committers with academic emails
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (11.9%) to scientific vocabulary

Keywords

accessibility alt-text deep-learning image-captioning twitter twitter-bot
Last synced: 4 months ago · JSON representation

Repository

Twitter bot for generating photo descriptions (alt text)

Basic Info
Statistics
  • Stars: 23
  • Watchers: 1
  • Forks: 2
  • Open Issues: 1
  • Releases: 0
Topics
accessibility alt-text deep-learning image-captioning twitter twitter-bot
Created over 5 years ago · Last pushed over 4 years ago
Metadata Files
Readme License

README.md

Title GRAMTION on the background of a black and white photo of clouds in the sky

Twitter bot for generating photo descriptions

According to Twitter support bot account has been suspended due to violations of rules: * Creating serial and/or multiple accounts with overlapping uses * Evading a permanent suspension by creating or using another account * Cross-posting Tweets or links across multiple accounts * Aggressive following, particularly through automated means

Twitter: https://twitter.com/GramtionBot

Source Code: https://github.com/lRomul/gramtion

This repo contains the source code of the Twitter @GramtionBot for generating photo descriptions.
Use cases and intends: * Help visually impaired Twitter users. Good image descriptions (alt text) will help them understand what is happening in an image. Instagram and Facebook use deep learning for image captioning. Twitter users can only add custom alt text descriptions themselves. Automation of alt text generation will help Twitter be more accessible. * Collect dataset for image captioning (legal issues for this use case are yet to be discussed). Annotations can be done by creating polls about prediction quality and getting corrected descriptions from users. Twitter API v1.1 has not the ability to create polls, but it will be added in API v2.

How to use

Tweet photo with mention @GramtionBot or reply with mention to a tweet with a photo and the bot will send you an auto-generated image description.

Dependencies

Gramtion is mainly built from ready-to-use third party libraries: * Image captioning model taken from self-critical.pytorch. * Evaluate text and image similarity with CLIP by OpenAI. * OCR and image labels by Google Vision Ai. * Bot written with Tweepy. * Configuration settings implemented with pydantic. * Docker image based on Dokai.

Current issues

  • Some descriptions may be confusing. Annotations may be created by using polls about prediction quality and getting corrected descriptions from users. Twitter API v1.1 has not the ability to create polls, but it will be added in API v2 endpoint POST /2/tweets.
  • For drawings and some other types of images, the predictions are pretty random.
  • Some results may reflect inherent gender and racial biases of open datasets.

Run own bot

To run an instance of the bot you need to install Docker and create Twitter API auth credentials.
If you have a Twitter developer account, but don't want to use it as a bot account, you can authenticate a new user that’s not has a developer account with twurl.

  • Create .env file with credentials.

    CONSUMER_KEY={{ consumer_key }} CONSUMER_SECRET={{ consumer_secret }} ACCESS_TOKEN={{ access_token }} ACCESS_TOKEN_SECRET={{ access_token_secret }}

  • Setup Google Vision AI and create account key (link). Copy key as google_key.json.

  • Run Docker container with running the bot

    bash docker run -d --restart=always \ --env-file .env \ -v google_key.json:/workdir/google_key.json \ --name=gramtion \ ghcr.io/lromul/gramtion:0.0.5

  • Open logs

    bash docker logs -f gramtion

  • Stop container

    bash docker stop gramtion docker rm gramtion

Owner

  • Name: Ruslan Baikulov
  • Login: lRomul
  • Kind: user
  • Location: Moscow, Russia

Deep Learning Engineer

GitHub Events

Total
Last Year

Committers

Last synced: 10 months ago

All Time
  • Total Commits: 126
  • Total Committers: 1
  • Avg Commits per committer: 126.0
  • Development Distribution Score (DDS): 0.0
Past Year
  • Commits: 0
  • Committers: 0
  • Avg Commits per committer: 0.0
  • Development Distribution Score (DDS): 0.0
Top Committers
Name Email Commits
Ruslan Baikulov r****3@g****m 126

Issues and Pull Requests

Last synced: 10 months ago

All Time
  • Total issues: 1
  • Total pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Total issue authors: 1
  • Total pull request authors: 0
  • Average comments per issue: 1.0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 0
  • Pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 0
  • Pull request authors: 0
  • Average comments per issue: 0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • jake-arkinstall (1)
Pull Request Authors
Top Labels
Issue Labels
Pull Request Labels

Dependencies

.github/workflows/build-push.yml actions
  • actions/checkout v2 composite
  • docker/login-action v1 composite
.github/workflows/linter.yml actions
  • actions/checkout v2 composite
  • actions/setup-python v2 composite
Dockerfile docker
  • ghcr.io/osai-ai/dokai 20.12-pytorch build
requirements.txt pypi
  • Shapely ==1.7.1
  • bottle ==0.12.19
  • ftfy ==6.0.1
  • google-cloud-vision ==2.0.0
  • python-dotenv ==0.14.0
  • regex ==2021.4.4
  • tqdm ==4.60.0
  • tweepy ==3.9.0
  • yacs ==0.1.8
tests/requirements.txt pypi
  • flake8 ==3.9.2 test