Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/ammarlodhi255/image-captioning-system-to-assist-the-blind

An image captioning system that is able to predict and speak out a caption of an image taken by visually impaired.

https://github.com/ammarlodhi255/image-captioning-system-to-assist-the-blind

Science Score: 26.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ○
    Academic email domains
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (12.6%) to scientific vocabulary

Keywords

bidirectional-lstm cnn computer-vision computer-vision-algorithms css3 deep-learning html5 image-captioning javascript javascript-es6 lstm-neural-network ml-web nlp nlp-machine-learning resnet resnet-50 vgg16
Last synced: 5 months ago · JSON representation

Repository

An image captioning system that is able to predict and speak out a caption of an image taken by visually impaired.

Basic Info
  • Host: GitHub
  • Owner: ammarlodhi255
  • Language: Jupyter Notebook
  • Default Branch: master
  • Homepage:
  • Size: 10.7 MB
Statistics
  • Stars: 9
  • Watchers: 1
  • Forks: 7
  • Open Issues: 1
  • Releases: 0
Topics
bidirectional-lstm cnn computer-vision computer-vision-algorithms css3 deep-learning html5 image-captioning javascript javascript-es6 lstm-neural-network ml-web nlp nlp-machine-learning resnet resnet-50 vgg16
Created over 3 years ago · Last pushed over 1 year ago
Metadata Files
Readme

README.md

Image Captioning System to Assist The Blind

Table of Contents

About

The goal of the project is to develop a system using deep learning techniques to assist visually impaired individuals in obtaining information by describing images taken by them. The system uses a CNN model and an NLP model to create a single image captioning system that takes image features as input and generates a text sequence describing the image.

Incorporated state-of-the-art pre-trained models, such as ResNet50, VGG16, and VGG19, for image feature extraction and LSTM and Bidirectional LSTM for text generation. Evaluated various models to determine the best-performing model with a BLEU-score of 0.61 and deployed it using Flask and pyttsx3 for web and text-to-speech functionality in the app.

Getting Started

These instructions will get you a copy of the project up and running on your local machine.

  1. Clone the project repository from GitHub:

bash git clone https://github.com/ammarlodhi255/image-captioning-system-to-assist-the-blind.git

  1. Navigate to the project directory:

bash cd image-captioning-system-to-assist-the-blind

  1. Create a virtual environment for the project:

bash python3 -m venv env

  1. Activate the virtual environment:

bash source env/bin/activate

  1. Export the Flask app:

bash export FLASK_APP=app.py

  1. Run the Flask app:

bash flask run

Screenshots

Dataset Split

Dataset Design

Model Anatomy

Model Anatomy

Project Workflow

Project Workflow

Results

Results Table

Final Outcome

Home Interface Browse Selected Choice Generating Generated

Additional Outputs

Output1 Output2

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

  1. Fork the repository
  2. Create your feature branch (git checkout -b feature/AmazingFeature)
  3. Commit your changes (git commit -m 'Add some AmazingFeature')
  4. Push to the branch (git push origin feature/AmazingFeature)
  5. Open a Pull Request

Owner

  • Name: Ammar Ahmed
  • Login: ammarlodhi255
  • Kind: user
  • Location: Sukkur, Pakistan

A computer scientist at heart, interested in AI, software development, and space.

GitHub Events

Total
  • Issues event: 1
  • Watch event: 2
Last Year
  • Issues event: 1
  • Watch event: 2

Issues and Pull Requests

Last synced: 10 months ago

All Time
  • Total issues: 1
  • Total pull requests: 15
  • Average time to close issues: N/A
  • Average time to close pull requests: 19 minutes
  • Total issue authors: 1
  • Total pull request authors: 4
  • Average comments per issue: 0.0
  • Average comments per pull request: 0.13
  • Merged pull requests: 11
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 1
  • Pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 1
  • Pull request authors: 0
  • Average comments per issue: 0.0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • harshk-89 (1)
Pull Request Authors
  • adnanalisolangi (4)
  • AbdulManaf12 (4)
  • ammarlodhi255 (1)
  • filza-a (1)
Top Labels
Issue Labels
Pull Request Labels