Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/aveek-saha/transformer

A TensorFlow 2.0 Implementation of the Transformer: Attention Is All You Need

https://github.com/aveek-saha/transformer

Science Score: 23.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ○
    .zenodo.json file
  • ○
    DOI references
  • ✓
    Academic publication links
    Links to: arxiv.org
  • ○
    Academic email domains
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (4.2%) to scientific vocabulary

Keywords

attention-is-all-you-need attention-mechanism attention-network keras tensorflow2 transformer
Last synced: 5 months ago · JSON representation

Repository

A TensorFlow 2.0 Implementation of the Transformer: Attention Is All You Need

Basic Info
  • Host: GitHub
  • Owner: Aveek-Saha
  • License: apache-2.0
  • Language: Python
  • Default Branch: master
  • Homepage:
  • Size: 39.1 KB
Statistics
  • Stars: 1
  • Watchers: 2
  • Forks: 0
  • Open Issues: 0
  • Releases: 0
Topics
attention-is-all-you-need attention-mechanism attention-network keras tensorflow2 transformer
Created over 5 years ago · Last pushed over 5 years ago
Metadata Files
Readme License

README.md

Transformer

A TensorFlow 2.x implementation of the Transformer from Attention Is All You Need (Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin, arxiv, 2017).

This is my attempt at trying to understand and recreate the transformer from the research paper. This is just for my own understanding of the subject and is by no means perfect.

In order to understand and implement the transformer I've taken the help of various tutorials and code guides, which I'll be linking in the resources section.

Requirements

  • tensorflow==2.1.0
  • numpy==1.16.5
  • tensorflow_datasets==3.2.1

How to run

python train.py

Resources

  • The original paper: https://arxiv.org/pdf/1706.03762.pdf
    • Input and training pipeline: https://www.tensorflow.org/tutorials/text/transformer
  • An useful article explaining the paper: https://mlexplained.com/2017/12/29/attention-is-all-you-need-explained/
  • Another useful article explaining the paper: http://jalammar.github.io/illustrated-transformer/
  • A tensorflow 1.x transformer implementation: https://github.com/Kyubyong/transformer/
  • The official implementation in tensorflow: https://github.com/tensorflow/tensor2tensor

Owner

  • Name: Aveek Saha
  • Login: Aveek-Saha
  • Kind: user
  • Location: Boston, MA
  • Company: @akamai

Cloud Computing, Machine Learning and Full Stack. SDE co-op @akamai. MSCS student @northeastern. Previously at @HewlettPackard, @altimetrik & @ IIT Kgp.

GitHub Events

Total
Last Year

Issues and Pull Requests

Last synced: 11 months ago

All Time
  • Total issues: 0
  • Total pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Total issue authors: 0
  • Total pull request authors: 0
  • Average comments per issue: 0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 0
  • Pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 0
  • Pull request authors: 0
  • Average comments per issue: 0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
Pull Request Authors
Top Labels
Issue Labels
Pull Request Labels