Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/kyegomez/aoa-torch

Implementation of Attention on Attention in Zeta

https://github.com/kyegomez/aoa-torch

Science Score: 26.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ○
    DOI references
  • ○
    Academic publication links
  • ○
    Committers with academic emails
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (6.1%) to scientific vocabulary

Keywords

ai artificial-intelligence gpt4 machine-learning multi-modal multi-modality research

Keywords from Contributors

projection interactive multimodla serializer measurement cycles gemini packaging charts network-simulation
Last synced: 6 months ago · JSON representation

Repository

Implementation of Attention on Attention in Zeta

Basic Info
Statistics
  • Stars: 4
  • Watchers: 2
  • Forks: 0
  • Open Issues: 4
  • Releases: 0
Topics
ai artificial-intelligence gpt4 machine-learning multi-modal multi-modality research
Created about 2 years ago · Last pushed 6 months ago
Metadata Files
Readme Funding License

README.md

Multi-Modality

Attention on Attention Implementation

This is a practice implementation after randomly finding it on Lucidrain's repo, I'm implementing the model architecture just for practice!

Basically the architecture is: x => q, k, v -> multihead attn with residual q -> concat -> 2 linear projects ->sigmoid -> mult -> add -> norm -> ffn -> add -> norm with residual of first add and norm

Install

pip3 install --upgrade aoa-torch

Usage

AoA Module

```python

import torch

from aoa.main import AoA

x = torch.randn(1, 10, 512) model = AoA(512, 8, 64, 0.1) out = model(x) print(out.shape)

```

AoATransformer

```python import torch from aoa.main import AoATransformer

x = torch.randint(0, 100, (1, 10)) model = AoATransformer(512, 1, 100) out = model(x) print(out.shape)

```

Citations

bibtex @misc{rahman2020improved, title = {An Improved Attention for Visual Question Answering}, author = {Tanzila Rahman and Shih-Han Chou and Leonid Sigal and Giuseppe Carenini}, year = {2020}, eprint = {2011.02164}, archivePrefix = {arXiv}, primaryClass = {cs.CV} }

bibtex @misc{huang2019attention, title = {Attention on Attention for Image Captioning}, author = {Lun Huang and Wenmin Wang and Jie Chen and Xiao-Yong Wei}, year = {2019}, eprint = {1908.06954}, archivePrefix = {arXiv}, primaryClass = {cs.CV} }

License

MIT

Todo

  • [ ] Create sample training code using enwiki8

Owner

  • Name: Eternal Reclaimer
  • Login: kyegomez
  • Kind: user
  • Location: Miami
  • Company: Automated Public Assistance Company

Leader of Agora, the open source Multi-Modal AI research lab join our community here: https://discord.gg/hCJpnhA5aP

GitHub Events

Total
  • Delete event: 15
  • Issue comment event: 26
  • Pull request event: 29
  • Create event: 14
Last Year
  • Delete event: 15
  • Issue comment event: 26
  • Pull request event: 29
  • Create event: 14

Committers

Last synced: 9 months ago

All Time
  • Total Commits: 15
  • Total Committers: 3
  • Avg Commits per committer: 5.0
  • Development Distribution Score (DDS): 0.2
Past Year
  • Commits: 0
  • Committers: 0
  • Avg Commits per committer: 0.0
  • Development Distribution Score (DDS): 0.0
Top Committers
Name Email Commits
Kye k****e@a****m 12
dependabot[bot] 4****] 2
Eternal Reclaimer 9****z 1
Committer Domains (Top 20 + Academic)
apacmediasolutions.com: 1

Issues and Pull Requests

Last synced: 6 months ago

All Time
  • Total issues: 1
  • Total pull requests: 70
  • Average time to close issues: 2 months
  • Average time to close pull requests: about 2 months
  • Total issue authors: 1
  • Total pull request authors: 1
  • Average comments per issue: 1.0
  • Average comments per pull request: 1.13
  • Merged pull requests: 1
  • Bot issues: 1
  • Bot pull requests: 70
Past Year
  • Issues: 0
  • Pull requests: 39
  • Average time to close issues: N/A
  • Average time to close pull requests: about 2 months
  • Issue authors: 0
  • Pull request authors: 1
  • Average comments per issue: 0
  • Average comments per pull request: 1.08
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 39
View more stats
Top Authors
Issue Authors
  • dependabot[bot] (1)
Pull Request Authors
  • dependabot[bot] (70)
Top Labels
Issue Labels
dependencies (1) github_actions (1) no-pr-activity (1)
Pull Request Labels
dependencies (70) python (41) no-pr-activity (36) github_actions (29)

Dependencies

.github/workflows/code_quality_control.yml actions
  • actions/checkout v3 composite
  • actions/setup-python v4 composite
  • ad-m/github-push-action master composite
.github/workflows/cos_integration.yml actions
  • actions/checkout v2 composite
  • actions/setup-python v4 composite
.github/workflows/docs.yml actions
  • actions/checkout v3 composite
  • actions/setup-python v4 composite
.github/workflows/docs_test.yml actions
  • actions/checkout v3 composite
  • actions/setup-python v4 composite
.github/workflows/label.yml actions
  • actions/labeler v4.3.0 composite
.github/workflows/lints.yml actions
  • actions/checkout v2 composite
  • actions/setup-python v4 composite
.github/workflows/pr_request_checks.yml actions
  • actions/checkout v2 composite
  • actions/setup-python v4 composite
.github/workflows/pull-request-links.yml actions
  • readthedocs/actions/preview v1 composite
.github/workflows/pylint.yml actions
  • actions/checkout v4 composite
  • actions/setup-python v4 composite
.github/workflows/python-publish.yml actions
  • actions/checkout v4 composite
  • actions/setup-python v4 composite
  • pypa/gh-action-pypi-publish f8c70e705ffc13c3b4d1221169b84f12a75d6ca8 composite
.github/workflows/quality.yml actions
  • ./.github/actions/init-environment * composite
  • actions/checkout v4 composite
.github/workflows/ruff.yml actions
  • actions/checkout v4 composite
  • chartboost/ruff-action v1 composite
.github/workflows/run_test.yml actions
  • actions/checkout v4 composite
  • actions/setup-python v4 composite
.github/workflows/stale.yml actions
  • actions/stale v8 composite
.github/workflows/test.yml actions
  • ./.github/actions/poetry_setup * composite
  • actions/checkout v4 composite
.github/workflows/testing.yml actions
  • actions/checkout v2 composite
  • actions/setup-python v4 composite
.github/workflows/unit-test.yml actions
  • actions/checkout v4 composite
  • actions/setup-python v4 composite
.github/workflows/welcome.yml actions
  • actions/first-interaction v1.1.1 composite
pyproject.toml pypi
  • python ^3.9
requirements.txt pypi