https://github.com/abudubai16/nlp-from-scratch
Science Score: 13.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
○.zenodo.json file
-
○DOI references
-
○Academic publication links
-
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (6.6%) to scientific vocabulary
Repository
Basic Info
- Host: GitHub
- Owner: abudubai16
- Language: Jupyter Notebook
- Default Branch: main
- Size: 25.4 KB
Statistics
- Stars: 1
- Watchers: 1
- Forks: 0
- Open Issues: 0
- Releases: 0
Metadata Files
README.md
NLP from Scratch (No Pretrained Models)
Approach:
• Download the dataset
• Use a tokenizer to convert the input sequence of words into integers.
• Create a model that embeds the semantic meaning of each individual word into a vector.
• Give the vectors positional embedding so that the semantic meaning of their positions are not lost.
• Using a simple pytorch nn.TransformerEncoderLayer and nn.TransformerEncoder get the contextual meaning of the input sequence.
• Using the contextual meaning of the previous sequence predict the next token in the sequence using a fully connected layer.
• Compare the predicted token to the target token and use CrossEntropyLoss for backpropagation.
Details:_
• For the tokenizer used hugging face's tokenizer library, and made a custom tokenizer for the dataset used here, because its not a general purpose tokenizer, the speed of encoding is much faster. The tokenizer the resulted for the dataset was a subword tokenizer, which leads to much greater inference.
• The dataset used is WikiText103, which is a dataset containing 103 million groups of sentences, each data point in the code is the single sentence in the dataset which is then padded to a specified length
• Because the dataset is too huge to load all at once, for the formation of proper embedding a small chunk of data is loaded and the model is trained on that. Once the training is done the next chunk of data is loaded, and so on.
• The model achieves english like sentences, however the complete coherence of sentences are still missing, perhaps more training is required or a bigger model is needed(unfortunately I do not have the resources to train a bigger model without spending hours).
Owner
- Login: abudubai16
- Kind: user
- Repositories: 1
- Profile: https://github.com/abudubai16
GitHub Events
Total
- Watch event: 1
Last Year
- Watch event: 1