Skip to content

This repo is for building a transformer model from scratch that is able to generate Shakespear-like literature

Notifications You must be signed in to change notification settings

yebyyy/Infinite-Shakepear

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

18 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Infinite Shakespear

This is a decoder only transformer, similar to a GPT structure that generates Shakespear-like text.

The Theoretical part of the model follows the decoder structure explained in the famous Attention is All You Need paper.

Other than the basic Multihead Self Attention and Feedforward Structure, also implemented LayerNorm, Residual Connections from Deep Residual Learning for Image Recognition, and Dropout from Dropout: A Simple Way to Prevent Neural Networks from Overfitting

lecture.ipynb follows the instruction by Andrej Karpathy's lecture Let's Build GPT

Final.py is the final Decoder transformer model with 10 million parameters and input.txt as all the shakespear work input.

About

This repo is for building a transformer model from scratch that is able to generate Shakespear-like literature

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published