Nano GPT

I am attempting to re create Andrej Karpathy's Nano GPT from scratch. In this process I have learnt how to implement Self Attention and Multi Head Attention from scratch.

Conda Environment

To set up the project environment, you can create a Conda environment using the provided YAML file:

conda env create -f environment.yml

Activate the Environment

conda activate nano_gpt

Here is some shakespeare text that is generated from my model

The code is running on cuda : GPU=NVIDIA GeForce RTX 3090
Resuming training from iteration 1500
step 0: train loss 1.4873, val loss 1.6777
step 500: train loss 1.3878, val loss 1.6048
step 1000: train loss 1.3184, val loss 1.5606
step 1500: train loss 1.2632, val loss 1.5244
step 2000: train loss 1.2160, val loss 1.5077
step 2500: train loss 1.1759, val loss 1.5014
step 3000: train loss 1.1378, val loss 1.4914
step 3500: train loss 1.1022, val loss 1.4938
step 4000: train loss 1.0682, val loss 1.4876
step 4500: train loss 1.0347, val loss 1.4992

Thy wrongring friends to excuse was,
To drown their soon soldiers: but I think
To devited my territy,
Which these-boroughs ladies hurl, and their own
hate them bring feed not that like hot well.
But have it doth commend up to the lighttle;
Or her deadly desire hour so ready;
So plead as steel how these perpily ones!
It in the enemy offerance
You to Bolphy Henry both and forsake,
Pardowned thy victory.

Nurse:
My woman, but we lords
You call me a clap of his ears cript and
No sbanish appetition;

FYI: I dont really understand shakespeare english, so I used chatGPT to translate. here is the translation:

Your mistaken friends made excuses
To drown their own soldiers, but I believe
They planned to divide my territory,
Which these ladies from the towns despise,
And their hatred won't diminish easily.
But I must admit it's well-justified;
Or her deadly desire is always so prepared;
She pleads like steel against these guilty ones!
It's a hostile act, an offer to the enemy,
You both betray Bolphy Henry and abandon,
Disgracing your victory.

Nurse:
My lady, these lords
You call me a slap in his face, scorned and
Not banished by desire;