Compact Transformers

[dmm]

For those of us (like myself) who'd like to experiment with changing Transformer architecture on a home personal computer.

Links are in the comments.

Comments

[dmm]

I just reproduced it on the latest PyTorch (a later version of PyTorch than their README suggests; I used Windows 10, but it should work elsewhere too).

The result was a bit lower than in their table:

[Epoch 200] Top-1 88.34 Time: 64.41
Script finished in 64.41 minutes, best top-1: 88.34, final top-1: 88.34

They had more luck and got 89.17 (the training is a bit jittery, but generally OK).