tiny-tf-transformer (yet another transformer implementation)

This is a tiny transformer implementation in tensorflow 2. It is based on the attention is all you need paper. The transformer implementation is based on the tensorflow tutorial. It is mainly used for my own learning purposes and also has some paper implementation related to transformers.

Installation

pip install .

Projects

• Transformer
• Pix2Seq: A Language Modeling Framework for Object Detection. Code
• ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision.
• BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension.
• T5: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.

Tests

pytest

References

• Attention is all you need
• Transformer tutorial