The main idea in the paper is that the performance of regular Multi-layer Perceptron (MLP) can be significantly improved if we use Transformers to transforms regular categorical embeddings into contextual ones.

The TabTransformer is built upon self-attention based Transformers. The Transformer layers transform the embed- dings of categorical features into robust contextual embed- dings to achieve higher prediction accuracy.

• Paper: https://arxiv.org/pdf/2012.06678.pdf
• Code: https://github.com/aruberts/TabTransformerTF
• Article: Transformers for tabular data: Tabtransformer deep dive