[N] Understanding & Fine-tuning Vision Transformers

A neat blog post by Mayank Pratap Singh with excellent visuals introducing ViTs from the ground up. The post covers:

• Patch embedding
• Positional encodings for Vision Transformers
• Encoder-only models ViTs for classification
• Benefits, drawbacks, & real-world applications for ViTs
• Fine-tuning a ViT for image classification.

Full blogpost here:
https://www.vizuaranewsletter.com/p/vision-transformers

Additional Resources:

• An Image is Worth 16x16 Words https://arxiv.org/abs/2010.11929
  
• Yannic Kilcher Discussion of the paper https://www.youtube.com/watch?v=TrdevFK_am4
  
• Generating Long Sequences with Sparse Transformers https://arxiv.org/abs/1904.10509
  
• Generative Pretraining from Pixels https://proceedings.mlr.press/v119/chen20s.html
  

I've included the last two papers because they showcase the contrast to ViTs with patching nicely. Instead of patching & incorporating knowledge of the 2D input structure (*) they "brute force" their way to strong internal image representations at GPT-2 scale. (*) Well it should be noted that https://arxiv.org/abs/1904.10509 does use custom, byte-level positional embeddings.