Attention in transformers (early view) (Patreon)
Content
Hey folks, here's a draft of the next chapter about Transformers focussed on the attention mechanism. (Note, I've now updated it based on many comments below, thanks for the feedback!)
Also, the first chapter is now public, so feel free to share that with anyone you think might like it, thanks again for all the feedback and input on that one.
I've gotten to most of the to-do's I had since the first draft I put up earlier this week. The only lingering item is the one below; with any video, there’s always a balance between adding more details to proactively answering viewer questions, vs. keeping things concise and unbloated, and right now I'm inclined to say things already being long enough.
Examples to motivate multiple layers: On the one hand, it might suffice to simply say "There are many attention blocks", but a longer version of the script for this video included a motivating example.
Consider the phrase "The glass ball fell on the steel table, and it shattered". How much should the words “ball” and “table” attend to the word “it”, say in an attention head trying to relate pronouns to the nouns they refer to? If this was the earliest attention block, only able to see the meanings and positions of the three words, in isolation, it’s not so clear.But suppose before reaching this attention head, these word embeddings had gone through an attention block which let "ball" imbibe some of the meaning from "glass", which let "table" imbibe some meaning from "steel", and which let "it" to take in some meaning form "shattered". After that, a later attention head now has much more of a chance to recognize that the connection to "ball" should be stronger, given the association between glass and shattering.
Worth including? I'm not sure.