Bahdanau Attention vs Luong Attention | by Navdeep Sharma | Jan, 2025

Greetings!

This weblog is in continuation of earlier weblog Demystifying Attention Mechanism: Revolutionising Encoder-Decoder Architectures. So, for those who haven’t learn it I extremely encourage you to learn it first. Let’s proceed our dialogue additional and focus on about two important kind of consideration mechanisms.

Bahdanau Consideration

Under is the temporary overview of the structure for Bahdanau Consideration.

Now allow us to get extra deep into the structure and examine its working.

Now allow us to see how α’s (alignment scores) are calculated in Bahdanau Consideration to lastly get c (consideration enter). For simplicity let’s assume our encoder hidden states (hj) and decoder hidden states (si) are of 4-dimension.

h1 = [h11, h12, h13, h14]

s0 = [s01, s02, s03, s04]

Now, under is the method for calculating consideration enter c1 (at timestamp 1).