Luc Cheung

Self-Attention in Detail

Let’s first look at how to calculate self-attention using vectors, then proceed to look at how it’s actually implemented – using matrices.

The first step in calculating self-attention is to create three vectors from each of the encoder’s input vectors (in this case, the embedding of each word). So for each word, we create a... See more

Simplicity is difficult because most of us are overcompensating for uncertainty. Adding something is easy. But removing something is hard, because it requires conviction. It’s easier to hedge against uncertainty, entertain multiple paths, and dilute your focus than to develop a strong opinion about what to exclude.

Falling into the trap of thinking other users will be like you is especially pernicious because the people building the product are usually among that early adopter cohort. The easiest north star for a product person is their own intuition. But if they're working on a product that requires customer segmentation, being in the early adopter cohort... See more