A friend of mine recently bombed MLE interview at NVIDIA, they asked: "We need to deploy a Llama-3 70B model on hardware with limited VRAM. You propose quantization. When is this a bad idea?" Here's how you break it down:

Not to say I saw this coming but the average ML engineer knows nothing about computers I’m not surprised people whose job it is to make hardware perform at its maximum is mogging them. This is the result of everyone hiring people who can solve leetcode hards in Python and don’t know the difference between big endian... See more