a quick note from @karpathy talk, which afterwards I was fully immersed in the talk I stopped writing down/taking notes: While it is interesting to think about LLMs in terms of "electricity" -> CAPEX to train an LLM (~= to build the grid) -> OPEX to serve intelligence over increasingly homogeneous API (prompt, image, to... See more

I'm writing a blog post on general advice for building a GPU machine learning rig for home use. WIP version in the post below https://t.co/xasO6MD6Y4

Before paying for high-end GPUs for LLM inference, understand your numbers first. For example, you can deploy most 7B models using AWS EC2 G5 or Microsoft NVadsA10v5 instances, but would you effectively saturate GPU utilization? To clarify this, I've created a simple visualization.___LINEBRE... See more