Sublime
An inspiration engine for ideas
🚀 ADAM in Action 🤖
Ever wonder why ADAM is still the go-to optimizer? In 2014, Kingma & Ba introduced Adaptive Moment Estimation (Adam), which tunes each parameter’s learning rate on the fly for speedier, more stable training!
🔹 Blends momentum & RMSProp
🔹 Corrects for early... See more
daily.ml.papersinstagram.comPeter Norvig, director of research at Google, told me at one point that it was the most widely used learner there, and Google uses machine learning in every nook and cranny of what it does. It’s not hard to see why Naïve Bayes would be popular among Googlers. Surprising accuracy aside, it scales great; learning a Naïve Bayes classifier is just a
... See morePedro Domingos • The Master Algorithm: How the Quest for the Ultimate Learning Machine Will Remake Our World
Efficient optimization in e.g. deep networks uses gradients, yet biological evolution uses random mutations - at least according to the textbooks.
Here I review classic evidence for, and recent evidence against, random mutations.
Blog: https://t.co/BlosxnGimg
Thread: 👇1/n... See more
Joram Keijserx.com
1/8 Second Order Optimizers like SOAP and Muon have shown impressive performance on LLM optimization. But are we fully utilizing the potential of second order information? New work: we show that a full second order optimizer is much better than existing optimizers in terms of iteration complexity (~5x over SOAP and ~15x over Muon).
I've been hacking with auto-generating KGs all week, and the tech is way more useful than I expected.
It is cheap enough to run, and it gives LLMs the ability to handle a new class of population-level queries they can't yet, like "Find all the founders that worked at Google." https://t.co/JuAwFVlSOw