AI - Ongoing Model Improvements

While LLMs continue to devour web-scraped data, they’ll increasingly consume their own digital progeny as AI-generated content continues to flood the internet. This recursive loop, experimentally confirmed, erodes the true data landscape. Rare events vanish first. Models churn out likely sequences from the original pool while injecting their own un... See more

o, alternative pathways to building Type-2 reasoning-capable AI systems, likely using neurosymbolic approaches, have become much more attractive. People like Gary Marcus have argued for neurosymbolic approaches for decades. Such approaches combine the pattern recognition of neural nets, like LLMs, with symbolic reasoning’s logic and rules. Vinod Kh... See more

The incentive to find breakthrough science that provides a performance pathway other than scaling is increasing. A GPT-6 class model will cost $10 billion and three years to improve on GPT-5 by some uncertain degree. That’s a ton of time and a lot of cash for an uncertain payout: in other words, a substantial prize for anyone who can figure out pro... See more

First, operationalisation takes a long time, often revealing hidden potential in existing technologies. As Jack Clark notes, “if we stopped all AI progress today, there’s a huge capability overhang”. Even without further model development, building the right scaffolding can unlock surprising capabilities. This scaffolding isn’t just software; it in... See more

One way of thinking about this is Daniel Kahneman’s simple model of thinking: System 1 and System 2. System 1 thinking is fast and intuitive. Current AI models’ pattern recognition and next-token prediction are good examples of this. System 2 thinking is slow and analytical, akin to genuine reasoning and understanding. It is System 2 thinking where... See more

LeCun points to four essential characteristics of human intelligence that current AI systems, including LLMs, can’t replicate: reasoning, planning, persistent memory, and understanding the physical world. He stresses that LLMs’ reliance on textual data severely limits their understanding of reality: “We’re easily fooled into thinking they are intel... See more

I still think that we are thinking about scaling of network-based intelligence looking at this particular network. An extended network of machines and humans in larger networks facilitated by smaller and increasingly embodied devices might in themselves add an AI-augmented neocortex to our (human-machine) systems. Intelligence will continue to emer... See more

He (Claude Shannon) had this great intuition that information is maximized when you’re most surprised about learning about something.” ~ Tara Javidi

“Whenever we are surprised by something, even if we admit that we made a mistake, we say, ‘Oh I’ll never make that mistake again.’ But, in fact, what you should learn when you make a mistake because you... See more

This is all early, and based on first impressions, but I think that voice capabilities like GPT-4o’s are going to change how most people interact with AI systems. Voice and visual interactions are more natural than text and will have broader appeal to a wider audience. The future will involve talking to AI.

Even though the underlying model is no different than the usual GPT-4o, the addition of voice has a lot of implications. A voice-powered tutor works very differently than one that communicates via typing, for example. It can also speak many other languages providing new approaches to cross-cultural communication. And I have no doubt people will hav... See more