[1] Examples of this kind of positive transfer: (1) pre-training on language helps speech generation, (2) pre-training on code helps in broader reasoning tasks, (3) LLMs improve on math and coding after multimodal training.[2] scGPT is an early example of an approach that can potentially scale much further.[3] In a recent paper, a group of biologis... See more

Towards Universal Simulation

October 11, 2024

DeepMind’s initial pitch was to “solve intelligence and then use it to solve everything else.” This has been the guiding philosophy of AGI labs for the past decade. I believe that rather than solving intelligence, we should be focusing on solving the “everything else” part first. That is, we should be bu... See more

Following this anthropomorphic approach, AGI labs often describe a future where their models can solve difficult scientific problems by suggesting wet-lab experiments to run and interpreting the results. This fundamentally misidentifies the main bottleneck, which is not coming up with ideas for experiments, but rather running them. The best way to ... See more

The simple idea of universal simulation is to train models on a far wider variety of data and modalities than we currently do, which may include human-generated data, but also observational data from scientific experiments and raw sensor data, all at the same time, with the goal of predicting an increasingly large percentage of all observations in ... See more

That is, we should be building systems for universal simulation.