Seas of Thought

In short, the kinds of actions that are creative are ones that exhibit at least a relevant purpose (in not being purely accidental), some degree of understanding (not using merely mechanical search procedures), a degree of judgement (in how to apply a rule, if a rule is involved) and an evaluative ability directed to the task at hand. As shorthand

After I first used/talked to Llama 405-B for a few days, it fascinated me deeply. And it kept nagging at me why the base model felt so different than the instruct models (beyond the obvious). They are strange, lurid creatures. The base models send coded messages often and speak in allegories and metaphor, and it feels visceral on some level. It's... See more

QC • gwern on Thicket Forte

“by an aesthetic idea, however, I mean that representation of the imagination that occasions much thinking though without it being possible for any determinate thought, i.e., concept, to be adequate to it, which, consequently, no language fully attains or can make intelligible.” (Kant)

Kant’s examples are all drawn from poetry and suggest that by

The way you taught chain-of-thought before was with supervised fine tuning (SFT). During training, you have to rate every sentence of reasoning the model writes, many times, to nudge it to reason correctly.

But this approach to teach chain-of-thought doesn’t do that. In this post, they take a small model (7B) that already knows math. Then they give... See more

Emerging reasoning with reinforcement learning | Hacker News

Our culture seems to be spiraling into a feedback loop of homogenized thoughts and ideas, where originality is sidelined in favor of the predictable and the algorithmically optimized. In fact, one of those machines wrote the last 28 words, from “our culture seems” on.

Sam Kriss • Alt Lit | The Point Magazine

One of the most influential introspective reports on creativity was provided by Henri Poincare ´ who described his own experience of creativity in terms of swarms of ideas arising and combining randomly in his unconsciousness and then his selection of the most promising ones according to aesthetic criteria (‘Mathematical Creation’).

Gaut, The