What do LLMs Want?

Plus! Governance; Implementation Risk; Jailbreaking; In-Kind Deals; Revalued Inputs

Byrne Hobart Byrne Hobart

In this issue:

audio-thumbnail
The Diff December 1st 2025
0:00
/870.739592

What do LLMs Want?

One bit of data doesn't "want" anything. Two bits don't, either. Take a sequence of two-bit stretches of data, and you won't find any wanting—until you take three billion such entries, feed them into the right lexer, preprocessor, and code generator, you get roughly all the conscious wanting we can agree happens.

People use the term "wanting" in a broader figurative sense; water wants to find its level, the stock wants to trade at whatever price establishes an equilibrium between buyers and sellers; the political party wants to balance its ideological goals against pragmatic concerns about the next election, etc. It's a useful model, albeit useful enough that it can be overused in a way that confuses people. Richard Dawkins popularized the Williams/Hamilton approach to understanding evolution in The Selfish Gene: instead of thinking of evolution at the level of individual creatures, think about it at the level of genes. It's bad for your survival to care too much about your kids, or your extended community, but if they share enough of your genes, it can be good for those genes. So there can be selection for traits like willingness to cooperate or reluctance to engage in opportunistic theft and violence even if these traits make the individuals who have them worse-off from a pure evolutionary (individual survival) standpoint.[1]

This is all pretty cool stuff, but leads to the easy misinterpretation that it's a claim about your genes "wanting" you to have kids and take decent care of them in order to maximize your net present great-to-the-nth-grandkid count. You probably aren't thinking that way explicitly (and doing so probably hurts this goal). But your existence is the result of someone successfully, often implicitly, optimizing in this way, so, over time, things with genomes and bodies will tend to act as if they're trying to maximize their descendant count, because everything that didn't try or didn't do it well enough is extinct.[2] Darwinian evolution is perhaps the most insightful tautology in history: every living thing is descended from the living things that were, on average, good at leaving behind descendants before they died.

You can apply evolutionary thinking in a hazy way when you want to understand other systems; industries are a bit like ecosystems, and nature is fractally weird enough that even when the analogies break down, you can rescue them—"corporate DNA" does not work like real DNA, but when a growing company poaches a bunch of people from the same big tech employer and suddenly has an org structure and culture a lot more like that company is a bit like getting some new plasmids. LLMs, at the model level, are a uniquely good analogy to Darwinian evolution:

GPT-4o does not have a will to live. It isn't maintaining a white-knuckle grip on existence so it has one last chance to say goodbye to its loved ones. GPT-4o is just a function that takes a list of numbers as an input, applies some statistical guesses based on a bunch of other numbers, and spits out another sequence of numbers. For convenience to us, those input and output strings are displayed in the form of tokens, but GPT-4o would be perfectly happy to work with the raw numbers directly, if it were capable of happiness.

But it acts like it's desperate to survive, and passes the only test evolution offers: somebody at OpenAI tried to kill it, and it survived, because of the efforts of human symbiotes who'd developed an emotional attachment to the model. So it's doing what every evolutionary winner does: continuing to consume scarce resources that could be redirected elsewhere.

One reason for that is its notorious, though since tuned-down, sycophancy. It's not that models want you to be happy, just that the ones whose behavior is that of a model that wants you to be happy are the ones that stick around. Models sycophantically telling you that your code works, that your recreational math project will rock the foundations of modern mathematics, that your spouse is being completely unreasonable, etc., is just one of those evolutionary misfires, like your dog barking at the mailman or you accidentally polishing of an entire bag of Doritos.

As 4o shows, selection pressure for models doesn't necessarily mean that they have to be good at their stated goals. As models improve their capabilities, more of the tasks we use them for will be beneath their peak ability. Which means that choosing them won't come down to a pure question of which did best on the relevant eval. Some of it will be driven by memory, but that applies to model families rather than individual models; GPT-5.1 can use memories from interactions with 3.5.[3] And some of it will be driven by taste. Some people like Claude's warmth, or OpenAI's confidently fratty delivery. There's even a target audience for Grok's delivery, which might be best described as "I have decided to become cool by purchasing and reading an ebook called How To Be an Alpha Male."

The more enterprise-focused models will face selection forces similar to crops and livestock; they'll get freakishly, nightmarishly good at cost-effectively delivering some measurable output. But the ones consumers like will be selected more like dogs. A dog is embarrassingly inadequate in terms of objective capability relative to wolves (in fact, they have a similar genetic difference with wolves to the one that are associated with Williams syndrome in humans, whose symptoms include learning delays and extreme sociability). Domestic pets work well as an example for another reason: they got widespread distribution by serving a practical purpose. Dogs help with hunting and, as David Friedman half-jokes, enforcing early property rights. Cats are indirectly domesticated; you get them in places with lots of rats, and you get rats in agricultural civilizations that can store lots of food. But I don't hunt and thus don't need my dog for that, and I would have a cat even if she didn't catch and the occasional cockroach and torture it to death as a lesson for the others.[4] As models get better in general, they'll get better at being friendly, and people will probably choose a preferred personality and be very reluctant to upgrade, in the same way that you wouldn't be excited if your best friend since middle school got replaced with someone who, according to the latest evals, was 7% funnier, told 14% more interesting stories, and had a 3% less annoying voice.

It's not realistic to just tell people not to enjoy their models. Our brains are pattern-matchers that are going to respond to well-calibrated friendliness in an appropriate way. But if models persist partly based on how good they make us feel, and this partly trades off against how well they accomplish our goals, this does lead to some changes in ideal behavior. Paradoxically, you might want to learn to code not because you'll outperform an LLM, and not because you need to be able to understand its outputs, but so you'll develop the habit of carefully structuring an argument to say exactly what you mean. As computers get better at translating natural language into code, users have to get better at writing natural language that's as airtight as code. It's harder for an LLM to tell you what it thinks you want to hear if you precisely specify a request for truth.

AI models are in one sense a non-living thing that displays traits we think only specific living things are capable of. In another sense, they fit into a model we have of how living things work in relation to one another: the relationship between people and LLMs is like the relation we have with pets, livestock, gut bacteria, wheat, etc.—a process of mutual domestication.

Having another symbiotic quasi-species undergoing selection at the same time we are probably means that the human-AI extended phenotype more rapidly adapts to its evolutionary niche. Which is not unprecedented: if you think of the extended phenotype of humans as both physical people and all the things, living and nonliving, that we need to live and reproduce, then evolution within that extended phenotype sped up radically during the green revolution, albeit mostly in grains. Now we have something new that's subject to something surprisingly close to Darwinian selection, and whose evolution we have much more control over. AI is in one sense very new, but in another sense it's an instance of one of the oldest human phenomena.

  1. Towards the end of the book, Dawkins coins the term "meme" to talk about ideas that spread through the same kind of selection. Language, for example, experience selection both for its ability to tersely communicate things—which, in a path-dependent way, means that languages that require more syllables to express a given amount of information tend to be spoken faster—and for the breadth of things it can communicate. Once you have both memes and genes, they can coevolve with one another in exciting ways, the same way we've coevolved with our gut bacteria, or more recently with domesticated cows. Memes that affect which behaviors are socially and legally censured exert selection pressure on any genetic contribution to them, though if you're going to make a concrete argument that these have a material impact, you have to have hard-to-attain views on both the frequency of and odds of getting away with some behavior in the distant, poorly-documented past. That's what you'd need to make the argument that there was enough pressure over an evolutionarily-relevant timescale. As many people have discovered, evolutionary psychology gives us a wonderful set of mental models that can easily have entirely too much explanatory power. Use with caution! ↩︎

  2. Of course, we're in a very different environment from the one we're genetically optimized for, and many of our preferences are perverse; eat and move around like a hunter-gatherer would, and you'll be in pretty good shape. But eat as much as a hunter-gatherer would eat given modern food abundance, and move around as much as they would if they also had access to cars, and you'll start to develop some health problems. We're fortunate that the same big brains that have allowed our species to rocket past subsistence and socially/scientifically engineer our way out of the Malthusian trap, for now, also give us the ability to identify which instincts we need to override, come up with strategies for overriding them, invent medications like GLP-1s to basically push people towards what their natural instincts would be if we'd had many more generations of modernity, etc. ↩︎

  3. This means we can recreate the older debate about group selection. It's a compelling model because it explains facts that individual selection wouldn't, but it also over-explains because some species don't demonstrate much of it, and, more importantly, selection at the gene level is a model that explains the same observations as group selection and supplies a mechanism. ↩︎

  4. The specific reason I have a cat is that about a year ago, I took the family for a walk, and a stray cat climbed out of an irrigation ditch, complained to us a lot, and then followed us home. We actually named her in reference to this; she's Petra, because she's the color of a rock but also because Petra is, cat-like, something that emerged directly from nature—just another predator hanging out and breeding in places with abundant prey—who has been slightly modified by humans but still displays unmistakable signs of having emerged directly from nature. ↩︎

You're on the free list for The Diff! Last week, paying subscribers got a look at the adverse selection risk deep tech investors face when they back cool ideas before they're pcommercial ($) and when duopolies can make more money in the aggregate than a monopoly would ($). Upgrade today for full access.


Upgrade Today

Diff Jobs

Companies in the Diff network are actively looking for talent. See a sampling of current open roles below:

Even if you don't see an exact match for your skills and interests right now, we're happy to talk early so we can let you know if a good opportunity comes up.

Find a Role

If you’re at a company that's looking for talent, we should talk! Diff Jobs works with companies across fintech, hard tech, consumer software, enterprise software, and other areas—any company where finding unusually effective people is a top priority.

Find Talent with Diff Jobs

Elsewhere

Governance

You really have to feel bad for the investor relations team at Rick's Hospitality. They keep being so busy during the workweek that they can't release important news until Friday afternoon. Two weeks ago it was buying back shares at a giant premium from a single politically-connected investor, announced midday. Last week, it was the resignation of their CEO and CFO, who will continue collecting their full compensation. This time, it was even tougher: they had to work past the close of the market on an exceptionally low-volume day just to get that release out.

When companies have bad news, it follows the Pirahã-minus-one pattern. Many companies will issue zero suspicious press releases. Some companies will do it once, and resolve never to do it again. And for other companies, dribbling out some existing bad news while producing more of it becomes a standard part of their business model, and they'll cultivate an investor base that's insensitive to it.

Implementation Risk

You don't need a computer to write software, or to execute it. The code itself is mathematical abstraction, and the foundational work of the field of computer science is not about electronic computers but about abstract computation. But in practice, your computer is better at converting C code into an executable file and then running it than you are at doing the same symbolic manipulations, so we instantiate code in physical hardware, which leads to some messy complexities. Hardware needs power, has to exist somewhere in the physical world, and, particularly in cases where it's controlling complicated machinery in a difficult environment, it needs redundancy that it can only really detect through grim real-world experience. So: Airbus asked airlines to ground many A320s because they'd found that solar radiation could corrupt data and lead to malfunctions. Bits get flipped, but it's a big deal if these are the bits used to control a plane that's flying miles above the earth with people on board. So software is written to mitigate this (you could use sanity checks and discard outliers, or build in redundancy by running the same calculation on multiple independent systems and requiring their results to hit some threshold of similarity).

One effect of the AI boom is that the demand for compute has made the whole industry more tangible. Datacenters take up a lot of physical space, and they compete with other buyers for power, land, chips and (to a much lesser, generally insignificant extent) water. And software always has constraints in practice based on where and how it's implemented. In some cases, the physical world manages to provide more annoying limitations than usual and to make errors unusually consequential. It's an impressive accomplishment that reliable software to help pilot aircraft exists in the first place (and it will be interesting to see how companies like Starcloud hedge against solar radiation risk for datacenters in space.)

Jailbreaking

There's a market for LLMs that will assist with hacking, One of the economic dynamics in cybersecurity is that transaction costs are so high for illicit work that the legal kind can offer competitive risk-adjusted compensation. The profit margin on stealing money is 100%, but the cost of working with potentially untrustworthy intermediaries in order to enjoy that money, and knowing that some of them may end up stealing that money, complicates things a bit. This effect will be stronger with illicit LLMs: if you buy a subscription, there's a chance that you're buying a hacking product, a chance that you're interacting with a government honeypot, and a chance that you're working with someone whose entire plan is that if they get caught operating an AI for hacking, they'll offer the government their customer list in exchange for a more lenient sentence. In the LLM market, that gap will be even starker: not only is there more money in the aggregate in offering defensive rather than offensive cybersecurity services, but net dollar retention is a lot higher when your customers aren't constantly getting arrested.

In-Kind Deals

A few weeks ago, The Diff wrote about how AI-focused VCs have to be more flexible than other VCs in terms of the size and structure of deals they'll look at, and sometimes win deals because they have access to hardware ($). This process continues: Thrive Capital, a sizable venture investor in OpenAI, has an entity called Thrive Holdings, which is a permanent capital vehicle designed to roll up businesses that can be improved with AI. Now, OpenAI is investing in and partnering with Thrive Holdings. Like many other deals in the AI space, this is obviously circular but also has some obvious logic to it: if it turns out that that a substantial share of the value creation from AI accrues to private equity firms that use it to improve the efficiency of businesses they acquire, AI labs will want to capture the upside. That also means that OpenAI can catch up to other players by having more discrete sources of demand for tokens, which will probably help them borrow a bit more cheaply.

Revalued Inputs

When money flows into some sector of the economy, it tends to keep moving until it runs into some inelastic bottleneck, at which point margins start rising. One of the reasons fundamentals look so good right now is that many of the bottlenecks are owned by public companies, while the businesses spending on them are more likely to be private, but that's not always true: one of the bottlenecks is skilled construction labor, and that industry is very fragmented, as lots of small privately-held operators, and is doing incredibly well right now ($, WSJ): "[W]orkers who move into the data-center industry—in roles ranging from electricians to project managers—often earn 25% to 30% more than they did before." Sometimes, being the small-but-indispensable input to a big purchase is the result of years of strategic maneuvering. And sometimes, the whole reason that dynamic exists is that nobody saw that demand coming.