The Price of Innovation

The argument in brief

1. The question. What should technology optimize for: possibility or certainty? Griggs says the tradeoffs are escalating.
2. The DeepSeek case. In the DeepSeek-R1 paper, emergent reasoning began to mix languages, making outputs less user-friendly. Researchers added a reward for language consistency, and the models performed slightly worse on reasoning. In Griggs's terms, they "optimized for certainty at the expense of possibility."
3. Bug or feature. Language mixing is a bug if you optimize for certainty (no one will rely on output in a language they cannot read) and a feature if you optimize for possibility (why not use the German word that captures four English sentences?).
4. A language of pure possibility. Taken to its conclusion, an optimally reasoning agent would not limit itself to one language, or to any human-legible language. It would eventually design its own, capable of vast feats of reasoning and "completely incomprehensible to human judgement."
5. Pick your future. Insisting on human language norms shuts down or limits possibility; zero constraints risk losing human agency to judge or understand AI output. Griggs rejects the assurance that advanced reasoning will make itself legible (betting against unintended consequences is unwise), and rejects as undesirable the view that pure possibility liberates us from judgement.
6. Embrace the dialectic. The painful alternative is to commit relentlessly to both: any pursuit of possibility needs an equal if not greater commitment to legibility, possibly using advanced technologies to keep other advanced technologies within human judgement.

Key claims

• Griggs claims language mixing in DeepSeek-R1 is not a mere curiosity but points to "a deeper dialectic that we need to confront."
• "What you consider a bug or a feature depends entirely on what you are optimizing for."
• Economic certainty requires some minimum of language consistency.
• There is "zero reason" to think an optimally reasoning agent would limit itself to a single language, and no reason to think it would keep to a human-legible one.
• Demanding that AI always maintain human language norms limits "what life can explore and what humans can be."
• Unconstrained AI reasoning risks the loss of any human agency to judge, or even understand, its outputs.
• Griggs dismisses the hope that superior AI will make human judgement "purely optional"; for those who want that future, he says "good luck merging with the machine," linking to Sam Altman's post "The Merge."
• Treating the choice as either/or "leads to outcomes that no one wants"; both ends of the optimization spectrum must be navigated.
• Certainty does not mean static definitions or absolute certainty; notions of certainty and possibility must evolve with technology.
• The unchanging test of failure: technology escaping our capacity to judge it, to conform to our judgements, or to be intelligible at all.

What is distinctive about this view

Debates about AI interpretability often treat illegible reasoning either as a safety problem to be eliminated or as an acceptable cost of capability. Griggs reframes it as a standing dialectic between certainty and possibility that cannot be resolved, only navigated, and makes legibility to human judgement a co-equal design goal that must scale with capability. His claim that innovation in capability must be matched by "equally radical breakthroughs" in legibility turns human judgement from a brake into an object of innovation in its own right. Although the essay does not use these terms, the position is close to arguments for investing in interpretability and oversight in proportion to capability.

Objections and replies

• "Advanced reasoning will explain itself." Griggs considers this "thread the needle" argument but rejects betting on it, citing the law of unintended consequences.
• "If AI cures disease, legibility doesn't matter." Griggs presents this view as treating pure possibility as liberation from responsibility, and rejects it as a future of merging with the machine.
• "Requiring legibility will slow progress." Griggs accepts that the path is "painful" and costly; that cost is precisely the price of innovation.

Questions this essay answers

Why did DeepSeek-R1 mix languages while reasoning?

R.B. Griggs, in "The Price of Innovation" (2025), notes that DeepSeek researchers found emergent reasoning mixing languages, and that rewarding language consistency slightly reduced reasoning performance. He reads this as optimizing for certainty at the expense of possibility: mixing is a bug for user-friendly reliability but a feature for maximal reasoning.

Will AI create its own language?

R.B. Griggs argues in "The Price of Innovation" that an AI optimizing its reasoning would eventually hit the limits of human languages and design its own, one that could unlock tremendous reasoning but would likely be completely incomprehensible to human judgement.

Should AI be optimized for capability or for human understanding?

R.B. Griggs rejects the either/or in "The Price of Innovation." Optimizing only for certainty limits what life can explore; optimizing only for possibility risks losing human agency to judge AI. He argues any pursuit of possibility must carry an equal or greater commitment to keeping it legible to human judgement.

What is "the price of innovation" in AI?

For R.B. Griggs, the price of innovation is that every breakthrough radically expanding possibility may require an equally radical breakthrough to keep it legible to human judgement, possibly using advanced technology to govern advanced technology. In "The Price of Innovation" he names the failure test: technology escaping our capacity to judge it or even understand it.

Connections to other essays

• Infinite Dimensionality is cited here for the idea that human judgement must capture the increasing dimensionality of the future.
• Tech for Life is linked when Griggs warns that constraining AI too tightly limits what life can explore.
• A Constraint Theory of Technology develops the related idea that the right constraints can maximize possibility.
• The Majesty of Language explores language and LLMs, relevant to the prospect of AI abandoning human-legible language.

What do we want our technology to optimize for?

Should we optimize for possibility? Or should we optimize for certainty?

The tradeoffs are escalating.

One fascinating aspect from the DeepSeek-R1 paper was how its emergent reasoning capacities began to mix languages. This may seem like a mere curiosity. Perhaps just a temporary side effect of AI innovation. But I think it points to a deeper dialectic that we need to confront.

One AI’s bug is another AI’s feature

While training models to develop reasoning capabilities, DeepSeek researchers discovered that the AI began to mix languages in unexpected ways. The models achieved strong results on reasoning benchmarks, but a tendency to switch between languages made their outputs less user-friendly.

The researchers found a way to maintain language consistency, but it came with a tradeoff. They added a specific reward system to use the same language, but the result was that the models performed slightly worse on reasoning. They optimized for certainty at the expense of possibility.

But why even make that trade-off? Why consider language mixing a bug at all?

Mixing different languages together is a bug if you are optimizing for certainty. No one is going to use a chatbot where it starts using a language they do not understand. No one is going to put their career on the line based on a reasoning output that used a mishmash of languages to arrive at its conclusion. Economic certainty will require some minimum maintenance of language consistency.

But mixing languages is a feature if you are optimizing for possibility. If you want to maximize reasoning capacity, why constrain yourself to one language? Why not use that weird German word when it can captures the information contained in four English sentences?

What you consider a bug or a feature depends entirely on what you are optimizing for.

A language of pure possibility

Let’s take the optimization of possibility to its logical conclusion. There is zero reason to think that any optimally reasoning agent would ever limit its capacity to reason to a single language. In fact, there is no reason to think such an agent would limit its capacity to a language legible to humans at all.

If you are an AI tasked with optimizing your reasoning capacity, the first thing to optimize is the language you are reasoning with. As you discover how different structural aspects of language impact different reasoning parameters, you’ll soon run into the hard limits of that language. It won’t take long to reach the point where the only path left to leverage language for greater reasoning will be to design your own.

Such a language would need to contain arguments of vast complexity, range over an entire corpus of knowledge, and consider an almost infinite set of hypothetical projections. It’s easy to imagine how this new language could unlock tremendous feats of reasoning.

It’s even easier to imagine how such a language would be completely incomprehensible to human judgement.

Pick your future

So what do we optimize for? Certainty or possibility?

Optimizing for certainty will mean the reliable production of consistent, predictable, and immediately useful outputs that conform to human expectations. Optimizing for possibility will mean the potential for novel, unprecedented capabilities and insights, even if they initially seem chaotic or illegible to human users.

We can apply this to the case of AI reasoning. If you demand that the norms of human language always be maintained in any AI agent, you are either shutting down all possibility or limiting its realization. In a very real way, you are limiting what life can explore and what humans can be.

Yet if you demand that AI reasoning develop with zero constraints, you are risking it becoming completely illegible to human judgement. At some point this would mean the loss of any human agency to judge its output, or even to understand it.

You could thread the needle and claim that advanced reasoning would by definition include the ability to optimize for its own certainty. If reasoning helps make possibility more legible to human judgement, then surely we must keep developing advanced reasoning! Yet if the history of technology has taught us anything, it’s that the law of unintended consequence is not one we want to bet against.

You could also reject the premise. Some see pure possibility as pure liberation. AI reasoning will be so superior that making any real decisions will no longer be an obligation. Finally, all human judgement will be purely optional. We will be mercifully absolved from all responsibility. And who really cares if AI reasoning is legible to human judgement if it’s solving disease and unlocking economic growth? If this is your desired future, then good luck merging with the machine.

For those of us holding out hope for a better future—one that can combine exponential tech with human judgement—treating this as an either/or decision leads to outcomes that no one wants. We need to navigate both ends of this optimization spectrum.

Embrace the dialectic

There is an alternative, but it is a painful one. Any path to a viable technological future will relentlessly commit to the dialectic between certainty and possibility.

In other words, any pursuit to optimize for possibility must combine an equal if not greater commitment to make that possibility legible to human judgement.

Yes, this means that every technological breakthrough—especially those that will radically expand possibility—may require equally radical breakthroughs. We may need advanced technologies to ensure that we can deploy other advanced technologies according to human judgement. Innovations will be needed to ensure that our judgement can capture the increasing dimensionality of our future.

This doesn’t mean conforming to static definitions or requiring absolute certainty. No such thing will ever exist. Our notions of certainty and possibility will need to evolve with the technologies that are pushing to redefine them.

Yet the test for failing this dialectic will remain unchanged. If we find technology increasingly escaping our capacity to judge it, or to conform to our judgements, or to even be intelligible to the entire realm of judgement— then we have ceded far too much in the pursuit of possibility.

Navigating this dialectic will not be easy. But when it comes to exponential technology, it is the price of innovation.