The Optimization Problem That Ate the 20th Century
Give two machine learning engineers the same impossibly complex loss function: maximizing human welfare across millions of features. Call the first engineer Adam Smith. He uses stochastic gradient descent, with thousands of independent agents making noisy local updates from the little information each has. The second engineer, Karl, insists on batch gradient descent. He'll collect all the data, compute the perfect solution—or rather, the gradient—and make one coordinated global update.
The loss function stands for the entire economy, the parameters for resource allocation decisions, and the engineers for economic systems. What I'm calling the Communist Constraint is the mathematical impossibility of centrally optimizing complex systems in real time.
Stochastic Capitalism: The Noise of Markets
A single update in Stochastic Gradient Descent might send you in completely the wrong direction. It's a noisy process, and it makes mistakes. Average those updates over thousands of iterations, though, and it converges to solutions that batch methods can't touch.
Markets work in exactly this way. Every transaction makes a noisy gradient update:
- A farmer switches from wheat to corn → price signal propagates → resources reallocate
- A startup fails → capital flows elsewhere → innovation continues
- A group of consumers choose Netflix over cable → the entire industry pivots to streaming
Random sampling helps SGD avoid getting trapped in local minima. That is what makes the "stochastic" part useful. The "irrationality" of markets serves the same crucial purpose: bubbles, crashes, Masayoshi's seemingly stupid investments, and Jane Street squeezing markets all explore the solution space.
The dot-com bubble is usually described as "wasteful" because billions went into pets.com and other failures. But it was also a massive distributed compute job. Thousands of Internet business models were tried at once. Most failed; the survivors found global optima that no central planner would have predicted.
Batch Communism and the Curse of Coordination
Collecting all the available information, computing the theoretically optimal update, and executing it flawlessly sounds like a sensible approach. With complex problems, putting batch gradient descent into practice is a nightmare.
The Soviet Union made history's most ambitious attempt at batch optimization. Its central planners would:
- Collect data from across the economy
- Compute "optimal" resource allocation
- Issue coordinated directives to all economic actors
- Repeat every 5 years
The approach runs into the same fatal flaws as batch gradient descent:
Stale Gradient Problem — The loss landscape has shifted by the time you've collected all the data and computed the gradient. Soviet five year plans relied on information that was already months or years old. Markets keep updating as this happens. A price change sometimes propagates in milliseconds rather than months.
The Local Minima Trap — Batch methods are very good at finding local optima and struggle to explore beyond them. The USSR became extraordinarily good at producing steel and tractors, a form of local optimization. It missed entire technological revolutions in personal computers, telecommunications, and biotech because it had no mechanism for exploring radically different solutions.
The Computational Complexity Barrier — Millions of agents make billions of decisions in a real economy. The Soviet planning apparatus employed millions of bureaucrats just to collect and process information, and they still couldn't keep up. Modern deep learning uses SGD precisely because batch methods do not scale.
The Convexity Delusion
Karl's approach assumes that the economic loss function is convex, with a single, computable global optimum. For convex optimization, batch gradient descent is provably superior to SGD. If the problem stayed still and computational resources were infinite, central planning would work.
Economies aren't convex. They're full of:
- Multiple equilibria — both Detroit and Silicon Valley can be optimal industrial centers
- Network effects — the value of a phone network depends on how many people use it
- Creative destruction — today's optimal becomes tomorrow's obsolete tech
That non-convexity makes Hayek's "knowledge problem" a computational complexity problem. Having information isn't enough for optimal central planning; processing the information it needs is computationally intractable.
The Antifragile Advantage
Nassim Taleb's idea of antifragility fits optimization theory perfectly. SGD benefits from noise and volatility: each random perturbation helps it escape local minima. That makes it antifragile. Batch methods are fragile because they assume perfect information and stable systems.
In capitalism, this antifragility takes the form of creative destruction, optionality, and adaptive exploration. Market forces automatically explore new regions of the solution space.
The Soviet system made itself brittle by optimizing for stability and control. Changes in oil prices, technology, and demographics shifted the loss landscape, and the computed solution couldn't adapt.
The Resource Constraint
We choose SGD over batch methods partly because we can't afford to process an entire dataset for every update. The computational constraint is part of the decision. Economies face resource constraints too, and these make central planning impossible.
The "Communist Constraint" also concerns the fundamental trade off between exploration and exploitation when resources are scarce. Markets naturally balance it. Profitable ventures receive more resources for exploitation, while venture capital funds the exploration of new possibilities.
Central planners have to choose explicitly between exploration and exploitation. The Soviet Union chose to perfect existing technologies rather than explore new ones. Exploitation worked at first, producing rapid industrialization, but it failed as the economy matured and needed innovation.
Where Batch Methods Actually Work
The parallel has limits. There are contexts where batch optimization excels:
- Convex problems — Military Logistics, infrastructure planning
- Stable environments — public utilities, emergency response
- Clear objectives — space programs, pandemic response
Singapore's mixed economy uses both approaches. Market mechanisms support innovation and growth; central planning provides housing and infrastructure. Choosing a method depends on the problem you're asking it to solve.
The Meta-Optimization Problem
Choosing between economic systems becomes an optimization problem of its own. Which sectors should we plan with SGD, and which with batch methods?
| Context | Best Method |
|---|---|
| Simple problems | Batch methods |
| Complex problems | SGD |
| Stable environments | Batch methods |
| Dynamic environments | SGD |
| Safety-Critical | Batch methods |
| Innovation driven | SGD |
Modern economies are converging on hybrids. They use market mechanisms for innovation and resource allocation, with government intervention to address market failures and provide public goods.
The Algorithmic Future
AI might make central planning more viable as it advances. If artificial intelligence can solve the computational complexity problem, perhaps Karl was right?
Probably not, because computation is only part of the difficulty. Human preferences, technological possibilities, and emergent behaviors keep changing the economic loss function. No algorithm can optimize for objectives that don't yet exist.
The future likely belongs to systems that combine both approaches. AI assisted markets would use machine learning to improve price discovery and resource allocation, while keeping the exploration and antifragility that only decentralized systems can provide.
Conclusion
The Communist Constraint explains why every attempt at comprehensive central planning has failed while market economies have thrived, obvious flaws and all. The explanation comes from mathematics, rather than ideology.
Markets are noisy, inefficient, and prone to bubbles and crashes. For all those faults, they're the only optimization algorithm we've found that can handle the complexity, non-convexity, and constant change of real economies.
When someone proposes a "rational" alternative to market chaos, ask how they've solved the computational complexity problem. Until they do, Engineer Adam's invisible hand remains the most powerful optimization algorithm in the social sciences.
"You can't maximize two competing objectives simultaneously - you can only find optimal trade offs between them".
"Every optimization problem is ultimately single-objective, but the objective might be a complex function of multiple competing goals".