Richard Sutton's Bitter Lesson has been evangelized so much in Silicon Valley to the point of cliché. Yet outside of the Twitterverse of pundits and VCs, you often find the bitter truth has yet to truly sink in. The lesson's apparent simplicity belies how difficult it is to actually put into practice, as people continue to mistake incremental progress for methods that truly scale with compute.
The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin. [...] Most AI research has been conducted as if the computation available to the agent were constant [...]. Seeking an improvement that makes a difference in the shorter term, researchers seek to leverage their human knowledge of the domain, but the only thing that matters in the long run is the leveraging of computation.
As a first-year comp bio PhD student, reflecting on this made me think a lot about how research is done in biology -- in some sense the least Bitter Lesson-pilled of all domains. Much of the scientific process feels artisanal: researchers develop hypotheses, carefully design experiments, and interpret results. While this approach has produced extraordinary breakthroughs, its pace is still largely constrained by human judgment and experimental labor. Many of the most important discoveries like CRISPR have emerged almost by accident, stumbled upon by lucky or especially shrewd researchers.
Because of this, biology often feels like exploring a maze: each interesting path branches into many others, rewarding deeper specialization and incremental progress within shallow subdomains, rather than general methods that scale across problems. The consensus today in the AIxBio field is that we should invest more resources in pushing deeper into the maze, replacing humans with AI agents that can work many times faster. But few stop to zoom out and ask whether the maze is even the right place to look.
For Bitter Lesson pundits, protein models like AlphaFold or ESM represent a major triumph. Train on vast databases of protein sequences and structures and you get models that can predict how proteins fold or which amino acids fit a given sequence. But I would argue these models, like LLMs trained on the Internet, have yet to fully embrace the Bitter Lesson. They learn the paths through the maze that evolution has already traced, rather than asking how the maze itself was designed. This works remarkably well until you ask them to venture beyond the distribution they were trained on. And the space of possible proteins (or molecular structures), for that matter, is quite, quite vast.
I saw this firsthand in experiments I ran this week. With directed evolution, we engineered novel iGeoCas9 variants that beat the original by swapping a bunch of amino acids in a particular loop. Yet ESM-2 consistently assigns low likelihoods to these beneficial mutations, and even fine-tuning on 80% of our experimental data fails to produce good predictions on the remaining 20%. The model favors what nature has already produced, but struggles to recognize what works beyond it.
In computer vision, there has been a similar pattern. Early methods conceived of vision as searching for edges, or generalized cylinders, or in terms of SIFT features. But today all this is discarded. Modern deep-learning neural networks use only the notions of convolution and certain kinds of invariances, and perform much better.
A model trained only on things that could previously fly might insist flight requires feathers or a beak, or other merely incidental evolutionary traits. But planes today are not built by imitating birds; rather, they're built by understanding the fundamental laws of physics and engineering around them. For centuries, inventors attempted to design flying machines by replicating the way that birds flew, but it was only through the Wright brothers understanding aerodynamics that flight was finally cracked.
Biologists often act as though biology has no such organizing principles. But bio is not exempt from physics any more than airplanes are. We have a ground truth to work from: atoms and the equations that describe their interactions. Through simulation, we can recreate the molecular machinery of life and probe how it works. Rather than blindly mutating proteins and selecting what works, we can understand what a variant does and why it works, something a protein model will never be able to conclusively tell us.
One thing I've learned from my CS curriculum in college is that computer scientists like to ask is what would happen when we assume a sufficiently good thing is said to exist. A sufficiently accurate simulation could, in principle, model any biological system. Biology’s complexity arises from simpler parts interacting chaotically under the same physical rules. If we can capture those interactions faithfully, modeling how they work together just becomes a question of scale and compute.
One thing that should be learned from the bitter lesson is the great power of general purpose methods, of methods that continue to scale with increased computation even as the available computation becomes very great. The two methods that seem to scale arbitrarily in this way are search and learning.
Simulations are very computationally expensive. Designing entire simulations of complex enzymes in silico is no easy task, and very demanding on GPUs. But as compute scales and becomes more abundant, I believe this will ultimately be the easier path than relying on painstakingly hand-crafted experiments, forever bottlenecked by the hours you have in a day and the number of pipette stations in a lab. The Bitter Lesson does not say that the computational approach will be easier at first. It merely says that you must have faith in the ability for compute to win out over human-designed rules and processes in the long run. Exponentials always look slow in the beginning.
Another intrinsically valuable property of simulations is their ability to generate meaningful data from computation alone. This is exceptionally rare in the ML world, where models are usually constrained by availability of real-world data and frontier labs pay top dollar for high-quality human datasets. But for physics-based simulation, this is precisely the appeal: huge supercomputers can run intensive quantum-mechanical calculations to produce extremely accurate descriptions of molecular interactions. Neural network-based approaches like MLIPs can then train on this data, approximating those calculations at a fraction of the cost and enabling larger, longer simulations with shocking generalization. Compute itself becomes the engine of progress, just as the Bitter Lesson so desires.
Ultimately as compute scales, so does the search space we can explore. We could test billions of molecular configurations and protein variants in silico, on a scale unimaginable in a wet lab. We could learn the interactions that give rise to molecular function and build a world model of molecular biophysics grounded in first principles. From there, we could design entirely new molecules nature has never seen, refine them in simulation, and finally bring them into the real world ... the Drexlerian pipe dream. This could revolutionize medicine, energy, agriculture, and countless other fields, harnessing the power of scalable computation for social good.
The [...] general point to be learned from the bitter lesson is that the actual contents of minds are tremendously, irredeemably complex [...] They are not what should be built in, as their complexity is endless; instead we should build in only the meta-methods that can find and capture this arbitrary complexity. [...] We want AI agents that can discover like we can, not which contain what we have discovered. Building in our discoveries only makes it harder to see how the discovering process can be done.
Maybe this is true not only of minds but of biology as well. Biology is irredeemably complex, full of mysteries that no amount of human intuition could ever fully unravel. Rather than trying to understand it all by hand, we should seek the fundamental rules that give rise to this complexity, the physical laws that govern all proteins, genes, and life. Through simulation, search, and learning, we can explore possibilities far beyond what humanity has already discovered. We could transform biology from a game of guesswork into a discipline of engineering.