I was thinking about Occam’s Razor tonight in my drive home after swapping a couple of emails with Tim Wolters, the CTO of my company, on an issue that involves the use of Occam’s Razor at work. Occam’s Razor is usually stated like this: “Whenever there is more than one hypothesis to explain something, the simplest hypothesis is usually correct.” This principle has wide applicability throughout all the sciences, in philosophy, in law, in statistics, and in day to day life. In physics, postulating that momentum and energy are conserved was able to explain most of the things we can observe without specialized equipment. When that proved inadequate, assuming that the speed of light is constant in all inertial frames of reference took care of the really big things, and assuming that energy is quantized took care of the really small things. In biology, evolution through natural selection is a powerful framework which serves as a basis for all the knowledge about living things we have today. Even something as innocent sounding as natural selection applied to groups is completely unacceptable as a hypothesis in biology if individual natural selection can explain it. In philosophy, you are more likely to be correct if your system does not postulate unobservable entities such as fairies, gnomes, UFOs, gods, demons, gremlins, and honest politicians. In a murder case if you are at the scene of a crime, your prints are on the murder weapon, the victim’s blood is in your car, and you have a motive for a murder, it usually means that you are the killer (unless you have Johnny Cochran as your lawyer). In statistics, fitting a line to a bunch of data points is usually better than fitting a quadratic, a cubic, or a higher order polynomial. In day to day life, you will have more luck predicting the weather based on atmospheric data than trying to guess the mood of the gods of rain and thunder.

So this got me thinking, why is this true? Why is it that the simpler hypothesis is usually correct? Why does nature prefer simplicity? And to go even deeper, what is simplicity anyway? A differential equation is the essence of simplicity to an educated person, but it looks horribly complex to a person unschooled in mathematics. This seems to support the notion that complexity is in the eye of the beholder. If this is the case, it makes even less sense that nature would behave in ways that look simple to us.

One possible explanation is that our brains have been fine-tuned to nature by evolution. So the laws of physics are not inherently simple, but our brains perceive them as simple because the brains of those who didn’t caused them to die before they could reproduce and pass that trait on. If this was true, then we should only be fine-tuned to the things that were observable when we were evolving, and which occur in a human scale. In other words, we should not think that the cosmos, subatomic particle, or phenomena that span millions of years are simple. However, successful theories in all these areas pass Occam’s Razor, so the explanation above is not sufficient.

Another possibility is that nature is made of simple components which interact in a deterministic fashion, providing predictable results. This is the standard assumption behind most sciences. This hypothesis is a good candidate because all we have to postulate is simple components and determinism. If this is the case, as we investigate smaller things we should be able to explain them with simpler hypotheses. Anybody who has studied quantum mechanics knows that this is not true, so the simple components hypothesis doesn’t seem to hold at the subatomic level.

Yet another explanation comes from a computer science analogy. For any observation we want to explain, there is an infinite number of hypotheses that could explain it. Let H be the set of all hypotheses that can explain that particular observation. Since all hypotheses are contained in H, any modification of a member of H leads to another member of H. This means that we can define a graph connecting all the hypotheses with the number of edges between any two of them representing the number of modifications that hypothesis A needs to go through in order to become hypothesis B. Given that we have this graph, let’s pick an arbitrary hypothesis as the simplest. The best way to find the correct hypothesis without any heuristics is to do a breadth first search. A depth first search would get us lost in the infinite hypothesis space, whereas a breadth first search would eventually get to the hypothesis we seek, or would find one that is close enough to be acceptable. Under this view, Occam’s razor is just a breadth first search of the hypothesis space starting from an arbitrary point that our brains perceive to be the simplest. What happens if the correct hypothesis is really far away from the point we started from? Then it would take a long time to get to the true hypothesis, so the explanation above would only make sense if we had some sort of heuristic that allowed us to start at a point close enough to the true explanation.

Since the three simple hypothesis above on why Occam’s razor seems to work don’t succeed at explaining it, let’s combine the biology based and the computer science based explanations. If we assume that our brains were fine-tuned to the human experience by evolution, that means that they are somewhat fine-tuned for the things we bother to try to explain. After all, only things that relate to the human experience raise our interest in the first place. If this is the case, whatever looks simplest to us is a good heuristic to start our breadth-first search and starting from the simplest hypothesis should put us in a much closer place to the to the correct explanation than starting from a random point. As we advance towards the true hypothesis, we keep adding complexity from our point of view, but since the hypothesis is closer to the truth, its explanatory power is better. In the end, when we find the true hypothesis we will know because any more complexity will decrease its explanatory power. Right now our scientific theories are probably still far enough from the truth that we can safely keep adding more complexity.

This explanation seems satisfactory, so I’ll stick with it. We are fine tuned by evolution to see the explanation closest to the truth as the simplest when we deal in a human scale. When we move away from the human scale, we are not really that far removed, so the true hypothesis is not too far from the simplest. Occam’s Razor is just an algorithm that guarantees a breadth first search in the hypothesis space starting from our initial, simplest hypothesis, so our search is much faster than if we used a depth first search, or a different starting point. I could try to add the simple components hypothesis to the mix, or even change the breadth first search to an A* search, but it doesn’t seem to add much more insight in exchange for the addition of more complexity. So I think I’m going to follow Occam’s Razor, and quit while I am ahead.