How to get the bone

Sometimes working on big projects can be frustrating. I think that's mainly because it's easy to get so immersed in detail that you lose sight of the big picture. A big project tends to break down recursively into sub-projects, and sub-sub-projects that you only realize are necessary after the "direct approach" turns out to be not a shortcut but an illusion, something that starts out looking like a shortcut and turns out to be a dead-end or actually a very long way around (e.g., fast short term results that are unsustainable in the long run). Then you have to turn back. That can be emotionally painful because on your original map of the problem it looks like instead of moving toward your goal you're actually moving away from it.

You have to make a conscious, rational effort to override that feeling, and you have to get used to doing that because it happens a lot in a big project, especially at the early stages. I suspect this is one of the most common failure modes for ambitious new projects. I've heard other people calling it the point in the life cycle of a new project in which uninformed optimism transforms into informed pessimism. Informed realism if you're lucky.

If the above is a bit too abstract for you, here's an analogy that explains it better: Imagine you are a dog that wants to reach a bone. You run toward the bone, but then you realize the bone is behind a fence. If you're a dumb dog, you'll probably start barking in frustration. Maybe you start digging, but you're a dog, not a mole so thats unlikely to be successful. Eventually if you're a smart dog you'll start looking for some way around the fence. Maybe there is a break in it somewhere, but to look you have to physically move away from the bone. On one level you are moving away from your goal! But because of the fence, proximity in physical space, which is usually a good heuristic for a dog is actually the wrong level to evaluate. On the right level, the problem space, by moving away from the bone in physical space you are actually moving toward your goal in the problem-space.

I've come to suspect many problems in the real world are a more elaborate human version of this simple canine scenario. But the basic principle is the same. You start out with a vastly simplified internal model of the problem space. You just made this model up to psych yourself into getting started. Inevitably your model begins to break down when it slams into reality (I.e., you start testing your assumptions). If you realize this before your resources run out, and you have the strength to override loss aversion then you go back to the drawing board to find a new and improved model. And more often than not, in the new model moving forward actually looks like moving backwards (or sideways) in the old model. Rinse, cycle, repeat.

I think it's important to realize this is how things really work because it allows you to let broken models fail more quickly and efficiently, and also it should make you wary of accepting generalized cookie-cutter advice from anyone that hasn't made the effort to understand the problem space in detail.

Incidentally, I think this is one of the big advantages open source projects have over proprietary projects. Building something sustainable takes time and more importantly patience and the willingness to do things right. Often proprietary projects don't do that because investors are rushing them to "be first to market" and "go big or go home". Under that kind of pressure they don't solve problems by being very clever because brilliance is unpredictable and harder to manage. Instead upper management focuses on resource allocation. If they want to accelerate development, they'll try to dump more resources on the project, often in a counterproductive way (more people rather than higher quality higher paid people). In other words, they try to steamroll their way through the simplified, broken model of the problem space. Unfortunately, dumping more resources often has nasty side effects beyond the intended goal of shortening timescales. You build up your overhead and cost structures. You build systems which are much more complex than they have to be. Your technology becomes expensive to develop and maintain. Unwieldy. In the rush a company may find that it simply can't hire the kind of quality people it might like to, so it starts cutting corners with the people you bring on board. Compromises are made, you start hiring according to buzzwords. Why would we want to use that buzzword non-compliant language? .NET and Java skills is what a "real" company looks for. Programmers for that are a dime a dozen!