Survival of the fittest does not mean designing the smartest of solutions

Having followed my husband’s, Anders Hesselbom’s and Marcus Ahnve’s discussion about the evolution saying of “survival of the fittest” and it’s baring on software development, I want to clarify some misunderstandings concerning “survival of the fittest”.

Many believe that survival of the fittest equals the winning of the best solution. The best and the strongest survives the competition. But that is not what survival of the fittest mean in software development.

Look at the vagus nerve in a human. It starts up in the brain, goes down under the aortic arch and then back up to the throat. Is this the most elegant possible solution to building that nerve? No, only an idiot would design the nerve like this if he started from scratch. A person on which the nerve goes directly would probably be considered as better or stronger. If you consider that the same type of problem of the rerouted nerve exists in for example giraffes, you can see the extent of the problem.

So, what is survival of the fittest? When the ancestors way back in time lacked our long necks, one or a number of individuals mutated so that their necks became slightly longer. The nerve became a tiny bit longer too, but the upside of the longer neck was bigger, so the group with the neck survived long enough to reproduce in greater amount than those without it. And step by step, the necks became longer. The side effect of the elongated nerve was always a lesser disadvantage than the longer neck. Of course, after many generations we had that absurd situation we find today, but the chance that a mutation surfaced where the nerve was instead directly routed and the rerouting was successful is very slim. And since the nerve as is does not result in an enough big disadvantage in reproduction, the design remains.

If you’re a software developer, close your eyes and relax. Then think about which system and which code the solution of the vagus nerve remind you off. Is it that neat and beautiful solution you’re so proud of? Well, I hope not. It is probably some kind of legacy code or legacy system. It was probably a good solution way back, when someone made a quick and dirty solution to a simple problem. The solution was just temporary, but remained and steadily became a bigger and bigger headache.

What you really would want is to cut that nerve, rebuild it in a smarter and more future safe way. But your manager might not see it as a problem. It works, doesn’t it? Which are the risks with cutting the nerve? Are you sure that we will survive and that things will be better afterwards?

Evolution is The Greatest Show on Earth. Wonderful to see, amazing when it comes to giving birth to amazing solutions, but don’t be so sure that you have the nicest and best design.

When a single feature results in an exponential increase of scenarios

Have you ever met a person involved in software development who thinks that the correlation between number of features and complexity looks something like this`

I mean, who does not understand that adding new features does add to complexity.

Then there are the slightly more informed ones who thinks that when you add 1 new feature, you add 1 complexity unit of one. You can see this as test cases. So, these guys thinks that one new feature is about one more test which you have to run.

But have you ever been in this situation (X axis is number of features and Y axis is number of test cases or what ever complexity unit you prefer).

You have a new system to wish you slowly add new features, things aren’t that complicated and then someone comes up with a brilliant idea. It shouldn’t be that hard to implement. But the new feature starts an exponential increase of complexity and now you’ve created a monster. The initial development cost was perhaps small but that little thing made all the upcoming features so much harder to implement.

We also have this scenario. Everything runs smoothly and suddenly you start an exponential growth of complexity. But someone stops the line, recognize what is going on. And does something of the architecture.

If you’re really lucky, you can perhaps even stop and decrease the complexity. Refactoring should lead to something like this:

But you also have another solution and that is removing features. Features which are heavy in complexity and which business value does not match the increased complexity it creates. I’ve done this a couple of times and all of those occasions there has been a tremendous debate over the issue. But when the feature was actually removed, no one noticed it.

So, what do I say. Two things:

1. Evaluate the complexity effects of a new feature before implementing
2. If you have a run away train of complexity, do something about it right away. And that can be refactoring or removing the feature(s) which causes the complexity.

Debunking the holy system entity

All that works with software development have been there. Having that discussions with customers/users and stakeholders about an entity or attribute which you cannot fully grasp. You have heard from the previous discussions that this attribute/entity, let us call it MustHave, is really really important and crucial Perhaps you can see it in the contract, the users brought it up during the first meeting and they keep coming back to it.

But you cannot understand why this is so important. For you it’s just another entity or more often an attribute. But for many of the users it’s the very foundation of the system.

In many cases it’s some kind of ID or number. The users know these numbers by heart and often there is one or two who can name all 250 of these unique numbers. But for you… It’s just a number. Why should the users see or know that?

The problem is that if you bring up the issue, stating that MustHave is not so crucial at all; that you can make it visible, mandatory, what ever, but that you see it as anything else in the system, you risk becoming crucified. You get a long lecture about how important Mustave is and how long they’ve been using MustHave in the company.

An you start to think about the new people coming into the company who have to learn all that. And perhaps you try to talk back.

But stop right now and think. Don’t think about how important MustHave is to the system and to the company. Think about how important is it to the users. MustHave can be the one thing that today tell a senior member of staff from a newbie. You are accepted in the group when you know all there is to know about MustHave. MustHave is an initiation rite in the organization.

The users probably does not see is as that. They are so into the ritual that they cannot get above that knowing long number series are no longer important.

What you do when you start treating MustHave as anything else in the system is that you take it from the secret knowledge of the seniors and you make it public knowledge. And the knowledge of all those who have spent decades mastering MustHave becomes useless.

The fastest person in the group is not those who have been there the longest and knows MustHave, it can be anyone who masters the systems.

When you demystify something like MustHave, you change the hierarchy and the definition of a competent employee.

But what should you do? My answer is: Agree in the communication but disagree in the system architecture. Comfort everyone involved by stating how important and special MustHave is but don’t make it special in the system. In all cases I’ve used this strategy, the MustHave of the system becomes debunked by itself. But without stepping on anyone’s toes and by letting people learn a new way without anyone discarding their previous competence.

Is it a project, or a product, or both?

I guess it’s not an uncommon problem. You have a project with a fixed and very important deadline which you cannot miss. But the stuff you’re building, you really want this to work for the future so that you can re use the stuff for other projects.

Since this is a scrum project, we only have one product backlog and all the items have a unique priority. So, how do you prioritize the long term value? In agile and lean software development, you want to build stuff as simple and easy as possible. But in this not uncommon scenario, “simple as possible” is not only regarding the project vision but the future projects. And these are not in the far future; one of these projects are probably starting half way through this project. So, this is something we need to plan for and take into consideration.

The best way to view this is from a product company’s point of view. The same scenario is common if you have a company who sells products but also custom built applications.

Here goes. You work at company which have the product XCV:P. And in walks a customer who wants to buy a custom application; Alpha. As a product company you can see that the output of this project can be used for XCV:P. The investment is big, so you really want the customer to pay but he only wants to pay for the stuff he needs for Alpha. If you end the project and the requirements of Alpha isn’t met; the customer won’t pay. But it would be crazy of you as a customer not to adjust the content so that you can’t use it for XCV:P.

So, how do you do it?

First, you need a project manager who understands both the need from Alpha and the need from XCV:P. But he must never risk Alpha to meet XCV:P. This is very very hard and require constant discussions and feedback. Second, the developer’s must grasp these long term and short term needs. You’re in for chaos if some of the developers think the objective is Alpha and others XCV:P. They need to know where the cash come from but they should also understand how beneficial it would be for the company if we made things nice for XCV:P too. Don’t try to hide this complication; expose it and let the developers handle and discuss it. We will probably need a product vision and a project vision.

In my non fictional project struggling with this complexity, I will use the allegory of the product company when presenting the project to the developers and I will keep the idea in my head when I work on the probably hardest task on any project: priorities. This is a project, and a product and we will need to handle that. But this is also what makes this a really fun and interesting project as well.

Avoid accidental architecture using scrum

The CIO at my company often say really wise stuff, things that get me thinking (and I don’t say that because he’s one of my bosses since I don’t think he reads my blog). It not for nothing that he was voted CIO of the year in 2007 by CIO Sweden. I guess he’s not the only one who says this, but for me to hear this from a manager is music to my ears:

We must make decisions on architecture. Either the organization have decided on architecture, or the architecture just happen. And the result will be a really bad architecture.

Developers make decisions on architecture almost every day and if they have nothing to base their decision on, the architecture will just happen and be accidental.

In a scrum project where the stuff with the most business value always should be prioritized, it’s easy to select everything else but removing technical debt and introducing a better architecture. This is why you must not only discuss income but also removed costs when you talk about priority. Still it’s hard when the sales department stand banging on your door, demanding the stuff which makes your product the best in the world.

So, you need a strategic decision that we work on our architecture. And yes, we give it a budget, plan for it and execute the changes. Try including the why clause to the architectural items, so that you can argue for the items and get yourself educated.

During ordinary working time. Architecture being improved on someone’s free time is not an option for a serious business. For a team to be proud in their work, their need of good solutions must also be addressed. So, letting the team have their saying about what in the architecture needs refining is a good reward and recognition that you believe in them as competent team members.

So, now when I’ve perhaps sold the concept of putting the architectural stuff, I’ll leave the details on how you can do that to Boris Gloger. You don’t have to be a developer to read this excellent article and you need not be a manager, business guy at all.