Flash back to old blog articles...some links may not work
How to stop your code going broke - Monday, May 15th, 2006
This is a talk/discussion I gave at CJUG May 10th 2006. These are effectively my talk notes. We did a recording (podcast) of the discussion which you can find with the slides at the cjug site. (direct links mp3 and pdf)
A discussion on when to take the plunge and do a “radical” Refactor, and how to choose when to deviate from a safe change. What tools can help you make serious changes without stopping everyone else on the team doing updates. How to manage team angst and the fear of change. How to smell your code (and prevent smelling mistakes)…. and much more.
Radical ReFactor – What? Why? When?
A refactor according to Wikipedia “Refactoring is the process of rewriting a computer program or other material to improve its structure or readability, while explicitly keeping its meaning or behavior.”
Radical refactor? A refactor that scares people.
What scares people? Rule of thumb, anything that changes more than three things (objects) at a time. Or something that a person can’t comfortably comprehend. Something that changes a lot of code or code structure significantly so that you can’t know all the consequences.
Experienced programmers and system designers hold a “map” of the structure and many details of a program in their head, and can judge what a change will involve. As the number of changes required in the code increases the natural uncertainty associated with a judgement tends to add up, leading to fear that they don’t know what will happen.
Less experienced programmers tend to rely on diagrams and documents of
structure to figure out what’s going on, but only hold a small map-page
in their heads at any one time. They live in constant fear of change
because they have to learn something new. And the documentation will
probably be out of date for weeks! (Better order toner for the printer
As you can see for a refactor to be considered radical it doesn’t need to be very large.
Simply put, shit happens. OK some examples. If you’re using agile development process or iterative development chances are you iterate from the ground up, gaining an understanding of the problem you’re trying to solve, and the code that might solve it as you go. The idea here is pretty much “learn as you go” or “learn from your mistakes”. You reach the end of an iteration and you carefully analyse what you’ve done and where you’re going. This means of course constant change and refactoring. Most refactoring will be small but eventually you’ll come to a point where you say, “hmm well we meant this but have this…”.
OK a metaphor: using agile/iterative development we’ll build a table. You have to start from the bottom up and build a functional table. Right we need legs. Hmm stability is important, so lets have three, and a top. Only needs to be functional so a small top is fine, just a bit or cardboard will do. Ah it works. OK iteration two add the extendability feature. This adds weight and when the table gets really long it’s looking a bit wobbly, but it works. Next iteration, add the anvil feature on the left far corner. Oh bugger, we need another leg, and they need to be stronger and the top is a bit too flimsy – bingo radical refactor.
When you have to, but sooner rather than later.
You’ll know when you really have to do a radical refactor, and by then it’s going to hurt. It’s a fine balancing act to determine when/if. Especially in iterative/agile development where you can’t say exactly what the product is going to end up like, if you jump too soon you could refactor for a problem that won’t exist in the next iteration. You need to say well can this be done quickly and simply to see how it works before making the code maintainable. And if so, can you see all the implications of the quick fix? Do they matter for this iteration?
You also need to judge if the “quick and simple/dirty” way will actually take you longer to implement. You see a well thought out and defined refactor normally doesn’t take as much time as you would estimate, and the process improves your understanding of the code. Refactoring by definition should simplify your code for the problem at hand. (Importantly a refactor does not add features)
OK lets look at some of the problems and issues again:
A “radical” refactor is by definition considered to be a big change, because a big change is hard to judge. Why don’t we like big changes?
A big change is not good for the team, look at the size of the commits to your version control system (you do have a version control system, don’t you).
- In a small commit it is easy to see the what changed, it’s easy to see the intent of the change and it makes it easier to review.
- The bigger the team, then the smaller your commits (and therefore the more frequent you can commit) the better the impact on other users. The bigger a commit, or the longer you wait to commit, the harder and harder it gets to commit because of merging conflicting changes.
These are points are very valid, far more than the fear of change. There are a couple of ways of looking at this but the first is to realise that the radical refactor is the exception not the rule, it should not be too common (or you really need to start planning shit). So when it does happen it’s like doing your taxes, not something you look forward too, but if you get in there and do it you can stop worrying.
To help with the above problems with big commits you should try and break your refactor into smaller functional commits if possible or branch the RCS for the refactor an then merge the branch when done. That way you can commit smaller changes regularly even though they break a build (so people can see where you’re going). You should also make sure everyone knows of the refactor so they don’t work on unnecessary code that will be replaced/removed when merged.
This is where good code repository tools come in. Most IDE’s have code repository support that lets you commit and look at diffs, add tags, merge code etc.. (PLUG)Tools like Fisheye give you a web-based view of your source repository with easy navigation, powerful search, historical reporting, configurable file annotation and diff views, change-set analysis, RSS feeds, and integration with your issue tracker.
A big change brings with it untested waters, you need to do all your testing again! (well you should be testing everything again anyway.) Importantly you have to refactor the testing too.
How can you sense a refactor is needed radical or otherwise?
Well you need some form of code metrics. The problem is it tends to take an experienced eye to spot problems in code that lead to diseased code. Experience comes from making the mistakes, and you’re trying to avoid mistakes, or pick them up early. One answer to this are Code Smells.
From Kent Beck and Ward Cunningham’s site http://xp.c2.com/CodeSmell.html
Mika Mäntylä has put together A Taxonomy for “Bad Code Smells” at http://www.soberit.hut.fi/mmantyla/BadCodeSmellsTaxonomy.htm
So how do you weigh up if you need to refactor or just do a hack? One way is to use technical Debt a metaphor developed by Ward Cunningham.
From Martin Fowler, http://www.martinfowler.com/bliki/TechnicalDebt.html
“In this metaphor, doing things the quick and dirty way sets us up with a technical debt, which is similar to a financial debt. Like a financial debt, the technical debt incurs interest payments, which come in the form of the extra effort that we have to do in future development because of the quick and dirty design choice. We can choose to continue paying the interest, or we can pay down the principal by refactoring the quick and dirty design into the better design. Although it costs to pay down the principal, we gain by reduced interest payments in the future.”
“The metaphor also explains why it may be sensible to do the quick and dirty approach. Just as a business incurs some debt to take advantage of a market opportunity developers may incur technical debt to hit an important deadline. The all too common problem is that development organizations let their debt get out of control and spend most of their future development effort paying crippling interest payments.”
“The tricky thing about technical debt, of course, is that unlike money it’s impossible to measure effectively. The interest payments hurt a team’s productivity, but since we Cannot Measure Productivity, we can’t really see the true effect of our technical debt."
Managing team angst
- What constitutes team angst when it comes to refactoring. It’s different things for different people. Managers see it as non-productive work, you’re not adding a new feature, the pay-off comes in the later savings (paying off the principle)
- Programmers see it as a lot of work to re-implement something they’ve already done and not fun, “aw mum do I have to clean up my room”
So to manage this
team angst there needs to be clear objective reasons why you have to do
it. For the manager you need to show that there will be a later pay off
in productivity. This is pretty easy if you can say we can’t implement
x without this refactor, or we can implement x in 2 days with this
refactor or 7 weeks with-out it.
To manage the programmers angst:
- If they’re concerned about the changes you need to discuss them, someone doesn’t understand the code/problem well enough and it might be you!
- If it’s boring then something’s wrong, you’re not implementing it right. Programmers enjoy the challenge to get code right and have it work, it’s a creation, we’re creative people.
- Tools tools tools. Tools do two things, remove mechanical (boring) tasks and make sure you don’t miss stuff. They can put programmers back in their comfort zone knowing that a mistake is less likely and they won’t have to do heaps of debugging.
Tools like eclipse, netbeans, IntelliJ all have refactoring support, take the time to learn what they do and how they work.
Remember CVS/SVN <insert your RCS here> is not just a common directory to store code, it’s a tool that lets you manage changes. Tags and branches mean you can try stuff out, and revert back to a known good state.
Also remember that work that is done that is “thrown away” is not wasted, it’s part of the process, this is a learning experience, that’s what development is.
A proposed formula for determining Fear
During my talk at CJUG it was suggested by Frank Van Praag that there should be a formula for determining the certain amount of uncertainty, or fear, people have when making a change and we came up with this:-
F=((L x n-1)/?)^f-u
F = fear
L = number of lines of code
n = number of people on the team
m = number of managers on the team
f = number of function points
u = the number of (functional) unit tests
? = something to do with the underlying architecture of the system