Fear of the new (how TDD and DI encourage bad practice)

When I first started doing TDD back in the mid-naughties it changed the way I programmed dramatically.  It took me a while to get it at first, how can you test one unit? until I discovered the tricks of dependency injection.  Now my code was shaped in loosely-coupled wonderness and I felt invincible.  I thought I had reached the pinnacle of OOP. Everything suddenly became so easy, it was well tested, easy to change, easy to wrap, swap implementations, decorate, whatever, I was throwing the best design patterns out with such ease that I barely needed to refer to the Gang-of-Four anymore.

I went on in this vein for a good year or so.  And as the millenium's first decade entered early evening and its bright lights began to fade I took a step back and looked at my code.  My god, I thought, this is terrible, what the hell have I been doing? And it was terrible, hundreds of interfaces named somebody's Manager, Persister, Service (oh so many classes called Service), Validator etc. etc., all with a single namesake class implementing them.  There were tens of classes whose purpose seemed to be to co-ordinate these little buggers - for they were little, ten or so lines each (oh how I thought I was doing a grand job) - and they would hand off to other similar classes who co-ordinated other classes and it was turtles all the way down.  And in amongst all of it there were about a dozen classes that had real names like Customer, Account (but even they had interfaces too) and, aye how the sting of memory pains me to this day, they only had getters and setters (it's the shame, you never get over the shame).

It was with great pain I realized for the last year of my professional life I had been producing procedural code with all the idealistic misguided joy of a card holding Russian worker pouring cheap cement to build pointless roads.  I knew the culprit alright: TDD and DI.  I had gone decoupling, unit testing mad.  The result was an irrational fear of the new keyword.  As far as I was concerned the new keyword was banished to the dark depths of Containers, it was a clumsy language feature never to be used again.  But what I had instead was a dependency graph made up almost entirely of stateless singletons.  A simple Ruby script could have replaced every injected dependency with calls to static methods and sure the application would have lost its testability, it would no longer be decoupled, but essentially, at it's essence, and in terms of structure, it would be the same.  TDD and DI had simply given me a fancy way of decoupling static classes.

The new keyword is a wonderful thing.  It lies at the heart of OOP.  Its sole objective is to create a new instance of an object.  If you don't use new you can't create new objects and if you can't create new objects you can't write object orientated code.  And the other fundamental thing about objects is they encapsulate behavior and state.  Singletons don't.  Singletons are procedural.  Regardless of whether they are loosely coupled and injectable.  Real OO classes have private fields (the fewer the better) and methods which do things based on those private fields, sometimes based on arguments passed in, but always, always in the context of those private fields.  And the only way, the one single true way you get values into those private fields is via the constructor and that means using the new keyword.  Let's put it clearly: this 'new ValidatedCustomer(customer).IsEmailAddressValid' is OOP. This 'customerValidator.Validate(customer).IsEmailAddressValid' is procedural.

Now I was writing code using TDD and DI that was object orientated code.  But my fear of the new was still there.  In order to maintain the injectable, loosely-coupled gorgeousness I had begun to do a horrible thing.  I started injecting factories everywhere.  Sure it was an improvement but there was still something horribly smelly going on.  After all the factories were essentially static, singletons whose sole purpose was to delegate to the new statement.  I mean there's single responsibility and then there's craziness!

So I started doing something I thought was really bad: I created objects in my classes! But I wanted to keep loosely coupled and all of that wonderfulness.  This required a dramatic shift in thinking.  Remember when you first did TDD and how it really hurt because you had to structure programs in a different way?  Well it was like doing that all over again.  Every time I started doing something I had to spend half an hour thinking, how do I make this work?  I know I'm right but how?

I learnt a few lessons:

• Not all classes are about interactions, sometimes they are about results.  Mocking everything out for the sake of it doesn't gain you anything.  Use state based testing to assert the end results not interaction testing to assert what is happening to achieve those results.  
• It's OK for your collaborators to new up something if you ask them, so you can say fileSystem.NewFile(name) if you need to.  
• Collaborators don't always have to be passed in the constructor: they can be passed as arguments to methods as well, so you can say new File('myfile.txt', 'Some text').SaveTo(filesystem).  
• If a class is unavoidably all boilerplate and co-ordination consider using integration tests, after all mocking the interactions of boilerplate code doesn't tell you anything useful at all.  
• The container shouldn't contain everything, it should be the things that generally require configuration or are likely to change: databases, filesystems, web services etc. Rarely do core domain classes or business principles need to be 'switchable'.

Desktop Mashup

The concept of mashups generated a leap forward for web applications. By exposing data and functionality in an open and standard way other sites could use it, mashed up with other data using the functions from a different site, to provide services beyond the imagination of the original owners. It presented a significant shift in thinking: that data and function was more powerful - and arguably more valuable - if applications outside your own could leverage it.

This shift in thinking is not only limited to the web; desktop applications can also leverage these ideas. Locally running rich client desktop applications can be enabled to expose their features to other local, or even distributed, applications.

In fact, in Unix, this concept has been around for decades in the very simple, yet immensely powerful, form of software pipelines. By chaining the output of one program's data (via stdout) another program, with it's own distinct functionality, can then manipulate the result. This simple technique has enabled Unix developers and administrators to solve an infinite number of problems from a small set of tools.

Between these two extremes is the idea of desktop mashups. But instead of following the pull nature of the web, or the push nature of software pipelines, desktop applications can take advantage of the event driven nature of GUI systems and thus create a rich desktop experience. Thus separate programs, with distinctly different functional offerings, can leverage each other to provide functionality beyond the others original intention.

For clarification consider this simple example: you have two separate applications: one which rips DVDs, another which catalogues them ready for playbook (e.g. iTunes). When the DVD ripping applications completes it sends a message to say a new movie is available. The cataloging application - which has no direct knowledge of the first, receives the event and interrogates the new movie and adds it, and its metadata to the catalogue.

Of course this is nothing more than Event Driven Architecture which is hardly a new concept. The move to mashups comes when we create new applications from the existing apps functionality. So, in our simple example, we create a new application which fetches film reviews from imdb. When the user inserts a DVD from ripping it fetches the review so the user can decide whether they should go ahead. When the user does rip the DVD the review application updates the catalogue application so the review is part of the movie's meta.

Enterprises are ripe for desktop mashups. A significant number of problems for users is the lack of integration between applications, whether they are different internal apps or even third party. Another advantage, especially in large enterprises, is that development teams can focus on developing small applications that do their jobs well, or integrating with legacy apps, and mashing them together rather than struggling to develop all-in-one god apps.

Immutable Wrapper

I am a huge fan of immutables, especially for value objects. Even in clumsy static languages like C# and Java that seem to throw every obstacle in your way to deter you from using them I find the benefits outweigh the costs.

Sometimes though a mutable makes sense. Whether that's because of the restrictiveness of a framework or the paradigm of the language. But what if we don't want to loose the power of immutables behind the scenes?

Introducing The Immutable Wrapper Pattern!

The Immutable Wrapper is a wonderfully simple pattern because there are only two things you need to do.

1. Create a class with only one mutable field
2. Wrap the immutables functions.

Here's an example. You've got a copy-on-write list implementation but your binding it to some GUI control that gives you a big headache if you try and point it to a different data source.

Here's your immutable:

  class CopyOnWriteList
 {
   private readonly IEnumerable items;

   CopyOnWriteList Add(item)
   {
     return new CopyOnWriteList(new List(items, item));
   }
 }

Now wrap it:

  class WrappedCopyOnWriteList
 {
   CopyOnWriteList list;

   void Add(item)
   {
     list = list.Add(item)
   }
 }

There, it couldn't be simpler. So simple in fact I don't know what else to say. Enjoy!

The Singleton Killer

We all hate singletons so here's another useful refactoring pattern: The Singleton Killer.

Here's a common snippet of code we all love to hate:

class Transfer
 def funds(from_number, to_number, amount)
   from = AccountRepository.instance.get(from_number)
   to = AccountRepository.instance.get(to_number)
   from.balance -= amount
   to.balance += amount
 end
end

So how do we kill this beast? Simply follow these idiot proof steps:

Step One: Introduce Field

Stop asking the singleton for it's reference and store your own by creating a field:

class Transfer
 @repository = AccountRepository.instance
 def funds(from_number, to_number, amount)
   from = repository.get(from_number)
   to = repository.get(to_number)
   ...
 end
end

Wow: it's starting to look like normal code!

Step Two: Initialize in constructor

Move the initialization to the constructor:

def initialize()
 @repository = AccountRepository.instance
end

This is good but wouldn't it be better if the dependancy could be injected?

Step Three: Chain constructor

In languages where dependency injection is useful you will need to chain the constructors:

public Transfer() : this(AccountRepository.instance)
public Transfer(AccountRepository repository)
{
 this.repository = repository
}

Now the AccountRepository can be injected which of course means we can mock it out!

Step Four: Extract interface (for statically typed languages)

Except statically languages: they'll need to extract an interface:

public interface IAccountRepository
{
 Account Get(string number);
}

public Transfer(IAccountRepository repository) ...

Good: now you can start mocking and testing.

Step Five: Introduce container

If you haven't already got a container then get one (else ensure you a wiring up your dependencies in one place). Then get that to wire up the Transfer object for you:

container.Add(AccountRepository)
container.Add(Transfer)
transfer = container.Get(Transfer)
transfer.funds("YOUACCNUM", "MYACCNUM", 50000)

Step Six: Remove singleton

Once a container is doing all your injection for you that singleton is a waste of space: get rid of him by removing that horrible instance accessor.

Choose your strategies with this refactoring wisely: if your objective is to get a class tested then first repeat steps one-four within the same class until it is clean of all singletons. This is a good exercise in it's own right - even if you're not going to move on to test or remove the singleton completely - as you start to move all the dependancies to the constructor you gain a better picture of the classes collaborators (and the quality of design). On the other hand if your objective is to remove the singleton repeat steps one-three on every usage you can find then finish up with steps four-six. Another technique is to keep pushing the useage of the singleton up through the code so each client then takes on responsibility for passing the singleton down when it creates an instance of the class. Like so:

  Transfer.new(AccountRepository.Instance)

Then repeat steps one-three on the class you've just pushed into, then push the singleton up again up and so on until you can't push it anymore. This can be an interesting exercise in itself as it starts to push out the paths through the code base as you move through the layers and may also give you a clear point where your container needs to be. Be warned though: it can get messy (though strangely satisfying)!

On larger, messier codebases none of these approaches may be realistic in which case opt for the guerilla approach and pick them off as you find them. Though I would encourage trying to at least clean out a whole class using steps one-three even if you won't end up testing it right away. With all your dependancies in one place you'll still have a better picture of the design.

Beware Metanarratives

Now my darlings gather round while I tell you a story. But not just any story. A story about stories. About BIG stories.

Not so many decades ago, when the French still measured their GDP on the combined weight of their philosophical works, a particular man, with the amusing name of Lyotard, came up with the concept of Metanarratives (or Grand Narratives).

A Metanarrative, simply put, is a big story, often idealistic in tone, essential a grand, universal theory which is then applied to every aspect of life. There are many good examples of metanarratives from religious narratives, humanist, to economical and social theories communist, capitalist, freemarket etc.

The IT industry is brimming with metanarratives. Whether it is Stallman's completely open source future, no software patents, where everyone collaborates and all is wonderful, or the visions of an open web which overflows with information causing the end of totalitarianism, poverty etc. or the whole proprietary vs Open Source battle of the metanarratives.

Even as we move closer to home in the ever rational world of Enterprise Software Development we too find it overspilling. Waterfall has a wonderfully convincing metanarrative. And if SOA isn't a metanarrative (of Joycean composition) with it's land of milk and honey then I don't know what is. Probably the fastest growing metanarratives in IT is that of the Agile and Lean movement.

We humans love a good metanarrative to pin our life story on. They make us feel safe and cosy; it's nice to comprehend the world through the view of a single cohesive story with a beginning, middle and happy end (which is why marketing departments love forming them). We look around us finding more and more things to embellish and support our narratives. As we gain experience we see it as evidence to the greater truth of our chosen metanarrative.

But metanarratives are not a good thing - especially in an ever shifting landscape like IT. They make us lazy and prevent us from considering other stories. And if they are considered it is as a threat (e.g. MS' campaign against Open Source). Which is a problem in it's own right: if someone has subscribed to a metanarrative then it is almost impossible to win them round to any other story. Put one metanarrative against another and your blocked - queue visions of an evolutionist and a creationist locked in a room for eternity - monkeys would have written the works of Shakespeare before either changes their mind.

Metanarratives ultimately disappoint - especially in the ever shifting landscape of IT. Not so much because they are wrong but because they cannot be right. Everything, especially in IT, cannot fit into one nice story. The world is too culturally rich and diverse. As with the thousands of varying religions, The British Empire, Communism, the American Dream, DNA architecture, eventually the world shifts into a state where the narrative becomes ridiculous to apply. Yet people will keep doing so - the continued prevalence of Waterfall in some businesses being a case in point.

My concern is that we are guilty of promoting Agile as a metanarrative. Evangelists have moved it, from being a local story about a collection of problems in software development, to a model for the entire way businesses are run. Now, let's be clear, there are many elements of Agile I strongly subscribe to and promote, and I often agree with the basis of many of these claims. However the danger occurs when the Agile movement (or any other movement) seeks to displace any other narrative and claim itself universal. That ALL software should be developed this way, and ALL businesses should model themselves to support this. That's just crazy speak!

Interestingly (in my opinion) in their original forms Agile and Lean are post-modern in their attitudes towards metanarratives. Both attempt to promote fresh thinking and new ideas and move away from grand prescriptivism and dogma, especially of those found in the industry at the time. Most importantly both movements promote the fact that there isn't one universal way. For many people it was this breaking away from the metanarrative of the time and creating space to form our own stories which made Agile so attractive. Perhaps it is the inherent paradox in the post-modernists view of metanarratives that have caused them to, ironically, become metanarratives themselves.

The world of IT is too rich and diverse and full of too many stories (some old, some new) to support metanarratives. There is nothing wrong with some of these stories (such as Lean and Agile) when viewed in isolation; some are full of rich culture and ideas, some are simply misconceived and naive. On their own they are harmless and together with other stories they are invaluable. But as soon as people start turning them into something with which to narrate the whole Industry they are deadly.

So my warning is this: beware of metanarratives. If you notice that you are tending to frame the IT world around one story, whether that is Scrum, SOA, BDUF etc. please consider whether you have been caught in a metanarrative. Equally if you find yourself embattled due to someone's particular metanarrative, don't try and slam another one around their head. Instead try and appreciate their story at a local level whilst keeping your own stories localized and nonthreatening - they may even begin to fit it into theirs.

Killing the Config

Most applications require config of some description and it is considered good practice to wrap any calls which retrieve configured values in a config class and pass that around rather than splattering the codebase with direct calls to the frameworks inbuilt static methods.

With all the applications configs neatly wrapped in one place the code starts to become more maintainable. Yet are we just replacing one evil with another, if somewhat lesser, one? The problem I often see is this Config class finds itself passed from class to class flouting it's promiscuity and leading writers of articles on config objects down paths dangerous and full of inappropriate metaphors more commonly favoured by the ruby crowd.

There is no doubt that it is useful to collate all of the configurable elements together in one (or more) common classes. What should be avoided is leaking this concept through the whole domain and kicking it from object to object like a dirty old data bag which everyone rumages through, unchecked, pulling out stuff on their own whim. Config is an implementation detail which the majority of the application should have no concept of and (like any other such implementation details) it needs to be locked firmly away like a lump of kryptonite in a lead cased layer so it can no longer leak and permeate and weaken your SuperApp.

Let's think about this in a BDD fashion. When your class want's a ghostbusting url who's it gonna call (they have a webpage now dinyuknow)? string Config.GhostBusterUrl? Or perhaps the GhostBusterUrl class? If your class needs a typed collaborator with certain logic and behaviour (even if it is just supplying a valid url) then that's what your class should get. Not some smelly generic config object that dishes out dirty unparsed strings. The tests should be enough to tell you I'm telling you the right thing: several lines of priming the mockConfig are replaced with the simple supply of a GhostbusterUrl.

So how do we get the GhostBusterUrl from a line in config to those poor desperate classes who are being terrorized by spooks? With a little Tell don't ask and a container. Turn the config object inside out, make its role to inject into the application the dependencies it wants to supply. If your config believes it has a GhostBusterUrl then get it to construct one and register it (along with anything else it thinks it might have of use) into the container when the application tells it to. Not only does the config object gain some of its dignity back (some other classes were calling it nasty names, I know, it's the new millennium, but some people still hold the old fashioned ideas) and is safely kept right to the very edges of your application where no one can get at it.

Another positive side effect is that now your config object is giving birth to loads of lovely typed objects that actually do real stuff rather than having wandering hands pluck at its primitives (I know but I did warn it was a dangerous path laden with tasteless innuendos, I was bound to trip). Though of course, be careful not to pollute your codebase with a million classes that simply represent strings and do nothing else (it's a fine line and one you'll need to argue between me, myself and I). It's also one of the easiest refactorings you can do to your code. Give it a go (and if you object on the grounds of their being loads of small types then please find the closet crumbling procedural bridge and throw yourself into the polluted sequential river below).

Value Driven Development

At the beginning of the year I blogged about using value as a measure of productivity. Since that time I have had the opportunity to refine my thinking and be influenced by some other ideas. I want to revisit the original idea of using value as a measure. I may be guilty of repeating myself a little but it's a small cost for clarity.

The concept focuses around one very simple idea: the primary objective of a project is to deliver value. The more value we deliver the more successful the project is. If you agree with this then my next idea shouldn't be too hard to grasp: the primary measure of a project is value. Not velocity, not story points, flow, throughput, cycle time, capacity, cyclic complexity, code quality, but value, pure and simple. Of course we can use the other measurements to help us discover more effective ways of delivering value but on their own they tell little about how much value is being delivered.

Strangely, despite the fact that agile and Lean and Kanban and all of it's derivatives tell us how important value is none of them propose explicitly measuring it. Agile proponents often argue that it is their focus on delivering value early and often which makes them unique over conservative methodologies but how do they know? I've never picked a story off the wall and been able to tell what worth the customer gains by delivering it - ten, a hundred, a thousand, a millions pounds? Hell, who knows? The BA? The PM? The product owner? And how many projects out there can raise their hands to the question "How much value has your project delivered in the last week/month/year?" So if we're all so very value focused in our lean and agile worlds how are we not answering these simple questions? Are the statements about delivering more value pure assumption? How do we know that when we make 'improvements' to our processes the effect is to deliver more value? Or is at case of substituting value with story points mapped against velocity? If so how do you know that twenty-five story points to improve online advertising generated any revenue? Or is it a case of take the points and run?

Before we can confidently improve the amount of value we deliver we need to be measuring the value we are delivering. The first problem that presents itself is how do we do this? Value is hard to quantify and even harder to estimate. Which sound awfully similar to the problems with size so why not transfer the techniques used to estimate of story size to estimating value. Start simple, use relative sizes, but instead of t-shirts maybe use precious metals (aluminium, bronze, silver, gold, platinum etc.) or any other metaphor that helps the team visualize features in relative worth and as with size estimation it's an estimate, it doesn't have to be exact, the important thing is that we have some sort of gage and that we track it and refine it.

Once stories have value estimates they can go through the normal delivery process: estimate size, queue them up, stick them in iterations (or do some funky Kanban pulling on them). Except now every story card has it's most important piece of information branded to it: it's value. During prioritisation the platinum card which is the size of a small T-Shirt should obviously be delivered before the XXL bronze one. When stories are worked on people know that it's worth putting that extra effort into putting the plating on the platinum one but perhaps the aluminium story requires a more pragmatic approach than spending two days of cross functional refactoring to perfect it's design. During standups if someone says a platinum card is blocked the whole team knows it's important to get this sorted and coming up to a release with three gold stories complete but a stubborn bronze proving difficult to get across the line you may be more tempted to make the call to leave it out this time and hell to the size points. Very quickly every part of the process focuses around the amount of value being delivered, every decision is being influenced by the value of the story, not by it's size, not by how cool it is, not how much can be crammed into an iteration, but by how much value this piece of functionality is actually going to bring to the customer. Now that's being agile!

The purpose of putting values on stories is to measure how much value has been delivered. Now let's be clear about what delivered means: out there, in production, available to everyone who should have it, in full use. Basically, not dev complete, not in UAT, not in live pilot to a handful of select users, but out there live and in concert, fully functional, fully delivered, bringing value. Until then the functionality is worthless; until it is delivering value then you ain't delivered any value. Once it is though start totting it up: add the value points onto the delivery (note: delivery not iteration - iterations cannot deliver value unless they release), repeat for the next delivery plotting the value delivered on a chart and voila: the project's true velocity: how much value, for each delivery, it has been delivering.

But what if the value didn't quite get delivered? What if the functionality was released but with a bug which somehow hindered it's full capability? Maybe it is too slow, too difficult to use, doesn't do quite the right thing etc. Anything that prevents the application from delivering all of the value it promised is an issue (regardless of whether it is a bug or 'a feature'). However by resolving the issue the lost value can be reclaimed. By fixing the bug, performance or usability problems, the project claws back the value it lost when it didn't get it quite right first time.

It's important the project's value velocity reflects this. So when a fix is raised it is given a value which is then deducted from the velocity: this is because the full value of that feature was not delivered. Once the fix is in use and active then the value points go back on. Personally I find this one of the most powerful ideas behind measuring the velocity of value as the bad habits of delivering bug ridden functionality iteration after iteration by dismissing fixes in order to keep earning the points will not get you anywhere as the value velocity either struggles to move or even, in extreme cases, goes backwards.

I think there are many other benefits to be gained from using value as the primary measure that are yet to be discovered (possibly placing value onto technical debt? Though I'm not yet convinced of that one). Explicitly placing value at the forefront of the project gives teams the power to optimize themselves and discover new ways to improve their efficiency. And it isn't necessarily limited to individual teams: projects themselves should be measured in this way with teams 'contributing' to value - not simply earning points for delivering 'their part' - breaking down silo attitudes by finding ways to effectively work as a whole so value can be delivered.

I think it's pretty exciting stuff. Of course all the other metrics out there may or may not find use, dependant on whether they are contributing to the delivery of value or not, and holes will be found in the technique, but overall I think it shifts the debate and discussion forward to a entirely new level of thinking. It will allow us to reach a whole new level in the Value Enlightenment.