Pynchon's Theorem

Here's a bit of Agile wisdom from Thomas Pynchon's postmodern classic Gravity's Rainbow (page 275 in the Penguin edition):

And yet, and yet: there is Murphy's Law to consider, that brash Irish proletarian restatement of Goedel's Theorem - when everything has been taken care of, when nothing can go wrong, or even surprise us...something will. [...] when the laws of heredity are laid down, mutants will be born.

Faux OOP

It may look like OOP and be written in an OOP language and it may have some of the characteristics of OOP such as classes and inheritance but it may, in fact, be Faux OOP.

Fortunately Faux OOP is easy to spot as it is actually just procedural code organized into objects. Tell tale signs of Faux OOP are a separation of data and behaviour with anemic data 'objects' manipulated by stateless methods (or procedures) in other classes. Faux OOP is often littered with classes named with agent nouns (*Manager, *Helper etc.) or *Service which tend to be stateless (and often singletons or static classes) which hold common variables (better known as global variables).  The rest of the code is built from data classes which consist mainly of getters and setters and no, or rudimentary, behaviour which are poked and prodded by their *Manager counterparts. The end result is a program consisting of tasks which act upon data structures and use basic program flow (ifs, switches, loops) to execute the tasks in order.

Faux OOP is an anti-pattern promoted by some of the biggest players in the industry: Microsoft's N-Tier architecture, popular with VB and early .NET promotes Faux OOP by prescribing a Data layer, Business Layer and Presentation layer. The presentation layer binds to data objects provided by the data layer which are then validated and actioned via the business layer. The SOA craze has also helped promote the procedural style where people have reduced their systems to procedural service calls.

Though be careful, you may think you have Faux OOP but really you've got Faux Procedural.

Faux Procedural
It may look like procedural because the program has methods and data structures but if you look closer it may be sequential code organized into methods.

Tell tale signs of Faux Procedural are long methods with no distinct separation into common tasks. These long methods often contain significant repetition and the heavy use of temporary variables. Common tasks are not abstracted into smaller, reusable methods and common groups of variables have not been abstracted into data structures.

Fortunately, with heavy refactoring, Faux Procedural can be refactored into Faux OOP and Faux OOP can be refactored into Real OOP(tm).

Agent nouns are code smells

Class names ending in agent nouns are a code smell. An agent noun is:

any noun that denotes someone or something carrying out the verb's action, typically words ending in -er or -or

Classes with agent nouns are, the majority of the time, a sign of procedural thinking - especially agent nouns such as "manager", "helper" etc.   As the definition says it denotes the class is carrying out the verbs action.   This is contrary to good OOP where the verbs belong to the nouns themselves; classes which represent things are responsible for their own doings, not someone else. This reflects the real world where I am a Person able to do Programmer tasks, the verbs of programming are executed by me not some CodeProgrammer object which sits next to my desk.

Agent nouns are useful for describing roles which makes them good for interface names. A SpellChecker interface on a Dictionary class gives clear definition of the role (to check spelling) and allows the Dictionary to implement the verbs (check_spelling "word"), likewise a SynonymProvider sits well on a Thesaurus class.

So beware agent nouns; they are a language trick which fools you into believing that a class is a first class concept when really it's stealing someone's verb.

Brain Rules

I stumbled across the Brain Rules website while browsing Presentation Zen.

It's a great site and goes through a nice little set of flash slides on how our brains work and how to maximize performance. I'm sure knowledge based workers (such as developers) could learn a few tricks from this.

The Product based build

Anyone who has worked on a project of significant size would have grappled with maintaining a build which continuously spins out of control. It feels like targets become confused, randomly change or seem to be repeated multiple times with only slight variations: "run", "run-without-tests", "run-wth-tests", "redeploy-run" etc. etc. It doesn't take long for the build to become unwieldy.

People who use the xAnt family or others with XML based languages such as MSBuild will have felt the pain more than those using dynamic language libraries such as Rake as XML is predisposed to becoming quickly unwieldy, is difficult to refactor and expensive to extend (basically XML sucks as a programming language). Yet, XML or not, most build frameworks share a common flaw: they are task based. The build becomes defined in terms of a set of steps to be executed and in which order and fairly soon you find you want subtle variations in those steps which result in more tasks.

But builds aren't linear; we don't always want to run the same dependent steps to achieve the same end result which is why we end up with masses of subtly different targets. One approach to reduce this is to remove dependancies from targets and have high level targets which simply list a set of steps. This can lead to flexible systems, as targets can easily be stitched up arbitrarily from the command line as needed, but it requires an intimate knowledge of how the build needs to be executed (which breaks encapsulation) and moves even further down the road of defining a list of steps.

So how should a build work? In this post I hope to outline what I believe to the be core metaphors and functions of an effective build system.

Product based
If we go back to basics and look at the purpose of a build we find it is to produce specific products: a deployable artifact, verification (by running tests), a deployed application, installers etc. Instead standard build tools promote thinking in terms of linear dependent tasks, for example creating an installer would be defined as "clean, compile, test, create-installer", the build is described in terms of steps, most of which say nothing of the products created.

A product based build is different: the build is defined in terms of products which are built by producers (or factories to use the GoF pattern). Products represent the artifacts of the build (binaries, installers, deployed application) and producers contain the logic on how to assemble the products.

Dependancy based
Similar to task based builds, product based builds are dependancy based. Where product based builds differ is they view the build process as a factory line where each product is produced from other dependent products. For example the "installer" product has a producer which packages binaries from the "verified-binaries" product which has a producer which runs tests from a "binaries" product which has a producer that compiles the code from the "fresh-destination" product which has a producer which creates new, clean directories.

Describing dependancies in terms of products rather than tasks creates a more descriptive build where each product can express its dependancies in a specific way, for example an "acceptance-verified-binaries" product produced from "unit-verified-binaries" expresses a different set of rules from being produced from "binaries".

Encapsulated
When we run the build we shouldn't have to understand the steps required to reach the end result but because of the nature of task based builds they often require an intimate knowledge of their internal workings: for example "run" may depend on "clean, compile, test" but "run-again" may have no dependancies but instead depends on a previous build. By focusing on products this thinking is broken. As an example consider the following: in a task based build you can retrieve a set of targets, with their descriptions, from the command line. To begin to use the build you must understand how the targets relate and what steps are being executed for each target, and from the descriptions (or the task names) work out what the end result may be. A product based build would give you a list of products but not the steps required to produce them. This allows the user to quickly understand the goal of the build and how to achieve the desired product without requiring any knowledge of how it is produced.

Non-repetitive
Task based build systems are prone to repeating already satisfied dependancies, for example if you ran "build compile, create-installer" and the "create-installer" target had a dependancy on "compile" most build systems would re-execute the "compile" target wasting your time. To work around this you must create more tasks and again require an intimate knowledge of the build. Instead when running "produce binaries installer" the product build would be satisfied that it had already produced binaries when it requires them for the input of the installer.

Order independent
As task based builds execute as a series of steps they are order dependent. Running "build compile test" has different results from "build test compile" (which would either fail or give invalid results depending on whether a previous build had compiled). As a product based build focuses on products and not steps it is order independent so specifying the following would give equal results:

  > produce binaries verification
  > produce verification binaries

Presuming that the "verification" product relies on the "binaries" product with the first case the build system would re-use the one produced from the already executed "binaries" request and in the second case it would be satisfied that during the production of the "verification" product it had already been produced.

Polymorphic
By focusing on products rather than tasks a product based build system can become polymorphic. This allows for a very flexible build removing the need for the subtle variations on targets which plague task based builds. For example you could define a product of "running-application" which is produced from the "binaries" product. The "binaries" product could be inherited by two other products "verified-binaries" and "unverified-binaries". By giving the "running-application" product an input of the base "binaries" product it can be produced from both "verified-binaries" and "unverified-binaries". This allows the user to easily control the build:

  > produce verified-binaries running-application
or
  > produce unverified-binaries running-application
or (as we are order independent):
  > produce running-application verified-binaries
or (which would use the default binaries product)
  > produce running-application

We can also extend "verified-binaries" to "unit-verified-binaries" or "acceptance-verified-binaries" etc. creating support for a build pipeline, or create a "reused-binaries" product to avoid recompiling. By allowing a build to become polymorphic it becomes both highly flexible and intent is clearer.

Stateful
Though not absolutely critical state is something most build systems lack. Every time a build is run the results of the previous build are completely discarded and any valid products are re-created. State is a difficult concept to implement in task based builds as tasks would have the responsibility of deciding whether the artifacts are still valid, this is further complicated by the fact that multiple tasks may depend on the same artifacts and would therefore have to make the same decision again. In a product based build each product can decide whether it is valid and the build system would simply reproduce it if it isn't. Furthermore the build system can analyze the product line and understand that if a product further down the line (say binaries) is invalid then all products produced from it need to be reproduced.

To conclude by changing the build metaphor from tasks to products we can solve many of the issues around build systems and create clearer, more flexible and more meaningful builds for our applications.

Decisions, Decisions, Decisions

During any development project thousands of decisions will be made. Every individual in a team will make large numbers of decisions whether about details such as variable names or cross cutting concerns such as architecture. The success or failure of a project is the sum of all these decisions.

There are two key factors in making decisions: information and risk. The more information we have the better informed the decision is and every decision made - and not made - carries a risk. Empirical methodologies (such as Agile and Lean) have contrasting methods to decision making than up-front plan based methodologies such as waterfall. Fixed-plan methodologies attempt to manage these factors by collecting as much information as possible and then committing to decisions before implementation starts where as empirical methodologies promote delaying decisions until there is enough information available.

Both approaches recognize the importance of information and of managing risk yet they have contradictory views on the best way of dealing with them. Where fixed-plan methodologies believe that by committing heavily to the informed decision prior to execution you reduce risk empirical methodologies accept that, in reality, every decision that you make ahead of time carries the greatest risk of all: it could be wrong. The chances of wrong decisions are increased because the information that fixed-plan methods derive their decisions from is essentially highly detailed speculation which in turn is based on a limited amount of real information which carries the risk of being incorrect or out-of-date or even irrelevant. The strength of commitment increases the impact of a wrong decision even more. Agile and Lean acknowledge this fact and tackle it head on by managing risk with two techniques: they delay decisions as much as possible until the maximum amount of real information is available - what Lean calls the last responsible moment (as Lean acknowledges there is also risk in delaying a decision too much) - and they attempt to de-risk decisions by lowering commitment - essentially by allowing change.

Agile and Lean acknowledge that upfront decision making carries the need to cover a great number of hypothetical eventualities than may or may not actually occur. The crystal ball gazing of fixed-plan methods is an attempt to de-risk the unavoidable gaps in information by trying to account for any possible variation with more pre-made decisions. Essentially imagine trying to plan out an entire chess game before you play it: every possible move your opponent does or doesn't make would have to be accounted for in the plan. The end result? An over complex and convoluted plan (which translates to over engineered, complex software). Again because Agile doesn't rely on pre-packed decisions, instead acting on the real information, it removes the need to make the "what if" decisions - this is the foundation of the maxim "You Ain't Gonna Need It".

Another aspect that traditional methods fail to tackle is decision paralysis: having one decision to make is hard enough but having to make all decisions infallibly and at once requires omnipotence. Although some developers believe they are gods Agile recognizes their limitations by giving techniques to allow focus on as few decisions at a time: TDD is a great example of this and so is the maxim "The simplest thing that could possibly work".

Agile and Lean promote both the overall reduction in decisions and minimizing the number of decisions to be made at one time (essentially by spreading them out across the project). Pushed to an extreme, decision making can be reduced to being purely reactive. The purely impractical "URL Driven Development" demonstrates this by only making decisions when the event which creates the need for a decision occurs. Essentially when someone hits a URL you write just enough HTML to satisfy their request. Despite being impractical it demonstrates the principles of Just In Time decisions well and should not be used to negate their value within the development cycle where decisions can safely be reduced to being purely reactive. Test Driven Development uses JIT decisions extensively: you write a test, an event occurs (unsuccessfully) which forces a decision to write just enough code to satisfy that one event, rerunning the test recreates the event and validates your decision and then you move on to add another event forcing another decision.

Aggressively minimizing the number of decisions which need to be made by simply delaying them or eliminating them altogether has an overall positive effect on a project. Ensuring that there are few small decisions which require minimum commitment and have the maximum amount of information increases the chances of making good decisions and reduces risk of committing to bad ones. In short: prefer fewer good decisions than many potentially bad ones.

Legacy Index

A legacy index measures the number of releases a software product has until it becomes legacy. It can be used to measure and communicate design quality: the more releases on a code base before change becomes difficult and it gets demoted to a legacy system then the better the design.

The target should be a high legacy index or to reach Legacy Max. An application that becomes legacy after its first release would be a Legacy Zero app. The lowest index is awarded to applications which become legacy before they are even released: Legacy minus One! 

Applications can still have further releases once they've hit their legacy index: these releases would be identified by consisting mainly of bug fixes and only minor changes, this would be called "pushing the legacy index" as each release would not increase the index. Also an application may be able to reverse its legacy status by refactoring it to a point where change is again possible. 

By measuring and plotting an applications legacy index against time you can clearly see the health and state of an application. If it has a steady and consistent incline then you know that the design is sound and the application is healthy but if the index has a trend of slowing or is continuously pushing the legacy index then it is clear that there is a problem. 

Though quite humorous legacy indexes may have a genuine use in describing the state of an application, you can simply say "this is a Legacy Zero app" or "it's hit its legacy index" and all is clear.