Tuesday, 12 August 2008
Brain Rules
It's a great site and goes through a nice little set of flash slides on how our brains work and how to maximize performance. I'm sure knowledge based workers (such as developers) could learn a few tricks from this.
Friday, 1 August 2008
The Product based build
People who use the xAnt family or others with XML based languages such as MSBuild will have felt the pain more than those using dynamic language libraries such as Rake as XML is predisposed to becoming quickly unwieldy, is difficult to refactor and expensive to extend (basically XML sucks as a programming language). Yet, XML or not, most build frameworks share a common flaw: they are task based. The build becomes defined in terms of a set of steps to be executed and in which order and fairly soon you find you want subtle variations in those steps which result in more tasks.
But builds aren't linear; we don't always want to run the same dependent steps to achieve the same end result which is why we end up with masses of subtly different targets. One approach to reduce this is to remove dependancies from targets and have high level targets which simply list a set of steps. This can lead to flexible systems, as targets can easily be stitched up arbitrarily from the command line as needed, but it requires an intimate knowledge of how the build needs to be executed (which breaks encapsulation) and moves even further down the road of defining a list of steps.
So how should a build work? In this post I hope to outline what I believe to the be core metaphors and functions of an effective build system.
Product based
If we go back to basics and look at the purpose of a build we find it is to produce specific products: a deployable artifact, verification (by running tests), a deployed application, installers etc. Instead standard build tools promote thinking in terms of linear dependent tasks, for example creating an installer would be defined as "clean, compile, test, create-installer", the build is described in terms of steps, most of which say nothing of the products created.
A product based build is different: the build is defined in terms of products which are built by producers (or factories to use the GoF pattern). Products represent the artifacts of the build (binaries, installers, deployed application) and producers contain the logic on how to assemble the products.
Dependancy based
Similar to task based builds, product based builds are dependancy based. Where product based builds differ is they view the build process as a factory line where each product is produced from other dependent products. For example the "installer" product has a producer which packages binaries from the "verified-binaries" product which has a producer which runs tests from a "binaries" product which has a producer that compiles the code from the "fresh-destination" product which has a producer which creates new, clean directories.
Describing dependancies in terms of products rather than tasks creates a more descriptive build where each product can express its dependancies in a specific way, for example an "acceptance-verified-binaries" product produced from "unit-verified-binaries" expresses a different set of rules from being produced from "binaries".
Encapsulated
When we run the build we shouldn't have to understand the steps required to reach the end result but because of the nature of task based builds they often require an intimate knowledge of their internal workings: for example "run" may depend on "clean, compile, test" but "run-again" may have no dependancies but instead depends on a previous build. By focusing on products this thinking is broken. As an example consider the following: in a task based build you can retrieve a set of targets, with their descriptions, from the command line. To begin to use the build you must understand how the targets relate and what steps are being executed for each target, and from the descriptions (or the task names) work out what the end result may be. A product based build would give you a list of products but not the steps required to produce them. This allows the user to quickly understand the goal of the build and how to achieve the desired product without requiring any knowledge of how it is produced.
Non-repetitive
Task based build systems are prone to repeating already satisfied dependancies, for example if you ran "build compile, create-installer" and the "create-installer" target had a dependancy on "compile" most build systems would re-execute the "compile" target wasting your time. To work around this you must create more tasks and again require an intimate knowledge of the build. Instead when running "produce binaries installer" the product build would be satisfied that it had already produced binaries when it requires them for the input of the installer.
Order independent
As task based builds execute as a series of steps they are order dependent. Running "build compile test" has different results from "build test compile" (which would either fail or give invalid results depending on whether a previous build had compiled). As a product based build focuses on products and not steps it is order independent so specifying the following would give equal results:
> produce binaries verification
> produce verification binaries
Polymorphic
By focusing on products rather than tasks a product based build system can become polymorphic. This allows for a very flexible build removing the need for the subtle variations on targets which plague task based builds. For example you could define a product of "running-application" which is produced from the "binaries" product. The "binaries" product could be inherited by two other products "verified-binaries" and "unverified-binaries". By giving the "running-application" product an input of the base "binaries" product it can be produced from both "verified-binaries" and "unverified-binaries". This allows the user to easily control the build:
or
> produce unverified-binaries running-application
or (as we are order independent):
> produce running-application verified-binaries
or (which would use the default binaries product)
> produce running-application
We can also extend "verified-binaries" to "unit-verified-binaries" or "acceptance-verified-binaries" etc. creating support for a build pipeline, or create a "reused-binaries" product to avoid recompiling. By allowing a build to become polymorphic it becomes both highly flexible and intent is clearer.
Stateful
Though not absolutely critical state is something most build systems lack. Every time a build is run the results of the previous build are completely discarded and any valid products are re-created. State is a difficult concept to implement in task based builds as tasks would have the responsibility of deciding whether the artifacts are still valid, this is further complicated by the fact that multiple tasks may depend on the same artifacts and would therefore have to make the same decision again. In a product based build each product can decide whether it is valid and the build system would simply reproduce it if it isn't. Furthermore the build system can analyze the product line and understand that if a product further down the line (say binaries) is invalid then all products produced from it need to be reproduced.
To conclude by changing the build metaphor from tasks to products we can solve many of the issues around build systems and create clearer, more flexible and more meaningful builds for our applications.
Sunday, 27 July 2008
Decisions, Decisions, Decisions
During any development project thousands of decisions will be made. Every individual in a team will make large numbers of decisions whether about details such as variable names or cross cutting concerns such as architecture. The success or failure of a project is the sum of all these decisions.
There are two key factors in making decisions: information and risk. The more information we have the better informed the decision is and every decision made - and not made - carries a risk. Empirical methodologies (such as Agile and Lean) have contrasting methods to decision making than up-front plan based methodologies such as waterfall. Fixed-plan methodologies attempt to manage these factors by collecting as much information as possible and then committing to decisions before implementation starts where as empirical methodologies promote delaying decisions until there is enough information available.
Both approaches recognize the importance of information and of managing risk yet they have contradictory views on the best way of dealing with them. Where fixed-plan methodologies believe that by committing heavily to the informed decision prior to execution you reduce risk empirical methodologies accept that, in reality, every decision that you make ahead of time carries the greatest risk of all: it could be wrong. The chances of wrong decisions are increased because the information that fixed-plan methods derive their decisions from is essentially highly detailed speculation which in turn is based on a limited amount of real information which carries the risk of being incorrect or out-of-date or even irrelevant. The strength of commitment increases the impact of a wrong decision even more. Agile and Lean acknowledge this fact and tackle it head on by managing risk with two techniques: they delay decisions as much as possible until the maximum amount of real information is available - what Lean calls the last responsible moment (as Lean acknowledges there is also risk in delaying a decision too much) - and they attempt to de-risk decisions by lowering commitment - essentially by allowing change.
Agile and Lean acknowledge that upfront decision making carries the need to cover a great number of hypothetical eventualities than may or may not actually occur. The crystal ball gazing of fixed-plan methods is an attempt to de-risk the unavoidable gaps in information by trying to account for any possible variation with more pre-made decisions. Essentially imagine trying to plan out an entire chess game before you play it: every possible move your opponent does or doesn't make would have to be accounted for in the plan. The end result? An over complex and convoluted plan (which translates to over engineered, complex software). Again because Agile doesn't rely on pre-packed decisions, instead acting on the real information, it removes the need to make the "what if" decisions - this is the foundation of the maxim "You Ain't Gonna Need It".
Another aspect that traditional methods fail to tackle is decision paralysis: having one decision to make is hard enough but having to make all decisions infallibly and at once requires omnipotence. Although some developers believe they are gods Agile recognizes their limitations by giving techniques to allow focus on as few decisions at a time: TDD is a great example of this and so is the maxim "The simplest thing that could possibly work".
Agile and Lean promote both the overall reduction in decisions and minimizing the number of decisions to be made at one time (essentially by spreading them out across the project). Pushed to an extreme, decision making can be reduced to being purely reactive. The purely impractical "URL Driven Development" demonstrates this by only making decisions when the event which creates the need for a decision occurs. Essentially when someone hits a URL you write just enough HTML to satisfy their request. Despite being impractical it demonstrates the principles of Just In Time decisions well and should not be used to negate their value within the development cycle where decisions can safely be reduced to being purely reactive. Test Driven Development uses JIT decisions extensively: you write a test, an event occurs (unsuccessfully) which forces a decision to write just enough code to satisfy that one event, rerunning the test recreates the event and validates your decision and then you move on to add another event forcing another decision.
Aggressively minimizing the number of decisions which need to be made by simply delaying them or eliminating them altogether has an overall positive effect on a project. Ensuring that there are few small decisions which require minimum commitment and have the maximum amount of information increases the chances of making good decisions and reduces risk of committing to bad ones. In short: prefer fewer good decisions than many potentially bad ones.
Thursday, 24 July 2008
Legacy Index
A legacy index measures the number of releases a software product has until it becomes legacy. It can be used to measure and communicate design quality: the more releases on a code base before change becomes difficult and it gets demoted to a legacy system then the better the design.
The target should be a high legacy index or to reach Legacy Max. An application that becomes legacy after its first release would be a Legacy Zero app. The lowest index is awarded to applications which become legacy before they are even released: Legacy minus One!
Applications can still have further releases once they've hit their legacy index: these releases would be identified by consisting mainly of bug fixes and only minor changes, this would be called "pushing the legacy index" as each release would not increase the index. Also an application may be able to reverse its legacy status by refactoring it to a point where change is again possible.
By measuring and plotting an applications legacy index against time you can clearly see the health and state of an application. If it has a steady and consistent incline then you know that the design is sound and the application is healthy but if the index has a trend of slowing or is continuously pushing the legacy index then it is clear that there is a problem.
Though quite humorous legacy indexes may have a genuine use in describing the state of an application, you can simply say "this is a Legacy Zero app" or "it's hit its legacy index" and all is clear.
Wednesday, 11 June 2008
10 things that should get devs fired
So in typical hard-nose, no messing Sir Allen style, here's the list in no particular order (and if you have any good ones please comment):
You've limited all design and architectural decisions down to one vendor regardless of suitability, cost effectiveness or productivity pushing up the cost of projects and potentially being left with an unsuitable solution.
Risking the success of a project by restricting technology choice to one vendor: you're fired!
2. Prohibiting Open Source Software
Despite the fact that an Open Source product may be the best solution available a prohibition on OS software has prevented its use on projects.
Dismissing viable solutions with potential benefits to your business: you're fired!
3. Single language
You've limited the performance of your development by prescribing that all development be done in one single language. The fact that the language may not be suitable for the job, will perform poorly or be more expensive is ignored over complying with an arbitrary 'strategic' decision to unify all development.
Running up costs because you can't choose the best tool for the job: you're fired!
4. "I'm a Java developer I don't do Ruby" (replace with any languages)
Your language defines your role and you only work in that one core language. You have no interest in other languages and believe your language is the one true language.
I have no place in my development team for one-trick pony developers: you're fired!
5. Documented not automated
You'd rather produce a 15 page document with screenshots on how to deploy your application than spend less time automating it. You place value on creating loads of documentation rather than producing things that actually work.
Wasting money on something that will be immediately out-of-date and no-one will read: you're fired!
6. No source control
Projects or critical dependancies have never been added to source control or even worse there's no source control at all.
This is wholly unacceptable: you're fired!
7. Artifacts built off of developer boxes
Deployment means opening up Visual Studio, pressing F5, zipping up the dlls and handing them over to IS to install (with a 15 page document).
You are a cowboy, this is simply unprofessional and amateurish: you're fired!
8. No automated tests
You never write automated tests and simply rely on the old "run and click about" or the "run the test console and check the results" methods of testing.
No way of verifying your changes, there's no room for hackers: you're fired!
9. No CI
There is no visibility of the state of the code base and as long as it runs on your machine then that's OK by you.
No way of understanding the status of the code base: you're fired!
10. You're an architect
What more can I say? You probably disagree with this whole list especially because it doesn't come with a Visio diagram and can't be orchestrated in BizTalk. You're fired!
Friday, 23 May 2008
It's ladies night
Friday, 25 April 2008
The Snake and The Emerald in The Mesh
Fairly soon the dust settled over the pamphlets and everyone forgot about the great multi-language dream, .NET was as polorized as British politics: there was Labour and Tory, C# and VB, no other language got a look in and those that existed were merley academic exercises of no importance to your MVP. There were many critics of the importance of multi-language support in .NET and they were feeling proven right.
At the same time another important trend was occuring: the rise of static languages: Java, C#, VB.NET are all strongly typed. Now even VB was strongly typed MS developers felt confident to dismiss dynamic typed languages as toys for script kiddies, web designers and network engineers. Grown-up enterprise development required things like type safety and checking.
So now the world is strongly typed and .NET settles around VB and C# and Java is just Java, the battle lines are clear: who is the king of the strongly typed. Then all of a sudden, out of nowhere, the alpha geeks decide that dynamic languages are the thing: Ruby, Python and Groovy, people are re-realising that dynamic languages can do very, very cool things which strongly typed languages struggle to express.
A new level starts in the arms race between Sun and Microsoft as people start trying to implement dynamic languages on top of the JVM and CLR. Java gets Jython and JRuby and a glut of projects attempt dynamic languages on the CLR including Microsoft's own IronPython but then all of a sudden people realize something: the CLR wasn't designed to do dynamic languages: sure you can build languages on top of the CLR but gone is the dream of having C# assemblies seemlessly call Python assemblies without even knowing it.
Fortunately Microsoft had the guts (or rather Jim Hugunin did) to sort this problem out and get .NET back to where it should be and the Dynamic Language Runtime was born. The DLR sits ontop of the CLR and allows all the dynamic goodness of languages like Ruby and Python (and now VB 10) to work as equals alongside C# and VB.NET code. The DLR will allow a new phase of the polyglot dream to emerge: C# calling Python calling Ruby calling VB and every combination in between.
Microsoft are going to start pushing IronPython and dynamic languages hard. It is one of the main features of Silverlight 2.0 and being expanded to ASP.NET. Interestingly MS is positioning dynamic languages firmly in the web and UI arena (which makes sence as this is where they are traditionally strong). In reality dynamic languages aren't restricted to the UI layer and entire applications can be written in them, though I think Microsoft aren't pushing this point because they don't want to scare or alienate their core developer community. Any how the people who would want to do will try and do it.
There are a few interesting issues arising out of the dynamic languages race:
When is Ruby/Python not Ruby/Python?
As languages run on the the CLR can talk to the .NET framework and other .NET libraries they break compatability between other interpreters: though it's still legal Ruby/Python code it isn't going to run on any other interpreter. DLR languages can also be compiled to .NET assemblies breaking all compatability with anyone else. Some see a risk here of MS "restandardising" pointing to HTML as an example.
Why run IronRuby/Python when you could just Ruby/Python?
Technically there are a few answers about integration with .NET etc. but they are pretty weak when it comes to writing apps in one language. Leveraging the power of dynamic languages from within a .NET app (i.e. polyglots) is the main selling point, otherwise, for those who are simply going to write pure Ruby/Python code, then the big thing is ratification: for those .NET shops out there who still believe the MS FUD of previous years simply putting MS in front of the language makes it acceptable.
What about open source?
Both Python and Ruby are open source languages so isn't this a bit incompatable with MS? Interestingly both IronPython and IronRuby are open source (under the Microsoft Public License) and because the DLR came from the work on IronPython it is also open source. IronPython is hosted on CodePlex but IronRuby - in an attempt to engage the Ruby community - is hosted on RubyForge.
About Me
- Peter Gillard-Moss
- West Malling, Kent, United Kingdom
- I am a ThoughtWorker and general Memeologist living in the UK. I have worked in IT since 2000 on many projects from public facing websites in media and e-commerce to rich-client banking applications and corporate intranets. I am passionate and committed to making IT a better world.