Time for a change
February 6, 2010
I’ve been in the IT industry for about 20 years now, and for nearly the last 8 years I’ve been a consultant with the rather excellent ThoughtWorks. Being anywhere for that long in our industry is quite uncommon, and to spend that long in a consulting role, with all the travel and disruption that implies, is even moreso. I’ve had a fantastic time with ThoughtWorks and I feel I’ve grown tremendously during the time I was there. Eventually something had to give though, so at the end of last year I decided it was time to try something else.
In particular I found I had moved away from the things I really enjoyed – writing software that matters and building high-performing software teams – more towards big organisational change, which, while it arguably has a bigger impact on an organisation, isn’t really where I wanted to be. So my criteria for what to do next came down to: writing business-critical software in a small, high-performing team, in an organisation that trusts its people and encourages them to excel. Having a great relationship with the consumers of that software and having them closely engaged with its delivery would be a huge plus.
Luckily for me such organisations do exist. One of them is a proprietary trading firm (basically a bunch of partners trading their own money) called DRW Trading, and it works a bit like this: The partners want to trade, so they have traders. In order to trade well the traders need to work with good financial models, so they have analysts who build those models. In order to trade effectively with these models they need good software, so they have programmers who build the software. That’s about it. The software development value stream is something like:
- I’ve had an idea for something that will make money. - Ok, here’s some software. - Thanks.
The procurement process seems equally onerous:
- I need a Thing. - You should get it then.
So there you have it. I’m taking a break from consulting, at least for a while, in order to rediscover good old-fashioned software delivery. I’m going to be doing a lot less on the conference circuit, although I don’t intend to vanish altogether, and hopefully I’ll be blogging and writing more now I’m doing less travelling and feeling less exhausted all the time.
Obviously I no longer have a ThoughtWorks email address, but you can contact me at dan@dannorth.net or I’m occasionally on Twitter as @tastapod. We now return you to your regular scheduled programme.
The lady in the taxi – a parable of metrics
November 12, 2009
Once upon a time there was a lady in a taxi. It took such a long time for the lady to get to her destination in the taxi that she went to the town hall and told the man from the council. The man from the council wanted to figure out why the taxi journey was so slow, so he placed cameras at all the traffic lights in the town to measure how many cars went past, and how quickly. The traffic light cameras would click every time a car went past the lights.
He wanted to speed up the rate of cars, so he changed the layout of the town. He figured if he introduced a one-way system the traffic flow would be more efficient. This confused the taxi drivers and they started to get lost. The taxi driver would go past a light, click, discover he couldn’t go the way he wanted, try to find a way through and find himself going back past the same light, click, realise he had been this way before, turned around and drive back through the light, click, and would still not find his way to the lady’s destination. The new one-way system was good at moving cars around – it just wasn’t very easy to navigate. And just as the taxi drivers were learning the new layout, the man from the council would try a new layout just in case.
What a lot of clicking, thought the man from the council, and what a lot of cars must be driving through my town. How efficient this is! I shall invite more cars into this town because it is so efficient at moving cars around.
So he invited more cars into the town, which of course just clogged up the streets. Every time the light would go green, a line of cars, nose-to-tail, would crawl through the lights: click, click, click! When they were past the lights, they would sit stuck in traffic. Sometimes the taxi drivers would get so fed up they would just abandon the journey and make the lady get out of the taxi. Of course she would have to pay for the journey so far. And wait for another taxi. And get in and try to resume the journey. (She often had to go a way back up the road to find another taxi.) Same lady, same journey, different taxi, back through the same lights. Click, click.
The taxi drivers realised they were losing money by spending all day in traffic jams, so they decided to have two kinds of tariff. When the taxi was moving they would charge by the mile. When the taxi was stopped, they would charge for waiting. What a clever idea!
This made the lady very upset. It is taking me longer than ever to get to my destination, she thought, and it is getting more and more expensive because it is costing me money just to sit here.
She sighed and looked out of the taxi window, and saw the cameras at the traffic lights. Then she realised what was happening. The poor man from the council thought that each time the same taxi went past the light, it was a different vehicle! He thought that when different taxis were taking the same lady to the same destination, that it was different jouneys! He probably figured that having lots of cars going through the lights meant they were travelling quickly!
Then she had an idea. I shall take a camera in the taxi she thought, and I shall show the film to the man from the council. So she took a camera in the taxi (and cleverly recorded the taxi meter at the same time). Look at this, she said to the man from the council. Ths shows you my experience as a passenger in the taxi. I move from red traffic light to red traffic light, crawling through the lights in a little batch of cars, queued up behind the next traffic light. I don’t mind paying for the mileage, but I don’t think I should have to pay just to sit waiting to go forwards. And to make matters worse, the journey to my destination is taking longer and longer!
Oh my! said the man from the council. I’ve been looking at the wrong thing all along. Instead of trying to maximise the amount of cars that go through a particular light, I should try to minimise the amount of time it takes you to get to your destination! How silly of me.
Oh, and perhaps I should pay you if you have to sit there in a taxi because my town is all backed up with traffic. At least then there would be an incentive for me to work on the most blocked-up parts of the town. Perhaps they are the only places I should be concerned with anyway, because by unblocking the most constrained parts I will probably have a better flow of traffic altogether. And perhaps when a particular street is backed up, I should stop more traffic coming in and causing traffic jams.
Thank you! said the lady. I now feel like you really are going to be able to help me to get to my destination quicker. I know it won’t happen overnight, but I am sure that over time my journeys will be faster. That’s quite alright, said the man from the council. Thanks for teaching me to look from the point of view of a passenger in the taxi, and not just to take snapshots from the different stages of the journey.
And they all lived happily ever after.
Thanks to Oliver Schreck for the idea that led to this story.
New translations
October 12, 2009
I currently have a backlog of about 15 blog articles I am failing to finish. The most embarrassingly laggy one dates from around the end of 2007. Now I know I’m a slacker.
However, others have been far more industrious than me.
What’s in a Story in Japanese and Introducing BDD in Korean
The industrious Yukei Wachi has followed up his translation of my Introducing BDD article by producing a Japanese version of What’s in a Story.
Inspired by Yukei’s translation, a Korean developer HongJoo Lee has written a Korean translation of Introducing BDD. How cool is that?
Enormous thanks to you both for your hard work.
Update: HongJoo has also translated What’s in a Story? into Korean.
Introducing BDD in Japanese
August 24, 2009
I am delighted to announce the official Japanese translation of Introducing BDD.
It’s easy to forget how big the Internet is and how small the world can be. Last year I gave a BDD talk at QCon in San Francisco that made its way onto InfoQ.
A Japanese programmer called Yukei Wachi watched it and decided he wanted to know more about BDD. He read my Introducing BDD article and felt it was worth translating into Japanese. He then translated several thousand words in what seemed like no time at all.
Yukei asked me for a couple of sentences to introduce the Japanese translation, and then with typical Japanese modesty decided to drop the part where I thank him. So here is the introduction again with the first sentence intact.
I am thankful to Yukei Wachi for translating this article from English for you, and I am grateful to you for your interest in Behaviour-Driven Development. I hope you find this article useful, and that you enjoy your journey into BDD as much as I have.
Thanks Yukei.
Welcome to my brain
August 13, 2009
I’m delighted to be taking part in a In the brain of… session organised by the folks at SkillsMatter.
When the SkillsMatter folks actually checked inside my brain and heard the rattling noise, they realised I would need some help, so Liz Keogh – BDD pioneer and project lead for JBehave – will be co-presenting with me. (If you hold Liz up to your ear you can hear The Cure.)
SkillsMatter is a fantastic training organisation that believes in creating and nurturing communities. This event kicks off a London BDD community, which is very exciting for me because it means there is a genuine momentum behind BDD as a movement. This is the first of a series of (mostly free) talks around BDD and automated testing taking place throughout the rest of the year.
The event is free to attend and it takes place at SkillsMatter’s London office on Monday 17 August from 6:30 pm. I expect we’ll go for beer afterwards.
Please register online so we know how many chairs to put out, and I look forward to seeing you there.
The perils of estimation
July 1, 2009
Business people want estimates. They want to know how much it’s going to cost them to get a solution, and they want to know how likely it is to come in on time and on budget. And of course quality is not negotiable.
Agile teams I encounter are at best nervous about estimates and at worst simply evasive. “You don’t need estimates if you’re doing Agile,” they say. “It will be ready when it’s done. We’re constantly adding value so we don’t need to commit to a date.”
We’re missing the point of release planning
My favourite exchange goes something like:
“We’ve done an inception and broken down the entire project into stories and measured it, and it’s come in at 400 stories, estimated at 865 story points.”
“865 what?”
“865 story points.”
“So how big is a story point?”
“We don’t know yet, we’ll let you know in a few weeks.”
At a governance and funding level the business could care less about story points. They don’t actually care about stories except that we shove them in their faces with our release plans. They care about solving a problem. They came to us and asked us a) how much will it cost to solve the problem, and b) how confident are we about that number?
So how do we approach that? We go through some sort of inception process that looks something like this:
1. Identify some personas
2. Identify some process flows
3. Start breaking the flows down into stories
4. Lots and lots of stories
5. Lots and lots and lots and lots of stories
6. Spike some technical ideas that came out of the stories
7. Estimate the stories
8. Roll up all the estimates and call that our project estimate
The part where we estimate the stories is a real chore (c’mon – we’re estimating 400 stories here), so we cut corners. We do a first pass as t-shirt sizes (small, medium, large) and then take a representative sample (sounds suitably scientific) and do a “detailed” estimate of those. This involves a bunch of people estimating lots of important-sounding metrics: minimum, likely and maximum size, clarity, volatility (eh?) and whatever else, and then multiply it up to provide a WOOOOAAAAAHHHH! hang on a minute! What were we trying to do again?
All they wanted to know is: How long will it take, and how confident are you about that number?
Redefining success – in a bad way
By introducing a comprehensive story list – with or without the notion of story points – we have unwittingly reframed the project. The business started out by defining success as solving the problem, but now we have redefined success as delivering this list of stories. However we frame it, that’s what the business will believe. The project will start and and the business stakeholders will start counting down the list of stories until you get to zero.
So now we have the worst of both the Agile and plan-driven worlds: the business expects delivery of a fine-grained list of requirements (whether we call it a Product Backlog or a Master Story List), and we have only taken a half-hearted attempt at it compared to the big up-front analysis we used to do. From here on we are on the back foot, constantly negotiating with the business to manage scope, when it’s our own fault they even care about the story-level detail. They see the story backlog and mentally turn it 90 degrees and think of it as a Gantt chart. Happy days!
Back to project basics
There are two observations to make here. Firstly the business wants accuracy and we’re giving them precision. If you tell me it will take 4.632 months and it takes 8 months, that’s worse than useless. If you tell me it takes “about six months” and it takes seven months, I should still be onto a winner. (If the return on investment is small enough that the extra month stops the proposition being viable, I’d have been better off investing in something else in the first place. Spending $60,000 to realise a return of $70,000 is risky to say the least.) I’m simplifying here of course because the real RoI varies over time, and its value may be particularly time-sensitive.
Secondly we know that requirements vary on an agile project over time, for good reason. Hopefully we are learning as we go, which means we will discover new requirements and decide others are no longer worth pursuing. If we assume about a third of the requirements will be delivered as described (this is generous in light of the Standish Chaos reports), another third will be delivered but with changes, and the last third won’t be delivered at all – but replaced by other features – then we have just wasted all that time and effort in the inception coming up with detailed, high-precision data for a pile of stories we will never deliver.
To compound this, it turns out that estimation is fractal. The more fine-grained you break down the requirements, the more “edges” you will discover. This means that the more detailed you estimate, the more the total will tend towards infinity, simply due to rounding errors and the fear factors that we multiply into fine grained estimates.
Use the inception for deliberate discovery
So what should we be doing during an inception instead of doing the fine-grained story breakdown? Taking it back to first principles we simply want a rough idea of size and an understanding of certainty. There is uncertainty in everything, so the purpose of the inception is to understand the potential landscape we are delivering in.
When we start the inception we know nothing. We want to come out of the inception knowing as much as we could reasonably expect to learn in the time we have allocated. This discovery is along several axes: technical areas such as the technology stack, potential architectures, integration points and external services; domain questions such as how well we understand the problem and whether it is more about research than solving a clearly-articulated problem; people and process challenges like the path to production, identification of stakeholders, how co-located or distributed the team is and how much of this kind of delivery they have done before. For some of these areas, breaking broad requirements into finer-grained detail is a great way to discover more. But not for all of them, and certainly not at the expense of other discovery activities.
There are good arguments from both the Kanban and Real Options folks about deferring the decomposition of feature sets into features and stories until the “last responsible moment”. This means the information is freshest and you aren’t holding an inventory of atrophying information. You might want a couple of weeks of story-level detail – to promote a consistent flow and avoid starving your process – and beyond that a few features identified that will be broken down into the next candidate stories, but beyond that you shouldn’t be worrying about that level of granularity. The experienced members of the team should be estimating feature sets of the order of person-weeks (or better yet, pair-weeks), not going down to the level of individual pair-days. The less experienced team members should be using the exercise as a learning opportunity.
So please, let’s move beyond this cargo cult approach to inception where we slavishly trot out hundreds of stories with their associated estimates, and remember that we are engaging in a process of deliberate discovery.
Its purpose is firstly to convey to our stakeholders and ourselves an order-of-magnitude sense of size – to quote the Pragmatic Programmers, is it larger than a breadbox and smaller than a house? – and secondly to present the risk landscape in which to understand that estimate.
One day DDD track at SkillsMatter
June 15, 2009
There’s a one day domain-driven design event happening at SkillsMatter this Friday, 19 June in London. I’m not speaking this time so I get to sit back and enjoy some talented folks talking about really applying DDD rather than just theoretical stuff.
Eric Evans will be kicking things off at 10am then there are five talks throughout the day. Two sessions I’m particularly looking forward to are Gojko Adzic – who spoke on my “Architecture for Architects” track at QCon earlier this year – talking about DDD in a distributed environment, and mild-mannered superhero Phil Wills from The Guardian talking about how domain-driven design helped them rebuild the http://guardian.co.uk site.
In the interests of full disclosure I’ve been given a complimentary ticket, but in any case the event is well worth the cover price and I’m happy to help publicise it. As I write this, places are still available and you can book online.
Next week I’m doing a new talk at the Better Software Conference in Las Vegas about learning models, where I was planning to talk about various learning styles and about how ineffective and systemically flawed most school systems are. Then I read up on Edward de Bono’s Six Thinking Hats model (I’ve linked to Liz Keogh’s write up because it was her who introduced me to it), which I’ve subsequently used to facilitate a workshop, and was amazed to say the least. So much so that it caused me to turn the Learning to Learn talk on its head. I still believe our schools are ineffective and systemically flawed but now I know why! de Bono goes beyond suggesting we should learn how to learn, and suggests ways of learning how to think. These strike at the heart of Western thinking models, which by and large haven’t moved forward since the days of the three amigos of Aristotle, Socrates and Plato, although arguably the scientific method adds something to the mix.
Briefly, five of the six hats represents a different “style” of thinking – yellow is positive, black is critical, green is creative, white is factual and red is emotional – and the idea is to get everyone thinking in the same style at the same time. The sixth blue hat is a kind of meta-hat or “process” hat, which the facilitator wears. You can either run a workshop with a fixed timeline of which hats you will wear when, or you can choose to switch hats reactively during the session when you think a different mode will be more useful or enlightening. The important thing is that everyone is in the same mode at the same time.
Edward de Bono calls this parallel thinking, as opposed to the usual mixed-up scrappy thinking we do most of the time, and has discovered it is far more powerful than the usual dialectic approach of taking a stance and trying to beat the other guy into submission. It also gives you a construct for talking about thinking styles. If someone is being critical or obstructive – usually for what they believe are good reasons – you can describe that as “excellent black hat thinking”, and ask them to note it for the next black hat segment (“but right now we have our yellow hats on, so assuming nothing could possibly go wrong, what would the best possible outcome be for this situation?”). It allows people to feel acknowledged without having the conversation derailed by either defensiveness or emotion. Of course you have to follow through and allow sufficient time with the appropriate hat later on. The same technique allows you to acknowledge and deal with emotive issues in a safe way, rather than pretending to suppress the emotions or risk having them take over.
Liz introduced the six thinking hats in a workshop with a client a few months ago, and I decided I wanted to find out more about it. I discovered that the original Six Thinking Hats book is now available as a Penguin classic, which means you should be able to find it for peanuts in your local bookshop. It’s well worth a read (not least because it finally gave me a decent definition of lateral thinking), and some of the anecdotal evidence is very persuasive – usually involving groups of people reaching consensus far quicker than they expected to.
One of my favourite observations is that often a good answer almost presents itself, and in a non-threatening way. Shifting the emphasis from pitting my idea against your idea to collaboratively trying to find the best idea makes it safer and easier for a group to arrive at a sensible conclusion. It even worked when I split the group of around 20 people into smaller groups (four groups of five, tackling different aspects of the topic). We would all still change hats in sync and it kept the flow going.
Oh, and I found that coloured paper party horns are a great substitute for real hats: they can be heard above a room full of people, and you can wave the hat around to emphasise the new colour.
RSpec book in beta
April 24, 2009
It’s finally happening – I’m writing a book! Well ok, the remarkable David Chelimsky is writing a book. It’s called Behaviour Driven Development with RSpec, Cucumber and Friends and myself and a few other folks are contributing in varying degrees. The book is already in beta, which means you can buy the PDF now from the Pragmatic Press and you’ll get the print version as soon as it’s ready – and still get change out of $50, which can’t be bad.
The folks at the Pragmatic Press have been brilliant – especially the ever-patient Jackie Carter – and my second chapter will hopefully be in the next beta (or if not, definitely the one after that). It’s also surprisingly satisfying to be writing in vim rather than Microsoft Word!
Well, that’s enough for now. With any luck I’ll be back blogging a bit more frequently. Next up, learning about learning, thinking about thinking and why we are so crap at estimating.
JAOO Australia
April 24, 2009
A friend of mine has a Far Side desk calendar that he describes it as a barometer for how busy he is. Some days he finds himself tearing off a whole bunch of pages because he’s been too busy or distracted to tear one off each day.
I noticed a couple of weeks ago that I haven’t blogged anything for over six months. It’s been hectic. I’m planning to be blogging more over the coming weeks and months. I’ve got a ton of things to write up, ranging from planning and estimation – it’s no wonder we are useless at it – we don’t even know where to start!, to architecture – why you wouldn’t want to go on a date with Maven, by way of build and release engineering – why Ivy means Ant is still the king of cool.
JAOO Australia
I’m in Sydney at the moment where I’ll be speaking at JAOO Australia which is in a couple of weeks time. It’s like a pocket-sized, portable version of the European JAOO conference. In fact it is two conferences, one in Sydney (May 5-8) and one in Brisbane (May 11-14). There are two days of tutorials and two days of conference proper, and it looks like it’s going to be great fun. Both conferences are small enough (a few hundred people) that there will be ample opportunity to hang out with speakers and ask the questions you really want answered. And buy them beer.
I’ll be Pimping my Architecture and doing tutorials on BDD (principles and methodology) and TDD in Java (technical workshop), the latter with the lovely Erik Doernenburg.
If you can get to either city on those dates I would highly recommend it. As of writing neither event has sold out so there is still time. JAOO is one of the best geek conferences I know, with a great mix of technology (languages and frameworks), methodology (with an Agile flavour), architecture (from cloud to iphone) and soft stuff (Your brain is deceiving you. Yes you. Even though you know it is!)
