Automation is not helping ship reliable Software quicker

Microsoft announced that their long overdue Vista will be delayed even further.

This annoyed a number of Microsoft employees, some who don’t work with Windows, some who don’t even work with Microsoft anymore (ex). But seriously, the effects are far-reaching. I am sure there was considerable thought put inot the decision.
Anyhow, one voice piped in that automation of tasks, including Quality Assurance wasn’t helping: (from Mini-MSFT)

In the last 18 months this org:

1) Cut the number of testers (several times) from approx 50 to now much less than a dozen. Of
course, many top performers also left MS entirely because of middle mgmt in this org.

2) Hired more PMs
3) Cut the scope of testing (anyone done any real code coverage testing lately?)

4) Cut the number of promotions in the test orgs – nothing like a little ‘de-incentivization’ to increase ‘bad attrition’

5) Dictate that everything can and should be automated.
(Ignore that eyeballs catch more in less time…) way to go Darren. Of
course, you
were probably lied to by your underlings, so it’s not entirely your
fault. Uhh, yes it is – you made the call.

6) Hire only a small handful of devs to write automation
code. Oh, and don’t forget to swamp them with added process and have
embittered leads review their code…

7) Hire more PMs

8) Outsource all testing to non-accountable and barely trained CSG firms
overseas (Ever try to translate/clarify a bug written not by a tester, but by their lead based on notes? )

Limit the number of heads
the abovementioned overseas firms can use. > Fewer testers, less
experienced, with little training, a much (ahem) ‘slower’ approach to
testing. Results: Client appcompat % hovering at <40% (GASP –
INTERNAL INFO… better moderate this one out!!!!)

an anomaly for PM’s to ‘splain away. If automation is such a great
tool, why is it not finding more bugs than a small handful of testers
in a lab on the other side of the planet?

Very Interesting observation. The largest software projects are
traditionally proving grounds for automation techniques. I wish the
authors of Windows Vista come out with a sequel to The Mythical Man-Month.

Updated: San Jose’s Mercury News also cites the same book and the Vista delay but in an altogether different context.

The challenge of big software projects was probably best
described by Frederick P. Brooks Jr. in his classic 1975 book, “The
Mythical Man-Month.” Brooks, a professor of computer science at the
University of North Carolina at Chapel Hill, stated then what he called
Brooks’ Law: “Adding manpower to a late software project makes it
later.” The need for latecomers to get up to speed and communicate
with their
predecessors, Brooks says, takes up more of the team’s time than what the added workers contribute.It’s even theoretically possible to reach a kind of software
gridlock, where the team is so big that all its time goes to
communicating among
each other and making revisions — with the project never reaching


2 Comments on “Automation is not helping ship reliable Software quicker”

  1. rishabht says:

    Hey Santosh, there are a few things to ponder over here.

    1. Anyone who thinks that automation is supposed to find bugs hasn’t done their homework. – Automation is supposed to find regressions and not bugs. Bugs and regressions are two different things.

    2. You cannot expect automation to provide short-term benefits, specialy if automation is written from scratch while the product is being is being developed or in this case undergoing major overhaul with more than 3000 new features and a significantly different architecture. The problem is that automation cannot be written till the features don’t come online – How could you possibly automate something that hasn’t been written yet? And by the time automation finaly comes online, it is already too late in the product cycle to put it to use. Moreover since the bug bar is already high, most of the regressions the automation finds don’t seem to get noticed or its impact is not felt as much, even though it is doing what it was supposed to do i.e. finding regressions before actual check-ins at a time when cost of fixing regressions is pretty high.

    3. Automation has a huge payoff in the long term – The next version of Windows is going to benefit a lot from the automation written now. It will help dev and test identify regressions from M1, when not many testers are focused on integration testing. The earlier you identify regressions and fix them, the more cost-effective your development process becomes, so automation is like a long term investments whose benefits you reap over time

    On a side note, The Mythical Man-Month was the first thing that came to my mind when I blogged Vista’s slip on my blog :-). I have a copy of the book in my bookshelf at work, I have yet to see a copy in anybody else’s bookshelf yet… We should make it a mandatory reading for everyone across company just like writing secure code.

  2. Santosh says:

    Thanks for stopping by Rishabh,

    Definitely, you can only hope to uncover regression bugs through automation.

    I also spoke to a couple of Windows devs around town and they also felt that it was important for people to know the insides of the product that is being tested. They felt, more white-box guys are sorely needed.

    As you commented in your blog, the 8 weeks slip will only help to ship a reliable version of Vista.

    I am interested in taking home lessons to help me ship software in a timely manner. For that, I think we both agree that sticking to the Mythical Man Month is just fine.