Sprint Planning Pt. 1: Stop with the Points Already

"Our culture thrives on black-and-white narratives, clearly defined emotions, easy endings, and so, this thrust into complexity exhausts." --Caroline Knapp

I've been doing a lot of reading and thinking about story points lately, and over time I have been edging toward what I now see as a necessary conclusion. Story points are meaningless. Even... Wait for it... POINTless. Yes, I just did that.

To recap, when planning a development sprint, it's customary to assign a user story a point value as a measure of... Well, to be honest, I really can't tell you what it's measuring. I can recap what people say about what it's measuring, however I've become skeptical of whether or not points are measuring what people think they are. And even moreso of whether or not it's a good use of time to do so.

Definition of a "Point"

Here's the root of my problem with story points. What are they and what do they measure? These aren't simple questions, either. If you read here, you get the idea that points are a measure of complexity. However, this StackExchange answer tells the reader that points need to be a measure of size, not complexity. Both will tell you that points are not a measure of time, but everywhere I've run into the practice, someone has always defined a point something along the lines of "What one developer can do in one day". Which certainly implies that points are used to give some measure of time. So I have a hard time getting started here, because I don't even know what I'm supposed to be measuring. Time? Size? Complexity? The amount of effort it takes for two women to have one baby? 

What's the Point, anyway?

I promise- I'll stop doing that now. Maybe talking through why we need points, we can come to an agreement on what they are. We need them for estimating, right? But estimating what? The amount you can do in a two week sprint? The amount of time you need to finish a certain set of features in a product? After all, you assign points, add them up, and divide by about what your velocity has been, right? Because that works.

Except, of course, it doesn't. Addressing the latter poi ISSUE ::ahem:: first, we all know that doesn't work. Regardless of what some may say, points do not mean you don't need to re-estimate. Software development is a process that builds on itself with every build adding to the whole an impacting the effort/size/complexity/time needed for future changes. I'd be very dubious of the point value of any story not going into the next sprint. I'd be even more dubious of any time estimates derived from arithmetic on points, velocity, and time.

So, maybe it's a good way of determining what can be worked on in the coming sprint. Problem is, without definitive answers to the question "What does a point measure", using points to determine what stories can fit into a sprint is a terribly uncertain process.

If we take the definition of points being a measure of the size of the story, then we truly have no idea how long a story will take. In fact, the StackExchange user in the link I referenced illustrated this very thing. He gave examples of the complexity of a 5-point sized story varying by a number of factors, leading to wildly different completion times. I don't really see that as a practical way to plan a sprint. More importantly, I don't think the people who expect me to plan sprints will see that as a practical way of planning sprints.

If we take the definition of points measuring complexity, we run into a different problem. How do you measure "complexity"? It's not like defining the length of a metre or defining a joule. the only way is to compare the complexity of one thing to the complexity of other things until you get something that feels like a match. Inaccurate, sure. But that's why it's called "estimating" and not "precision". I'd be basically okay with this except for a couple of things:

1. That which is complex to one developer may not be to another. Planning Poker is supposed to account for this, but it doesn't. In fact, it ironically exacerbates the problem it claims to solve. My 5 point complexity story may be another's 2 point complexity. To assign it 5 or 2, then, is meaningless outside the context of the developer who will be working on it. And splitting the difference to make it 3 or 4 is just compounding the uncertainty of an already uncertain process.
2. Complexity has nothing to do with the amount of time it takes to do the task. Planning Poker assumes that every developer handles every type of complexity in the same way and at the same pace. Maybe one guy tackles a complex story by diving in and coding until he grinds to a solution. Maybe another takes time to read more on the subject, asking questions, and only starting when he feels he has a better understanding of the subject. Given this, complexity can only correlate with time if every developer approaches complexity the same way.

Either way you look at it, you're trying to objectively quantify a series of subjective data points. Logic 101 tells us that this is a highly inaccurate way of measuring anything.

Velocity

Finally, we get to Velocity, which I'm pretty sure is the whole reason we bother to keep track of points in the first place. Velocity is nothing more than the sum of the points of the stories in a given sprint. It purports to measure the amount of work done in a sprint so you can compare to other sprints and then plan a timeline for future features.

Where to start.

First off, all this gives you is a comparison of how you've been objectively quantifying the subjective sprint over sprint. But as a comparison of the actual work done? All you're doing is comparing arbitrary numbers to arbitrary numbers and pretending it's a meaningful measure of productivity.

Secondly, using this as a future planning tool is a recipe for disaster. In order to make that work, you'd have to give point estimates to everything in your backlog and then continually refine those estimates in light of what the previous sprints have accomplished. Do that for long enough, and sure. your velocity will become a more and more accurate tool for predicting the amount of work the team can handle. But at what cost? At what time commitment?

Conclusion (WHEW!)

I'm not saying we need to stop estimating anything. Projects, stories, releases, versions, whatever unit of work that is needed, all need to come with some idea of what it takes to accomplish them. What I am saying is that we need to re-think the whole point estimation bit and ask ourselves if this isn't just an overly complicated process. Effort, complexity, productivity- these are all complex issues that are impossible to quantify in a simple context. Maybe we need to stop trying. Because I really hate unnecessary complexity. I'm not even much of a fan of necessary complexity. I'm lazy and don't want to work any harder than I have to in order to solve problems.

Next article, what I see as the real root of the problem and what I have started doing about it.

New Challenges

"Be careful of what you wish for. You might get it".

Hard to say where that comes from, although some cite it as a misquote of Goethe. The image, of course, is from the amazing mind of Bill Watterson and his Calvin and Hobbes comic.

Regardless, that is kind of the source of why I haven't been writing much lately and why I'm starting again. I tend to write about what I'm seeing and experiencing. Which, over the last few years, has become fairly routine. After writing a good bit about my thoughts on Software Architecture, it became the task of taking my own advice.

So what changed? A chance to get what I asked for, namely a chance to go from helping insure quality in a software system to helping insure quality in a development team. The company I work for has a relatively new development team that operates fundamentally differently than our other teams. This team doesn't support a product. Instead, they handle smaller projects that have a more rigid "Definition of Done", as opposed to continued development and support of a product. This has come with more than a few challenges. Challenges that I have a pretty free hand to solve, but challenges that will take a lot of solving.

All this has put me in an interesting opportunity. Processes are getting built from the ground up. We have leave to borrow from other teams the processes we feel are helpful, as well as to ignore the things we feel won't benefit us. The end goal is increasing the rate of shipping quality software, and with very few exceptions I don't think the head of software development much cares how it happens. This gives me the freedom to take my biggest piece of advice to anyone designing software. Or, for that matter, anyone currently living and interacting with the world in any way:

“This, what is it in itself, and by itself, according to its proper constitution? What is the substance of it? What is the matter, or proper use? What is the form, or efficient cause? What is it for in this world, and how long will it abide? Thus must thou examine all things that present themselves unto thee.” --Meditations, Marcus Aurelius

In other words, "Question everything". Some ideas are worth leaving alone. Questioning the necessity of engagement between the business and development is a lot like questioning gravity. Feel free, just stay away from ledges. But how do you do these things when the business units aren't used to that engagement and don't understand the value in it. And that's not looking down on anyone- how can you know the value of something until it's been shown to you?

Beyond that, what traditional roles in Agile projects do we need? Which can we live without, and which should we live without? What do these roles mean, and how do we make sure everyone knows what an iceberg is?

What about the team? What common practices do we need? What offers us the best value in our situation. Not just what others are doing or what generally works well, but what works well for us?

These are the questions I'm asking myself and the ideas I'm exploring. I'm sure we'll go down some odd paths, and probably even the (hopefully) occasional bad path. And hopefully, my philosophy that understanding the world leads to understanding software still holds.

Agile Isn't Dead

"Best practices are useful reference points, but they must come with a warning label : The more you rely on external intelligence, the less you will value an internal idea." ― Gyan Nagpal, Talent Economics: The Fine Line Between Winning and Losing the Global War for Talent

Agile isn't dead. It's not even on life support. However, to extend an analogy that everyone in the world should stop using, it does have a couple of CAT scan anomalies it should get checked out.. I've read a few tech opinion pieces to the contrary lately, so I thought I'd throw my hat in the ring.

+Hayim Makabee wrote an excellent piece on the subject, but one that I feel missed the mark somewhat. Jim Bird wrote on a finer grained level on the subject, and if you take them together an interesting picture starts to emerge. Starting with Jim Bird's piece, he lists several "Agile" practices that aren't needed. Skipping by a few points on which I disagree, his overall point is sound. Not all common practices are necessary. Hayim Makabee takes this a step further, rightly pointing out how "Agile" or "Scrum" consultants "over-simplify the software development process and underestimate the real complexity of building software systems". And I'm not really arguing that this "one size fits all" approach is a problem. I believe that the problem's source hasn't quite come out yet.

In short, the problem is IT decision makers. Not Agile, which is just a set of principles. Not Agile consultants, who are just selling a service people have asked for. And while we can discuss the moral culpability of someone "just selling a service", the fact of the matter is that people are buying the service. And those people are IT decision makers in pursuit of a concept that has made me cringe for some time now. Best Practices. "Best practices" are like the Dark Side of the Force. Quick, simple, seductive, and once you walk down it's path, forever will it dominate your destiny. "Best Practice" is shorthand for "What everyone else is doing", and while it's the sort of thing one ought to take into account, I've seen enterprise after enterprise implement iron-clad procedures based solely on "Best Practices", or rather "Just do what the neighbors across the street are doing". And before too long, you have Citogenesis, that curious phenomena where people think something is correct because people think something is correct.

Agile is a set of principles, and as far as principles go, they're pretty darn good. Agile isn't dead because these principles are still the best way to create software. But it really needs that dark spot checked out.

Healthcare.gov failures in leadership

"Good management consists in showing average people how to do the work of superior people." --John D. Rockefeller

"Good management is the art of making problems so interesting and their solutions so constructive that everyone wants to get to work and deal with them." --Paul Hawken

Yep. I'm back at this well. Partially because it's a great way to boost hits (I don't use ads, but I do have an ego) but mostly because I've been there. Not in a project this size, but I've lived the nightmare. Maybe someone that can affect this boondoggle

reads this and listens. Likely not. I have an ego, but I'm a pretty small fish. More important to me is that the developers and IT staff affected by projects like this understand the failings so that they know when to update their resume.

This one is coming from a New York Times article titled Inside the Race to Rescue a Health Care Site, and Obama. And again, whether Ms. Stolberg and Mr. Shear realize it, they paint a picture familiar to many IT veterans.

Failures of Testing

I've written about the technical failures of the development teams but what has been written in this article serves to underscore something all development teams know, but fewer do. Testing. From the article:

"To do that, they would have to take charge of a project that, they would come to discover, had never been fully tested and was flailing in part because of the Medicare agency’s decision not to hire a "systems integrator" that could coordinate its complex parts." 

"The website had barely been tested before it went live, so a large number of software and hardware defects had not been uncovered." 

"'There’s so much wrong, you just don’t know what’s broken until you get a lot more of it fixed,' Mark Bertolini, the chief executive of Aetna, said on CNBC."

Regression testing. Unit testing. User Acceptance testing. I can't think of a better way to talk about the dangers of not budgeting time to properly test your project.

"In Herndon, as engineers tried to come to grips with repeated crashes, a host of problems were becoming apparent: inadequate capacity in its data center and sloppy computer code, partly the result of rushed work amid the rapidly changing specifications issued by the government."

Code reviews and proper architecture concern would have prevented the "sloppy code" issue. The rest of this segueways nicely into my major point, thought.

Failure to understand a software development project

Clearly, project management didn't know how to manage a large-scale project. I've said before that the evidence could not be clearer that this project was managed Waterfall-style. And while the far-preferable Agile development style can also struggle with changing requirements, the iterative workflow, combined with constant checkpoints with the business stakeholders allows for better options for dealing with the fact of life that is spec change. Here are a few examples from my playbook:

"No, we will not change specifications in mid-sprint. If you want this change, submit it to the Icebox and give it a priority. We will then do the design and estimation work and add it to a future sprint."

"No. This is a two week sprint. We will not finish early. We will not rush. We have a planned amount of work that will take two weeks to deliver properly."

"No, we will not add to this sprint. The sprint covers an amount of work that can be accurately delivered in two weeks."

"No, we will not include this work into a sprint until there is proper user acceptance criteria attached. The delivered code will then conform to the acceptance criteria listed. No more, no less. I'm here to help you write good acceptance criteria."

Enforcing this kind of project discipline is crucial. It should be well-communicated up front. Everyone should know what an iceberg is. And once the expectation is set, that expectation is enforced. But even with that, the problem runs far deeper. Some quotes from the article really chilled me:

"Out of that tense Oval Office meeting grew a frantic effort aimed at rescuing not only the insurance portal and Mr. Obama’s credibility, but also the Democratic philosophy that an activist government can solve big, complex social problems." 

"'We’re about to make some history,' she (Ms. Sebelius,) said.

"...reveals an insular White House that did not initially appreciate the magnitude of its self-inflicted wounds"

As any regular readers know, my overall philosophy comes from Marcus Aurelius:

“This, what is it in itself, and by itself, according to its proper constitution? What is the substance of it? What is the matter, or proper use? What is the form, or efficient cause? What is it for in this world, and how long will it abide? Thus must thou examine all things that present themselves unto thee.”  

By itself and in its proper constitution, Healthcare.gov is a website that allows people to purchase health insurance. It is not a justification of a political philosophy. It's not about making history. Those are incidental. You cannot make good decisions if you don't understand the context for those decisions. You can't make good decisions about a development project if you see it as anything other than a software development project. And don't think for a second that minimizes its importance somehow. Most developers I know care a good deal more for their code than they do your politics. But if management mistakenly sees a project as some kind of ideological movement or as setting their legacy then they aren't making project management decisions.

Management culture

Finally, what may be the worst failing of the management of this project. It's culture. The standards management sets for its operations. The only place I've ever heard of management leading so poorly is in Dilbert. For example:

"For weeks, aides to Ms. Sebelius had expressed frustration with Mr. McDonough, mocking his “countdown calendar,” which they viewed as an example of micromanagement."

Mocking other stakeholders. Ignoring, for a moment, the notion of expecting professionals to not act like spoiled children, why was this behavior considered acceptable? Management sets the standard for how people act. A good manager understands this. A good manager sets an expectation of professionalism. Complaining is one thing. Open mocking should be the sort of thing people are embarrassed to do, or even to listen to.

"Contractors responsible for different parts of the portal barely talked to one another, hoping to avoid blame."

There's enough written about how a management culture of blame creates a toxic environment. Problems aren't addressed. People pass the buck rather than address problems. The concern is to not be left holding the bag, rather than making sure work is done well. Fear does not create quality. And clearly, this is not known here.

 "Mr. Obama, meanwhile, was under assault. After years of telling Americans, “If you like your insurance plan, you can keep it,” he was being accused of lying. On the night of Oct. 28, Ms. Jarrett, one of Mr. Obama’s closest confidantes and a guardian of his personal credibility, took to Twitter to defend him — and to shift the blame. 

“FACT,” she wrote. “Nothing in #Obamacare forces people out of their health plans. No change is required unless insurance companies change existing plans.”

The tweet touched a nerve; it was not the first time the Obama White House had used the insurance industry as a scapegoat. Ms. Ignagni’s (chief executive of America’s Health Insurance Plans,) members were furious. “Here it comes — we knew it would happen,” one executive recalled thinking."

 The Obama administration built support for the ACA by building hostility to the insurance industry, with whom they must now work. So, in essence, the Obama Administration has taken every opportunity to publicly condemn the insurance industry and is now relying on that same industry to launch not only it's centerpiece website, but the validation of their political philosophy. A management team that does not treat their vendors with respect will find themselves very lonely once they need their vendors. UnitedHealth and CIGNA have already "mostly shied away from the online marketplace" (From the article). To assume this has nothing to do with the administration's clear contempt for their vendors is folly.

I have a difficult time summing up my shock at the sheer number of leadership failures involved here. Not just in knowing how to run a software development project- clearly Medicare and the Dept. of Health and Human Services considered this a "Fire and Forget" project and that they were absolved of all responsibility to be involved once the project started. More disturbing, though, is the culture that management allows. Childish behavior and finger-pointing should not be acceptable. And that attitude starts from the top.

The Unambiguous Measure of Success

"[T]he presence of an unambiguous measure of ex-post success (profit) serves to harness the natural tendency toward overoptimism that otherwise would almost certainly be present when someone else’s money is being spent." --Robert Wagner, "Economic Policy in a Liberal Democracy"

Every once in a while I'll come across a quote or an article that makes me think about software development. Often that's because I tend to read a lot of material related to software development, but sometimes it's not. Such is the case of Donald Boudreaux's Quotation of the Day for October 23rd. The full quote is:

[T]he presence of an unambiguous measure of ex-post success (profit) serves to harness the natural tendency toward overoptimism that otherwise would almost certainly be present when someone else’s money is being spent.  The necessity of putting one’s money on the line and of being responsible for the ultimate outcome surely has a sobering effect on the assessment of the prospects for such projects [that governments typically undertake], an effect that is weakened when tax money is used in a setting where no judgement about profitability has to be faced.

I'm no economist and I don't pretend to be one. And this isn't a post about economics, anyway. What caught my eye was the idea that putting one's money on the line and being responsible for the ultimate outcome by setting an unambiguous measure of success. Even though we all know, often through painful experience, why clear and unambiguous project goals are a necessity, I think it's interesting to look at it with an economist's viewpoint.

Instead of "someone else's money is being spent", let's use "someone else's resources are being spent". In other words, not just the salaries of development staff but also time and infrastructure. If the burden of this is borne solely by the development staff, then the tendency of the customer stakeholders is toward overoptimism. Features, both initial and scope creep, and timelines all trend toward pushing the limits of what the development staff can reasonable accomplish. At least, that's been my experience.

A couple of things happen when the customer is expected to give a clear and unambiguous measure of success. Not just in terms of a requirements document, but close involvement in the development process both in defining clear acceptance criteria for user stories and in reviewing the results of development sprints. The customer is now spending their resources on the project. Their staff has to be available for clarifying requirements. Their staff is has committed their time to insuring that the "measure of success" is being met. And their staff also has to budget, and therefore use effectively, their time for the project in relation to the time needed for other tasks. The tendency of those with a stake in the game is to be more careful with how those resources are spent and to insure that resources aren't wasted. I guess this makes sense, really. People tend to spend money more frivolously when not using their money, or when it doesn't look like they are using their money. It's why managing credit can be tricky and it's why casinos use chips instead of currency. Why should spending resources on a project be any different?

I like this quote. It's a truism of software projects, for that matter projects in general, that you can't finish a project if you don't know what "finished" looks like. I'd never stopped to think about how a clearly defined measure of success affects the customer in a project.

Health Care Exchange Project Pt. 3

"One test is worth a thousand expert opinions." -Wernher Von Braun

And finally the last piece of the puzzle. As a software developer, I find the previous types of project problems maddening. However, this last category boggles my mind. Perhaps I'm just an idealist, but I truly want to believe that this sort of thing doesn't happen anymore. Sadly, I read The Daily WTF far too often to really believe it. We are now down to technical failures.

Again, quoting from From the Start, Signs of Trouble at Health Portal

"Others warned that the fixes themselves were creating new problems, and said that the full extent of the problems might not be known because so many consumers had been stymied at the first step in the application process."

"'So much testing of the new system was so far behind schedule, I was not confident it would work well,' Richard S. Foster, who retired in January as chief actuary of the Medicare program, said in an interview last week."

We all know that maintenance, especially bug fixing, is the true bulk of any software development work. And we all know that testing is the heart of finding, and therefore fixing, bugs. This is not under dispute. However, I've been associated with so many development projects that ignore this basic principle that I want to weep sometimes. And it's always the same. "We don't have time to test because we're busy building features". Or "We'll focus on testing later". Or the worst, "We'll worry about bugs when they're reported by users". (Yes- I've been told that)

To be clear. Unit tests insure that a unit of code has a consistent result at any point in time. It doesn't insure that the code does what it is supposed to. It insures that what the code does hasn't changed due to other factors. Unit tests are how you do regression testing. At least, how you do it without resulting in Cthulhu-level madness.

User Acceptance Testing insures that the users can actually perform the tasks called for in the specifications. This doesn't happen at the end of the project. This happens at planned stages throughout the project so that testing happens on a manageable set of features. A set of features that can be easily documented, easily described, and easily managed. Failure to do this step before rollout is inexcusable.

And while we're at it, what about Exception handling testing? What effect does any given exception have? How is it reported, both to support and to the user? How are exceptions tracked? Testing isn't just about making sure the application works well. It's about insuring that it fails gracefully.

"The biggest contractor, CGI Federal, was awarded its $94 million contract in December 2011. But the government was so slow in issuing specifications that the firm did not start writing software code until this spring (Em. mine- MO), according to people familiar with the process."

I'm pretty okay with most of this, but the failure is so bad that it's worth mentioning. The award amount doesn't bother me. I'm also pretty okay with two years of requirements. This isn't like turning on a switch and watching everything work. This is a serious development project- far more serious than anything I've participated in. I would have been more surprised to see the award amount or the planning time significantly lower.

But read the bit I emphasized. Development didn't start at any point during planning. In other words, a project with this level of work and complexity was attempted Waterfall-style. Not Agile. Waterfall. In a project like this, the technical leadership deliberately passed on the ability to easily reach to changing requirements. And the ability to work on completed requirements as they become available. And on increased involvement between development and stakeholders. And continuous testing.

I'll take the heat for saying this- Agile isn't a buzzword. It isn't a topic for bloggers to discuss. And it isn't an alternative methodology. It's the only sane way of approaching a development project of any more than a trivial size. You simply can not anticipate everything ahead of time, and attempting to do so harms the project more than it helps. Case in point.

All of the problems the NYT article describes are serious. Any more than one or two of them will probably sink a project. The fact that people are reporting this many fundamental mistakes makes this an example everyone familiar with software development should understand. If only to protect your career.

Health Care Exchange Project Pt. 2

“No matter how good the team or how efficient the methodology, if we’re not solving the right problem, the project fails.” - Woody Williams

In my previous article I started talking about the New York Times article From the Start, Signs of Trouble at Health Portal. See previous article for the disclaimers that hold here as well.

In Part Two, I want to talk about the parts of the article that, to me, describe a major failure in project leadership. Not to be confused with executive leadership. In this case, the described failures in those responsible for the actual project leadership.

"Failure to plan is a planning to fail". It's a cliche for a reason. Starting a development project without a plan, or with an obviously flawed plan, is a massive waste of time and money. From the article:

"Dr. Donald M. Berwick, the administrator of the federal Centers for Medicare and Medicaid Services in 2010 and 2011 'The staff was heroic and dedicated, but we did not have enough money, and we all knew that,'"

From the beginning, we have a serious issue. The project wasn't funded.  The only way this works is if you have a plan to scale back what you can't pay for. The quotes in the previous article regarding executive leadership make this an impossibility, however.

 "Some people intimately involved in the project seriously doubted that the (Medicare and Medicaid) agency had the in-house capability to handle such a mammoth technical task of software engineering while simultaneously supervising 55 contractors."

"The political people in the administration do not understand how far behind they are." 

Well, now we have some insight into some of the executive leadership issues. Project management is supposed to be responsible for insuring that the correct groups are responsible for units of work and are responsible for reporting progress. It very much sounds to me like the project management team fell flat here. This is by no means unique to ACA, nor even to government development projects. I've seen, far too often, project managers who think their job ends with the kickoff meeting. Or who only schedule meetings and do little else. A good sign of a sinking project is negligent project management. Worse is project management that is unfamiliar with what the role entails.

"A round-the-clock effort is under way, with the government leaning more heavily on the major contractors"

"Worried about their reputations, contractors are now publicly distancing themselves from the troubled parts of the federally run project."

"Senior executives at Oracle, a subcontractor based in California that provided identity management software used in the registration process that has frustrated so many users, defended the company’s work. 'Our software is running properly,' said Deborah Hellinger, Oracle’s vice president for corporate communications."

How often have you seen this story play out? Lack of planning and poor leadership lead to "crunch times". As a result, development staff is required to work late. Demands raise past what is reasonable and soar up to the ceiling of what is possible. The result?  Discontent and CYA. The quotes above tell me that the contractors have already given up on the project and its leadership. Worse yet, the "blame game" has gotten into full swing. Blame doesn't happen when the project staff sees the project as salvageable. Blame only happens when the project is seen as a loss and people only want to salvage their careers. In a very real way, the blame game prevents problems from getting fixed.

Project management problems are a warning sign that is difficult to see. Oftentimes, poor project leadership isn't obvious until the project is well under way. At that point, recovering the project can be problematic. Tasks have already been assigned inappropriately, Reporting is either far behind or nonexistent. And at the worst, although I didn't see any evidence of this in the article, poor project management often involved a lack of clear project goals. This last is, in my opinion, the worst way project management can fail. If a project without clear end goals is allowed to continue, failure is the only possible result.

Health Care Exchange Project Pt. 1

“I have witnessed boards that continued to waste money on doomed projects because no one was prepared to admit they were failures, take the blame and switch course. Smaller outfits are more willing to admit mistakes and dump bad ideas.” - Luke Johnson

Let me start off saying that this isn't political commentary. If you want to talk about whether or not the Affordable Care Act (ACA) is a good idea or should be repealed, please go elsewhere and don't pollute my blog with political commentary.

Earlier today, I came across NYT article headlined "From the Start, Signs of Trouble at Health Portal" and written by Robert Pear, Sharon LeFraniere, and Ian Austin. Before going on, I highly recommend reading the article. I also found a very readable companion piece written by Megan McArdle.

The reason it caught my attention isn't that it is about ACA but rather the amount of project failures that I can relate to. The signs of common project management mistakes are so obvious, anyone experienced with software development projects will find themselves nodding their head with nearly every paragraph. This article doesn't describe a mere failing project. The failures described here are so blatant, so numerous, that I have to wonder if we're looking at deliberate sabotage. Or possibly the article is merely social satire.

The point of this isn't to slam the ACA, let me make that clear. The desired end of the project isn't what caught my attention. Nor am I trying to "discredit" the ACA. This is a real world study of how to run a software project into the ground and guarantee its failure before the first line of code is written. This article should, absent all else, serve as a warning to those undertaking software development projects, and indeed projects of any kind.

As this is far longer than my usual blog posts, I'm breaking it up into three parts, by what I see as logical categories of the project's failures. For the first one, lack of executive leadership.

It's frustrating when software projects are crippled by executive-level politicking. Executive-level leadership is foundational to a project's success. And like a building with a flawed foundation, poor executive leadership will destroy a project in unexpected ways. I have been involved in several projects that lacked executive-level leadership. All collapsed. Quoting the NYT article:

"Politics made things worse. To avoid giving ammunition to Republicans opposed to the project, the administration put off issuing several major rules until after last November’s elections. The Republican-controlled House blocked funds. More than 30 states refused to set up their own exchanges, requiring the federal government to vastly expand its project in unexpected ways."

"Administration officials dug in their heels, repeatedly insisting that the project was on track despite evidence to the contrary." 

"Mr. (Henry) Chao’s (Chief digital architect) superiors at the Department of Health and Human Services told him, in effect, that failure was not an option, according to people who have spoken with him. Nor was rolling out the system in stages or on a smaller scale, as companies like Google typically do so that problems can more easily and quietly be fixed." 

I see a few serious warning signs here. First, not all stakeholders were on board with the project. I don't care what rationale you use, if the stakeholders aren't on board, the project is doomed. This is as close to political as I want to get here. Neither the reasons the Democrats had for steamrolling the Republicans on the ACA nor the reasons the Republicans have for trying to kill it matter in this context. The unassailable, unarguable, truth is that if a major stakeholder want to kill a project then the project is dead. Ideology does not change this, nor does intent. Whether the result of the project is necessary, helpful, or detrimental is also irrelevant. And it is inexcusable for executive leadership to begin a project like this, through intent or ignorance of This Law, with such high level opposition.

Just as bad, however, is the unwillingness to acknowledge that the project is in trouble, but rather relying on the "Failure is not an option" method of resuscitating a project. Take a moment and count the number of times someone told you "Failure is not an option" or "Just make it happen" and it actually helped. Go ahead. Raise your hand if you got a number above zero. Anyone out there have their hand raised?

Didn't think so.

Attention all managers- these phrases do not solve problems, nor do they create an environment in which problems can be solved. For the love of all that is holy, stop using them.

Several reasonable suggestions are on the table. Postponement. Small scale rollouts, Google-style. The reasons given for ignoring these options are just about the worst possible. Executive-level politics.

Executive leadership enables development projects in ways that many people on the project never see. Because of that it's difficult to see when executive leadership is failing. Sadly, the results are less hidden.

Finding Valrhona, or Habits of Effective Architects

I have very strong opinions on the subject of software, architecture, and quality in general. Coors is not beer. A Hershey bar is not chocolate. Neither Velveeta nor Kraft Singles are cheese. Starbucks does not serve anything I identify as coffee. "Cowboy Coding" is not software development". This really isn't about my quirks in food quality. More a list of items that I find helpful in making sure I'm helping to deliver Valrhona.

Understand what you need to deliver

Before you select technologies, before you start with the design patterns, and certainly before you put hand to keyboard, make sure you understand what pain point you're relieving for your customer. Software development is about solving problems. So many times, I see projects skipping right over the problem to be solved and heading right to implementing a solution. Oftentimes, talking through the problem to be solved makes a murky solution obvious. If you're stuck, ask yourself "What problem are we solving?" If you don't know, you know what to do next.

Solve for realistic problems

If you don't need a full enterprise-y solution with distributed widgets, Something As A Service, and Fully Configurable Everything, don't build it. This is kind of an extension of the previous point, but for every architecture decision you make ask yourself "What value is this adding to the solution?" if you don't have an answer then you don't need the Thingy.

On the other hand, understand that no software project is ever finished and that no set of requirements stay static. Especially once development starts. As Helmuth Graf von Moltke said, "No campaign plan survives first contact with the enemy". There are well established patterns for solving common problems. A thorough understanding of the Gang of Four's design patterns will go a long way in helping you avoid common pitfalls. Don't use them just to use them, but don't avoid them just because they're common. They're common for a reason.

Stop. Collaborate and Listen.

Okay, for those of you who get the reference, I apologize. Note there's no link. I don't want to infect those of you who don't get it. But, origin aside, it's good advice for the architect. Even if you're sure of yourself, get feedback. A development team is more than the sum of its parts, and several smart developers working together produce far better results than several smart developers working separately. Capitalize on this. Ask for comments and then listen to them. Especially the criticism. The worst that can happen is that you'll feel more confident in your design.

Along those lines, keep up on the current trends in software developemnt. I'm not saying you have to be KanBanAgileScrumTDD just because others have written about how shiney they are. But you won't know how these concepts can, or can't, help if you aren't familiar with them.

Strive for Elegance, but understand what it means

To me, an "Elegant" solution is not necessarily overly-clever. It does not have to solve problems in a new way. And it certainly doesn't take Donald Knuth to understand. To me, "Elegance" makes the solution look easy. Sure, maybe you come up with a better way of solving a problem. But maybe you recognize that some techniques are "Tried and True" for a reason. Either way, your result shouldn't look like a bunch of work. It should look obvious.

Know when to say when. And when not to.

Understand that at some point in your career (or, in my case, at some point in your day), the pursuit of higher quality will conflict with the overall effort in such a way that the pursuit does more harm than good. be able to recognize that time and let go.

Understand that at some point in your career, you will be expected to sacrifice quality for the overall effort in a manner that does more harm than good. Don't dig in. Don't get stubborn. Learn to present your case in terms that the decision makers understand. You will not always get your way, but you will become known as an asset that is always looking out out for the overall project.

Note that there isn't a lot of actual code advice here. Sure, I could tel you that if you're instantiating different types of classes depending on context, consider an Object Factory or even an Abstract Factory. Or if you have a somewhat complex process that other processes interact with, or a subsystem that might change, consider a Facade. I could give you the ol' "Design from the Interface" advice or even tell you that if you find yourself considering recursive queries maybe you should step back a bit. But I think that if you really take the above to heart, everything else is just details.

Projects and Icebergs

Once, I watched a class being taught to a small group of children. The subject was Aikido, an ancient martial art which utilized much inner energy, or ki. The instructor used an analogy to show internal versus external strength. ‘Ki’, he said, ‘is much like an iceberg. There is a tip, which is visible, much as external strength, which uses muscles; then there is the internal strength, which is at once much greater, and yet hidden.

…

When he had completed his explanation he asked, ‘Are there any questions?’ A small boy of perhaps four or five years raised his hand. ‘What’s an iceberg?’” --Steve Perry “Matadora”

Software development projects hinge on many things. The makeup of the development team. Resources that can be committed to the project. Management and stakeholder buy-in. However, the biggest problem I've seen is communication and the biggest communication problem I've seen is defining terms so that everyone is using a common lexicon.

Some years ago, I was part of a development project that was a miserable failure. The six month project was still in development after two years and eventually a competitor beat us to market. Looking back on it, the core problem could be distilled down to the fact that no two groups of people agreed on what any phase of the project meant. It’s not that anyone wanted to skip defining the project work or goals, or wanted to skimp on testing, or even that no one wanted to go through the time to meet to discuss project progress. The problem was that no one agreed with anyone else on how to define these aspects of a project or what should be done with them.

To some, “Testing” meant developers testing their work, and nothing else. To others, testing meant focus groups. Some thought that the project goals could be adequately communicated in a Project Charter document, others wanted to have agile-like meetings to discuss goals. And no two groups agreed on what “Finished and ready to deploy” meant. One group thought we were building an ecommerce site while another was adamant that the site should not handle payment transactions. One group defined “Deployment” as a full release that would allow all customers to begin using the site. Others defined “Deployment” as a limited release, similar to a beta test.

Even worse, there was no common definition of project roles. Sure, there was a project manager, product owners, executive stakeholders, a dev lead, and QA staff. We hit all those bullet points. The problem was that no two people agreed on what these roles meant. Because of that we had the dual problems of nobody performing certain critical tasks while stakeholders were fighting over how other tasks should be handled because multiple groups believed that the task was theirs to do.

As a result, work stagnated as developers began undoing work that one group had requested. Frustration levels ran high and developers began dragging their feet on work they didn't see a point in doing, knowing that someone else would soon have them undo the work. Testing was a mess as some groups refused to test functionality, thinking that was the developers job, and others submitted the application to a focus group and expected any feedback to be addressed. Finally, the project died when developers started leaving for other jobs.

I want to take a moment and underscore that last point. Not only did a lack of common definition of concepts cause the failure of this project, but it hindered future development projects due to development staff leaving the company.

In contrast is the project I’m on now. Development is going to take at least a year and a half and the resulting platform will have implications far beyond the current business need. We are coordinating three different business units with similar, but not identical needs. Work will be performed by four parallel development teams working separately but coordinating with each other. And this is all possible because before we even began gathering requirements we defined every project concept we could think of so that there was no misunderstanding. The definition of each project role. What "Scrum" and "Agile" meant in the context of our project. And, of course, the definition of "Project Complete".

As a result, milestones have become easier to hit. Not easy, mind you, just easier. We have agreed on what "Acceptance Criteria" and "User Story" mean so that when we do requirements gathering we all know what information is to be presented and in what form. Development milestones are easier to communicate, since we all know what to expect of code that is “Ready for sprint review”. When we talk about testing, we have agreed on what that means. Now everyone knows what has happened at each testing stage. As a result, the state of the code is clear at any given step. We have agreed on what it means to release code to production and we have a clear definition of what it means for this project to be finished.

Better yet, we have a common agreement on all project roles. Because we took the time to define “Product Owner”, we know what to expect from that role. Those responsibilities are clearly documented. As is the meaning of “Project Manager”, “Project Architect”, “Development Lead”, and “Business Analyst”. We know who does what, what to expect from whom, and to whom to look for any needed deliverable.

Of course, a common definition does not guarantee success. Agreement on terms doesn't remove the work of meeting project milestones. Because of the common lexicon, though, we can take a look at any part of the project and make a determination of whether or not it meets the definition it is supposed to. If two parties disagree on whether or not the definition has been met, we know exactly who makes the final decision. If that decision is that the definition is not properly met, we know what needs to be changed and exactly who is responsible for making the change.

Communication is one of the biggest single points of failure in a software development project. Nothing can tank a project faster than misunderstandings or people not clearly communicating their understandings and expectations. But before you can clearly communicate, everyone on the project must know what an iceberg is.