A New Perspective

"I believe everyone should have a broad picture of how the universe operates and our place in it. It is a basic human desire. And it also puts our worries in perspective." --Stephen Hawking 

"Everything we hear is an opinion, not a fact. Everything we see is a perspective, not the truth." --Marcus Aurelius 

This  article falls more heavily under the "Musings" title of my blog. I'm less making a point than I am thinking out loud. As always, feedback and insights are always welcome.

My job responsibilities are changing. I phrase it like that because I doubt my actual title will change, merely the meaning of that title. I rarely handle project level work anymore. Rather, I'm more involved with enterprise level architecture decisions. I haven't implemented a design pattern in quite awhile. I find myself, instead, setting the standard of what patterns are best to use or avoid in certain situations. Or what frameworks we will be using or whether we will use an off the shelf solution or build our own. In other words, my implementation decisions are becoming less important than my opinions and experience with those decisions. I find this a very new perspective and more than anything else, I find myself more and more writing about new perspectives on old ideas.

For instance, take our current investigation into unit tests. The discussion started with "What mock object framework should we use?" We quickly boiled down to "Which framework will be unduly burdensome to the development staff?" This actually eliminated a couple of frameworks at the beginning. But when we settled on two that are, more or less, of equal use the conversation quickly changed to unit test standards. I have a few strong opinions on the matter. I believe that unit tests should cast a wide loop so that behavior consistency can be assured. If that means mocking Internal methods to assure their consistency, then so be it. If that means only using strict mock objects, despite their fragility, then so be it. In fact, I like fragile unit tests. If the behavior of the class changes then the tests should break. I realize that my opinions are not shared by the community at large, and I'm okay with that. I'm always open to debate, but I approach things somewhat differently than normal. I think I've made that clear. Now, of our enterprise level architects, one disagrees with me and the other is still weighing arguments and that's great. That kind of conversation is a new perspective for me.

So what's this new perspective and what does it give me? Because I'm no longer considering patterns and practices for a given set of circumstances, but rather to be followed in the enterprise, I have to more seriously consider the pros and cons of those patterns and practices. I find that thinking of effects of standards on the enterprise at large makes me think differently about the effects of my design decisions at the project level. Not just "Does this work here" but "Would this work in other, similar, situations and if not, why not?" If the answer to the second question is "No", then should I reconsider my decisions at the project level. Note that I'm not offering concrete conclusions here. I'm expanding my perspective and thus expanding the pool of questions I ask myself before making a decision.

Maybe that's my point here, although I expect I'll be getting comments about my approach to unit tests. That's fine, too. The day I stop listening to others is the day I stop being useful.

The Unambiguous Measure of Success

"[T]he presence of an unambiguous measure of ex-post success (profit) serves to harness the natural tendency toward overoptimism that otherwise would almost certainly be present when someone else’s money is being spent." --Robert Wagner, "Economic Policy in a Liberal Democracy"

Every once in a while I'll come across a quote or an article that makes me think about software development. Often that's because I tend to read a lot of material related to software development, but sometimes it's not. Such is the case of Donald Boudreaux's Quotation of the Day for October 23rd. The full quote is:

[T]he presence of an unambiguous measure of ex-post success (profit) serves to harness the natural tendency toward overoptimism that otherwise would almost certainly be present when someone else’s money is being spent.  The necessity of putting one’s money on the line and of being responsible for the ultimate outcome surely has a sobering effect on the assessment of the prospects for such projects [that governments typically undertake], an effect that is weakened when tax money is used in a setting where no judgement about profitability has to be faced.

I'm no economist and I don't pretend to be one. And this isn't a post about economics, anyway. What caught my eye was the idea that putting one's money on the line and being responsible for the ultimate outcome by setting an unambiguous measure of success. Even though we all know, often through painful experience, why clear and unambiguous project goals are a necessity, I think it's interesting to look at it with an economist's viewpoint.

Instead of "someone else's money is being spent", let's use "someone else's resources are being spent". In other words, not just the salaries of development staff but also time and infrastructure. If the burden of this is borne solely by the development staff, then the tendency of the customer stakeholders is toward overoptimism. Features, both initial and scope creep, and timelines all trend toward pushing the limits of what the development staff can reasonable accomplish. At least, that's been my experience.

A couple of things happen when the customer is expected to give a clear and unambiguous measure of success. Not just in terms of a requirements document, but close involvement in the development process both in defining clear acceptance criteria for user stories and in reviewing the results of development sprints. The customer is now spending their resources on the project. Their staff has to be available for clarifying requirements. Their staff is has committed their time to insuring that the "measure of success" is being met. And their staff also has to budget, and therefore use effectively, their time for the project in relation to the time needed for other tasks. The tendency of those with a stake in the game is to be more careful with how those resources are spent and to insure that resources aren't wasted. I guess this makes sense, really. People tend to spend money more frivolously when not using their money, or when it doesn't look like they are using their money. It's why managing credit can be tricky and it's why casinos use chips instead of currency. Why should spending resources on a project be any different?

I like this quote. It's a truism of software projects, for that matter projects in general, that you can't finish a project if you don't know what "finished" looks like. I'd never stopped to think about how a clearly defined measure of success affects the customer in a project.

Health Care Exchange Project Pt. 3

"One test is worth a thousand expert opinions." -Wernher Von Braun

And finally the last piece of the puzzle. As a software developer, I find the previous types of project problems maddening. However, this last category boggles my mind. Perhaps I'm just an idealist, but I truly want to believe that this sort of thing doesn't happen anymore. Sadly, I read The Daily WTF far too often to really believe it. We are now down to technical failures.

Again, quoting from From the Start, Signs of Trouble at Health Portal

"Others warned that the fixes themselves were creating new problems, and said that the full extent of the problems might not be known because so many consumers had been stymied at the first step in the application process."

"'So much testing of the new system was so far behind schedule, I was not confident it would work well,' Richard S. Foster, who retired in January as chief actuary of the Medicare program, said in an interview last week."

We all know that maintenance, especially bug fixing, is the true bulk of any software development work. And we all know that testing is the heart of finding, and therefore fixing, bugs. This is not under dispute. However, I've been associated with so many development projects that ignore this basic principle that I want to weep sometimes. And it's always the same. "We don't have time to test because we're busy building features". Or "We'll focus on testing later". Or the worst, "We'll worry about bugs when they're reported by users". (Yes- I've been told that)

To be clear. Unit tests insure that a unit of code has a consistent result at any point in time. It doesn't insure that the code does what it is supposed to. It insures that what the code does hasn't changed due to other factors. Unit tests are how you do regression testing. At least, how you do it without resulting in Cthulhu-level madness.

User Acceptance Testing insures that the users can actually perform the tasks called for in the specifications. This doesn't happen at the end of the project. This happens at planned stages throughout the project so that testing happens on a manageable set of features. A set of features that can be easily documented, easily described, and easily managed. Failure to do this step before rollout is inexcusable.

And while we're at it, what about Exception handling testing? What effect does any given exception have? How is it reported, both to support and to the user? How are exceptions tracked? Testing isn't just about making sure the application works well. It's about insuring that it fails gracefully.

"The biggest contractor, CGI Federal, was awarded its $94 million contract in December 2011. But the government was so slow in issuing specifications that the firm did not start writing software code until this spring (Em. mine- MO), according to people familiar with the process."

I'm pretty okay with most of this, but the failure is so bad that it's worth mentioning. The award amount doesn't bother me. I'm also pretty okay with two years of requirements. This isn't like turning on a switch and watching everything work. This is a serious development project- far more serious than anything I've participated in. I would have been more surprised to see the award amount or the planning time significantly lower.

But read the bit I emphasized. Development didn't start at any point during planning. In other words, a project with this level of work and complexity was attempted Waterfall-style. Not Agile. Waterfall. In a project like this, the technical leadership deliberately passed on the ability to easily reach to changing requirements. And the ability to work on completed requirements as they become available. And on increased involvement between development and stakeholders. And continuous testing.

I'll take the heat for saying this- Agile isn't a buzzword. It isn't a topic for bloggers to discuss. And it isn't an alternative methodology. It's the only sane way of approaching a development project of any more than a trivial size. You simply can not anticipate everything ahead of time, and attempting to do so harms the project more than it helps. Case in point.

All of the problems the NYT article describes are serious. Any more than one or two of them will probably sink a project. The fact that people are reporting this many fundamental mistakes makes this an example everyone familiar with software development should understand. If only to protect your career.

Health Care Exchange Project Pt. 2

“No matter how good the team or how efficient the methodology, if we’re not solving the right problem, the project fails.” - Woody Williams

In my previous article I started talking about the New York Times article From the Start, Signs of Trouble at Health Portal. See previous article for the disclaimers that hold here as well.

In Part Two, I want to talk about the parts of the article that, to me, describe a major failure in project leadership. Not to be confused with executive leadership. In this case, the described failures in those responsible for the actual project leadership.

"Failure to plan is a planning to fail". It's a cliche for a reason. Starting a development project without a plan, or with an obviously flawed plan, is a massive waste of time and money. From the article:

"Dr. Donald M. Berwick, the administrator of the federal Centers for Medicare and Medicaid Services in 2010 and 2011 'The staff was heroic and dedicated, but we did not have enough money, and we all knew that,'"

From the beginning, we have a serious issue. The project wasn't funded.  The only way this works is if you have a plan to scale back what you can't pay for. The quotes in the previous article regarding executive leadership make this an impossibility, however.

 "Some people intimately involved in the project seriously doubted that the (Medicare and Medicaid) agency had the in-house capability to handle such a mammoth technical task of software engineering while simultaneously supervising 55 contractors."

"The political people in the administration do not understand how far behind they are." 

Well, now we have some insight into some of the executive leadership issues. Project management is supposed to be responsible for insuring that the correct groups are responsible for units of work and are responsible for reporting progress. It very much sounds to me like the project management team fell flat here. This is by no means unique to ACA, nor even to government development projects. I've seen, far too often, project managers who think their job ends with the kickoff meeting. Or who only schedule meetings and do little else. A good sign of a sinking project is negligent project management. Worse is project management that is unfamiliar with what the role entails.

"A round-the-clock effort is under way, with the government leaning more heavily on the major contractors"

"Worried about their reputations, contractors are now publicly distancing themselves from the troubled parts of the federally run project."

"Senior executives at Oracle, a subcontractor based in California that provided identity management software used in the registration process that has frustrated so many users, defended the company’s work. 'Our software is running properly,' said Deborah Hellinger, Oracle’s vice president for corporate communications."

How often have you seen this story play out? Lack of planning and poor leadership lead to "crunch times". As a result, development staff is required to work late. Demands raise past what is reasonable and soar up to the ceiling of what is possible. The result?  Discontent and CYA. The quotes above tell me that the contractors have already given up on the project and its leadership. Worse yet, the "blame game" has gotten into full swing. Blame doesn't happen when the project staff sees the project as salvageable. Blame only happens when the project is seen as a loss and people only want to salvage their careers. In a very real way, the blame game prevents problems from getting fixed.

Project management problems are a warning sign that is difficult to see. Oftentimes, poor project leadership isn't obvious until the project is well under way. At that point, recovering the project can be problematic. Tasks have already been assigned inappropriately, Reporting is either far behind or nonexistent. And at the worst, although I didn't see any evidence of this in the article, poor project management often involved a lack of clear project goals. This last is, in my opinion, the worst way project management can fail. If a project without clear end goals is allowed to continue, failure is the only possible result.

Health Care Exchange Project Pt. 1

“I have witnessed boards that continued to waste money on doomed projects because no one was prepared to admit they were failures, take the blame and switch course. Smaller outfits are more willing to admit mistakes and dump bad ideas.” - Luke Johnson

Let me start off saying that this isn't political commentary. If you want to talk about whether or not the Affordable Care Act (ACA) is a good idea or should be repealed, please go elsewhere and don't pollute my blog with political commentary.

Earlier today, I came across NYT article headlined "From the Start, Signs of Trouble at Health Portal" and written by Robert Pear, Sharon LeFraniere, and Ian Austin. Before going on, I highly recommend reading the article. I also found a very readable companion piece written by Megan McArdle.

The reason it caught my attention isn't that it is about ACA but rather the amount of project failures that I can relate to. The signs of common project management mistakes are so obvious, anyone experienced with software development projects will find themselves nodding their head with nearly every paragraph. This article doesn't describe a mere failing project. The failures described here are so blatant, so numerous, that I have to wonder if we're looking at deliberate sabotage. Or possibly the article is merely social satire.

The point of this isn't to slam the ACA, let me make that clear. The desired end of the project isn't what caught my attention. Nor am I trying to "discredit" the ACA. This is a real world study of how to run a software project into the ground and guarantee its failure before the first line of code is written. This article should, absent all else, serve as a warning to those undertaking software development projects, and indeed projects of any kind.

As this is far longer than my usual blog posts, I'm breaking it up into three parts, by what I see as logical categories of the project's failures. For the first one, lack of executive leadership.

It's frustrating when software projects are crippled by executive-level politicking. Executive-level leadership is foundational to a project's success. And like a building with a flawed foundation, poor executive leadership will destroy a project in unexpected ways. I have been involved in several projects that lacked executive-level leadership. All collapsed. Quoting the NYT article:

"Politics made things worse. To avoid giving ammunition to Republicans opposed to the project, the administration put off issuing several major rules until after last November’s elections. The Republican-controlled House blocked funds. More than 30 states refused to set up their own exchanges, requiring the federal government to vastly expand its project in unexpected ways."

"Administration officials dug in their heels, repeatedly insisting that the project was on track despite evidence to the contrary." 

"Mr. (Henry) Chao’s (Chief digital architect) superiors at the Department of Health and Human Services told him, in effect, that failure was not an option, according to people who have spoken with him. Nor was rolling out the system in stages or on a smaller scale, as companies like Google typically do so that problems can more easily and quietly be fixed." 

I see a few serious warning signs here. First, not all stakeholders were on board with the project. I don't care what rationale you use, if the stakeholders aren't on board, the project is doomed. This is as close to political as I want to get here. Neither the reasons the Democrats had for steamrolling the Republicans on the ACA nor the reasons the Republicans have for trying to kill it matter in this context. The unassailable, unarguable, truth is that if a major stakeholder want to kill a project then the project is dead. Ideology does not change this, nor does intent. Whether the result of the project is necessary, helpful, or detrimental is also irrelevant. And it is inexcusable for executive leadership to begin a project like this, through intent or ignorance of This Law, with such high level opposition.

Just as bad, however, is the unwillingness to acknowledge that the project is in trouble, but rather relying on the "Failure is not an option" method of resuscitating a project. Take a moment and count the number of times someone told you "Failure is not an option" or "Just make it happen" and it actually helped. Go ahead. Raise your hand if you got a number above zero. Anyone out there have their hand raised?

Didn't think so.

Attention all managers- these phrases do not solve problems, nor do they create an environment in which problems can be solved. For the love of all that is holy, stop using them.

Several reasonable suggestions are on the table. Postponement. Small scale rollouts, Google-style. The reasons given for ignoring these options are just about the worst possible. Executive-level politics.

Executive leadership enables development projects in ways that many people on the project never see. Because of that it's difficult to see when executive leadership is failing. Sadly, the results are less hidden.

Finding Valrhona, or Habits of Effective Architects

I have very strong opinions on the subject of software, architecture, and quality in general. Coors is not beer. A Hershey bar is not chocolate. Neither Velveeta nor Kraft Singles are cheese. Starbucks does not serve anything I identify as coffee. "Cowboy Coding" is not software development". This really isn't about my quirks in food quality. More a list of items that I find helpful in making sure I'm helping to deliver Valrhona.

Understand what you need to deliver

Before you select technologies, before you start with the design patterns, and certainly before you put hand to keyboard, make sure you understand what pain point you're relieving for your customer. Software development is about solving problems. So many times, I see projects skipping right over the problem to be solved and heading right to implementing a solution. Oftentimes, talking through the problem to be solved makes a murky solution obvious. If you're stuck, ask yourself "What problem are we solving?" If you don't know, you know what to do next.

Solve for realistic problems

If you don't need a full enterprise-y solution with distributed widgets, Something As A Service, and Fully Configurable Everything, don't build it. This is kind of an extension of the previous point, but for every architecture decision you make ask yourself "What value is this adding to the solution?" if you don't have an answer then you don't need the Thingy.

On the other hand, understand that no software project is ever finished and that no set of requirements stay static. Especially once development starts. As Helmuth Graf von Moltke said, "No campaign plan survives first contact with the enemy". There are well established patterns for solving common problems. A thorough understanding of the Gang of Four's design patterns will go a long way in helping you avoid common pitfalls. Don't use them just to use them, but don't avoid them just because they're common. They're common for a reason.

Stop. Collaborate and Listen.

Okay, for those of you who get the reference, I apologize. Note there's no link. I don't want to infect those of you who don't get it. But, origin aside, it's good advice for the architect. Even if you're sure of yourself, get feedback. A development team is more than the sum of its parts, and several smart developers working together produce far better results than several smart developers working separately. Capitalize on this. Ask for comments and then listen to them. Especially the criticism. The worst that can happen is that you'll feel more confident in your design.

Along those lines, keep up on the current trends in software developemnt. I'm not saying you have to be KanBanAgileScrumTDD just because others have written about how shiney they are. But you won't know how these concepts can, or can't, help if you aren't familiar with them.

Strive for Elegance, but understand what it means

To me, an "Elegant" solution is not necessarily overly-clever. It does not have to solve problems in a new way. And it certainly doesn't take Donald Knuth to understand. To me, "Elegance" makes the solution look easy. Sure, maybe you come up with a better way of solving a problem. But maybe you recognize that some techniques are "Tried and True" for a reason. Either way, your result shouldn't look like a bunch of work. It should look obvious.

Know when to say when. And when not to.

Understand that at some point in your career (or, in my case, at some point in your day), the pursuit of higher quality will conflict with the overall effort in such a way that the pursuit does more harm than good. be able to recognize that time and let go.

Understand that at some point in your career, you will be expected to sacrifice quality for the overall effort in a manner that does more harm than good. Don't dig in. Don't get stubborn. Learn to present your case in terms that the decision makers understand. You will not always get your way, but you will become known as an asset that is always looking out out for the overall project.

Note that there isn't a lot of actual code advice here. Sure, I could tel you that if you're instantiating different types of classes depending on context, consider an Object Factory or even an Abstract Factory. Or if you have a somewhat complex process that other processes interact with, or a subsystem that might change, consider a Facade. I could give you the ol' "Design from the Interface" advice or even tell you that if you find yourself considering recursive queries maybe you should step back a bit. But I think that if you really take the above to heart, everything else is just details.

Is Open Source Any Help

"Courage is what it takes to stand up and speak; courage is also what it takes to sit down and listen." --Winston Churchill

I've had some thoughts about Open Source Software percolating for some time now. Before getting to them, though, I want to lay out what Open Source is, especially for the non-programmers reading this. As a programmer myself, the concept of source code is so deeply embedded in what I do, it's a little like trying to explain "air". So I turned to Google. And while the linked search brings back a lot of sites that offer good definitions, I like the one returned by Google at the top of the page. "A text listing of commands to be compiled or assembled into an executable computer program." "Open Source" is, then, software that makes the source code publicly available. Some Open Source Software is free, some is not. Some allows others to make changes to the source code, some do not. But, by definition, all Open Source Software makes the code available for public perusal.

Part of the reason I'm thinking about it is due to an opinion piece written by Open Source advocate Richard Stallman. Now, personally, I think Stallman relies too heavily on hysteria and exaggeration to make his points, but agree or disagree, this piece got me thinking. His point is that all software should be Free Open Source (FOSS). "Free", in this case doesn't refer to cost to use, but rather the freedom for others to use the source of a piece of software for their own purposes.

The other thing that got me thinking was Healthcare.gov making all of their code Open Source and even freely available for others to use. It's a bold step, darn near unprecedented.

To start out my thought process, I'm going to quote, verbatim, my comment on +Matt Quackenbush's post, where he linked the article. While my opinion has wandered a bit, I want to start out with an accurate historical record of where I started.

Yay. Another Stallman rant. SIGH

Okay, here's the thing that Stallman either doesn't realize or just doesn't care about.

To most people, source code is useless. Even to most programmers. He wants to liken code to language? Then source code is a book. For any important software, reading it would be akin to reading the entire Encyclopedia Britannica. Doable, even informative. But hardly a priority, and rarely worth the effort.

So what you end up with is similar to RMS' beef with Ubuntu's Amazon search. You end up with one batch of so-called experts yelling "IT'S DANGEROUS" and another yelling "NO IT'S NOT!". For everyone else, all that can be done is decide is more credible: Someone known for blatant histrionics and exaggeration or a company looking to defend their product. The truth is likely somewhere in between and likely more shaded than either side wants to admit. But how am I to know?

My point was that making a software's source freely available is only useful to those with the ability to read the code and the willingness and ability to devote the time to really understanding it. And programmer reading this understands the difficulty of this, but for non-programmers it's important to point out that when we programmers understand software, we have to hold the entirety of the code in our head. Paul Graham explained this in the best way I've ever seen, and I encourage all to read this essay of his. My point here is that even for an experienced developer, holding a significant part of, say Windows 7 or Adobe Acrobat, represents a serious amount of  time and effort. Time and effort that is often better spent elsewhere.

So where does that leave the rest of us? Before I answer that, let me pose a few questions. Raise your hand if you're a trained and educated Climatologist. Figuratively, but literally if you wish. For those of you with neither your figurative or literal hand in the air, how much do you know, of your own experience and knowledge, on the issue of climate change? Have you done an exhaustive reading of the current research? If you have, how much of it do you understand? What about peer reviews of research on both sides? In other words, what do you really know? If you're like me, darn little. Not really.

That leaves us in the position of judging credibility based on... Well, based on what? Largely, I suspect, our own biases. Likely a bit of, "Well, that just makes sense to me", which is a terribly way to judge scientific research since results can often defy what we would see as "common sense results". Please note- this is not about what you believe to be true. It's about the fact that you believe something to be true, rather than know something to be true.

The same is true of FOSS. Personally, I don't have the time to put in a serious review of the Ubuntu operating system. On a practical level, it doesn't matter to me if it is FOSS or closed source. All I can do is try to critically judge the credibility of people who have and decide who I believe. I can read people's opinions, although I freely admit that it is not practical to do an exhaustive enough study to give me enough information to make an accurate determination of the credibility of the people involved. And this is coming from someone that understands the issue here. Those that have chosen other paths in life don't even have that advantage.

That's where I was as of yesterday. Today, I read something that extended my thoughts on the matter, namely the fact that HealthCare.gov has embraced FOSS to the point of making their back-end code, called APIs (Application Programming Interface), available to those that want to use it. HealthCare.gov is not an exception to the issue I outlines above. However, when I read their statement I found myself thinking "It takes real courage to put yourself under that kind of microscope". People will be reading their source code, people will be judging not only its quality but what it actually does. So Kudos to you HealthCare.gov. Not for finding a way to make your source truly open to all, but for having the courage to stand up and be judged by those that can and will. Sometimes courage is its own reward, and I hope it is for you guys.