Certain Uncertainty

"There are known knowns. These are things we know that we know. There are known unknowns. That is to say, there are things that we know we don't know. But there are also unknown unknowns. There are things we don't know we don't know." --Donald Rumsfeld

--Ed. Note: Changed to link to Gene Hughson's G+ profile rather than just mention his name. Should have thought of that the first time.

+Gene Hughson , author of a blog that is shortly going to be added to my "What I Read" section, has a fascinating article on the dangers of Premature Certainty. It's quite long, but well worth reading. Worth reading a couple of times, in fact.

Right out of the gate, he gets to the meat of it:

"Premature certainty, locking in too soon to a particular quality of service metric or approach to a functional requirement can cause as many problems as requirements that are too vague. For quality of service, also known as “non-functional”, requirements, this manifests as metrics that lack a basis in reality, metrics that do not account for differing circumstances, and/or metrics that fail to align with their objective. For functional requirements, premature certainty presents as designs masquerading as requirements (specifying “how” instead of “what”) and/or contradictory requirements. For both functional and quality of service requirements, this turns what should be an aid to understanding into an impediment, as well as a source of conflict."

There are several important points here that every software architect needs to keep in mind. The temptation in a software project is to start talking about performance and quality of service- the dreaded Non-Functional Requirements, or NFRs- far too soon. It's an understandable temptation as it's near and dear to the business unit's heart. It's also a subject largely misunderstood by the business unit's heart. When Donald Knuth said "Premature optimization is the root of all evil (or at least most of it) in programming.", he was talking to programmers more than the business unit, but both sides ought to take heed of his warning.

So why is this so dangerous? Generally, isn't focusing on optimization and performance A Good Thing? Well, sure, but there's a great difference between focusing on performance and deciding on how to measure it. Especially if you decide on how to measure it before you're finished.

The problem is that NFRs are more like decorating a living room than building one. When you build a living room, you have definite dimensions you conform to.  You have building materials that you need to use, you have building codes you need to adhere to, and these are all very measurable and easily tracked. In decorating a living room, however, you have a general set guidelines to work within, but few hard and fast rules, if any. "I want the room to be light and airy" or "I want a more modern feel" are both valid sets of instructions for room designers, but it would be absurd to tell a designed "I want a minimum of 5.8 on the Light/Airy scale and am willing to settle for a 3.2 Modernness level". Yes, the designer is expected to engage the room's owner on every step of the design process, but you can't really know the final result until you see the final result. And there's certainly no scale against which to measure the final result.

This is the difference between Qualitative and Quantitative requirements, and making sure that everyone understands the difference between the two, and into which any given requirement falls, is the difference between building meaningful and meaningless requirements. Hughson references Tom Graves' Metrics for Qualitative Requirements:

To put it at perhaps its simplest, there’s a qualitative difference between quantitative-requirements and qualitative ones: and the latter cannot and must not be reduced solely to some form of quantitative metric, else the quality that makes it ‘qualitative’ will itself be lost.

And therein lies the proverbial rub. The concept of the managed project so often comes from purely quantitative worlds. Construction is an excellent example. The projects I saw at a defense contractor are also prime examples of quantitative projects- you received a set of specs and you build to the specs. NFRs didn't factor in. Software development simply isn't like that and the disconnect comes in when we try to treat software development projects as if they were purely quantitative. As architects, it often falls on us to manage this disconnect because we're often caught between the business, which often hasn't considered the difference between the two types of requirements, and the development staff, who will almost certainly understand the difference but on a more experiential level that is more difficult to communicate to someone lacking the same experiences. And as the group that is often tasked with communication between the two groups, it's of critical importance that software architects understand the difference between the two kinds of requirements and can clearly communicate the difference and its importance to the business unit.

Hughson also goes on to discuss the importance of developing functional requirements that have a level of certainly to be useful while still keeping the level of uncertainty needed to stay flexible and allow architects to do their job. Also a good read, and also necessary to keep in mind. However, as I'm in the middle of attempting to manage the development of NFRs on a current project, I very much focused on the first part and wanted to write this out as an attempt to better understand the points Hughson was making. I think it helped me and I hope it helped you, too.