The Binary Fallacy

Two roads diverged in a yellow wood,
And sorry I could not travel both
And be one traveler, long I stood
And looked down one as far as I could
To where it bent in the undergrowth;
-Robert Frost, The Road Not Taken

Binary Thinking is the process of thinking of things in terms of two opposites. You can either do A or it's opposite B. But that's it. In Neuro-Linguistic Programming, it's a technique for manipulating someone into making the decision you want by framing the choice as a matter of either the thing you want them to do or the opposite, presented as a harmful choice. In Cognitive Behavioral Therapy it's a pattern to be broken to help people gain more control over their lives. In Software Architecture it's a warning sign.



If stuck between two different ways of designing something, the answer is always door number three.

Why do humans think like this? Why didn't Frost just keep walking straight through the woods and not worry about the roads? Why didn't The Clash stop worrying about staying and going and just Rock the Casbah? I don't really know. I'm not a psychologist. I'm just a software architect who's familiar with Analysis Paralysis. The phenomenon of getting nothing whatsoever done because you're locked up in the decision making process. And after years of trying to figure out how to avoid this, I came to one inescapable conclusion.

When you're in analysis paralysis it's because every path you're considering is wrong

In my opinion, this happens when you realize deep down that you're looking at the problem wrong and considering bad options. You can't decide because you know your options are both bad.

So what do you do? Well, that's the tricky part and adages about in which part of the box to think aren't as helpful as people who use them think they are. If thinking of something new was as easy as the realization that we need something new then Elon Musk wouldn't be quite the icon he's become. The important question is HOW? First, you just need to stop and accept the fact that you need to come up with something different. Your first decisions were wrong and you need to set them aside. Don't go back to them. Then I recommend you look at the great Stoic thinkers. You start by asking "What is this and what is it for?" I often tell developers to answer those two questions for every application, for every class they build, for every database and table they create. And you don't build the thing in question until you can answer those questions.

Then you ask if a thing is needed or just wanted. If it's not something you need, based on the answers to the above questions, then you leave it out. You strip away the irrelevant. Often I find myself locked up because I can't find a clean way of integrating the irrelevant into my design. Once I stop and realize the piece that doesn't fit doesn't need to fit, things get easier.

I've often said that software development is 90% thinking and 10% typing. If you understand how to think clearly, how to organize your thoughts, and how to tell the difference between the necessary and the unnecessary then there's little that can keep you from being a great software developer.

You've Got One Job

"If you choose to not deal with an issue, then you give up your right of control over the issue and it will select the path of least resistance." -Susan Del Gatto, Creating Balance in a World of Stress: Six Key Habits to Avoid in Order to Reduce Stress

Never forget that your only job is to provide a solution to a problem



Stick with any career long enough and you start to develop a set of principles that guide your approach to your job. Call it experience, call it being set in your ways, even call it wisdom if you dare. But we all do it. I've got about twenty of them, which I realized after an NCIS binge when I started writing them down Gibbs' Rules style. Rather than blasting them out all at once, over the next several articles I'm going to talk about each one. There's no particular order, other than the order in which I wrote them down. However, if I had to pick one to be the most important, it's the one I'm starting with.


Even at the beginning of my career, I never really thought of myself as a software developer. Rather, I thought of myself as a problem solver, albeit one who tended to approach solutions with code. But that idea has always stuck with me. Always be solving a problem.

Now, not all problems are created equal. Some are quite big. For instance, Linus Torvalds solved the problem of having to pay money for an operating system. Some seem almost insignificant. Recently I've been working on the problem of "I don't know Python as well as I'd like to". Something of import to almost no one other than myself. But that's the point. It's someone's problem, and it's getting solved.

Of course, what I'm really getting at is why you write software professionally. And while we all basically realize that we need to fulfill requirements, implement stories, or however we get the list of stuff we're supposed to write, it's easy to lose track of the big picture when you're in the weeds of the little picture.

The problem is that we, as software developers, don't just like to develop software. We like to develop cool software. However we define that, but usually as "the last new idea I read". Admit it- you do it, too. Goodness knows I do. We like to learn new things, we like to try new techniques, because we like, no we need to grow. This rule is to remind me not to take that too far. Ultimately, it's not about what I want, it's about what solves the customer's problem. Because if I'm not doing that, I'm wasting everyone's time.

I've also ran into situations where solving the problem didn't have anything to do with new software, custom software, or even software at all. I've on occasion been able to trim large portions out of a project with suggested business process changes. Or by pointing out that the requested software doesn't actually solve their problem, and on occasion might even compound it. A good software developer knows when to not develop software.

The idea of solving a problem is a big one, and one that spans several of the principles on my list. But this one gets its own item because it all boils down to this simple concept. You're not writing software to write software. You're solving a problem.

                                                                 

New Challenges

"Be careful of what you wish for. You might get it".

Hard to say where that comes from, although some cite it as a misquote of Goethe. The image, of course, is from the amazing mind of Bill Watterson and his Calvin and Hobbes comic.

Regardless, that is kind of the source of why I haven't been writing much lately and why I'm starting again. I tend to write about what I'm seeing and experiencing. Which, over the last few years, has become fairly routine. After writing a good bit about my thoughts on Software Architecture, it became the task of taking my own advice.

So what changed? A chance to get what I asked for, namely a chance to go from helping insure quality in a software system to helping insure quality in a development team. The company I work for has a relatively new development team that operates fundamentally differently than our other teams. This team doesn't support a product. Instead, they handle smaller projects that have a more rigid "Definition of Done", as opposed to continued development and support of a product. This has come with more than a few challenges. Challenges that I have a pretty free hand to solve, but challenges that will take a lot of solving.

All this has put me in an interesting opportunity. Processes are getting built from the ground up. We have leave to borrow from other teams the processes we feel are helpful, as well as to ignore the things we feel won't benefit us. The end goal is increasing the rate of shipping quality software, and with very few exceptions I don't think the head of software development much cares how it happens. This gives me the freedom to take my biggest piece of advice to anyone designing software. Or, for that matter, anyone currently living and interacting with the world in any way:

“This, what is it in itself, and by itself, according to its proper constitution? What is the substance of it? What is the matter, or proper use? What is the form, or efficient cause? What is it for in this world, and how long will it abide? Thus must thou examine all things that present themselves unto thee.” --Meditations, Marcus Aurelius

In other words, "Question everything". Some ideas are worth leaving alone. Questioning the necessity of engagement between the business and development is a lot like questioning gravity. Feel free, just stay away from ledges. But how do you do these things when the business units aren't used to that engagement and don't understand the value in it. And that's not looking down on anyone- how can you know the value of something until it's been shown to you?

Beyond that, what traditional roles in Agile projects do we need? Which can we live without, and which should we live without? What do these roles mean, and how do we make sure everyone knows what an iceberg is?

What about the team? What common practices do we need? What offers us the best value in our situation. Not just what others are doing or what generally works well, but what works well for us?

These are the questions I'm asking myself and the ideas I'm exploring. I'm sure we'll go down some odd paths, and probably even the (hopefully) occasional bad path. And hopefully, my philosophy that understanding the world leads to understanding software still holds.

The Tipping Point

"You can push that car just a little too far any Sunday afternoon. And if you break your neck in some damnfool wreck, they forget about you soon." --Charlie Daniels, "Stroker Ace"

I generally prefer to work at small to midsize companies. The reasoning is fairly simple, too. I like flexibility. Not a complete lack of defined policies, mind you. Policies are important. They give companies a road map for, if not efficiency, then at least predictable inefficiency. But smaller companies tend to have the flexibility to know when you need to stray off the beaten path. Large companies do not. And there's a certain logic behind that. The more you do, the more you rely on that predictability. Policies have saved my neck a time or two. Being able to say "I can't start this effort because our policies have not been followed" has forced projects to refine requirements and has gone a long way to prevent train wrecks. But there's a flip side, too.

At larger companies, it's harder to change policies that either no longer fit, or need refinement from the original draft in order to produce the intended result. There's also more risk in proposing new policies, I think. There generally tend to be more people that need convincing, which tends to mean more people that don't want to "Change what has been working for us". It often comes to a point where it's easier to just live with the inefficiency than try to change it. Or worse, at least in some ways, ignore policy and try not to get caught.

Okay. So far, I haven't told anyone something they don't know. This is one of the factors that need to be weighed when deciding whether or not to accept an offer or to leave a company. It's, hopefully, one of the factors you investigate when interviewing a company. As an aside, please note that I didn't say "interviewing with a company." You're as much deciding whether or not to offer them your services as they are offering you a position. Never forget, you're a contractor.

The point is, while working for smaller companies, I've noticed a reoccurring phenomenon, which I've started calling The Tipping Point. As smaller companies grow, they start to take on more work and start to do more. At this point, I have observed two results.

If the company does not at least have some policies guiding their work, the company tends to collapse. There comes a time when it's simply too late to gain control of source code, whether because nothing is in source control or there's no structure. When it's simply too late to implement good project policies because stakeholders aren't interested in becoming properly involved in projects and that disinterest is too entrenched to change. When projects are dangerous to deploy because no one knows what anyone else is doing. At this point, while the company may struggle on, it's no longer possible to fix the problems, and the only smart move is to leave.

The second result I've observed usually happens when there are a modicum of policies in place, and is usually hastened by bringing in An Expert. Perhaps a new CIO, perhaps a contractor. The buildup, again in my observations, tends to start slowly. No deploying projects without communicating with other groups. Nothing terrible there. Common sense, in fact. Issues, bugs, and defects need to be logged. Again- why would you not? And we need to get a sense of how much we're spending on projects, so we need people to log time spend on projects. Not my favorite activity, but I can't deny the importance.

But then things grow. Communicating deployments becomes getting signoff from other groups before deploying. The bug tracking system becomes a full-on change management system designed to integrate with every step of any process. Except yours, inevitably. Rather than turning in time tracking spreadsheets, either a web app gets build, or you're to use a built-in feature of the change management system.

Finally, comes the tipping point. I wish I could say this is an exaggeration, but in one case, I was entering time into an application that had a database lookup for projects, yet my projects never seemed to show up. I also had to email the same to my manager. The time tracker was in half hour increments, but my manager wanted 15-minute increments. Every change, even to dev environments, needed to be entered into five different systems (again- not exaggerating) and cleared in a bi-weekly Change Control Board meeting.

And all this doesn't even take into account well-documented problems when you mix in an Expert Scrum Consultant. I think I've made my opinions on the subject pretty clear, but it's pointless to deny what often happens when Scrum policies get blindly applied. And that's the problem- policies getting blindly applied. The "Monkey See Monkey Do" style of "Best Practices".

It's a rare gem when you find an organization with the self-discipline to recognize a policy that needs refining or simply removed. Or temporarily ignored on a non-precedential basis. A good set of policies are a balancing act. Too many and you get buried under their weight. Too few and you get buried under your own weight. That's The Tipping Point.

Life is Agile

"It's not what happens to you, but how you react to it that matters" --Epictetus

I'd like to pose a question to any parents reading this. When your child was born did you plan out every detail of your child's life? Did you attempt to account for every problem and develop a contingency plan before even leaving the hospital?  Do you parent in a vacuum, having no contact with other parents?

Similar question to those of you with active careers, be it in the job market or at home. Yes- being an at-home parent is a career. Did you plan out every step of your career before you began working? Do you work without interaction with others in your career? Did you plan for every problem to the most minute detail?

Of course not. To approach such a major endeavor in life is more like planning to fail. And this is why Agile is the only way to successfully complete a software development project of any consequential size. Because it mimics human behavior and human capabilities. At it's core, the Agile methodology is an admission that you can only do so much to plan for the future. It's a system built on the foundation of communication with others. It does not merely recommend regular and effective communication, it states that failing to do so will cause overall failure. It is built around doing what you need to now, while keeping an eye on the future. Similar to how you eat an elephant (one bite at a time), so do we manage our lives. Why should we assume that managing a development project should run so completely contrary to our fundamental behavior?

The converse is also true. It's been a very long time since I've had to defend Agile vs. Waterfall/BDUF. The reason appears very straightforward- the proof is in the doing. Agile has proven itself. It's not theoretically better, it simply is.

How, then, can we extrapolate this to our lives? Should you find yourself working at a company where you have no future, what do you do? Do you scrap the idea of working at your best capacity while you look for another job? Agile says differently in that a development team has the discipline to finish a development sprint, i.e. the work in front of them.

If you want something in your career, do you just keep working and hoping it happens? One of the foundations of Agile is communication. Do we keep grinding and hope, or do we approach others for advice? And then do we approach management and clearly communicate what we want?

Life is Agile is life. The lessons we learn from each can be applied to the other.

Dedication and Vision

"If you do not have an absolutely clear vision of something, where you can follow the light to the end of the tunnel, then it doesn't matter whether you're bold or cowardly, or whether you're stupid or intelligent. Doesn't get you anywhere." --Werner Herzog

Short article today.

Ran across something that really made me think about software development projects. Three developers in San Francisco built an alternative to Healthcare.gov in two weeks.

Two weeks.

Now, you can't purchase- just search and compare rates and plans. But again. Two weeks.

No complicated procurement process. No contractors. Certainly less than hundreds of millions of dollars. Just three dedicated guys with a clear vision.

A G+ friend of mine +Chaka Hamilton took this a step further:

I bet if the gov offered a darpa style challenge, they'd have a working website in half the time, and cost. They obviously have learned nothing from open source / crowd source community.

 Indeed. What if they had.

A New Perspective

"I believe everyone should have a broad picture of how the universe operates and our place in it. It is a basic human desire. And it also puts our worries in perspective." --Stephen Hawking 

"Everything we hear is an opinion, not a fact. Everything we see is a perspective, not the truth." --Marcus Aurelius 

This  article falls more heavily under the "Musings" title of my blog. I'm less making a point than I am thinking out loud. As always, feedback and insights are always welcome.

My job responsibilities are changing. I phrase it like that because I doubt my actual title will change, merely the meaning of that title. I rarely handle project level work anymore. Rather, I'm more involved with enterprise level architecture decisions. I haven't implemented a design pattern in quite awhile. I find myself, instead, setting the standard of what patterns are best to use or avoid in certain situations. Or what frameworks we will be using or whether we will use an off the shelf solution or build our own. In other words, my implementation decisions are becoming less important than my opinions and experience with those decisions. I find this a very new perspective and more than anything else, I find myself more and more writing about new perspectives on old ideas.

For instance, take our current investigation into unit tests. The discussion started with "What mock object framework should we use?" We quickly boiled down to "Which framework will be unduly burdensome to the development staff?" This actually eliminated a couple of frameworks at the beginning. But when we settled on two that are, more or less, of equal use the conversation quickly changed to unit test standards. I have a few strong opinions on the matter. I believe that unit tests should cast a wide loop so that behavior consistency can be assured. If that means mocking Internal methods to assure their consistency, then so be it. If that means only using strict mock objects, despite their fragility, then so be it. In fact, I like fragile unit tests. If the behavior of the class changes then the tests should break. I realize that my opinions are not shared by the community at large, and I'm okay with that. I'm always open to debate, but I approach things somewhat differently than normal. I think I've made that clear. Now, of our enterprise level architects, one disagrees with me and the other is still weighing arguments and that's great. That kind of conversation is a new perspective for me.

So what's this new perspective and what does it give me? Because I'm no longer considering patterns and practices for a given set of circumstances, but rather to be followed in the enterprise, I have to more seriously consider the pros and cons of those patterns and practices. I find that thinking of effects of standards on the enterprise at large makes me think differently about the effects of my design decisions at the project level. Not just "Does this work here" but "Would this work in other, similar, situations and if not, why not?" If the answer to the second question is "No", then should I reconsider my decisions at the project level. Note that I'm not offering concrete conclusions here. I'm expanding my perspective and thus expanding the pool of questions I ask myself before making a decision.

Maybe that's my point here, although I expect I'll be getting comments about my approach to unit tests. That's fine, too. The day I stop listening to others is the day I stop being useful.

Finding Valrhona, or Habits of Effective Architects

I have very strong opinions on the subject of software, architecture, and quality in general. Coors is not beer. A Hershey bar is not chocolate. Neither Velveeta nor Kraft Singles are cheese. Starbucks does not serve anything I identify as coffee. "Cowboy Coding" is not software development". This really isn't about my quirks in food quality. More a list of items that I find helpful in making sure I'm helping to deliver Valrhona.

Understand what you need to deliver

Before you select technologies, before you start with the design patterns, and certainly before you put hand to keyboard, make sure you understand what pain point you're relieving for your customer. Software development is about solving problems. So many times, I see projects skipping right over the problem to be solved and heading right to implementing a solution. Oftentimes, talking through the problem to be solved makes a murky solution obvious. If you're stuck, ask yourself "What problem are we solving?" If you don't know, you know what to do next.

Solve for realistic problems

If you don't need a full enterprise-y solution with distributed widgets, Something As A Service, and Fully Configurable Everything, don't build it. This is kind of an extension of the previous point, but for every architecture decision you make ask yourself "What value is this adding to the solution?" if you don't have an answer then you don't need the Thingy.

On the other hand, understand that no software project is ever finished and that no set of requirements stay static. Especially once development starts. As Helmuth Graf von Moltke said, "No campaign plan survives first contact with the enemy". There are well established patterns for solving common problems. A thorough understanding of the Gang of Four's design patterns will go a long way in helping you avoid common pitfalls. Don't use them just to use them, but don't avoid them just because they're common. They're common for a reason.

Stop. Collaborate and Listen.

Okay, for those of you who get the reference, I apologize. Note there's no link. I don't want to infect those of you who don't get it. But, origin aside, it's good advice for the architect. Even if you're sure of yourself, get feedback. A development team is more than the sum of its parts, and several smart developers working together produce far better results than several smart developers working separately. Capitalize on this. Ask for comments and then listen to them. Especially the criticism. The worst that can happen is that you'll feel more confident in your design.

Along those lines, keep up on the current trends in software developemnt. I'm not saying you have to be KanBanAgileScrumTDD just because others have written about how shiney they are. But you won't know how these concepts can, or can't, help if you aren't familiar with them.

Strive for Elegance, but understand what it means

To me, an "Elegant" solution is not necessarily overly-clever. It does not have to solve problems in a new way. And it certainly doesn't take Donald Knuth to understand. To me, "Elegance" makes the solution look easy. Sure, maybe you come up with a better way of solving a problem. But maybe you recognize that some techniques are "Tried and True" for a reason. Either way, your result shouldn't look like a bunch of work. It should look obvious.

Know when to say when. And when not to.

Understand that at some point in your career (or, in my case, at some point in your day), the pursuit of higher quality will conflict with the overall effort in such a way that the pursuit does more harm than good. be able to recognize that time and let go.

Understand that at some point in your career, you will be expected to sacrifice quality for the overall effort in a manner that does more harm than good. Don't dig in. Don't get stubborn. Learn to present your case in terms that the decision makers understand. You will not always get your way, but you will become known as an asset that is always looking out out for the overall project.

Note that there isn't a lot of actual code advice here. Sure, I could tel you that if you're instantiating different types of classes depending on context, consider an Object Factory or even an Abstract Factory. Or if you have a somewhat complex process that other processes interact with, or a subsystem that might change, consider a Facade. I could give you the ol' "Design from the Interface" advice or even tell you that if you find yourself considering recursive queries maybe you should step back a bit. But I think that if you really take the above to heart, everything else is just details.

Is Open Source Any Help

"Courage is what it takes to stand up and speak; courage is also what it takes to sit down and listen." --Winston Churchill

I've had some thoughts about Open Source Software percolating for some time now. Before getting to them, though, I want to lay out what Open Source is, especially for the non-programmers reading this. As a programmer myself, the concept of source code is so deeply embedded in what I do, it's a little like trying to explain "air". So I turned to Google. And while the linked search brings back a lot of sites that offer good definitions, I like the one returned by Google at the top of the page. "A text listing of commands to be compiled or assembled into an executable computer program." "Open Source" is, then, software that makes the source code publicly available. Some Open Source Software is free, some is not. Some allows others to make changes to the source code, some do not. But, by definition, all Open Source Software makes the code available for public perusal.

Part of the reason I'm thinking about it is due to an opinion piece written by Open Source advocate Richard Stallman. Now, personally, I think Stallman relies too heavily on hysteria and exaggeration to make his points, but agree or disagree, this piece got me thinking. His point is that all software should be Free Open Source (FOSS). "Free", in this case doesn't refer to cost to use, but rather the freedom for others to use the source of a piece of software for their own purposes.

The other thing that got me thinking was Healthcare.gov making all of their code Open Source and even freely available for others to use. It's a bold step, darn near unprecedented.

To start out my thought process, I'm going to quote, verbatim, my comment on +Matt Quackenbush's post, where he linked the article. While my opinion has wandered a bit, I want to start out with an accurate historical record of where I started.

Yay. Another Stallman rant. SIGH

Okay, here's the thing that Stallman either doesn't realize or just doesn't care about.

To most people, source code is useless. Even to most programmers. He wants to liken code to language? Then source code is a book. For any important software, reading it would be akin to reading the entire Encyclopedia Britannica. Doable, even informative. But hardly a priority, and rarely worth the effort.

So what you end up with is similar to RMS' beef with Ubuntu's Amazon search. You end up with one batch of so-called experts yelling "IT'S DANGEROUS" and another yelling "NO IT'S NOT!". For everyone else, all that can be done is decide is more credible: Someone known for blatant histrionics and exaggeration or a company looking to defend their product. The truth is likely somewhere in between and likely more shaded than either side wants to admit. But how am I to know?

My point was that making a software's source freely available is only useful to those with the ability to read the code and the willingness and ability to devote the time to really understanding it. And programmer reading this understands the difficulty of this, but for non-programmers it's important to point out that when we programmers understand software, we have to hold the entirety of the code in our head. Paul Graham explained this in the best way I've ever seen, and I encourage all to read this essay of his. My point here is that even for an experienced developer, holding a significant part of, say Windows 7 or Adobe Acrobat, represents a serious amount of  time and effort. Time and effort that is often better spent elsewhere.

So where does that leave the rest of us? Before I answer that, let me pose a few questions. Raise your hand if you're a trained and educated Climatologist. Figuratively, but literally if you wish. For those of you with neither your figurative or literal hand in the air, how much do you know, of your own experience and knowledge, on the issue of climate change? Have you done an exhaustive reading of the current research? If you have, how much of it do you understand? What about peer reviews of research on both sides? In other words, what do you really know? If you're like me, darn little. Not really.

That leaves us in the position of judging credibility based on... Well, based on what? Largely, I suspect, our own biases. Likely a bit of, "Well, that just makes sense to me", which is a terribly way to judge scientific research since results can often defy what we would see as "common sense results". Please note- this is not about what you believe to be true. It's about the fact that you believe something to be true, rather than know something to be true.

The same is true of FOSS. Personally, I don't have the time to put in a serious review of the Ubuntu operating system. On a practical level, it doesn't matter to me if it is FOSS or closed source. All I can do is try to critically judge the credibility of people who have and decide who I believe. I can read people's opinions, although I freely admit that it is not practical to do an exhaustive enough study to give me enough information to make an accurate determination of the credibility of the people involved. And this is coming from someone that understands the issue here. Those that have chosen other paths in life don't even have that advantage.

That's where I was as of yesterday. Today, I read something that extended my thoughts on the matter, namely the fact that HealthCare.gov has embraced FOSS to the point of making their back-end code, called APIs (Application Programming Interface), available to those that want to use it. HealthCare.gov is not an exception to the issue I outlines above. However, when I read their statement I found myself thinking "It takes real courage to put yourself under that kind of microscope". People will be reading their source code, people will be judging not only its quality but what it actually does. So Kudos to you HealthCare.gov. Not for finding a way to make your source truly open to all, but for having the courage to stand up and be judged by those that can and will. Sometimes courage is its own reward, and I hope it is for you guys.

What's the point of an architect

"I know I can't do everything myself. So I know I specialize in my melodies and I do some of my demo work. I pass it on to my producers who are much better at the production level." --Paul Taylor


I'm not asking why we need software architecture. Anyone reading this knows why. Insuring that standards are met. Insuring extensibility and code maintainability. Making sure that the design patterns used are necessary and that the necessary patterns are used. Choosing proper technology and it's use. This is not in question. I'm asking, "What's the point of an architect?" What do I bring to the table to justify my presence on the project, and indeed the salary and benefits I draw to do what I do, and only what I do. Couldn't all this be done by a senior developer or a development lead? In sort, what is the value of me.

Paul Graham once wrote that a software developer must hold a program in his head:

"A good programmer working intensively on his own code can hold it in his mind the way a mathematician holds a problem he's working on. Mathematicians don't answer questions by working them out on paper the way schoolchildren are taught to. They do more in their heads: they try to understand a problem space well enough that they can walk around it the way you can walk around the memory of the house you grew up in. At its best programming is the same. You hold the whole program in your head, and you can manipulate it at will."

This isn't to imply that programmers are necessarily smarter than average. It does mean, however, that we need to be able to visualize abstract concepts in our mind  More important that how well we think is how we think. The problem is that applications are becoming more and more complicated. N-Tier applications. Web services. Multiple platforms. And with more and more to keep track of, the percentage of the code space as a whole that a developer can hold in his head becomes less and less. 

That's where an architect comes into play. More and more I find myself not holding an application's problem space or code space in my head. I find myself holding the project problem space and its place in the enterprise in my head. Where the developers focus on visualizing code, I focus on visualizing the enterprise as a whole to help the applications fit into that space. 

Consider the evolutionary development of a surgical team. At one point in time all surgery took was a guy with a sharp knife and an understanding of the human body, such as it was at the time. Any modern surgeon would be horrified at the prospect of an operating theater running in such a manner. Now it takes surgeons- maybe more than one. It takes an anesthesiologist and nurses. It takes someone to coordinate the surgeries so that, as much as is possible, there is available surgical resources for high priority cases. Modern surgery is complicated and beyond what any one person, no matter how talented, can manage.

So, too, goes modern application development. It takes developers. It takes senior developers or dev leads- sometimes both- to manage and coordinate development activities and even to mentor more junior developers. It takes DBAs and perhaps other SMEs (Subject Matter Experts) to lend specialized knowledge in areas the development team may not have. And it takes architects to make sure that the application has its proper place in the enterprise, making proper use of existing tools, and insuring that the application being built will work in other contexts, if applicable. And just as the modern surgeon would be horrified at the prospect of the old west surgeon giving a patient a bottle of whiskey before digging out a bullet, I'm horrified at the idea of "One Project, One Developer". Good software development is more complicated than that, no matter how talented the developer.

As any regular readers should know (Thanks, guys!), I ask "What is this?" and "Why is this?" a lot and don't take a lot for granted. Questioning my role in a project- and again, not the role of architecture but the role of the architect itself- makes me more focused on what I should be doing, rather than what I may want to be doing. Because let's face it. I came from software development and at heart I will probably always be a software developer. But after asking "What?" and "Why?", I had an interesting conversation with the tech lead on my project. The specifics of the conversation are less important than the conclusion. I said (roughly)

"I'm starting to get that itch in the back of my architect's neck that means I'm getting in your way. You know the direction we need to take, and you know why. So I'm going to step out of your way and let you get it done as you know best."

Clay Pots

"Perfection is not attainable, but if we chase perfection we can catch excellence." --Vince Lombardi


I tried to find a link to the experiment, but could not. Perhaps it’s allegorical. However, I read once about an experiment done by a pottery teacher. She divided the class into two teams. She told the first team to make the perfect clay pot. She told the second to simply make as many clay pots as they could. At the end of the experiment, the perfect clay pot was indeed made, but not by team 1. As it turned out, the constant iterative practice by team two trumped the careful work of team one.

NOTE: Thank you to +Dave Aronson for pointing me to the link I couldn't find!
http://kk.org/cooltools/archives/000216
He also has some thoughts on the subject at http://www.dare2xl.com/2010/08/just-do-it.html

This is not an article about getting better at software development the more you develop software. If you’re reading this, you already know that. No, this is about software architecture and building the perfect design. Which, as I explained in my first article, doesn't exist anyway.

Every Software Developer/Architect/Engineer/Whatever that I've worked with has shared a couple of characteristics. They want to get their work done right, and they take time to think through what they're doing before they do it. Both of which are commendable. The problem comes when this leads to Analysis Paralysis. When the process of thinking things through in order to make the perfect design deadlocks the developer and he can't move on.

When that happens to you, remember the clay pots.

Agile methodologies such as Scrum and XP were developed, in part, to avoid analysis paralysis at the project level. With a focus on action and testing the results, agile methodologies seek to create the perfect pot by creating pots until they get it right. As it turns out, this technique works just as well at the individual level.

Sometimes the best way to break through design indecision is to just start writing it. Build the class stubs, make them interact, and build unit tests around them. How well does it work? How badly doesn't it work? Then consider what worked, what didn't, refine your ideas and start over. Wash, rinse, and repeat until you’re happy. Or at least satisfied. Or at least still on this side of “I’m so frustrated I’m about to throw my laptop through a window”. Seeing how the design plays out and forcing yourself to refine and retest can often lead to better results than trying to think through every detail in advance so that you create the "perfect" design the first time.

Don’t get me wrong. I’m not advocating against careful thought. I’m not saying “Don’t plan” or “Don’t think”. And I'm certainly not saying you should just throw code against the wall until you get something that looks workable.

Consider the T.V. show "Dr. House". His beliefs that there is one absolute right way of handling a problem is completely detrimental to software development. But one of the few things that I agree with Dr. House *in practice* is his insistence on thinking through a problem before acting on it. But if you remember the series, he follows the clay pot model. Think. Do. Refine. Think again. Continue until done. You won’t get it right the first time, and you should be very suspicious if you do, so don’t grind on it. And I love his attitude that making mistakes is expected. No one cares, as long as your end result is solid. 

Here’s the thing I tell architects and developers alike. There are no points for style. No one is counting code check ins, no one is counting compilations, no one is counting design iterations, and no one cares as long as the end product is a good one. Until then, if you have to slam it in with your knees, do so.

Often, you don't know what works until you've seen something that does not.

Layering It On

Once you know who you are, you know what you have to do.” --Steve Perry, The 97th Step

We all know that building your application in layers is important. Portability, separation of concerns, extensibility, and blog articles are all highly dependent on proper application layers. The problem I see isn't a lack of understanding the importance or disagreeing with it. The problem I consistently see is people not understanding how to layer their applications. Part of this is, of course, practice. My first attempt at building an application with a 3-Tier architecture was an epic disaster that would have made the Titanic step back and say, “DAMN- I thought this was bad.” My second one was also pretty terrible. But better than the first.

Practice become easier with understanding, though. Tips, circumstances, and examples are all limited in scope in that they only give you a small slice of the whole picture. But once you understand what a layer is, why it’s important, and how to look at it, the rest is just reinforcing your understanding with practical experience. As any regular readers will know (Have I been around long enough to have regular readers?), I see software architecture as applied philosophy. I know I've used this one already, but:

“This, what is it in itself, and by itself, according to its proper constitution? What is the substance of it? What is the matter, or proper use? What is the form, or efficient cause? What is it for in this world, and how long will it abide? Thus must thou examine all things that present themselves unto thee.” --Meditations, Marcus Aurelius

I originally used this in understanding classes and properly understanding what they do, but it applies to application structure as well. Once you understand what something is, be it a class, a layer, a carburetor, or a hammer, you know what to do with it. So let’s take a pretty typical application stack- Presentation, Controller, Model, and Persistence. We start by considering each layer as a real-world entity, with things it knows about, actions it knows how to take, and actions it does not know how to take. Then we ask ourselves Aurelius' questions about these entities.

Presentation

What is a Presentation layer, in itself and by itself? Not to put too fine a point on it, but the presentation layer presents data, both to the end user and to the model. That’s what it knows how to do. It knows how to arrange data in a way that makes your application usable and useful. It knows how to sort and filter data so the user can get to the important data without wading through the unimportant data. 

Is your presentation layer interpreting data for any reason other than how to properly display it or properly send it to the lower reaches of the application? Then your presentation layer is doing something it doesn't know how to do.

Controller

Of all the application layers, I've seen more misunderstanding about the Controller than any other. And this is a prime example of why understanding needs to come first, because this one is easy to get wrong if you don’t understand it. The Controller is a Switchboard Operator. Okay- there are a ton of more recent comparisons that are just as good, but I’m going with switchboard operator. The controller routes requests from one place to another, and that’s it. It knows where a request came from and based on that, it knows where the request goes next. A controller that routes the request to different receivers based on some conditional logic with the data itself is interpreting and attaching meaning to the data. It doesn't know how to do that.

Model

In and of itself, what is a Model Layer? What's its purpose? The model knows what the data means, how it should be interpreted, and how it should be used. Which is, admittedly, the meat of the application but there are a few things this layer doesn't do as a part of its purpose. It doesn't know where data comes from. It doesn't know where data goes when the it is done doing what it does. In this way, it’s a lot like an assembly line worker. A widget  shows up and the model performs a task on it. Then the widget moves on. Where it came from and where it goes next are not important. The task performed is the only thing that is.

Persistence

What is the form or efficient cause of the Persistence layer? Sure, this layer interacts with data, but the question is "What is the... *efficient* cause". In its most efficient form, the persistence layer retrieves the data it’s asked for and stores the data it’s told to. It doesn't know how to do anything else. If, for instance, you've asked your persistence layer to tell the model if the correct data has been retrieved, then you’re asking your persistence layer for something it doesn't know how to do. If, as is common, you’re asking your persistence layer to know whether or not data is correct before storage then you are also asking it for something it doesn't know.

Although this becomes much easier with practice, the underlying key to application layering is knowing what you want your layer to do, and making sure that it doesn't do anything else. Thinking about your application layers as specialists helps greatly in keeping in mind what they should, and shouldn't, be doing. You don’t call your pediatrician when your car dies and you don’t call a ticket box office when your roof leaks. Don’t call a model layer when you need to know how to display data.

What Is a Software Architect

"That's incidental. What's the first and principal thing he does, what need does he serve..." --Dr. Hannibal Lecter

As any regular readers know, (Thanks, by the way!) I pretty strongly believe that understanding a thing tells you want to do with that thing. And since I've been writing for some time now about being a "Software Architect", we should take some time and ask the question "What does it mean to be a Software Architect?" As I've said before, not agreeing on definitions leads to communication breakdown.

On the surface, a software architect designs software. he is responsible for the design, even if he doesn't necessarily go on from there to build it. Although he might go on to do just that, as in many organizations the architect and the developer wear the same skin. So, the architect creates not just the class diagrams but often the technologies as well, thus shaping what the final product will look like.

But, as Dr. Lecter would say, that's incidental.

In order to design a software application, an architect needs to know two things. What is needed of the application and what is the best way of fulfilling those needs.Without these pieces of information, the design will fail somewhere along the line. So first of all, the architect needs to be familiar with the business requirements at hand. Secondly, the architect needs to have a broad enough field of experience to know the proper technologies to implement the requirements. If the architect only has a shoe or a bottle to pound in a nail then the final result can't help but fail. So, a software architect is starting to sound a lot like a general contractor. You get the request, you choose the materials, plan your work, and then hand things off to subcontractors to fulfill.

That, too, is incidental.

Software doesn't exist in a void or for its own sake. It exists to fill a need. A real-life need that is both concrete and immediate. Whether the application is as innocuous as Angry Birds or as complicated as the Linux Kernel, it exists because someone needed it. In order to design a useful application, you need to understand that need. When you understand a need, you can provide a solution. If you provide a solution, then you can provide a spec doc that can actually be fulfilled. If you have a spec doc that can be fulfilled, then you can choose the right tools and build the proper design.

This is not incidental. This is the first and principal thing that you are and the need you serve. You're a problem solver. This has been my employment Elevator Pitch for years.

"What do you do?"

"I solve problems."

The Ant, The Tiger, and The Programmer

“You have power over your mind - not outside events. Realize this, and you will find strength.” --Marcus Aurelius

“My weaknesses... I wish I could come up with something. I'd probably have the same pause if you asked me what my strengths are. Maybe they're the same thing.” --Al Pacino

Consider the ant. While pound-for-pound one of the strongest animals out there, since it weighs in at a stunning 0.0003 grams, that metric is less than useful. No, the greatest strength of the ant lies in a colony’s sheer number of ants and the fact that they can act with a single-minded determination to get a task done. Very little short of poisoning the lot of them interrupts their task once they begin and they have the ability for hundreds, if not thousands, act as if one being.

Consider the tiger. Tigers are not pack animals. Add a second tiger to a tiger’s territory and you’ll likely end up with a dead tiger. They do not like competition. And yet, as hunters go tigers are frighteningly effective. They can get to be 4 meters long (13 feet), weigh 600 kg (1300 lbs), can run at about 90 kph (55 mph) nearly silently, and are powerful enough to bring down a rhino or an elephant. Where the tiger walks, things that don’t want to be lunch best move carefully.

Two remarkably effective animals. One that can only follow orders, but carries out its task with a determination and undeterrence rarely found in nature.  The other nearly unable to work with others of its kind, but with a frightening level of individual ability. Both achieve their goals but in opposite, and incompatible, ways.

Despite having just called the two “incompatible”, a development team must be a colony of tigers. Considering the dev lead as the colony “queen”, each developer must be able to accept their marching orders and complete their assignments with the level of determination of an ant. Yet they must not be the mindless workers that ant embody. They must have the individuality of a tiger and a tiger’s ability to successfully determine how to achieve a goal and then the ability to actually achieve it. And the “Colony Queen”, i.e. Dev Lead, needs to understand that the tiger is not a mindless implementer of tasks and if treated as such then the overall effort is endangered.

We all know developers that are like the ant. Practically useless on their own because they can’t hold an application in their head or understand how various objects should relate or even how various pieces of functionality impact each other. But hand them a task and they complete it. We all know developers that are like the tiger. Highly skilled, able to intuitively understand the task at hand and how best to implement it. But get in his way, critique his code, touch his check ins, or even talk to him when he’s in the middle of something and he’s going to bite.

As a developer it is critical to your career development to embody the strengths of both the tiger and the ant while taking on none of their weaknesses. You must be a colony member that hits his tasks reliably. You must be a tiger that can rely on your own skill and experience. However, when your ability and understanding seems to conflict with your direction, you must be able to do something that neither ant nor tiger can do. You must be able to communicate your results and opinions to team leadership, do so clearly and helpfully, and understand that your ideas will not always be taken.

Make no mistake. This is the difference between a successful software developer and one that is perpetually wondering why he can’t keep a job. It’s not ability. Every developer I've ever worked with is either a tiger or an ant. The successful ones are the developers that can maintain those strengths without succumbing to the weaknesses. Are you an ant? Don’t become so focused on carrying out tasks that you forget that you can contribute your knowledge and experience. Not just repository check ins. Are you a tiger? Remember that at some point, you’re going to be overruled. Learn why, rather than biting. Is there a non-functional limitation, such as time frame or lack of stakeholder buy-in, that simple cannot be controlled? Is your preferred path incompatible with another development team’s work? Remember that setbacks are learning opportunities and treat them with grace. Remember also that others have skill and experience as well and respect that. Especially if you disagree with them.

Neither the tiger nor the ant will ever be anything more than what they are. Their weaknesses insure that they are only useful in narrow circumstances. Take on the strengths of both and the weaknesses of neither and you will quickly find that your organization find more and more situations where you are considered useful, or even necessary.

Projects and Icebergs

Once, I watched a class being taught to a small group of children. The subject was Aikido, an ancient martial art which utilized much inner energy, or ki. The instructor used an analogy to show internal versus external strength. ‘Ki’, he said, ‘is much like an iceberg. There is a tip, which is visible, much as external strength, which uses muscles; then there is the internal strength, which is at once much greater, and yet hidden.

…

When he had completed his explanation he asked, ‘Are there any questions?’ A small boy of perhaps four or five years raised his hand. ‘What’s an iceberg?’” --Steve Perry “Matadora”

Software development projects hinge on many things. The makeup of the development team. Resources that can be committed to the project. Management and stakeholder buy-in. However, the biggest problem I've seen is communication and the biggest communication problem I've seen is defining terms so that everyone is using a common lexicon.

Some years ago, I was part of a development project that was a miserable failure. The six month project was still in development after two years and eventually a competitor beat us to market. Looking back on it, the core problem could be distilled down to the fact that no two groups of people agreed on what any phase of the project meant. It’s not that anyone wanted to skip defining the project work or goals, or wanted to skimp on testing, or even that no one wanted to go through the time to meet to discuss project progress. The problem was that no one agreed with anyone else on how to define these aspects of a project or what should be done with them.

To some, “Testing” meant developers testing their work, and nothing else. To others, testing meant focus groups. Some thought that the project goals could be adequately communicated in a Project Charter document, others wanted to have agile-like meetings to discuss goals. And no two groups agreed on what “Finished and ready to deploy” meant. One group thought we were building an ecommerce site while another was adamant that the site should not handle payment transactions. One group defined “Deployment” as a full release that would allow all customers to begin using the site. Others defined “Deployment” as a limited release, similar to a beta test.

Even worse, there was no common definition of project roles. Sure, there was a project manager, product owners, executive stakeholders, a dev lead, and QA staff. We hit all those bullet points. The problem was that no two people agreed on what these roles meant. Because of that we had the dual problems of nobody performing certain critical tasks while stakeholders were fighting over how other tasks should be handled because multiple groups believed that the task was theirs to do.

As a result, work stagnated as developers began undoing work that one group had requested. Frustration levels ran high and developers began dragging their feet on work they didn't see a point in doing, knowing that someone else would soon have them undo the work. Testing was a mess as some groups refused to test functionality, thinking that was the developers job, and others submitted the application to a focus group and expected any feedback to be addressed. Finally, the project died when developers started leaving for other jobs.

I want to take a moment and underscore that last point. Not only did a lack of common definition of concepts cause the failure of this project, but it hindered future development projects due to development staff leaving the company.

In contrast is the project I’m on now. Development is going to take at least a year and a half and the resulting platform will have implications far beyond the current business need. We are coordinating three different business units with similar, but not identical needs. Work will be performed by four parallel development teams working separately but coordinating with each other. And this is all possible because before we even began gathering requirements we defined every project concept we could think of so that there was no misunderstanding. The definition of each project role. What "Scrum" and "Agile" meant in the context of our project. And, of course, the definition of "Project Complete".

As a result, milestones have become easier to hit. Not easy, mind you, just easier. We have agreed on what "Acceptance Criteria" and "User Story" mean so that when we do requirements gathering we all know what information is to be presented and in what form. Development milestones are easier to communicate, since we all know what to expect of code that is “Ready for sprint review”. When we talk about testing, we have agreed on what that means. Now everyone knows what has happened at each testing stage. As a result, the state of the code is clear at any given step. We have agreed on what it means to release code to production and we have a clear definition of what it means for this project to be finished.

Better yet, we have a common agreement on all project roles. Because we took the time to define “Product Owner”, we know what to expect from that role. Those responsibilities are clearly documented. As is the meaning of “Project Manager”, “Project Architect”, “Development Lead”, and “Business Analyst”. We know who does what, what to expect from whom, and to whom to look for any needed deliverable.

Of course, a common definition does not guarantee success. Agreement on terms doesn't remove the work of meeting project milestones. Because of the common lexicon, though, we can take a look at any part of the project and make a determination of whether or not it meets the definition it is supposed to. If two parties disagree on whether or not the definition has been met, we know exactly who makes the final decision. If that decision is that the definition is not properly met, we know what needs to be changed and exactly who is responsible for making the change.

Communication is one of the biggest single points of failure in a software development project. Nothing can tank a project faster than misunderstandings or people not clearly communicating their understandings and expectations. But before you can clearly communicate, everyone on the project must know what an iceberg is.