The reason architecture was brought into question in Fowler’s paper is because when you are doing evolutionary-based design (ie. bottom-up design, where you articulate first-class entities as they reveal themselves, a lot like how I build my own org-mode headlines), then you aren’t designing a whole gestalt system of components all in one go, what an architect typically does. Instead, you are guiding the growth of the system, identifying boundary points as they reveal themselves. You are more a gardener than an architect.

Clearing the weeds and design or system calcifications, defining BoundedContexts and ApplicationBoundaries, refactoring as necessary.

The reason architects are needed in buildings is because they’re not structures that can be trivially changed. Most likely, you won’t be able to change your mind later, so you get it right the first time.

For software, refactoring is always possible, though it has a cost. So it’s more sculpting and closer to art than traditional engineering disciplines.

The value-add of the “architect” is refactoring direction.

Design == refactoring/gardening/cultivation.

Intermediating between chaos|entropy and order.

Defining the taxonomies as they grow, and managing them sanely such that they are well articulated to the people who use them.

Image by buwaneka | CC BY 2.0

This is a tricky one to navigate and think through. I’ll try my best to articulate my thoughts on Scrum, how it relates to GTD, and some new epiphanies I had about the roles they play in successfully creating software.

Org-mode and GTD are awesome tools for the individual because they clearly lay out all potential paths you could walk and has you start managing them via externalized memory and regular check-ins. In other words, these tools have you write down and outline all possible tasks and projects you may want to pursue. And in doing so, you directly face your list of all possible paths you can take, all possible action steps you can execute on.

And then you perpetually manage this system in regular intervals, always moving forward with your projects in some way.

A well implemented project management cycle is the same in any organization. Scrum tries to package itself as the project management solution to do this. Scrum attempts to be the organizational-level version of GTD. It attempts to externalize memory from the individuals on the team via wikis, issue trackers, and kanban boards. Sprint meetings and retrospectives are the Scrum analogues to GTD’s weekly reviews.

Some organizations are wildly successful with Scrum, while others are 100% mediocre.

Once you understand the value-add of both systems, you can begin to pick apart how users interact with both methodologies, why the methodologies fail in some instances, and why they succeed in others. Quick aside: Part of it is because GTD targets the individual, while Scrum targets the organization. And comparing the two reveals the values and weaknesses of each, particularly of Scrum (I’m not gonna delve too much into the dynamics here- but organizational level solutions almost always fail if the individual isn’t properly taken into account, which is why Scrum is more prone to weakness and failure than GTD when implemented).

So my epiphany is actually pretty simple- project management is probably a fractal pattern. It applies to the individual, the organization, and beyond.

While simple, there are some interesting conclusions on what this means for overall strategy.

At an ideal level, a developer is executing their own individual instance of project-management (eg. org-mode + GTD), the product-development department is executing their own organizational instance of project management (eg. project documentation + Scrum), and the business is doing it’s analogue as a whole (ie. the CEO + board of directors are managing their work as ideally as possible).

The major point is that the strategies align at every level of analysis. From the individual, to the organization, to the entire entity as a whole. The individual GTD-strategy can’t be compromised for the organization. But the organization can’t be compromised for the individual. All strategies have to be in harmony or it fails.

And thus, the key takeaway is that Scrum (or any organizational project management tool) must allow for the individual to implement their project management processes freely. The individual must be given the discretion to make seemingly arbitrary decisions. They must be given creative freedom. In short, they must allow the developer to execute on the tasks that they think are the highest value.

A couple of components of Scrum stick out as bottlenecks to this philosophy, particularly estimates and sprints. Creative freedom means developers aren’t locked into completing a particular subset of tasks, aka the sprint backlog, because they will naturally tackle the highest value tasks out of their own interest and affinity. Estimates are also a similar bottleneck in that they often establish an implicit contract for what gets completed and when. Which immediately eliminates the ability for the developer to work on necessary tangents in the project. AKA it removes the necessary discretionary and creative decisions developers must make and, as a result, will likely introduce it’s own form of technical debt, which leads to time and money costs and even further project delays.

Ultimately, a strategy aligned at the organizational and individual levels means that Scrum estimates don’t matter. If it means the developer doesn’t complete their assigned stories at all, then so be it.

Of course, it’s reasonable for stakeholders to know what was done instead of the assigned stories. If developers went off-the-road and worked on non-story tasks, it should result in exponential returns, since they are deliberately choosing to pursue higher-value tasks than the assigned stories. No one wants to spend their time on make-work, and if the assigned stories are later revealed to be make-work, then it’s better for developers to focus on higher-value aspects of the project. That’s the key value in giving developers creative freedom. That immediate ability to identify tasks that will be a waste of time, cull them, and move on to bigger and better project work.

Forecasting and estimates can play a role in giving semi-realistic projections that allow businesses to make decisions in other parts of the organization, so long as making these estimates is not more time-consuming than it needs to be and so long as all parties understand the volatility inherent in the forecast. (And if you’re interested in the non-linear dynamics of organizational strategy, this volatility can be used to your advantage. We don’t know what the reality of the marketplace is. We don’t know with 100% precision how the market will respond to our products. We don’t have an accurate picture as to the moves our competitors are going to make. By embracing volatility in our own product development, we are allowing our products to be developed in such a way that it will ultimately be a more accurate reflection of the marketplace than by the top-down imposition of what we guess it is).

Thus, based on what I’ve described here, as well as some of my observations of Scrum in general, I have a new attitude towards the methodology- Scrum isn’t inherently bad, but it seems to propagate mediocre dynamics just like the cubicle did. Instead of being leveraged for it’s strengths, and instead of culling out it’s weaknesses, Scrum is often used to impose predictability on an inherently unpredictable process. The results speak for themselves.

A good organization with quality project management is good in spite of Scrum. Because they’d be good anyway without it. The product leader “knows” the spirit of GTD even if he doesn’t know of GTD. AKA the same attitude and values that would drive him to adopt GTD if he knew of it, at least drive him to cull out make-work and to focus on high-priority tasks. He has the same values and therefore would inevitably implement a successful strategy, or find one that aligns with his values. Those same values would lead him to interfacing with the higher and lower levels of the organization with an acute awareness of the limits of forecasting. And if tasked with using Scrum as a methodology, he’d pick and choose the useful components of Scrum relative to the project’s needs and cull the weak parts without hesitation.

Turns out I don’t have an aversion to Scrum and to the idea of project management (hell, I love GTD). I just have an aversion to Scrum when it’s done in a nonharmonious fashion. AKA when it gets in the way of doing useful work rather than being a catalyst for it.

The effects of a dysfunctional Scrum project management cycle are worse for the individual in the short-term because it means the individual worker’s day-to-day life sucks and is closer to a psychological prison- no creative freedom, completion of backlog tasks that provide little-to-no real world value, inability to develop his skills as a developer (which otherwise would be compounding interest for the organization’s product development in general), etc. The organizational effects are worse in the long-term- while they might survive for 4-8 quarters on budgeting alone by “forcing predictability,” the effects will be more severe. The individuals in the organization learn to not complete the higher value work (since they don’t have the discretion to override the assigned tasks that in-development reveal themselves to be useless or make-work), and thus not build a product that more accurately reflects usefulness in the real-world. As a result, even if the project gets done on time, it will not be a product worth using. And thus, the organization, or at least that product, will probably die in the long-term.

In other words, by not embracing unpredictability and allowing for real-time micro-changes in the backlog, the organization will fail to get ahead of the real-world threats they do not perceive (competitors, market changes, etc.). They’ll force a model that grows stale and more wrong with each passing day. And they’ll do this to their peril.

Here is a quick software development methodology cheat sheet.

Note: There are many specific processes defined for some of the methodologies that are beyond the scope of this article. The main goal of this article is to give very high-level definitions of each methodology.

• Waterfall - You do ALL of one phase, then ALL of the next, then ALL the next (eg. design -> coding -> testing).
• Agile - Agile is a philosophy, not a step-by-step methodology¹. It is a set of attitudes about software development with the goal of improving communication and the overall development process for all parties involved. You are “agile” if you align yourself with the agile philosophy and make your actions in development reflective of that.
• Scrum - A template of steps to follow for the development cycle. It is an implementation of agile- it is supposed to follow the principles and attitudes of the agile philosophy. At the same time, it actually gives you the steps that you can follow. In this methodology, you have a prioritized todo list, and you incrementally and iteratively complete chunks of the todo list. By performing incremental and iterative development, you allow requirements to be flexible and changeable since you are not committing to completing one entire phase before moving to another one (waterfall). Thus, the development process can better cope with inevitable changes.
• Extreme Programming (XP) - XP is a methodology which implements a set of rules that you follow to optimize software development and make it easier to collaborate and work with a team while sustaining a high level of software quality. XP is an agile methodology as it aligns with agile principles.
• Kanban - A software development process that aims to manage the flow of items being worked on (for example, the flow of completion of each item on the project’s todo list). Kanban processes seek to identify and eliminate bottlenecks in the software development lifecycle.
• Lean - Product development principles that have been translated to the software development domain from lean manufacturing. There are 7 principles- eliminate waste, build in quality, create knowledge, defer commitment, deliver quickly, respect people, and optimize the whole.
• Cowboy Coding - Developers have complete control of software development. There is no formal structure that guides the project along. The easiest way to think of this is as a “free for all” where you can do whatever part of the project you want whenever you want.
• Iterative Development - The software is built knowing that future versions (iterations) will be used to refine the project.
• Incremental Development - A software project is divided into chunks and each chunk is developed by the team, one at a time.
• No Silver Bullet - A paper written in 1986 by Fred Brooks, in which he argues that /”there is no single development, in either technology or management technique, which by itself promises even one order of magnitude improvement in productivity, in reliability, in simplicity.”/

Let me know if I’ve missed anything and I’ll put it up!

So you’ve decided you want to be a developer. You want to understand code and be a master of computers. You want to tell them what to do instead of struggling to figure out what they may or may not be doing. But you don’t know where to begin.

I’ll help you out.