Thad Scheer
SPHERE OF INFLUENCE, INC. – software studios and services

 

 

With Agile Software Development a lot of people are confused about whether they should still be using design and requirements. Test Driven Design, Story Cards, and other well known elements of Agility make it difficult to figure out exactly what it means to have design or requirements anymore. People get a lot of peer pressure to ignore design, we are told, because the code is the design. There’s also pressure to not have requirements because they interfere with change.

 

I’m no stranger to Agile Software Development and yet I’ll admit that our delivery teams rely heavily on both design and requirements.  I’ll also recommend that you might do the same if your situation is similar.  Of course, I define design and requirements differently than most people.

 

Design is about fitting a product to the humans that use, own, and service it. Design is not flowcharts, UML diagrams, or other technical modeling. That errant notion of what design is came about many years ago when Systems Engineers established the early software lifecycle processes. 

 

Similarly, requirements exist to constrain design elements.  Requirements should not be used to articulate product design nor should they be used as a closed specification of what to build. Many software developers believe requirements exist to describe “what to build” and they think design is “how to build it”. This is a classic systems engineering viewpoint, and it is wrong when we are talking about the vast majority of software products.

 

The truth is most people working in software have misunderstood what purpose is served by requirements and design, this includes most Agilists.  The misunderstanding is mostly due to the way requirements and design have been applied to software for the past 40 years or so.

 

Blame Systems Engineers. Depending on where you’ve worked you might not be familiar with the term Systems Engineer, but no other discipline has had more of an influence on the manner in which software is built.  The Waterfall model from Winston Royce, the Spiral model from Boehm, and all derivative works such as Iterative/Incremental (and to an extent Agile) have their roots in the system lifecycle process from Systems Engineering. This influence has defined what requirements and design mean for the software industry.

 

The view that requirements describe “what to build” results from 40 years of Systems Engineering influence, and it is hogwash in any product-oriented context.  Systems Engineers use requirements to control product content and they believe delivery teams need to be held accountable for every requirement, exactly as specified. They get downright irate when developers go “rogue” by exercising creativity that is outside the requirements framework, and thus anything in the software that isn’t a requirement is by definition out of scope – and bad!!  This view is obsolete for software and it is exceptionally detrimental for any product that will be used, owned, and serviced by humans.

 

With Systems Engineering traditions a product’s concept is locked down by the requirements, making it difficult to innovate or create key product elements after the requirements have been written. This is not exactly the model of creativity and innovation people expect from high technology!

 

Systems Engineering traditions have worked their way into the DNA of software project management, software process, and software methodology. In my opinion this explains much of the mediocrity we witness in IT projects, even the “successful” ones. 

 

After 40 years of Systems Engineering influence, the landscape is finally changing.  Software developers have become frustrated by the conventional treatment of requirements while the Agile community has all but abandoned the term “requirement” and done its best to give the “system lifecycle” the boot…and Systems Engineers with it. 

 

Most developers think design is an engineering activity.  Say “software design” to most developers and their heads explode with visions of UML.  Once again, I blame Systems Engineers because for nearly 40 years their traditions had a dominant influence over the industry’s views regarding system development.  From a Systems Engineering perspective, requirements describe what you need to build and design sets forth a plan for meeting the requirements.  A design becomes “correct” when it can be “verified” through two processes: 1) analysis by comparison to the requirements; 2) technical assessment of implementability, affordability, testability, correctness, and completeness.

 

Software professionals have been trained to think of design in terms of design-vs-implementation or design-vs-requirements, rather than being trained to understand design as a standalone art.  Even within the Agile community the discussion of Test Driven Design exposes an interpretation of design that has more to do with the system lifecycle definition than the true definition.

 

Conventional traditions are wrong. The Systems Engineering view works when software is merely an internal component of a system, much the way a transistor is an internal component to a computer.  The system is the product and software is the part. Realistically, this situation is seldom applicable anymore. Perhaps this view still holds true for missile guidance software and similar devices, however, for the vast majority of projects the software is the system not just one of its parts. Software does the work, it’s what users see, and it’s what breaks and needs fixing.  More importantly, for many types of products software design is the #1 factor in whether a product is liked and appreciated by the people who own and use it. 

Let’s define this term, “design”. If you think software design is UML then you are wrong.  If you think software design is the interfaces, classes, types, events, and callbacks then you are wrong.  If you think software design is a set of project files then you are wrong.  That’s all engineering.

 

Design is the act of adapting a product to fit the human world.  It is appropriate to engineer technology but we should always design for humans.

 

With this lead-in you are no doubt anticipating that I will describe design as user interface development, which it isn’t.  Design is about the whole human experience with a product, not just the user interface.  Products are defined by the experience and the experience is the product.  If you would like to deep-dive into this you should read the first half of Bill Buxton’s exceptional book, “Sketching User Experiences – getting the design right and the right design”.

 

Software design is not software engineering any more than industrial design is industrial engineering. If you are still caught in the conventional mindset I urge you to stop thinking about software design as UML!  If non-developers can’t appreciate it then it’s not design, it’s something else.

 

Design is the what not the how. The true function of design is actually similar to what you might have previously thought requirements were for.  Design is about inventing ways to satisfy previously unmet needs for the buyers, users, administrators, and other humans involved with each product.  What a product is to humans, how it works, how it feels, and how it looks are all part of its design.  This has traditionally been regarded as the role for requirements, but it is not!  Requirements are for something different.

 

A designer (yes that is a role on a project) is concerned with shaping technology to fit the human world.  Designers balance what a product does (its functionality) with the user experience, the data center, the support environment, etc. Some designers have black turtleneck and beret personalities, but most don’t.  The most important skills for being a designer are the ability to visualize the human experience, conceptualize ways to adapt technology to enhance that experience, and communicate ideas.  Many of the best designers have some engineering training, but that is not essential or even necessary.  Most good designers have an engineer they work closely with.

 

Design begins with an unmet need. These days most business problems already have existing software implementations, therefore when a new system is commissioned the unmet need is usually in the area of making a “better” implementation than what was already there.  “Better” can have several meanings.  In some cases replacing a technology stack (e.g. porting from COBOL to C#) is sufficient to make the product better.  However, in most cases the existing technology has failed to meet the expectations or desires of humans.  This is not indicative of a technical flaw with the existing system, it is a design flaw.  The way to address this is with better design.  Design should be used to solve a functional problem by fitting a functional solution to the human world while minimizing things such as frustrations, unnecessary steps, maintenance problems, and etc.

 

Design should offer humans the conveniences of innovation and modernity.

 

The language of design is still evolving, and I expect within a few years the software industry will develop an improved breed of tools for expressing design. Buxton points out that designers (in other industries) have relied on the “sketch” as the universal language of design for hundreds of years.  Sadly, for software designers, sketches are insufficient to capture many important aspects of design.  A sketch can convey what software will look like but it cannot reproduce the real experience of using a product.  Could you fully appreciate the novelty and elegance of the iPhone from a sketch?  No.  Tool vendors, we are ready for you!

 

Let’s talk about product innovation.  Where does product innovation come from? There are two main sources: 1) Research/tinkering; 2) Design.  Some innovations are highly technical, pushing the limits of mathematics or science; innovations of this type arrive through focused research or tinkering.  However, the vast majority of product innovations are design innovations.  Designers use a creative process (not an engineering process) very similar to what marketing studios and other creative places use.  Designers emphasize activities that I’m sure good Systems Engineers do internally, but the act of an explicit creative process is a tremendous improvement.

 

The front-end of the design process consists of these steps:

• Idea Generation
• Idea Competition
• Idea Selection

Notice the term “idea”. Remember, design is about discovering what to build.  A big part of design is discovering the right design, and that means designers need to work in the language of ideas.  Such explicit handling of ideas causes creativity. The funneling of ideas through a selection process elevates creativity to the level of innovation. 

 

Design changes the overall software delivery process. Design comes before engineering, and yes, you should do design (at least a lot of it) up front before major engineering iterations start.  Agilists please don’t freak out! We all understand why Big Design Up Front (BDUF) is a horrendous idea.  Keep in mind, however, most of the compelling arguments against BDUF assume design means “technical specification”, i.e. UML diagrams, and assumes there is no unifying product design.  Agile recommends an approach that the design community calls “participatory design”, wherein the users, buyers, and other stakeholders participate in determining product content.  This is accomplished through delivering working software frequently with intense customer involvement.

 

Unfortunately, as Erik Stein points out in his Cutter IT Journal article “Innovation: Agile with Intent”, the Agile approach does not necessarily yield more innovative products, nor does it yield products that are more likeable by users even though users have a hand in driving the design.  Giving someone exactly what they ask for does not necessarily make them happy or satisfied.

 

While there is nothing wrong with participatory design, it should not be used in place of a formal design process led by a trained, experienced, and accountable-for-results designer.  Intense user involvement works better to refine design elements than it does to lay them out initially, so it is important to introduce participatory design at the appropriate point in the schedule.

 

Before discussing requirements, I will state explicitly that product features still need to be iteratively elaborated.  Design is not meant to address a product at a feature-by-feature level of granularity.  Design should outline the themes and broad concepts that a product adheres to.  Design should call out the top differentiating elements and features of a product, not specific functionality. This practice helps us get away with doing design up front, because the specific feature elements will be added iteratively as working software evolves.  Agilists, you can sit back down now.

 

Requirements are constraints on features. At the point when a feature is added to a product the feature needs requirements. For functional features I prefer to see constraints expressed in the language of executable test rather than English – however that is just a preference. It is irrelevant to this discussion how a requirement is recorded.

 

A requirement is simply a description of record that defines correctness for a feature.  Projects need this because every feature should have objective pass/fail acceptance criteria. Moreover, at a later date and time there could be a dispute about whether a feature is working correctly or not.  Even small projects have staff turnover that can make determining correctness an archeological dig if there is no description of record.

 

Projects sometimes need detailed feature descriptions to communicate intents to engineers, requirements give engineers something concrete to work against.  In this case the requirements will come first, before the engineering.

 

It should be mentioned that requirements do not necessarily need to precede engineering. Sometimes a designer will work with an engineer to flesh out exactly what a feature needs to do or what blend of features will create the best overall experience.  In such cases requirements should not be written in advance (as a Systems Engineer would do), they ought to be written after the experimentation period.

 

As an Agilist I’ll point out that feature elaboration is something that should be done iteratively and incrementally throughout development.  Although a bulk of design work needs be completed up front (my definition of design – not the conventional definition), the feature-by-feature layout of a product happens iteratively. Requirements should be written then, when the features are specified in detail. It takes a lot of orchestration to make sure requirements are done “just in time” so that they don’t hold up engineering.

 

Some requirements will be known in advance.  Not all requirements should be iteratively recorded, some can be captured upfront. For example, if you know a product needs to be Bluetooth 2.0 compatible, you can immediately capture the requirements that describe compliancy. The same goes with other system-level interfaces, government regulations, or things not likely to change during the period of development.  We only want to delay capturing a requirement if there is a chance the requirement will change or go away during development.

 

Neither requirements nor design should be overdone.  I firmly believe great design equals great products.  Sadly most software projects, particularly in enterprise IT, don’t use anything I would describe as design, much less great design. That said, all good things need to be administered in the appropriate dosages. Overdosing on black turtleneck and beret design can shift a project’s focus from practical concerns to creativity for creativity’s sake; producing things that while creative are completely useless and unlikeable.  Similarly, requirements should be captured and maintained with moderation. Overspecification interferes with the art of design and slows projects to a crawl. The right amount of specification happens “just in time” and captures key aspects of features that are essential to consistency and quality control.  Requirements volumes, consisting of A-spec, B-spec, and C-spec still have their place in Systems Engineering, but should generally be avoided in any product that has human users, owners, and administrators.  There are, of course, exceptions to this.