PM Blog: Alan S. Koch

Little ITIL, Big Results

2011-12-20T19:31:41-08:00

Step 4 of 10: Talk About the Score

by Alan S. Koch, PMP

In our last article, "Little ITIL®, Big Results; Step 3 Know the Score," we continued looking at our 10 steps to starting an ITIL effort in a small IT shop. (Listen to our webinar on this 10-step process!) Step 3 was about objectively measuring what is important, not only the internal technical metrics, but also external customer-oriented metrics.

Back in Step 2 -- "Talk About Getting Started" -- you probably heard a lot of complaining from your constituents. And if so, those complaints were likely broad generalizations that had more of a basis in emotion than in metrics. The metrics you started collecting in Step 3 will provide a tool for moving those conversations away from emotion and toward consideration of facts.

This fourth step of the process is focused on restating the complaints you are hearing in terms of metrics. You do that by showing both your customers and your IT Staff what the metrics say, and by probing to see if other metrics are needed to address some complaints.

The point of this is not to prove to the complainers that they are wrong or over-stating the problems. Rather, it is to replace emotion with a reasoned basis for a rational conversation so all of you can agree on problems that need to be solved, and how you will know when things get better.

Starting Conversations About the Score

Having opened a communication channel with each of our constituents in Step 2, we have a built-in basis for these new conversations with them. With each person, we want to begin with the things we heard last time, which demonstrates that we were listening, and move the relationship forward by bringing the relevant metrics into the discussion.

Talk About the Score With Your IT Staff

It may be tempting to start by showing our customers the "real data," but we have a more important place to start -- within IT. Every member of IT needs to understand (in terms of hard data) what he or she does and how it affects the customers and end users.

Internal Technical Metrics -- We start with the internal, technical metrics (like server performance, network down-time, or person-hours per service request). They provide a basis for understanding and talking about how each component of our IT shop is performing and identifying opportunities to improve that performance.

If our staff members have complained about anything other than customers and end users (be it hardware reliability, software performance, or process efficiency), we will start there, by bringing out and discussing the metrics that clarify those issues. Compare staff members' perceptions to the hard numbers. Are things as bad as they seemed? Worse? Is there room to improve? Would the effort needed to improve be justified? What changes might we make to improve things?

But let's not limit our focus to complaints. Often people live with problems that they fail to articulate. Look at all of the available metrics. Do any of them indicate problems? Is anything out of line with expectations? Are opportunities for improvement apparent from the numbers?

This is the beginning of an important pattern we want to establish with our staff -- that of confirming or illustrating problems with metrics, and examining metrics to identify hidden problems. The more our people think in terms of measuring the IT environment to identify how to make things better, the more improvements we will be able to make.

External Customer-Oriented Metrics -- After looking at the technical metrics with our staff, we need to turn their attention to the customer-oriented metrics (like application response times as experienced by end users, down-time as experienced by end users, and hold-time when calling the service desk). Any of their complaints about customers or end users must be viewed through the lens of these metrics. Again, we want to confirm where problems exist, and understand them in a way that will allow them to be solved.

Then we must examine all of the customer-oriented metrics to look for issues. Much of what we have begun measuring at this point was prompted by customer complaints, so this will be our opportunity to translate complaints our staff may be hearing from their customers into quantitative form. It will provide all of us with insight into our customers' and end users' experience with our services, and will be the starting point for addressing the problems.

Your IT staff are in a unique position that allows them to draw connections between these customer-oriented metrics and the more familiar technical metrics. After using the customer-oriented metrics to confirm and understand each customer complaint, they can use the technical metrics as the centerpiece in root-cause analysis. This understanding of the causal connection between what goes on within IT and how our customers experience it is key to making meaningful improvements.

Talk About the Score With Your Customers

After exploring all of the metrics with our staff, we are ready to have similar conversations with our customers. But in this case, we will not talk about technical metrics (except with members of our customer community who are interested in them). With our customers, our focus will be on the customer-oriented metrics.

External Customer-Oriented Metrics -- As with our staff, we will start our discussion with each member of our customer community by revisiting the complaints they have already raised. We will demonstrate that we were listening to them, then bring out and discuss the metrics that clarify those issues. Compare the customer's perceptions to the hard numbers. Are things as bad as they seemed? Worse? Is there room to improve? What levels of service would meet their needs?

Again, we won't limit our focus to their complaints. We will look at all of the available metrics. Do any of them indicate problems? Is anything out of line with expectations? Are opportunities for improvement apparent from the numbers?

It is important that we not make any promises that we cannot be sure of meeting. At this point in our process, that likely means that we can't promise to do anything more than investigate how we might make appropriate improvements in the relevant metrics. (Indeed, that will be our next step!) For now, we have achieved the intermediate goal of translating their complaints into an actionable form.

As with our staff, this is the beginning of an important pattern we want to establish with our customers -- that of confirming or illustrating problems with metrics, and examining metrics to identify hidden problems. The more our customers think in terms of measuring their need for IT services, the better able we will be to actually meet those needs.

Talk With Your IT Staff About Your Customers

After exploring the metrics with our customers, we can return to our staff to discuss options. The issues have been quantified, and we have discussed with our staff how those customer-oriented metrics relate to the internal technical metrics. We are now ready to explore options for improving the services we supply to our customers.

This final step is important even if it seems clear to us what needs to be done. Every member of our staff must understand how the things they do and the metrics that they use contribute to the service levels our customers experience. Besides, they may have insights that we have overlooked! Clarifying how the changes we make within IT translate into service improvements to our customers closes a very important loop!

Little ITIL, Big Results

2011-10-11T13:56:15-07:00

Steps 3 of 10: Know the Score

by Alan S. Koch, PMP

In our last article, "Talk About Getting Started," we discussed the second of our 10 steps to starting an ITIL effort in a small IT shop. (Listen to our webinar on this 10-step process!)

Talking is all well and good, but we are not in this to hear ourselves talk! We need to do something, so let's get to work! But before we make any changes in our IT shop, we must lay some important groundwork. So, we continue in this article with Step 3, "Know the Score."

Step 3. Know the Score

What gets measured gets rewarded. And what gets rewarded gets done.

Having begun dialogs that will continue in all of the even-numbered steps, you now need to turn inward. You have heard about problems and opportunities for improvement from all sides. But before you can begin taking steps to improve the score, you need to know the score. It is said that, "all players know the score," so if you're going to be a player, you had better know the score. To carry the sports metaphor one more step, every player knows his or her own stats as well of the stats of the players they play with and against.

When there are many problems to address, it's tempting to dispense with the quaint idea of metrics and jump right into the important work of changing things. This impulse is foolhardy. Without baseline metrics, you lack an objective basis that will allow you to do four very important things:

Prioritize problems and opportunities to decide on the order of attack
Compare your IT shop with others to understand what they may be doing that you could capitalize upon
Judge the effectiveness of the supposed improvements you make to determine if they should be retained
Demonstrate that things are indeed improving to gain support from others for your continued efforts

Good baseline metrics (objective measures of how things are before you begin making changes) will enable all of those things as well as being useful in a host of other ways as your IT processes and practices become more mature.

But of all the things we could measure, which are the ones we should measure? Devising metrics, collecting measurements, then doing something with them all take time and effort. There is only so much time we can devote to metrics, so we must focus on those few things that will be valuable to us. The metrics you will need fall into two broad categories:

Internal technical metrics
External customer-oriented metrics

Internal technical metrics

These are the metrics we normally think of in IT: Utilization of CPU, disk and network; numbers of users, components, incidents, and requests; average effort to stand up a PC, resolve an incident, or fulfill a request; and many, many others. These sorts of metrics enable us to gain an objective understanding of our IT shop's current capabilities and performance, which will help us to recognize our opportunities for improvement.

The valuable internal metrics start with those that come for free. Our devices, systems, and software are capable of measuring and reporting on a variety of things. Those capabilities should be enabled and the data stored even if you have no immediate intention of using it. You will need some of it to diagnose problems at some point in the future, and having that data available to comb through sure beats beginning to collect it after it is needed.

Then there are the metrics that you will have to invest time and effort in defining, collecting, and using. How do we identify which of all those metrics are worth worrying about? Start with the concerns that are common in most IT shops (especially the small ones).

What chews up all the money? Capital purchases are obvious, but the other parts of TCO (Total Cost of Ownership) are also important. Service contracts, parts and supplies, training, consulting, and support or other help can add up.

The good news is that this data should be readily available from your bean counters. Whether you have budgetary responsibility or not, you should take it upon yourself to understand where the dollars go so you can make smart decisions or recommendations about future investments.
What chews up all of our time? Employee costs often are among the biggest IT costs, and they are certainly among the most controllable. Insight into where the time goes -- both your staff's and your own -- can help you to distinguish between the mere annoyances and the true time wasters.

But this data is difficult to collect. Most technical workers find it difficult to log where their time goes, and many have a downright negative reaction to the idea, fearing that the data will be used against them. Appropriate tools and processes can address the difficulty issue, but the fears must be handled with finesse. Lay the groundwork by making it clear that the data will be used to defend and support their jobs, not to threaten them, and demonstrate the behavior by logging your own time.

Then back those words with actions. Key among them: never, ever discipline an employee on the basis of the time data they log. This will be a difficult for you when the time log data shows a clear problem. But remember this: the moment you use data against those who collect it, it will cease to be real data, and will simply be manufactured to make them look good. (And the reality is that there will be other symptoms of the problem that you can use for disciplinary purposes.)
What causes our pain? This is where the information you gained in Step 2 comes in. The complaints you have heard, both from your customers and users and from your IT staff, often will point to things that should be measured. The first step in determining the legitimacy and priority of complaints is to measure them.

Of course, we can't give specific advice without knowing precisely the nature of the complaints. (We will address these topics in other articles on specific topics.) But some generic advice is still useful here.

For example, suppose that one of the complaints we are hearing from users is that problems or requests require multiple calls to the service desk before they are resolved. How shall we go about measuring this problem?
- First, capitalize on what you already measure. Often you can gain indirect insight into a problem through metrics you already have at hand. Better to capitalize on those than to invent something new.
  
  In our example, see what data the service desk already collects. Gross number of calls may be helpful if there is other data we can use with it. If each incident record includes the name of the person who called, you may be able to identify series of calls from the same users. And if there is a category or call-type attached to each incident, you may be able to see how often this multiple-call problem is actually happening.
- If you don't measure anything that is relevant, ask around. Someone else in IT or in some other part of the enterprise might have some usable data. The obvious place to start in this search is with the source of the complaint. But don't stop there. Be creative in your search, and you may be rewarded.<
  
  In our example, we would ask those who have made these complaints to give as many specific examples as they can recall. (We must be careful to ask in a way that does not sound like we are challenging the allegation. We must make it clear that we are seeking data to diagnose and fix the cause of the problem.)
- Finally, keep it simple. We technology people tend to want to do things rigorously and completely, and the costs involved in doing so are often not justified. For many kinds of problems there is a simple, easily available metric that will give you good enough insight. It may not capture all variables involved or quantify all aspects of the problem, but it will provide what you need to get started on improvements.
  
  In our example, it will be tempting to institute new procedures to ask callers if they are calling back about something they called about previously, and to collect specific data about the original call. Although this approach might give us precisely the data we want, we must consider the impact it will have on the callers as well as the productivity of our Service Desk personnel. A simpler approach (especially one that uses data you already collect) is likely to be good enough.

These internal, technical metrics are critical to your ability to gain control of what is happening in your IT shop. You can't manage what you don't measure.

External customer-oriented metrics

But you must manage more than the technical infrastructure and your IT staff. The other thing that is critical for us to manage is our relationship with our customers and users.

The metrics that enable us to manage what goes on within IT are not generally useful for managing these relationships, because for the most part they measure things that those people do not care about. (For example, they don't care how many person-hours the average request eats up; but they do care how long it takes to get what they requested.)

Your starting point for this is the list of services you provide for them. For purposes of communicating with your customer, you must be careful to define each service in the customer's terms.

For example, when your marketing department refers to CRM (Customer Relationship Management), they mean much more than the CRM application. We know that when a user clicks the CRM icon, many parts spring into action: their PC, the client side of the CRM application, the network, the server, the backend of the CRM application, the database server, the DBMS …. Those are the things we must manage. But they are mostly invisible to that end user, who only sees that either the CRM system works or it doesn't.

With this customer orientation in mind, it becomes clear why some of the services we routinely talk about within IT (like network services) and the metrics we use to manage them have little or no meaning to our customers and end users. But what can we measure that is relevant to them? Let's look at our services from the customer's perspective.

In the CRM example above, the marketing department cares about things like these:

Availability. By this, we don't mean the availability of the server or the network, or the DBMS, or any other individual component. The Marketing department cares about how often the CRM system is available to them, which is only true when all of the underlying components are working. We don't want to talk with them about the availability of the components that enable the CRM system. We must talk about the availability that the end users actually experience. Which means that we must find a way to measure that. (Not a simple feat!)
Performance. And just as above, we don't mean the performance of individual components, but the performance levels that the end user experiences. Again, we must find a way to measure from the customer's perspective.
Support. When things go wrong, or when they need something, how long must they wait? We must manage call pickup time, investigation time, escalation, and resolution. But, similarly to Availability and Performance, we must find a way to measure the customer's total downtime (the total elapsed time from when their ability to work is blocked to when they can get back to work).

It will take a while to understand your customers well enough to know which metrics will speak most loudly to them. (We will look more at this in Step 6.) But you already have enough information to begin rudimentary measurements that will be relevant to them. This is your start on metrics. You can't afford to measure everything to the nth degree, so don't try.

Putting Internal and External Metrics Together

The external, customer-oriented metrics give us a way to quantify and understand the service we are providing to our customers and to understand the impact they experience from problems with our services.

When the external metrics point us to a problem we need to address, we must turn to our internal technical metrics to understand how all of the pieces and parts contribute to the problem. This will give us the insight we need to know what to fix, as well as a way to understand the efficacy of any fixes we attempt.

When we believe that the problem has been resolved, we must return to the external customer-oriented metrics to confirm that the resolution had the desired impact on the service from our customer's perspective. Only after we have confirmed that the problem is resolved from the customer's perspective can we claim success!

Without these metrics, we may cast about blindly trying to solve problems, or we may even blame the customer. With these metrics, we will know precisely what is happening and what to do about it. There is no better basis for beginning to improve things than to know the score!

Little ITIL, Big Results

2011-08-02T09:37:05-07:00

Step 2 of 10: Talk About Getting Started

by Alan S. Koch, PMP

Wouldn't it be great if your IT shop operated like a well-oiled machine? No crisis. No hassle. No surprises. Click, whirrrrrrrrr. Click, whirrrrrrrrr. Happy staff. Happy users. Happy executives. Click, whirrrrrrrrr. Click, whirrrrrrrrr. Smooth. Efficient. Cost-effective. Click, whirrrrrrrrr. Click, whirrrrrrrrr.

If only reality could come close to that idyllic image! ITIL℠ (the Information Technology Infrastructure Library) tells us that moving toward that ideal is within the realm of possibility. But look at all that is involved! The processes. The books! The work!!! Who can find time to even begin such an effort?

In our webinar last week, we discussed the 10 steps to starting an ITIL effort in a small IT shop:

Step 1. Buy Some Time
Step 2. Talk About Getting Started
Step 3. Know the Score
Step 4. Talk About the Score
Step 5. Grab Low-Hanging Fruit
Step 6. Talk About Improvements
Step 7. Learn About Your Customers' Businesses
Step 8. Talk About Your Customers' Business Success
Step 9. Help Your Customers Improve Their Businesses
Step 10. Keep Talking

Let's take a closer look at step 2.

Step 2. Talk About Getting Started

Before you can get started on anything of consequence to improve your IT services and people's perception of them, you must connect with those people. Naturally, this will look quite different depending on your history with those people. So let's look at the two extremes: You have a long and contentious history to overcome, or you have no history, having been hired to fix all of those IT problems.

Extreme #1: You have been hired to fix all of those IT problems.

There are some significant positives about this situation. First, the fact that you got the job says that at least a few of the senior decision makers in the enterprise have confidence in your abilities. Also, as a new hire you will be afforded a “honeymoon” period during which mistakes will not be counted so strongly against you.

Although those positives are huge, you must also realize that you have significant disadvantages. The first is your lack of knowledge about the situation. Yes, you have heard stories and complaints as part of the hiring process, but you still know precious little about the history that underlies the situation. Taking what little you know as gospel and assuming that there is little that you don't know can be a fatal error.

The second disadvantage is in your relationship with your IT staff (or more properly, the lack of a relationship with them). While the newness of the relationship is generally a positive thing with the other stakeholders, it may be entirely different with the staff. When IT is seen as a problem by the organization, IT management and staff often hunker down in a defensive us-vs-them posture.

Now, their old IT manager has been fired or forced out, and you have been brought in to whip them into shape. They are likely to be hostile about their old manager's fate, and fearful about their own. They may see the entire set of developments in a very negative light. They may actively sabotage anything you do, or they may passively resist while they look for a new job to run off to.

Although you already have a plan (and you may have been hired specifically because of that plan), now is not the time to pontificate on what you intend to do. Your first job is to listen and learn and build relationships. You need to talk with every stakeholder you have. (And that is a lot of people!)

First is the community of people who use the IT Services—from VPs and managers, to power users and other users. Later, you will be coming back to these people with a specific agenda and questions. But at this point you are just laying groundwork. Tell them you are here to make things better, then shut up and take notes. You want to hear what is on their minds and start building the reputation of being someone who listens.

You will come away from these meetings with a relatively good idea of who your customers and users are, what services each uses, what they use those services for, and what pains they experience. You will not yet have a complete picture of any of this, but this beginning will give you what you need for Step 3.

The other people you need to talk with are your IT staff. Again, you are not there to tell them about how things are going to be different. You are there to listen and learn. Their perspective is diametrically opposed to that of the other stakeholders, and reconciling what you hear from them with what you hear from other people can be a valuable way for you to begin sorting through and making sense of it all.

This talking needs to happen both individually and with the various IT workgroups. You will be listening for their perceptions of the customers and end users of their IT services as well as perceptions of the others within IT. You will also be watching for dysfunctions that may be contributing to negative perceptions, both within IT and within the customer community.

And of course, the biggest thing you are doing with your staff is cultivating the idea that you are not the enemy, and that you are genuinely focused on making things better. You must build a positive relationship with them because you will be unable to do anything of consequence without their support and help.

Extreme #2: You have a long and contentious history to overcome.

If you are in this situation, I don't need to tell you about how painful it can be. The main problem is broken trust on both sides, and this second step is aimed at that problem. This is not to say that you can repair trust with a few meetings. That is not possible. But work on it must begin immediately so that relationships can begin to mend as soon as possible.

Just as above, you will need to meet with all of your stakeholders, both within IT and your customer and user community. The purpose of these meetings is to announce that beginning today things will be changing. But you do not want to start spouting specifics. You want to follow that announcement with as open a discussion as can be mustered about what they think should change.

Your external stakeholders may use this opportunity to skewer you and your staff, so be prepared to hold your tongue and simply take notes. By listening without defending, you may learn something about their situations and needs. Then again, you may find yourself simply weathering the same storm that has battered you before. The good news is that you are demonstrating the sort of civil discourse that must replace contention if the situation is ever to improve. Even if the civility is purely one-way, it is still a good first step.

With your IT staff, you want to begin brainstorming opportunities to do things better within IT. They are likely to focus on problems with the users and customers, and allowing this sort of venting is appropriate to a point. But you want to keep redirecting them to things that are under IT's control. What can we do better? How can we change? Even if there is more bellyaching than idea generation, it is still a good thing, because you are beginning a very necessary pattern that will take time to become ingrained.

Step 2 – A Springboard to What Follows

Beginning with Step 3, you will actually be making the changes that are necessary to improving how your IT shop runs. Step 2 opens the communication lines that enable all of the following steps. The criticality of Step 2 cannot be over-stated. You may think that you can just take command and do the right things, but your success will be limited (and may be completely scuttled) if your key relationships are not bolstered first.

How important is talking? Take a look at the 10 steps that we listed at the beginning of this article. (I'll wait while you scroll back …) Notice anything? That's right. Half of them involve talking. Hard work yields its best results when we forge strong relationships with our key stakeholders and we listen to them.

There's no time like right now to start talking!

Related Links
If you missed Alan's webinar, the recording is available for purchase here, along with Alan's detailed checklist, several supplemental templates, and instructions for claiming 1.5 Category A PDUs.

ITIL - Why Should I Care?

2011-05-24T19:20:27-07:00

by Alan S. Koch, PMP

If you haven't heard of ITIL®, you will. ITIL (the Information Technology Infrastructure Library) is a framework of good practice in IT Service Management. It has been widely embraced over the past couple of decades in Europe, and is now making significant inroads in the US and the rest of the world.

ITIL is one of the few widely accepted frameworks that directly address IT operational practices. Just as the PMBOK® (Project Management Body of Knowledge) has been widely accepted as a source of good practice for Project Management, and the CMMI® (Capability Maturity Model Integration) has been embraced as a source of good system engineering practice, ITIL is becoming the "go-to" source of good practices in IT operations.

You needn't look far to find articles and presentations that explain ITIL's Service Lifecycle Model, the Processes and Functions that comprise it and the ITIL certifications one can obtain. But all of those facts may leave you with many more questions. If you are like Cinda Voegtli, CEO and Founder of ProjectConnections, you may still be wondering what the point is.

Cinda and I engaged in a long-distance conversation about this. We invite you to listen in as she articulates her questions and I answer them on our way to answering the biggest question about ITIL - "Why should I care?"

Back and Forth

Cinda Voegtli: I have attended several presentations at PMI events on ITIL. They have talked about what ITIL is, using high level descriptions of content and rather general descriptions of benefits. They cover the framework and refer to all of the processes, but they don't tell me what I really want to know: Why should I care?! In all seriousness, from those ITIL presentations, I have not gotten a sense of what ITIL really looks like in practice, and the benefits expressed in a way that make me feel I need this process now.

Alan S Koch: There are a lot of reasons why people care about ITIL. Which one applies to you depends on which hat you are wearing when you ask the question.

If you are speaking as Cinda Voegtli, CEO of ProjectConnections, you will care about ITIL because ProjectConnections has a large IT operational investment. Some of those operations are managed by ProjectConnections personnel, and others are contracted from IT service providers. But either way, if they don't do what you need in a cost-effective way, you have problems.

Your objective is to run the company, and your internal and external IT service providers are partners in this endeavor. The success of ProjectConnections is dependent upon them and the maturity of their practices. ITIL defines all of the aspects of this partnership, providing a basis for capitalizing on those aspects that work well, and for addressing those aspects that are problematic.

Cinda Voegtli: How does ITIL help me in working with my outside partners? For example — does ITIL get specific about what exactly should be covered in a contract with a hosted web provider? Or about what should in general be included in a services vendor contract?

Alan S Koch: ITIL's Supplier Management process does not get into those types of specific technical topics. It is very similar to what the PMBOK and CMMI say on this topic. Where ITIL adds real value is in its approach to tying Supplier Management (what we expect of our suppliers) back to Service Level Management (what our customers expect of us) and ultimately back to our Service Strategy (what we expect of ourselves).

ITIL calls our agreements with suppliers "Underpinning Contracts". What do they underpin? They underpin our Service Level Agreements (SLAs) with our customers. Our ability to make specific commitments to our customers in enabled (underpinned) by the commitments we get from our suppliers. What must be included in our supplier contracts is not specified by ITIL; rather those things are naturally derived from the needs of our customers.

Alan S Koch: Then again, if you are speaking as Cinda Voegtli, Project Management devotee, you will be more interested in how ITIL defines the attributes of a mature context in which your project will thrive from the start.

Cinda Voegtli: For example… ?

Alan S Koch: For example, you will appreciate how some of the ITIL practices establish a basis for IT projects by defining objectives, business requirements and success criteria before the project is even initiated.

Cinda Voegtli: How is ITIL's take on that different than what we already know, or what various other standards say, about defining requirements? What is it that is better, more IT specific, or otherwise unique, about this aspect of ITIL?

Alan S Koch: ITIL's take on this is not different per se. Rather, this is one of the things I was referring to when I said that ITIL provides "a mature context". For most of us, a project begins with a problem or opportunity statement, and then we must expend a significant amount of effort doing requirements elicitation and developing a complete understanding of what is needed. (And we all know the statistics about how many projects are deemed to be challenged or failed due to requirements issues!)

In an ITIL environment, the need, objectives and requirements (both business and operational) are all researched and agreed to as a part of the Change Management process before a project is initiated. So instead of starting with a cocktail napkin-worth of "ideas", the project begins with a well-formed charter, as well as a fully researched and vetted set of business and operational requirements that are ready to be translated into the solution requirements and product design.

Cinda Voegtli: OK, so PMBOK talks about project initiation process and charters, and BABOK covers requirements-related processes. How does the ITIL intersect with those? I have this picture in my head of 3 different standards trying to cover that one part of the project — which makes me dizzy. How do I as a project manager understand what is covered by each standard, choose between them, or determine that a project needs all three in some way? Are they compatible or conflicting? Are they at different levels of detail and therefore useful in different ways?

Alan S Koch: In reality, you should not "follow" any of the three! PMBOK is written from the Project Manager's perspective. BABOK is from the Business Analyst's perspective. And the part of ITIL that I was just discussing is from the Change Manager's perspective. Are they compatible? I would go beyond "compatible" and say that they are complementary, each with useful perspectives and practices.

The best usage of these three sets of good practice is for the Project Manager (with PMBOK in hand), the Business Analyst (with BABOK in hand) and the Change Manager and Service Level Manager (with ITIL in hand) to work out precisely how projects should be initiated in their specific organization. They will find no conflicts among them; only points at which they must agree that one person's responsiblity ends and another's begins. The result will be a way of kicking off projects that is informed by three of the best sets of good practice to be found, but a results that is also designed for the specific needs of their own organization.

Cinda Voegtli: That's a helpful way of depicting how any standard could be used effectively for a project - I get the image of the people in those 3 key roles each bringing their domain's best practices to the table. How does this play out during the rest of a project?

Alan S Koch: Initiation is not the only part of the project that benefits from the ITIL processes. You will also appreciate how other ITIL practices address the delivery and hand-off aspects of your projects in a well-defined and systematic way.

Cinda Voegtli: My mind immediately goes to IT-specific challenges and complexities related to delivery and hand-off. Does ITIL go all the way to specifics for different types of IT projects, e.g. delivery and hand-off considerations for apps vs. infrastructure vs. business process improvement vs. web vs. ERP…?

Alan S Koch: It is not that ITIL provides specific guidance for every type of implementation you might be doing. Rather, the ITIL Release and Deployment process is structured to ensure that all of the right players are included, and all of the right questions are asked prior to actual rollout of the product.

For example, on an infrastructure rollout, deployment planning would include identifying how and when the deployment can be done without having a visible impact on the end users. And when that is not possible, it would include working with those users to identify the best (or least bad) time for the deployment-related service interruption.

In addition, the Release process includes ensuring that rollback plans are in place so that a failed deployment can be un-done quickly and effectively, minimizing the negative impact. And, it includes fully testing both the release and deployment mechanisms (including data conversion) as well as rollback procedures to be sure they will work correctly.

And if I may throw one more item into the mix: ITIL's Knowledge Management process works with the Release and Deployment process to ensure that everyone has all of the information they need about whatever is being released. This goes all the way from operations personnel knowing how to run a new system, to end users getting the necessary training and support during transition, and the Service Desk having the data they need to fully support it on day one.

You will appreciate the benefits of managing projects in an ITIL-enabled environment where your projects will be part of a well-defined whole, rather than an oasis of good practices in a sea of chaos.

Cinda Voegtli: Say more — what do you specifically mean by the oasis vs. the sea around the oasis?

Alan S Koch: The image I have in mind comes from what I have repeatedly experienced. We institute sound project processes (ala PMBOK or CMMI or Agile) to bring order to our world of projects, only to find that the rest of the organization doesn't get it. And not only do they not understand, but they seem to be working at odds with our good practices, almost like they want to make us fail. We had hoped that good practices would solve our problems, but instead they highlighted other dysfunctions around us. Instead of living in a world of good practice, we are merely creating an oasis. The chaos around our projects persists.

ITIL extends sound practices to our entire IT organization and beyond, to our customers. And it is not just a matter of good IT Service Management processes coexisting with good Project Management processes. It is about uniting them into a single process framework that is coordinated and focused on meeting our customers' needs.

Cinda Voegtli: To clarify ITIL's boundaries — do you mean that it extends beyond the IT team itself? How?

Alan S Koch: In a sense it does. ITIL's Service Level Management (SLM) process defines how we can do effective Customer Relationship Management (CRM) with IT's customers — the users of the IT Services we provide. Where our customers have mature business processes and good understanding of their needs, SLM is focused on coordinating our processes with theirs so that IT services go beyond merely supporting them, to fully enabling and optimizing what our customers do.

Where chaos exists, SLM actively works with customers to help them to define their IT needs (Service Level Requirements) and define how their relationship with IT should work to best meet their business needs. This is not a one-time task - this relationship is maintained and reinforced over time, which results in a gradual quelling of the chaos.

The other way the ITIL reaches beyond IT is in ITIL's whole approach to IT Service Strategy. IT Service Strategy is not a matter of IT coming up with a strategy; rather, it is mainly about connecting and coordinating the strategic fabric of the organization into a unified whole. ITIL does not talk about IT strategy supporting corporate strategy. Rather, ITIL expects that corporate strategy covers all phases of the enterprise including IT.

And again, in an organization where corporate strategy is ill defined, ITIL will have a chaos-fighting effect. The IT executives will be asking questions that will generate healthy discussions of strategic issues which will (hopefully) result in corporate strategy slowly coming into better focus.

Cinda Voegtli: Taking on any new process is an investment of thought and time. I'm a huge believer in finding ways to start adopting high-leverage new techniques — pieces of process, if you will — to get going with new best practices. But even then I need to understand the big picture. The examples you've given, of specific questions and practices ITIL calls for at different times in the project, absolutely resonate, and show me opportunities to leverage even just pieces of ITIL. But let me ask one final question, to make sure I truly understand ITIL's "big picture".

I come originally from a high-tech product development background. So I know NPD (new product development) methodologies extremely well. I understand the phases and why the work is chunked that way. I understand who should be on the team from all the different functions (manufacturing, service, marketing, engineering, quality and testing, etc.), when they should be involved, how, and why. I know all the typical technical and cross functional deliverables such as specs and plan. In all these areas I know from years of experience why those elements are there and why they contain what they do. For example, if we don't have manufacturing involved in reviews up front, the engineers could inadvertently design systems that are too complex or costly to manufacture at the desired price point and profit margin.

We also need to involve a number of different groups, including outside development partners, in certain planning and review activities, or we'll risk misunderstandings and bad handoffs. An NPD process is a mix of technical and project management techniques, brought together in a framework that integrates a set of complex work, in a way that helps ensure the business benefits of the project are ultimately met. The process framework can be adjusted and adapted for different sizes and types of NPD projects.

Is this, in something of a nutshell, what ITIL is for IT environments and projects?

Alan S Koch: Precisely!

Closing note from Cinda. My thanks to Alan for engaging in the back and forth dialogue that resulted in this article. I had truly gotten frustrated with ITIL presentations that had left me with more questions than answers. No organization takes on a new process lightly, and I want to understand just what it means to me and my teams on the ground, what benefits I can expect specifically, and how it relates to all the other good-practice processes so many of us are already trying to utilize. Just this one conversation with Alan cleared much of my personal ITIL fog; hope you've found it useful as well. I of course have more questions! But this conversation gave me a new base understanding of ITIL's purpose and scope and a sound basis for investigating ITIL further for my own organization.

If you have your own questions about ITIL — whether about the philosophy of it, details within it, or successfully implementing it, please feel free to post below or email them to me at cinda@projectconnections.com. And if you have your own perspectives to contribute, write away!

Quality = Business Value

2011-03-17T08:53:52-07:00

Part 2: Business Value as the Measure of Quality

by Alan S. Koch, PMP

What do we mean when we use the word "Quality"? In part 1 of this 2-part series, we explored the most common definitions of quality to understand how they fall short. Specifically, a product or service is "high-quality" if it

Conforms to Specification
Satisfies Users' Needs
Implements "Best Practices"
Is produced using defined processes
Is acceptable to the customer
Is driven by the customer
Fits cleanly into the operational environment
Does not fail in any serious way

Having done this, we are now ready to address the emerging new definition:

Quality is Delivering Business Value

As we do, we will see that it borrows the high points from each of the common definitions while leaving the baggage behind.

What is "Business Value"?

Every project must be justified in order to be approved and funded. Some of us as Project Managers are involved in this justification process, and others of us are given responsibility for projects after they have been justified and approved. But in either case, the justification should have been made and documented.

In most cases this justification can be summarized in terms of Return on Investment (ROI). The project requires some level of investment of money, effort and time, and that investment is worthwhile because it will pay back a return that is worth more than that investment. Sometimes that payback is monetary, so it can be measured more or less directly against the investment. (Invest $x so we can save $5x.) Sometimes the payback not monetary, or not primarily monetary, but the project is worthwhile because the decision makers deem that payback to be valuable in some other way. (Invest $x so we can improve customers' experience.)

So, while the Scope statement in our Charter calls out that the project will build X, do Y, and achieve Z, the "Business Value" is the net benefit that will be realized by the customer, and on which the project was approved and funded. A description of this expected Business Value should be included in a good Project Charter. It may be called the "Project Objective" or "Success Criteria," or something else. (But beware that those identifiers do not guarantee that they are talking about Business Value!)

If you find that Business Value has not been laid out in your Project Charter, your first order of business is to find out from the key stakeholders what is the Business Value they are expecting, and to document it as a part of your project's driving documentation.

How is "Business Value" measured?

We are used to measuring things on our projects. We can count requirements satisfied, or defects found. We can measure system performance and project Earned Value. And we can compute Schedule Variance and Cost Overruns. But the concept of Business Value is different from those.

The things we normally measure on a project are the "nuts and bolts" of what we are doing or building. But Business Value is on a different plane. Measuring Business Value involves stepping away from the details of project activity and looking at what is being done from the customer's perspective. It requires understanding what they are expecting from the project and why that result is important to them.

Sometimes the customer's statement of Business Value tells us precisely how to measure it. If the purpose of the Order Management System Upgrade project is to "Enable the sales department to process 50% more orders per day without increasing staffing levels," then there is no question about how to measure it when the upgraded system is deployed.

On the other hand, if the purpose of the Website Redesign project is to "Enhance customers' experience," we are left with many questions about how to measure it. The only way to get answers to those questions is through discussions with our key project stakeholders to understand their expectations. For example, if surveys of customer opinions are performed, they could provide a basis for measuring if the project has succeeded. Or perhaps there are specific features or capabilities that customers have been requesting. Perhaps there is a competing website that can stand as our benchmark.

In the end, we must measure the value produced by our project in the same way that our customer does. And for many of us, that is the crux of the paradigm shift. A successful project is no longer seen as one that merely delivers on time, on budget and with certain quality levels. Rather success means ensuring that our customer receives the value that they need as measured from that customer's perspective.

How Do We Deliver Business Value?

Even if we agree that delivering Business Value is our purpose, we are still left with a significant problem. How can we ensure that we are progressing toward that goal while we are in the project? Rarely will our customers' assessment of Business Value be possible before the project is over. So how do we measure our progress?

Taking a page out of the Agile playbook helps with this. The more collaboratively we approach the project, the more opportunities we will get to test our understanding of our customers' view of value, and the more opportunities we will have to get material feedback from them.

For example, after translating the project needs into a list of deliverables, we need to discuss that list with our customer. Is the list complete? Is it fully aligned with the Business Value the customer is expecting? How does each deliverable support that ultimate value? Asking these questions does far more than assure that the deliverables list is appropriate. The bigger win is the way our own understanding of Business Value (as estimated by our customer) will grow!

As we finalize the requirements, we need our customer to articulate how each of them enables the Business Value they expect. In addition to being a great way to validate the requirements, this will arm us with the customer-centric view that will enable us to translate each requirement into something that contributes to Business Value. And it will make it more likely that we will build the right product.

But "more likely" isn't enough. How to we ensure that we are actually progressing? Again, by working with our customer. Any time we build something that customers will be able to recognize and evaluate (like a component of the User Interface, or a data transformation), we get feedback on it. As the system becomes more and more complete, we will be able to demonstrate bigger chunks of it, and get better and fuller feedback on it. "Is this what you need?" "Will this work in your business process?" And the ever-present, "How will this support your realization of the Business Value you need?"

This kind of interaction on a regular basis will ensure not only that there are no unhappy surprises at the end of the project, but it will build a more positive relationship between customer and project team in the process.

Won't They Keep Changing Their Mind?

There is always the fear that this sort of unfettered visibility in to the project will lead the customer to continually escalate their demands. Although this is a possibility, it can be mitigated in two ways. The first is the observation at the end of the previous section: A more positive relationship contributes to more reasonable behavior. One of the major benefits of close interaction between customer and project team is that the customer quickly comes to realize that nothing is free.

Which leads to the other mitigation. Remember what we said at the beginning about Business Value and project justification! The project is only worthwhile if the value delivered exceeds the investment. Escalating demands mean escalating costs, which reduce ROI. When this dynamic is highlighted, the customer will temper his/her demands. But be prepared! If adding something to the project will improve the Business Value by more than it costs, it should probably be done!

What of the Traditional Measures of Quality?

Do we no longer need to worry about our list?

Conforms to Specification
Satisfies Users' Needs
Implements "Best Practices"
Is produced using defined processes
Is acceptable to the customer
Is driven by the customer
Fits cleanly into the operational environment
Does not fail in any serious way

Far from it. Defects, conformance and all of the others are important for us to manage the nuts and bolts of our technical work. A product that is riddled with defects will never deliver Business Value, no matter how good it looks!

We still need our focus on these things. But that focus must always be held in the context of the ultimate Business Value we are tasked to deliver. Doing something "well" for the sake of goodness does nothing for my customer. If that hot "Best Practice" doesn't benefit my customer, then it is not worth doing. A technically excellent product that doesn't meet the need is useless!

It's All About the Relationship

All of this boils down to building a very different relationship with our customer than we have traditionally had. If we are to deliver the value they need, we must understand them and their needs, which means that our connection with them must be richer and fuller. At the same time, if customers are to have assurance of receiving Business Value from us, they must ensure that we understand that Value, which means investing time and effort helping us to learn.

This relationship is an investment that is sure to pay a handsome return!

Related Links
Document your project's business value in a Project Charter or Opportunity Screening Worksheet. For a more Agile approach, try our technique brief on Project Value Models. Need a refresher on Agile PM techniques? Our paper Agile: Overview and Core Methods can help.

Quality = Business Value

2011-01-04T15:24:24-08:00

Part 1: Why common definitions of Quality fall short

by Alan S. Koch, PMP

How does your organization define "Quality"? Has a definition of "Quality" ever been formally adopted? Is there even a common understanding of what is meant by the word among all of the various people in your organization?

What about you? How do you define "Quality"? How many others in your organization would agree with you?

"Quality" is one of those words that is used all the time, but rarely defined. We are supposed to produce "quality" products (or to be more correct, "high-quality" products). But the precise meaning of that mandate is left to the imagination. Is it any wonder that we have so many disagreements about whether or not we have succeeded?

Even among those who have undertaken to define "Quality," there is more variety than consensus. For example, a product or service is "high-quality" if it…

Conforms to Specification
Satisfies Users' Needs
Implements "Best Practices"
Is produced using defined processes
Is acceptable to the customer
Is driven by the customer
Fits cleanly into the operational environment
Does not fail in any serious way

By examining the ways each of these traditional definitions falls short, we can appreciate an emerging new definition:

Quality is delivering Business Value

As we do, we will see that it borrows the high points from each of the common definitions while leaving the baggage behind.

How Common Definitions Fall Short

Conforms to Specification – This is perhaps the oldest of the definitions of Quality. It has been around since we began writing specifications decades ago. Since a specification is an agreement between producer and customer, the expectation of conformance is implied. And of course, when we promise something, ethics demands that we do what we said we would do!

The problem with using the specification as the definition of quality lies in the difficulty of writing specifications. Has everything that needs to be said been captured? And has it been written in sufficient clarity and detail to lead to the correct product? Can the customer and producer working together accurately forecast what will be needed? In what ways will the technology or business environment change between when the specification is signed and the product is delivered? (And many other problems.)

Although defining Business Analysis as a professional specialty has helped us to write better specifications, it has not proven to be a panacea (nor does it promise to be in the foreseeable future). Conformance to specification is generally a good thing, but sorry experience has proven it to be far from sufficient.

Satisfies Users' Needs – This definition came about primarily as a reaction to the problems identified above. In extreme cases, the concept of specification is abandoned altogether. But even where there is a specification, its focus is on the users' needs, and the project is more about those needs than the spec itself. What could be more appropriate than a people-oriented approach? Meet people's needs, and you have done your job.

The problem here starts with the definition of "User." Identifying all of those people is non-trivial, and subject to the same completeness issues as the specification itself. But the real problem here is that the users as a group do not represent all of the dimensions of quality. Many things that should be included in the product come from other Stakeholders who will never use the system itself (either directly or indirectly). So we are likely to end up with a myopic view of the needs and an insufficient product.

Beyond that danger is an even more important issue: Is satisfaction of every need of every user really appropriate? In many situations, this approach could be costly and result in a never-ending project that never quite reaches the goal of user satisfaction.

Implements "Best Practices" – As the decades have passed and more organizations have experienced various forms of success, we have seen a plethora of "Best Practices" being identified and promoted. These run the gamut from how systems should be developed to how various business activities should be done, and how technology should support those business processes. Following the patterns of successful organizations seems like a safe path to success.

The problem here is with the "Best Practices" themselves. I have adopted the rule of putting the term "Best Practices" in quotation marks because it has become clear to me (and I am not alone) that what is best for one organization may be a train wreck for another. And even if it does not constitute a train wreck, a practice may be merely passable and inferior to other options. As a friend and colleague of mine loudly proclaims, "There is no such thing as 'Best Practice'! There are only good practices that are appropriate in certain contexts." Applied to the wrong context, a "Best Practice" is a recipe for poor quality.

Produced using defined processes – The process movement of the last decade or more has resulted in this definition of Quality. The Software Engineering Institute's (SEI) CMMI® and the Project Management Institute's (PMI) PMBOK® are the best-known (though far from the only) examples of methodologies, models and BoKs (Bodies of Knowledge) purporting to tell us how to solve the difficulties we encounter on our projects. Although they are helpful, they can only take us so far.

Case in point: In the original CMM, the "Quality Assurance" Key Process Area was all about process quality, not the quality of the product! And although the new CMMI changed the name of that Process Area to "Process and Product Quality Assurance," the "products" are almost exclusively intermediate work products that result from the process, and "quality" is defined as conformance to standards.

The reality is that good processes can be employed to produce the wrong products. Process quality is an important measure of the development organization and how well it works internally. But as a measure of the quality of the result of our project, process quality has little utility.

Acceptable to the customer – This definition re-focuses on the customer. But in this case, it is not so much about the end users as it is the authority who is requesting and paying for the project. In this case, the "customer" is a manager or executive who has made the decision to request the product, and the final acceptance of the product by this person becomes the focus of the project. What could be more appropriate than this?

The first problem with this definition of quality is much like the problem with the "Users' needs" definition: Although the "customer" is a critical stakeholder, he or she does not constitute the entire stakeholder population. This person will be optimizing the project from one perspective, and is likely to make decisions that are not in the best interest of other stakeholders, especially end users or the organization at large. In the worst cases, this person may have an inadequate idea of what the true needs are, resulting in a product that he or she deems to be acceptable when it is actually useless.

When this "customer" is truly buying the product (either as a commercial transaction or as a budgetary transfer), the specter of budgetary considerations overriding others is a very real possibility. On the other hand, when there is no money (or funny-money) at stake, you run the risk of having a customer who never finds the product to be fully acceptable and a project that never ends.

Driven by the customer – The Agile methods attempt to remedy many of the problems we have discussed by making the Customer a member of the project team. And this "Product Owner" role is not merely a member of the team, but the primary driver of project activities. The Agile methods "time-box" the project to control costs by limiting schedule, and the Product Owner is given broad latitude to get as much he or she can from the project within those constraints.

This definition of quality has most of the same shortcomings as the prior one, "Customer Acceptance." While the customer is sure to get the product he or she wants, there is no guarantee that the needs of other stakeholders and the organization at large will be adequately met.

Fits cleanly into the operational environment – In some organizations, high availability and other operational issues are so critically important that they become primary considerations in product design and implementation. This approach remedies problems that many shops experience—operational issues not being sufficiently addressed during specification or development, only to surface during implementation. In a healthy organization, a new product's fit with the existing operational environment will always be considered.

This is only a problem when operational fit is the overriding or primary measure of quality. The trumping of customer need and other organizational concerns by operational issues fits the classic definition of "the tail wagging the dog." The Operations function exists primarily to support the organization and its customers, not the other way around. Operational fit should be a consideration in quality, not the consideration!

Does not fail in any serious way – Finally, we look at the common focus on defects as the measure of quality. Of course, testing (which is the primary tool employed to detect defects) is usually framed by the specification, so it begins where we did in this article—conformance to specification. But because no specification defines every detail, testers must go beyond the written document as they determine whether they are observing proper functionality or defects.

Defining quality in terms of defects is problematic in two ways. First, as we observed above, personal judgment becomes a significant part of the equation. Many disagreements between testers and developers come up, and it is inevitable that many of those disagreements are won by whoever is the stronger negotiator, more politically potent, or able to yell louder.

The second problem is that this tug of war between testers and developers leaves all of the other stakeholders out of the mix. The quality of the product is determined without input from people outside the project. This means that at the end of the project, when the developers and testers have declared the project to be of "good quality," other stakeholders could seriously disagree, and often do.

Quality is Delivering Business Value

Defining quality as delivering value to the business takes us a long way down the road to correcting the problems we have discussed in this article. It is not merely a matter of blending these definitions of quality. Rather it is a change of the point of reference from which quality is judged. The project is being undertaken in order to bring specific benefits (or value) to the organization. That value proposition becomes the touchstone for all decision-making in the project, not the least of which are the qualitative judgments about the product.

We will delve into what this looks like next time, in Part 2.

Blurring the Line between Dev & QA

2010-10-27T10:36:13-07:00

What's the difference between development and QA?

It's been decades since we began distinguishing between these two project roles, and in most organizations the fact that they are necessary and distinct from each other is taken as an article of faith. But new voices have arisen in recent years. Most of them do not suggest that we go back to the 1960s, but they do raise interesting questions about the Dev/QA dichotomy. How well has the traditional structure worked? What dysfunctions are crying out to be addressed? Can we make our projects more effective by re-thinking these two roles and how they relate to each other?

Credit where credit is due: When I spoke at the Pacific Northwest Software Quality Conference last week they were promoting an upcoming event in Portland, Oregon, which bears the title I have co-opted for this article. The 17 November 2010 session is being presented by the Software Association of Oregon in Portland, as a joint effort of their Developers and Quality Assurance Forums. (I wish I could be in Portland to participate!)

How Dev & QA Came to Be Distinct

Before the dawn of time (in the late 1800s), we saw the advent of the large corporation (with its need to process large amounts of financial data) as well as surges astronomical and other scientific research (with their need to process large amounts of scientific data). Thus, the computer was born. Large corporations and research organizations had "Computer rooms"—vast seas of desks at which people sat and scratched pencils down to nubs as they filled reams of papers with numbers. Their job title? "Computer."

A few decades later we developed machines that would eventually put those human Computers out of work, but there was no such thing as "software." The circuits in each digital computer were custom wired (by Electrical Engineers—EE's using wires) to perform the necessary functions. This practice was soon replaced by more generalized hardware and the concept of software (machine code) that the EE's used to encode the logic. As we progressed to Assembly language and then to higher-level languages like Fortran, the EE's began to branch and specialize. Some remained hardware engineers, and others transformed into coders of this new software stuff. Thus was born the role of "programmer."

Fun side note: The first computer "bug" was discovered in this time period under the direction of then-Lieutenant Grace Murray Hopper. A moth had been mashed by a relay in ENIAC, causing the relay to malfunction and the computer to stop working. When the moth's remains were found, the operators pulled it from the switch with a pair of tweezers, taped it into the logbook, noted dryly that the failure was the "First actual case of [a] bug being found," and reported that they had "debugged" the machine. (A picture of the log page is available on the Naval History and Heritage website as part of Admiral Hopper's biography.) We've been finding computer bugs ever since.

In the 1950s, it became clear that programming was a different profession from Electrical Engineering. Schools started providing education in this new arena, and ever so slowly, the ranks of programmers were filled by people who were not EE's. These people did all software work. They figured out what was needed, designed the systems, coded up the logic and data storage mechanisms, tested and "de-bugged" them, put them into production, operated them and kept them running.

From there we have seen a continuing parade of specialization. Operations vs. programming. Designing vs. coding. Developing vs. testing. Technical work vs. project management. And most recently, business analysis vs. system development. Each of these specializations has been driven by the fact that each of these jobs requires specialized capabilities and knowledge requiring different training, education, and experience paths.

It is generally recognized today that conceiving, designing, and building software requires a different set of skills than conceiving, designing, and performing tests of that software.

Separating QA from Dev

The distinction between developing and testing gave rise to the observation that "independence" of testing resulted in objectivity, which enabled the tester to see the system under test through a different lens than the developer. The tester did not share the developers' presuppositions, blind spots or biases. This made the tester more able to identify the defects that result from those failings. In addition, the tester felt no ownership of the code, and so had no reason to ignore or downplay issues with it.

The obvious advantages of independence gave rise to the idea that QA professionals on a project must be kept from working too closely with the developers. We moved the testers into separate departments and built walls between them and developers (literally and figuratively). We limited the opportunities for them to communicate across those walls, and built protocols for the communications that were allowed.

This degree of separation has been codified by regulators in some domains (for example, the US Food and Drug Administration—FDA), to the point that some regulators or auditors will cite an organization for not maintaining enough separation between developers and testers. And even in the majority of organizations, where this kind of structure is not required, separation is widely accepted as being an important contributor to product quality.

The Dysfunctions of Separation

While there is clearly value in recognizing testing as a separate specialty, and while the difference between the testers' and developers' views of the system under test enhances the effectiveness of testing, the ways in which we have separated developers and testers have resulted in a lot of dysfunction on projects.

Antagonism between testers and developers is common. Operating as two separate groups creates an "us vs. them" relationship. While "us vs. them" is not inherently bad, it can easily degenerate into an adversarial relationship. While some organizations do a good job of keeping this relationship from becoming too adversarial, many suffer the consequences of antagonism, often because of some of the other dysfunctions!

Misalignment of objectives results in developers and testers working at cross-purposes. When developers are rewarded for producing lots of code quickly, they are often motivated to minimize their own testing and other quality-oriented activities (e.g. design and reviews). This results in buggy (and sometimes dead-on-arrival) code being pushed into test, causing frustration for testers.

When testers are rewarded for writing lots of bug reports quickly, they are often motivated to be picky and report as a defect anything that strikes them as being questionable. This results in a flood of questionable defects reports washing back over the developers and frustrating them.

Respect is eroded when developers and testers don't appreciate what it takes to do each other's work. It is not uncommon for developers to view testing as beneath them. This idea comes from being unaware of the knowledge and skill required to do the job well. Testers sometimes view developers as not really caring about what they produce. This idea comes from having little (or no) experience grappling with the complexities of the systems that are being built.

Throwing things over the wall is the natural mode of operation when a wall separates people. This is a dysfunction to the extent that the thrower is unaware of the impact on the target, and the target doesn't appreciate the value of what is being thrown.

In the end, all of these dysfunctions boil down to the project not operating as a unified whole, but as two factions pushing against each other. Even in projects with well communicated and inspiring vision or mission statements, each faction will embrace and interpret the purpose of the project in a way that maintains, if not reinforces, the antagonism between them.

Blurring the Line

Much recent experimentation has been done with how we run software projects, and the experiences that are being reported call into question the traditional separation of Dev and QA. The Agile methods are one example of this experimentation.

The Agile methods as originally documented do not even mention the role of "tester." The development team is responsible for delivering what the customer needs in the form of a technically excellent product. This sounds like going back to the 1960s, before "tester" was a recognized role.

But a funny thing happened in the real world of the organizations who began adopting Agility. There were testers! And management (if not the developers) recognized that testers were an important part of the organization, so they asked what testers do on Agile projects. The official answer (based on the Agile books' silence about testers) was, "I don't know." And the experimentation commenced.

The Agile value of "Individuals and Interactions over processes and tools" meant that even if the testers operated as a separate team, there was significantly more communication between developers and testers than on traditional projects. On some projects, a tester or two was actually made a member of the Agile team, ratcheting the communication level upward.

In addition, the developers were now delivering small slivers of product to test at the end of each short iteration—sometimes even more often than that. Even in cases where there was still a wall to throw the product over, the dynamics of the regular volleys reduced the negative impact. And in the cases without walls (tester on the Agile team), there was a totally new cooperative dynamic that replaced throwing it over the wall.

The net effect reported out of many organizations is that many of the dysfunctions discussed above began to melt away. In working more closely together, testers and developers began to understand each other's work to an extent that improves respect. The complexities and difficulties of each job were better appreciated by the other professionals. Each began to appreciate the effects on the other party of what they do, leading to more cooperation and less antagonism.

But the biggest effect people are reporting is that the two teams operate less as two teams, and more as one team with one objective. We're all on this project together, and making it successful is our responsibility together. I cannot be successful unless you are, and vice versa!

But what of independence? How much do we lose when testers are more closely integrated with the development team? The emerging answer seems to be that not only do we not lose quality, but it is improved! The current experience seems to be that the need for separation was a myth. It's the existence of testing professionals on the project that brings the benefit. The separation added only dysfunction.

So What Can We Learn?

For those who are adopting Agility, the lesson is clear: Testers should be integrated into the Agile team and simply work shoulder-to-shoulder with the developers.

But for those of us who are not embracing Agility (or not yet), there is much we should ponder. What can we do on our projects to break down the walls between developers and testers? How can we align their objectives to avoid cross-purposes and encourage one-project thinking? What opportunities exist for increasing communication between development and testing? How can we build the respect of each for the other?

Whether or not your projects are Agile, the concept of blurring the line between Dev and QA is worth exploring. Let's continue experimenting, figure out what works in each situation, and lay the foundation for moving into the next generation of thinking about roles on our projects.

Related Resources
For a traditional look at the testing role, see our QA and Beta Test Manager Job Descriptions. Agile methodologies like Extreme Programming and provide a different perspective.

How to Estimate Program Size

2010-08-18T09:52:55-07:00

In my last article on Cost of Quality, I started out by blithely proposing, "Let's say we're going to write a system of 25,000 Lines of Code." Teri (a perceptive reader) called me on it! She wrote:

If a new system is built, how do you guess at how many lines of code there will be? You possibly can guess at the number of programs from looking at the requirements but how do you guess the number of lines involved for each program?

It's an important question without a quick and easy answer. This was what I told Teri.

Why Lines of Code?

In my article I used Lines of Code (LOC) as the unit for measuring the size of software. Some people embrace this unit and others are adamantly against it, preferring Function Points. So the first step is to decide if you will measure the size of your software in LOC, Function Points, or some other unit. A good unit will have these attributes:

It correlates with the effort involved in producing the software, defect rates and anything else we would want to use the estimates for. Both LOC and Function Points correlate nicely with these things.
It can be measured objectively in existing code. LOC is quite easy to measure; LOC counters abound. Function Points cannot be measured using a tool, and counting them manually in existing code is not an exact science.
It can be estimated by professionals. Function Points were designed precisely for estimating systems before they are built. Using LOC for this purpose is less straightforward, but it can be done handily using the techniques I will describe below.

These things lead me to choose LOC as my unit of choice. (And I am not alone.) But it is a professional preference. If you prefer a different unit, then by all means, use it.

After we have chosen a unit of measure, we can use it to do two things: Compile our historical data, and use that data to estimate each new system we will build.

Compiling Historical Data

We all have libraries full of code written by our developers. Those libraries are goldmines! Let's start digging.

Measure your existing software using your chosen unit. You could measure each source file, but this is likely not your best option because there are usually multiple components in each file. Measuring each component instead (e.g. each function or class) gives you a lot of data for the next two steps, which is a good thing. It also allows you to be more precise, which will make for better precision in the estimates based on your measurements.

Categorize each component you measured. The categories need to be things developers can visualize based on requirements (e.g., an I/O function or a certain type of a class). How many categories should you have? Too many will make it difficult for developers to slot individual components. Too few will give you less precision. My recommendation: you should have around a dozen categories. Of course, the size of your library will be a factor, because you need to have several examples of each category for the next step.

Document the size ranges for each category. If your data set is large enough to use statistical techniques, compute the Mean and Standard Deviation for each category. For smaller data sets (the norm when you are just getting started), just capture the Min, Mean, and Max sizes for each category.

Now you have a list of a dozen or so categories of software component that you find in your systems, and for each, you have basic size ranges. This gives you the data you need to estimate a new project.

Estimating the Size of a System

This estimating process can only be accomplished by development professionals. (Sorry! Project Managers must stand by and watch, or facilitate!) It works best as a workshop in which the whole development team can discuss the requirements and come to consensus on the estimates. If the team has not yet been assigned, the estimates should be prepared by at least two competent development professionals. Based on the requirements, the developers compile two lists:

Components that will change. If the project is to change or enhance an existing system, or if a new system will be build using an existing one as a starting point, the developers must identify each existing component that will be changed. For each of these components they must:
- measure its existing size;
- estimate how much of it will be changed (only a ball park estimate, e.g. around 30%); and
- compute the size of the change by multiplying the existing component size by the estimated percentage change.
New components to be built. The developers then forecast the new components they expect to build. For each of these components they must:
- identify the category from your historical data that it best fits into;
- determine whether it would be a "very small", "small", "medium", "large", or "very large" example of that category; and
- assign an estimated size based on category, the small-medium-large judgment, and your historical data. This is where the size of your historical data set comes in. If you have enough data to use statistical methods, then your size estimate for each component will be:
  - "Very small" = Mean – 2 Standard Deviations
  - "Small" = Mean – 1 Standard Deviation
  - "Medium" = Mean
  - "Large" = Mean + 1 Standard Deviation
  - "Very large" = Mean + 2 Standard Deviations
If your historical data set is small so that you only have Min, Mean and Max values, then your size estimate for each component will be:
- "Very small" = Half of Min
- "Small" = Min
- "Medium" = Mean
- "Large" = Max
- "Very large" = Twice Max
Either way, the developers must apply their best judgment to the "Very small" and "Very large" estimates. If the estimates look unrealistic, the developers should adjust them.

With the developer estimates in hand, you can easily compute the expected size of the new system. Just add up the estimated new and changed component sizes to get the total project size. You may want to inflate the total estimate if the team believes they have not identified all of the components.

Better Estimates Next Time!

At the end of the project, you will want to add all of the newly created components to your historical data. (More data is always better!) Be sure to categorize them properly and use the actual final sizes, not the estimates! You will also want to update the historical data for any existing components that have changed.

Then re-compute the size ranges for each category. After doing a few projects (or one big one), you will likely have enough data to compute Mean and Standard Deviation for your categories, which will give you better estimates than Min, Mean and Max.

Finally, the developers can improve their ability to use this sort of estimating technique by comparing their initial list of components and estimated sizes with what they ended up with at the end of the project. They should specifically look for:

Components they built or changed, but did not identify during estimation
Components they identified during estimation, but did not build or change
Size estimates that were way off (e.g. anything that was estimated "Medium" but ended up being "Very Large")

These errors should all be documented as lessons learned on the project. Then those lessons should be reviewed immediately before the next project is estimated.

Using techniques like these, you too will be able to blithely comment on the estimated size of your projects!

Related Resources
How do I get estimates when the work is uncertain?

IT project templates – an index for project managers

Software Requirements Specification – Make sure you capture all the requirements

Project Tradeoffs Table for negotiating trade-offs when you can't possibly complete every requirement

How Much Quality Can We Afford?

2010-06-08T11:06:21-07:00

Sure, it might be nice to build a higher quality product. But how much quality can we really afford? Well, let's break out our Cost of Quality Calculator and try out the numbers.

What is Our Cost of Quality Today?

First the basics: Let's say we're going to write a system of 25,000 Lines of Code, and our data tells us that we inject about 50 defects per 1000 lines of code. So along with those 25 KLOC, we will also write 1250 defects.

			Defect Injection		Defect Removal		Defects in Product		Cost of Quality
Activity	Size		Rate	#	Yield	#	#	/KLOC	Rate	Hours
Design & Code	25	KLOC	50	1250			1250	50.00
Totals				1250		0	1250	50.00		0

For a system of that size, we usually end up with a 20-page requirements document and a 40-page high-level design specification. And since we usually inject 5 defects per page in our documents, that gives us 100 defects in the requirements and 200 defects in the design bringing our total to 1550 defects.

			Defect Injection		Defect Removal		Defects in Product		Cost of Quality
Activity	Size		Rate	#	Yield	#	#	/KLOC	Rate	Hours
Write Requirements	20	Pgs	5	100			100	4.00
High Level Design	40	Pgs	5	200			300	12.00
Design & Code	25	KLOC	50	1250			1550	62.00
Totals				1550		0	1550	62.00		0

Of course, we always review the requirements. We review about 10 pages of requirements per hour, finding 40% of the defects in the document. We incur 2 hours of Cost of Quality to find and fix 40 defects.

			Defect Injection		Defect Removal		Defects in Product		Cost of Quality
Activity	Size		Rate	#	Yield	#	#	/KLOC	Rate	Hours
Write Requirements	20	Pgs	5	100			100	4.00
Review Requirements					40%	40	60	2.40	10	2
High Level Design	40	Pgs	5	200			260	10.40
Design & Code	25	KLOC	50	1250			1510	60.40
Totals				1550		40	1510	60.40		2

And of course, we test. Our data tells us that we find 50% of the existing defects in Unit test and that we can do it at the rate of 5 defects found and fixed per hour. We find 40% of the existing defects in Integration (at a rate of 2 hours per defect found & fixed, or 0.5 defects per hour), and 30% in System test (at a rate of 10 hours per defect found and fixed, or 0.1 defects per hour). Unfortunately, we also inject about 2.5 new defects for every 100 we fix.

			Defect Injection		Defect Removal		Defects in Product		Cost of Quality
Activity	Size		Rate	#	Yield	#	#	/KLOC	Rate	Hours
Write Requirements	20	Pgs	5	100			100	4.00
Review Requirements					40%	40	60	2.40	10	2
High Level Design	40	Pgs	5	200			260	10.40
Design & Code	25	KLOC	50	1250			1510	60.40
Unit Test			2.5%	19	50%	755	774	30.96	5	151
Integration Test			2.5%	8	40%	310	472	18.88	0.5	619
System Test			2.5%	4	30%	142	334	13.36	0.1	1416
Totals				1580		1246	334	13.36		2188

So this spreadsheet represents our normal 25KLOC software project. Over a person-year of effort spent in Cost of Quality work, and hundreds of defects delivered. More than 13 defects per 1000 lines of code is pretty bad.

How much will it cost us to improve it?

What if we were to do a more formal inspection of the requirements? Let's say we could only inspect about 2 pages per hour, but would find 65% of the existing defects.

			Defect Injection		Defect Removal		Defects in Product		Cost of Quality
Activity	Size		Rate	#	Yield	#	#	/KLOC	Rate	Hours
Write Requirements	20	Pgs	5	100			100	4.00
Inspect Requirements					65%	65	35	1.40	2	10
High Level Design	40	Pgs	5	200			235	9.40
Design & Code	25	KLOC	50	1250			1485	59.40
Unit Test			2.5%	19	50%	743	761	30.44	5	149
Integration Test			2.5%	8	40%	304	464	18.57	0.5	609
System Test			2.5%	3	30%	139	328	13.14	0.1	1393
Totals				1580		1251	328	13.14		2160

We found more defects, but look! Our cost of Quality went down! Why did that happen? Because those defects we removed early in the inspection didn't get found later in testing when they are more expensive. Cool! Let's try inspecting our design document, too!

			Defect Injection		Defect Removal		Defects in Product		Cost of Quality
Activity	Size		Rate	#	Yield	#	#	/KLOC	Rate	Hours
Write Requirements	20	Pgs	5	100			100	4.00
Inspect Requirements					65%	65	35	1.40	2	10
High Level Design	40	Pgs	5	200			235	9.40
HL Design Inspection					65%	153	82	3.29	2	20
Design & Code	25	KLOC	50	1250			1332	53.29
Unit Test			2.5%	17	50%	666	683	27.31	5	133
Integration Test			2.5%	7	40%	273	416	16.66	.05	546
System Test			2.5%	3	30%	125	295	11.79	0.1	1249
Totals				1577		1282	295	11.79		1959

Yes! Same again! Fewer defects and less time spent in Cost of Quality! Our coding is our biggest source of defects. What if we inspect the code as well? Let's plan for the same 65% yield and say we can inspect code at 100 LOC per hour. (That's gonna be a lot of hours, but here goes!)

			Defect Injection		Defect Removal		Defects in Product		Cost of Quality
Activity	Size		Rate	#	Yield	#	#	/KLOC	Rate	Hours
Write Requirements	20	Pgs	5	100			100	4.00
Inspect Requirements					65%	65	35	1.40	2	10
High Level Design	40	Pgs	5	200			235	9.40
HL Design Inspection					65%	153	82	3.29	2	20
Design & Code	25	KLOC	50	1250			1332	53.29
Design & Code Inspections					65%	866	466	18.65	100	250
Unit Test			2.5%	6	50%	233	239	9.56	5	47
Integration Test			2.5%	2	40%	96	146	5.83	0.5	191
System Test			2.5%	1	30%	44	103	4.13	0.1	437
Totals				1559		1456	103	4.13		955

Jackpot! We improved our defect level by almost a factor of three! And we cut our Cost of Quality by more than half! We have achieved a respectable defect level of less than 5 defects per 1000 lines of code while investing less than half of the quality effort we started with.

So, What Do The Numbers Mean?

Of course all of this is theoretical and based on some basic averages that many organizations have reported. This exercise will be most interesting when you plug in your own organization's numbers.

What? You don't have these numbers? Better start measuring your Cost of Quality today!

P.S. If you want to use my spreadsheet instead of building your own, just download it here or from my website at www.ASKProcess.com/resources/CostOfQuality.xls.

Related Resources
Our Requirements Walkthrough Checklist can help you organize a productive inspection of requirements. Make sure everyone knows what they should look for when reviewing requirements.

The Agile Customer

2010-03-30T09:41:08-07:00

Making your supplier a bit more Agile

"We are adopting an Agile method for our internal development projects." This student in my Agile Project Management Boot Camp is really beginning to embrace Agility. But ... "But on my project, some of the development work is being done by an outside contractor who is decidedly non-Agile! How can we be Agile when they are not?"

Agility is relatively easy when you control all of the parts of the project. But when others are involved, barriers to Agility can begin to spring up. The best way to overcome those barriers depends upon your situation. So let's explore options for injecting some Agility in spite of the Waterfall raining down on you from your supplier.

Hire an Agile Contractor

The obvious best case would be for you to contract with suppliers who have already embraced Agility. They will understand what you are trying to achieve and work with you to make the iterative lifecycle work to your advantage.

Unfortunately, many of us find ourselves beyond this point in the process. The supplier has already been chosen (by us or someone else) and changing suppliers is either impossible or highly undesirable. Next time, we must be sure to choose an Agile supplier!

But in the mean time, all is not lost on this project. Even though we find ourselves in a less-than-optimal position, we can still make the best of it. We can work with this supplier in a way that will make the relationship a bit more Agile.

Write an Agile Contract

The contract defines expectations with our supplier, so we really need to start there. Most contracts are written with the Waterfall in mind. First there are requirements, then a design, then (at the end of the timeframe) they deliver a product to us for acceptance. If we are to interact with our supplier in a more Agile way, then the contract must not enforce waterfall!

But what if the contract has already been written and signed? Can you say "Contract Mod"? Few contracts go unchanged for their lives. If your contract does not support an Agile approach, then change it. Very few suppliers would balk at a time and materials contract with no fixed deliverables pre-defined. (And if your powers-that-be won't go for the concept of an Agile contract, then you have bigger problems than a Waterfall supplier!)

The key difference with an Agile contract is in the deliverables. An Agile relationship calls for completely different deliverables than the normal waterfall-ish ones. Our Agile contract must call for Agile deliverables. For example:

The supplier will collaborate with us to define a Requirements framework that draws a broad picture of the product they are to build. In addition, they agree to work out the requirement details collaboratively with us as the project progresses.
If we feel that up-front design work is necessary for the type of product they are building, then the supplier must deliver an architectural framework (without all of the design details). This framework must provide enough detail so we can ensure that anything we are building will fit together with what the supplier will deliver to us.
The supplier must deliver a product deliverable to us for acceptance every month or two, beginning no more than two months after the requirements and architectural frameworks are complete. If the supplier is stuck in the waterfall, we must ensure that the earliest product deliverables are things they are capable of building. For example, most suppliers can build the main User Interface or workflow screens with no actual processing code or database undergirding it.
At the beginning of each of those month-or-two chunks of work, the supplier will collaborate with us to define precisely what they are to deliver in that time box, including the details of the requirements.
If this supplier's product must integrate with other software that we are developing, then these interim product deliverables must build up to the right set of capabilities at appropriate times in the project.

An Agile contract must establish time boxes and budget limits, but leave requirements details as the variable for managing the relationship. Ultimately, the supplier will be paid a fixed fee for a fixed timeframe, and will work with us (their customer) to determine precisely what they will deliver.

Be an Agile Customer

We are used to being the supplier for our customers. But when it comes to our supplier, the shoe is literally on the other foot. They are the supplier, and we are the customers. That means that we are the "single wring-able neck" that Ken Schwaber talks about in the Scrum book.

If we are successful at establishing an Agile contract with our supplier, that means that they will be delivering what we asked for. And that means that we must be asking for the right things! If we end up with the wrong product at the end of the project, it is because we have not been diligent in providing feedback to our supplier about each deliverable.

The other part of being an Agile customer is making the hard choices. When things aren't going well (there are technical difficulties, or the things we want take longer to do that we anticipated), then we must decide how to make the project fit into the prescribed time box. What requirements are truly required? And how much (or little) gold plating do we really need? And in the end, if it really will take more time and money to get what we need, we must admit it and pay the piper.

As the project moves forward, we will quickly come to understand precisely what our supplier is capable of delivering, both from a technology and a quality standpoint. This allows us to take any corrective action that is needed early in the project. And in the end, it allows us to get the greatest benefit we can from the contractor relationship while mitigating the risks.

Related Links
Partner Contract Guidelines
When outside partners are employed to execute part or all of a project, the contract with that partner is critical for defining expectations, due dates, and responsibilities. This template provides guidelines for typical contents for such contracts.

Vendor Assessment Checklist
A set of questions to ask a company or contract resource you're thinking of hiring for your project. The questions are aimed at ensuring you have a common understanding and good fit in goals, skills and experience, processes, and priorities. Avoid disaster! Choose your partner well.

Agile Project Management Index Our Agile Project Management resources help you understand and apply agile principles, practices, and techniques to your projects.