Using Quantitative Methods
in Contract Management
By Dr. John O’Neill, Quality Assurance Manager
Dr. John O’Neill is a statistician and a college instructor in advanced statistics and business management. An Air Force veteran, he has decades of experience in quality systems management. He can be reached at .

CLICK HERE to download a PDF of this white paper

Introduction
Techniques for applying statistics and other quantitative methods in manufacturing are well documented, but these tools are far less known and practiced in service companies. This paper discusses the benefits of a quantitative approach, explains how statistical methods can be used, and shows various methods of applying these tools.

As an award-winning, ISO 9001:2008 registered government contractor that provides diversified services to both the civilian and defense markets, Phacil’s experience in applying quantitative methods in a service environment shows the value of this approach. Much as financial tools are used in the analysis of costs, Phacil uses statistical tools to understand the quality and effectiveness of services performed.

To be successful in a wide variety of different service areas, there has to be a recognition that each customer has different requirements and expectations. Sometimes, the same customer will even have unique requirements on different contracts. While each customer has a need for superior service, customers will vary in how they judge this.

Early in the process, the contractor has to define the deliverables.  Sometimes these are detailed in the statement of work, sometimes only in vague terms.  However there are always tasks that must be performed, and the successful completion of these tasks depends on subordinate activities being done right.  Therefore, there has to be some type of work breakdown and some way of measuring the work done.  Finally, there has to be a way to measure the customer’s perception of results.  While the customer may not always perceive “well done” the same way the service provider does, the accomplishment of all tasks is a necessary precondition of customer satisfaction.

Therefore, to track performance, a company must have a way of quantifying accomplishments, as well as a way of measuring the customer’s ultimate perception of these accomplishments. 

W. Edwards Deming, the noted quality guru, wrote “A numerical goal without a method is nonsense.”  The selection of metrics in a service environment must be related to the needs of the customer, but also under the control of the contractor, or at least subject to the influence of the contractor.   A measurement that shows only natural variability is not useful in itself.  If the company has some method that can be deployed, or even an approach that can be tried, then measuring the results can verify progress or indicate the need for other solutions.

Monitoring Performance On The Job
As each customer, and each contract, is different, there is no one set of measurements, or metrics, that apply to all. However, establishing what data is to be monitored, how it is collected, and what controls should be put in place as early-warning indicators, should be determined up-front. At Phacil, prior to the start of the period-of-performance, and at regular intervals, our Program Managers decide on how the contract will be monitored. This commitment is documented on a form that records the particular requirements for the contract and the specific metrics to be used. This acts both as a means of focusing attention, and as a tool for auditing.

Two basic kinds of metrics can be created to measure the quality of work performed. The first type of metric can be one that compares results achieved to the specified outcome. Examples of this type of metric include percentage completed, thruput per month, accuracy, time saved, funds recovered, system availability, and others. The second type of metric measures the factors that will affect the overall result desired. For example, system availability is impacted by changes in mean time to repair. The usefulness of an IT system is dependent on uptime, help desk response time, ease of use, and wait time.

After reaching a conclusion on what things are most relevant to measure, the company must decide on its measurement approach. How do you collect the data and verify its accuracy? How is the data presented so that it can be understood? Who analyzes the data, and how is the information communicated to the people who can control the underlying processes?

In short, what are the metrics and how do you use this for decision making?

Here are some examples of the types of metrics that are useful in a service environment.

Example 1: A Measure of Service Productivity  

Figure 1 A simple chart of output shows that peaks are handled and assigned workload is covered.

The chart in figure one is a measure of service productivity. This is data taken from a warehouse work center. It shows the fluctuation in workload, and how the employees are able to deal with peaks. As there are periods where the number of lines to be processed are relatively low, the program manager was able to plan quickly-deployable cross-training activities for these periods. When the workload was at its maximum, the team was able to successfully complete the work. The availability of cross-trained employees contributes to the flexibility in handling peak demands.

Example 2: Quantity & Quality of Monthly Deliverables

Figure 2 A table showing quantity and quality of monthly deliverables

This table above summarizes the number of documents reviewed and submitted to a government agency. The second column shows the number accepted, after government review, as having conformed to requirements. The government’s goal is for 100% accuracy, and this is met.

Example 3: Tracking Performance in a Software Application

Figure 3 An automatically-generated chart showing the time it takes users to access a portal Phacil created and manages.
This far exceeds specification, and contributes to higher-level goals.

The chart above is used to track performance in a defense software application. The agency has a high degree of interest in the software meeting its intended purpose. In addition to the portal content, the system must be user-friendly. The excerpted portion of this chart shown above, displays the wait time users experience in accessing the system. The wait time, in all cases, is a fraction of a second. This is monitored continuously, as is system uptime and other metrics.

Example 4: Attainment of a Goal

Figure 4 A graph showing attainment of a goal

Providing technical support can be measured in several ways. Upon beginning work on a contract, a team may experience unresolved issues, and may establish a goal to improve customer support. The chart above shows the outcome of a help-desk support activity. When a computer system user calls in to report a problem, a team member resolves the issue to the customer’s satisfaction, and then the ticket is closed. This goal is to reduce the number of open tickets, and the team reduced this to zero, and maintained this level.

Having a variety of key areas assessed in the same manner from a larger number of customers provides much information for analysis.

1.       A comparison of the individual categories alongside each other (a stock type chart: high, low, median) provides a graphic representation of the data, and helps identify areas of opportunity where the range is large, or a category is shifted below other categories.

2.        A year-to-year comparison shows the success of improvement activities. Tests of significance indicate how much credit should be given to an improved number.

3.       A regression gives an indication of factors that are associated or may be associated with some other factor. For example, there may be a few categories that have more of an impact on overall satisfaction, or with a non-survey result.

4.       Individual customer results can be compared. It must be remembered that a unique person has completed each survey, there are lessons to be learned from both the customers who give a relatively low rating, as well as the ones who give a high rating.

To illustrate the use of an internal performance-rating system a company could use, here is a stock type chart a company could use, showing the highest, lowest, and median scores in various categories, for a particular year. In this example, values have been assigned to show the effect of changes over time.

Example 5: Measuring Customer Perception

Figure 5 Illustration of the method of displaying Customer Survey Data (Hypothetical Case – Year 1)

The chart shows that, while the company in this example is generally well-regarded by customers, there are some low marks presenting areas for improvement. Analysis of the data suggests that there is a high correlation between responsiveness and business relations. One explanation is that if it is hard to get hold of the right people who can provide answers, or get back to the customer with a solution, the customer will give low scores to business relations as well as responsiveness. If the company does a root cause analysis and corrective action to improve communications, they should see an improvement in both business relations and responsiveness. The next chart, by way of illustration, shows what might be the next year’s survey results:

Figure 6 Illustration of the method of displaying Customer Survey Data (Hypothetical Case – Year 2)

The one customer who had (in our illustration) previously rated responsiveness as 4 out of 10 has noticed improvements that has partially mitigated the prior impression, and now has rated the company at a 7, and also gave a higher score to business relations. Further, the improved processes have have resulted in other customers seeing improvements, and the average in these two criteria have risen.

The internal survey is a useful tool for understanding the results of a company’s efforts. Additionally, our government customers employ two main types of assessment tools.  These assessments are available to other agencies to use in selecting potential suppliers.  Many civilian agencies use a Contractor Performance Rating, while defense agencies use a Contractor Performance Assessment Rating system. Unfortunately, the grading scales and the number of grades differ between the systems.  In order for this information to be most useful to the company, we normalize the data by establishing a point system (by procedure) where the highest possible score in each system is given 10 points.

By having a standard normalized score, we can compare customer evaluations between customers as well as compare results year-to-year.  A tracking system shows the normalized scores from each assessment method so the results can be compared and analyzed.

Conclusion
Dr. Deming wrote, “If you can't describe what you are doing as a process, you don't know what you're doing.” A process includes the steps followed to reach a desired outcome, as well as all of the input conditions, such as the character of the employees, the abilities of the leaders, the work environment, etc. In the service environment, we typically don’t have lathes, milling machines or blast furnaces as part of our inputs. Instead, we have people. Our job is to maximize the chance of success our people have in producing the results the customer deserves by optimizing our processes.

High customer satisfaction is highly related to successful performance in meeting task requirements, but success in meeting tasks is not the only important consideration; there must be a good business relationship and adaptability to changing circumstances.

Monitoring and measurement are continually required to assess and to adjust performance. Metrics are tailored to the specific needs of each customer. Data without analysis is of little value. The use of quantitative methods is an essential tool but only one tool. Processes must be understood and managed.

The customer’s evaluation of your performance is your future.

Phacil brings a wealth of experience in managing contracts for defense and civilian customers. Our mission is providing exceptional service through our total commitment to continuous improvement. Our professional team is dedicated to understanding our customers’ needs and actively managing the processes to meet and exceed expectations. Learn more at Phacil’s website http://www.phacil.com/ or by calling (703) 562-4250