Defense Acquisition: Best Commercial Practices Can Improve Program
Outcomes (Statement/Record, 03/17/99, GAO/T-NSIAD-99-116).

Pursuant to a congressional request, GAO discussed the best practices
that can improve the way the Department of Defense (DOD) buys major
weapon systems.

GAO noted that: (1) on the basis of the work GAO has done over the past
3 years, GAO believes the best practices of leading commercial firms can
be used to improve the development of technology and weapon systems in
DOD; (2) knowledge standards that are rigorously applied, coupled with
the practice of keeping technology development separate from product
development, stand out as key factors in the most successful commercial
examples; (3) these practices have put managers in the best position to
succeed in developing better products in less time and producing them
within estimated costs; (4) DOD programs, with some exceptions, proceed
with lower levels of knowledge available about key factors of product
development; (5) DOD allows technology development to take place during
product development; (6) these practices put DOD program managers in a
much more difficult position to deliver better weapons more quickly and
within cost projections; (7) getting better outcomes on weapon systems
programs will take more than attempting to graft commercial best
practices onto the existing acquisition process; (8) there are
underlying reasons and incentives for why such practices are not a
natural part of how weapon systems are bought; (9) environmental factors
encourage lower standards of knowledge and the acceptance of higher, but
unrecognized, risks; (10) what GAO offers to help the adoption of best
practices is not a cookbook recipe, but a series of actions aimed at
fostering an environment in DOD that encourages or rewards such
practices; and (11) these actions will put managers of DOD programs in a
better position to succeed, for GAO believes they are as informed and
capable as their commercial counterparts.

--------------------------- Indexing Terms -----------------------------

     TITLE:  Defense Acquisition: Best Commercial Practices Can Improve 
             Program Outcomes
      DATE:  03/17/99
   SUBJECT:  Weapons systems
             Procurement practices
             Procurement planning
             Defense procurement
             Private sector practices
             Defense cost control
IDENTIFIER:  DOD Critical Design Review
             F-22 Aircraft
             C-17 Aircraft
             Comanche Helicopter
             Navy New Attack Submarine Program
** This file contains an ASCII representation of the text of a  **
** GAO report.  This text was extracted from a PDF file.        **
** Delineations within the text indicating chapter titles,      **
** headings, and bullets have not been preserved, and in some   **
** cases heading text has been incorrectly merged into          **
** body text in the adjacent column.  Graphic images have       **
** not been reproduced, but figure captions are included.       **
** Tables are included, but column deliniations have not been   **
** preserved.                                                   **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
** A printed copy of this report may be obtained from the GAO   **
** Document Distribution Center.  For further details, please   **
** send an e-mail message to:                                   **
**                                                              **
**                    <[email protected]>                        **
**                                                              **
** with the message 'info' in the body.                         **
****************************************************************** GAO United States General Accounting Office

Testimony Before the Subcommittee on Readiness and Management
Support, Committee on Armed Services, U. S. Senate

For Release on Delivery on Wednesday, March 17, 1999 DEFENSE

Best Commercial Practices Can Improve Program Outcomes

Statement for the Record by Louis J. Rodrigues, Director, Defense
Acquisitions Issues, National Security and International Affairs


Page 1 GAO/T-NSIAD-99-116 Best Practices

Mr. Chairman and Members of the Subcommittee: Thank you for the
opportunity to submit this statement for the record, which
discusses the best practices that can improve the way the
Department of Defense (DOD) buys major weapon systems. With DOD's
annual research, development, and production spending for major

at about $85 billion, the Subcommittee's oversight of acquisition
policy can have a major impact on the value the taxpayer gets for
that expenditure.

For several years, the leadership in the Office of the Secretary
has been committed to instituting reforms to improve the outcomes
of the acquisition process. In particular, the Under Secretary of
Defense's (Acquisition and Technology) focus on shorter cycle
times is welcome.

Shorter cycle times make for a more agile acquisition process,
which is critical if DOD is to respond quickly in a national
security environment of unknown threats. Shorter cycle times are
also important for solving problems associated with readiness by
ensuring that the industrial base will be able to support systems
once they are fielded. Despite good intentions and some progress,
our ongoing reviews of DOD's major system acquisitions are showing
that these efforts at systemic change have not yet been reflected
in the management and decision- making on individual programs. The
flagship systems, as well as many other top priorities in each of
the services, continue to cost significantly more, take longer,
and deliver less than was promised. Our work for this Subcommittee
shows that lessons for major system acquisitions can be learned
from the best commercial practices and applied

in the DOD environment. Adopting these practices will require a
dramatic change in behavior a change that must be supported by
incentives. In this context, the Office of the Secretary of
Defense, the Congress, and the

services' organizations for requirements, research, and
acquisition, each play a critical role in getting the better
outcomes sought on major weapon system programs.

Results in Brief On the basis of the work we have done over the
past 3 years, we believe the best practices of leading commercial
firms can be used to improve the development of technology and
weapon systems in DOD. In particular,

knowledge standards that are rigorously applied, coupled with the
practice of keeping technology development separate from product
development, stand out as key factors in the most successful
commercial examples. These practices have put managers in the best
position to succeed in

Page 2 GAO/T-NSIAD-99-116 Best Practices

developing better products in less time and producing them within
estimated costs. DOD programs, with some exceptions, proceed with
lower levels of knowledge available about key factors of product
development, such as proof of design maturity and production
readiness. In addition, DOD allows technology development to take
place during product development. These practices put DOD program
managers in a much more difficult position to deliver better
weapons more quickly and within cost projections. Getting better
outcomes on weapon system programs will take more than attempting
to graft commercial best practices onto the existing acquisition
process. There are underlying reasons and incentives for why such
practices are not a natural part of how weapon systems are bought.
Environmental factors, such as the intense competition for funding
when a program is launched, encourage lower standards of knowledge
and the acceptance of higher, but unrecognized, risks. What we
offer to help the adoption of best practices is not a cookbook
recipe, but a series of actions aimed at fostering an environment
in DOD that encourages or rewards such practices. These actions
will put managers of DOD programs in a better position to succeed,
for we believe they are as informed and capable as their
commercial counterparts.

Following the discussion of our work on best practices, we present
information on the status of other work we are doing that is of
interest to the Subcommittee. Most of this work deals with DOD
initiatives related to acquisitions. The work includes acquisition
workforce training in best practices, best practices for test and
evaluation of weapon systems, pricing

of sole- source commercial items, spare parts price trends, the
Cost Accounting Standards Board, other transactions, and
government- wide information technology contracts.

A Best Practices Model for Acquisition

For an acquisition process that meets DOD's goal of developing and
producing militarily superior weapons in a resource- constrained
environment, we look to answer the basic question of how a
capability can best be provided to the customer. The
characteristics of best practices, as we have analyzed them,
suggest a process for developing new

capabilities whether they are commercial or defense products that
is based on knowledge. It is a process in which technology
development and product development are treated differently and
managed separately. The process of developing technology
culminates in discovery and must, by its nature, allow room for
unexpected results and delays. The process of

Page 3 GAO/T-NSIAD-99-116 Best Practices

developing a product culminates in delivery, and therefore, gives
great weight to design and production. Discipline is inherent
because criteria exist, tools are used, and a program does not go
forward unless the strong

business case on which the program was originally justified
continues to hold true. In the past several years, we have
examined best practices used by world

class commercial firms such as Boeing, Chrysler, Hughes, Ford, and
3M, and individual DOD acquisition programs for weapons such as
the F- 22, the C- 17, the Comanche, the New Attack Submarine, and
the Advanced Amphibious Assault Vehicle with the objective of
finding best practices for developing and producing major weapon
systems. Our completed work has examined best practices for
quality assurance, earned value

management, supplier management, and transitioning products from
development to production. A listing of these reports is included
in the appendix. We are currently reviewing best practices for

technology for inclusion in product development programs, best
practices for test and evaluation of weapon systems, and how well
DOD training supports the implementation of best practices. We
have learned that a knowledge- based process is essential to
getting better cost, schedule, and performance outcomes. This
means that decisionmakers must have virtual certainty about
critical facets of the product under development when needed. Such
knowledge is the inverse of risk. The commercial and military
programs we reviewed did not all follow the same processes in
their development cycles. However, at some

point, full knowledge was attained about a completed product,
regardless of what development approach was taken. This knowledge
can be broken down into three junctures that we refer to as
knowledge points: when a match is made between the customer's
requirements and the available technology, when the product's
design is determined to be capable of meeting performance
requirements, and when the product is determined to be producible
within cost, schedule, and quality targets. In addition, we

have identified metrics that indicate the knowledge levels
associated with best practices and can thus help forecast problems
as a development program progresses. An important corollary to
having a knowledge- based process is that technology development
should take place separate from an acquisition program and its
related product development process. The knowledge points and
their associated metrics are depicted in figure 1.

Page 4 GAO/T-NSIAD-99-116 Best Practices

Figure 1: Levels of Knowledge Attained in Best Practices for
Developing Technology and Products

The leading commercial firms we visited gained more knowledge
about a product's technology, design, and producibility much
earlier than DOD in the acquisition programs we reviewed. In fact,
product development in

commercial ventures is a clearly defined undertaking for which
firms insist on having the technology in hand that is needed to
meet customer requirements before starting. Once underway, the
firms demand and receive specific knowledge about the design
capability and producibility of a new product before production
begins. The process of discovery the accumulation of knowledge and
elimination of unknowns is completed well ahead of production.
There is a synergy in this process, as the attainment of each
successive knowledge point builds on the preceding one. In
contrast, DOD programs are started earlier and allow technology
development to continue into product development and even into
production. Consequently, the programs proceed with much less
knowledge available and thus more risk about required

design capability, and producibility. Proceeding with lower levels
of knowledge available explains much of the turbulence in DOD
program outcomes. Metrics, such as those associated with knowledge
points, show this to be a predictable consequence.

Technology Development Product Development Production

Knowledge Point 1

Technology readiness

levels Percent of

drawings complete

Percent of key production processes in

control Metrics

Knowledge Point 2

Knowledge Point 3

Page 5 GAO/T-NSIAD-99-116 Best Practices

Knowledge Point 1: Requirements and Technology Are Matched

Technology development has the ultimate objective of bringing a
technology up to the point that it can be readily integrated into
a new product and counted on to meet requirements. As a technology
is developed, it moves from a concept to a feasible invention to a
component that must fit onto a product and function as expected.
In between, there

are increasing levels of demonstration that can be measured. In
our ongoing review of best practices for including new technology
in products, we applied a scale of technology readiness levels
from one to nine pioneered by the National Aeronautics and Space
Administration and adapted by the Air Force Research Laboratories.
Without going into the details of each level, let me note that a
level four equates to a laboratory demonstration of a technology
that is not in a usable form. Imagine, if you will, an advanced
radio technology that can be demonstrated with components that
take up a table top. A level seven is the demonstration of a
technology that approximates its final form and occurs in an
environment outside the laboratory. A level eight is a technology
that has been proven to work in its final form and in its intended
operating conditions. The same

radio at this level would have been installed in the instrument
panel in the aircraft cockpit, integrated with other aircraft
systems, and flown under all expected conditions.

The lower the level of the technology at the time it is included
in a product development program, the higher the risk that it will
cause problems in the product development. According to the people
that use the technology readiness levels in DOD, level seven
enables a technology to be included on a product development with
acceptable risk. In applying these standards to leading commercial
firms, we have observed that the firms do not let a new technology
onto a product development until it reaches level eight. On weapon
systems that experienced cost and schedule problems, we observed
that they were started with key technologies at levels three and
four. By the time the same programs reached a point DOD considers
analogous to beginning product development, key technologies were
still at level five or lower.

We also observed that three factors contribute to the successful
maturation of technology for inclusion on a product development.
These are: flexibility in resources and requirements to allow for
the uncertainties of

technical progress; disciplined paths for technology to take
toward inclusion on products, with strong gatekeepers to decide
when to allow it onto a product development program; and high
standards for judging the maturity and readiness of technology.
The commercial technologies and a few of the DOD technologies we
reviewed exhibited these factors and were

Page 6 GAO/T-NSIAD-99-116 Best Practices

successfully included in product developments. Technologies that
caused problems in product development did not exhibit all of
these factors. In essence, these technologies were still going
through discovery in a delivery- oriented environment. As one
might imagine, the most difficult

situation occurs when a product development is launched with
inflexible requirements that can only be met with a new and
immature technology. Knowledge Point 2: The Design Will Perform as

Required The commercial firms we visited had achieved near
certainty that their product designs would meet customer
requirements and had gone a long

way toward ensuring that the products could be produced by the
halfway point of product development. Both DOD and the commercial
firms hold a critical design review (CDR) to review engineering
drawings, confirm the

design is mature, and freeze it to minimize changes in the future.
The completion of engineering drawings and their release to
manufacturing organizations signify that program managers are
confident in their knowledge that the design performs acceptably
and is mature. The drawings are critical to documenting this
knowledge because they are not

only precision schematics of the entire product and all of its
component parts they also reflect the results of testing and
simulation, and they describe the materials and manufacturing
processes to be used to make

each component. Both DOD and commercial companies consider the
design to be essentially complete when about 90 percent of the
engineering drawings are completed. Officials from commercial
companies such as Boeing and Hughes told us that they typically
had over 90 percent of these drawings available for the CDR. Two
DOD programs we reviewed had less than

60 percent one had less than one- third of the drawings done at
the time their CDRs were held. Thus, these programs had
significantly less knowledge available about their designs. The
programs did not get or were

not expected to get to the 90- percent level of completion on the
drawings until late in development or in production. Nonetheless,
at the time of the CDRs, the risks of proceeding with the rest of
development on these programs as planned were deemed acceptable.
Both programs encountered significant design problems in testing
that occurred after the CDR.

Page 7 GAO/T-NSIAD-99-116 Best Practices

Knowledge Point 3: Production Units Will Meet Cost, Quality, and
Schedule Objectives

Leading commercial firms reached the point at which they knew that
manufacturing processes would produce a new product conforming to
cost, quality, and schedule targets before they began fabricating
production articles. Reaching this point meant more than knowing
the product could be manufactured; it meant that all key processes
were under control, such that the quality, volume, and cost of
their output were proven acceptable. Commercial firms relied on
good supplier relationships, known

manufacturing processes, and statistical process control to
achieve this knowledge early and, in fact, had all their key
processes under statistical process control when production began.
The ability to establish control for key processes before
production began was the culmination of all the practices employed
to identify and reduce risk. All of the companies we visited
agreed that knowledge about technology and design early in the
process makes the control of processes possible.

One weapon system program that had been in production for nearly 9
years at the time of our 1998 review still had less than 13
percent of its key manufacturing processes in control. Another
program had 40 percent of its key manufacturing processes in
control 2 years before production was scheduled to begin, but was
not scheduled to have all key processes in control until 4 years
into production. Both programs experienced basic producibility
problems that were not discovered until late in development or
early in production. These risks went unrecognized even though the

DOD had established criteria for determining whether risks were
acceptable and whether enough knowledge had been gained to enter
the next development phase.

Impediments to Adopting Best Practices

The most important factors in the adoption of best practices are
the incentives the development or acquisition process offer to
managers of technology and product developments. The differences
in the practices employed by the leading commercial firms and DOD
reflect the different demands imposed on programs by the
environment in which they are managed. The way success and failure
are defined for commercial and

defense product developments differs considerably, which creates a
different set of incentives and evokes different behaviors from
the people managing the programs. Specific practices take root and
are sustained because they help a program succeed in its

Leading commercial firms launch a product development only when a
solid business case can be made. The business case basically
revolves around

Page 8 GAO/T-NSIAD-99-116 Best Practices

the ability to produce a product that the customer will buy and
that will provide an acceptable return on investment. The point of
sale occurs in production after product development is complete;
program success is determined when the customer buys the finished
product. If the firm has not made a sound business case, or has
been unable to deliver on one or more of the business case
factors, it faces a very real prospect of failure

the customer may walk away. Also, when each program is delivered
as promised, the company does not put at risk resources invested
in other products. Because the match between technologies and
product requirements is made before the product development is
launched, the cost and schedule consequences associated with
discovery are minimized. Production is a dominant concern
throughout the commercial product development process and forces
discipline and trade- offs in the design

process. This environment encourages realistic assessments of
risks and costs; doing otherwise would threaten the business case
and invite failure. For the same reasons, the environment places a
high value on knowledge for making decisions. Program managers
have good reasons to want risks identified early, be intolerant of
unknowns, and not rely on testing as the main vehicle for
discovering the performance characteristics of the product. By
protecting the business case as the key to success, program
managers in leading commercial firms are conservative in their
estimates and aggressive in risk reduction. Ultimately, preserving
the business case strengthens the ability of managers to say no to
pressures to accept high risks or unknowns. Practices, such as
maturing technologies to high

readiness levels before inclusion in a program, having 90 percent
of engineering drawings done by the CDR, and achieving statistical
process control before production, are adopted because they help
ensure success.

The basic management goal for a weapon system program in DOD is
similar: to develop and deliver a product that meets the
customer's needs. However, the pressures of successfully competing
for the funds to start and sustain a DOD acquisition program make
for a much different business case. Compared with commercial
programs, the DOD environment

encourages launching product developments that embody more
technical unknowns and less knowledge about the performance and
production risks they entail. Although DOD is attempting to create
more flexibility in how technical requirements are set for
programs, a new product development is encouraged to possess
performance features that significantly distinguish

it from other systems. Consequently, aspiring DOD programs have
incentives to include performance features and design
characteristics that rely on immature technologies. These unknowns
place a much greater

Page 9 GAO/T-NSIAD-99-116 Best Practices

reliance on maturing technology during product development than we
found on commercial programs.

Even though less information about a new product development is
available at the time of launch, the competition for funding
requires detailed projections to be made from what information
does exist. A product development deemed worthy cannot be launched
unless the program's development and production cost, as well as
timing, fall within available funding. Because DOD relies largely
on forecasts of cost,

schedule, and performance that are comparatively soft at the time,
success in funding competition encourages the cost and schedule
estimates to be squeezed into profiles of available funding.
Additional requirements, such as high reliability and
maintainability, serve to make the fit even tighter. As
competition for funding will continue throughout the program's
development, success is measured in terms of ability to secure the
next installment.

Untempered by knowledge to the contrary, the risks associated with
developing new technologies together with the product within tight
estimates are deemed acceptable. Production realities, critical to
matching technological capabilities with customer requirements on
commercial programs, are too far away from the DOD launch decision
to have the same curbing effect on technology decisions. Thus, the
environment for managing weapon system programs is a particularly
difficult environment for managing technology development. The ups
and downs and resource changes associated with the technology
discovery process do not mesh well with a program's need to meet
cost, schedule, and performance goals. Problems with developing
technologies, which are to be expected, can actually threaten the
support for a program if they become known.

These pressures and incentives explain why the behavior of weapon
system managers differs from that of managers of commercial
product developments. Problems or indications that the estimates
are decaying do

not help sustain funding support for the program in subsequent
years, and thus, their admission is implicitly discouraged. An
optimistic cost estimate makes it easier to launch a product
development and sustain annual approval; admission that costs are
likely to be higher could invite failure. Rewards for discovering
and recognizing potential problems early in a DOD

product development are few. Less available knowledge makes it
harder for program managers to say no. In contrast with leading
commercial firms, not having attained knowledge such as on the
full performance of a key technology or the true risks facing
manufacture can be perceived as

Page 10 GAO/T-NSIAD-99-116 Best Practices

better than knowing that problems exist. For these reasons, the
practices associated with managing to knowledge points such as
applying knowledge standards for technology, design, and
production maturity are

not readily adopted in DOD. Our ongoing work on technology
inclusion shows that in addition to the incentives provided by the
acquisition environment, some structural and budgetary impediments
exist that make it difficult for technology to be matured to a
high level of readiness before being included in a DOD

weapon system development. First, it is not clear that
organizations exist other than program offices that have the role
to take technology from a readiness level of three or four, where
new technologies are often put into a program, to the level seven
needed to be readily included in a product development. We found
that for the DOD cases in which high technology readiness levels
were attained before program launch, organizations stepped in and
played atypical roles in bridging the gap from science and
technology to product development. Second, budget realities the
fact that programs attract much higher levels of funding than

science and technology projects make programs a favored venue for
funding technology development through the higher and more
expensive readiness levels. Third, if science and technology
organizations became responsible for managing technology
development to the higher levels, they most likely would need
additional funding.

These observations about the differences between the commercial
and DOD environments should not be interpreted to mean that
commercial managers are somehow more skilled or knowledgeable than
their DOD

counterparts. Nor do DOD program managers act irrationally. They
see the acquisition of the weapons under their purview as aligned
with national interests, and they do what they believe is right,
given the pressures they face. All of the numerous participants in
the acquisition process play a part in creating these pressures.
In fact, the weapon systems acquisition

process asks much more of DOD program managers than commercial
firms do. Perhaps they are asked to do too much: develop advanced
technology, manage product design and production, and champion the
program's cause

through budget and other decisions over many years. In fact, one
commercial executive observed that it is unreasonable to expect
people to focus on a goal (such as production start- up) that is 4
or more years away. Commercial program managers are put in a
better position to succeed; they have to worry only about product
design and production within the cost, schedule, and performance
demands of the business case.

Page 11 GAO/T-NSIAD-99-116 Best Practices

Charting a Course for Adopting Best Practices to Get Better

Commercial practices for gaining knowledge and assessing risks can
help produce better outcomes on weapon systems. Collectively, such
individual outcomes will help attain DOD's modernization goals and
improve funding stability for programs. For such practices to
work, however, the knowledge they produce must help a DOD program
succeed in its

environment. Thus, the DOD environment must become conducive to
such practices. At least two factors are critical to fostering
such an environment. First, program launch decisions must be
relieved of the need to overpromise on performance and resource
estimates. The pressure to amass broad support at launch creates
incentives for new programs to embrace far more technology
development than commercial programs do.

Separating technology development so that it does not have to be
managed within the confines of a weapon systems program would go a
long way to relieving this pressure. The objectives of technology
development, as well as what is demanded of knowledge and
estimates, differ from those of product development. Clearly, DOD
has to develop technology, particularly the technology that is
unique to military applications. However, by separating technology
development from weapon programs, DOD could insist on higher
standards for knowledge on its programs and get better results
when those programs transition to production.

Second, once a program is underway, the participants in the
acquisition process must make it acceptable for managers to
identify unknowns as high risks so that they can be aggressively
worked on earlier in development. Commercial firms insist on
knowledge measured against a

criterion for assessing risk. Firms then make decisions to
preserve the business case by eliminating risks. The result is
discipline provided from within. We believe that if the Congress
and DOD weighed program launch decisions and subsequent progress
on weapon systems by applying a common set of knowledge standards,
like those gleaned from leading commercial firms, they could
create a better business case for weapon

systems. By developing technology separately to high readiness
levels before including it in a program and by adhering to
knowledge standards in product development, DOD program managers
can be put in a better position to succeed in the timely design
and production of weapon

systems. The real test of the participants' resolve to get better
outcomes by applying best practices will be the decisions made on
individual weapon systems, such as for launch and funding. These
decisions define what success means in DOD and what practices
contribute to success. Decisions made

Page 12 GAO/T-NSIAD-99-116 Best Practices

by DOD or the Congress to advance or fund programs that do not
have enough knowledge to meet agreed- upon standards send signals
to managers that not having the necessary level of knowledge is
acceptable. On the other hand, decisions to not start new programs
that need

technology advances to meet unforgiving requirements or to
recognize early that a change in a program is necessary to attain
desired knowledge levels merit support.

Managing technology development differently and applying knowledge
standards to both technology development and product development
will have implications for organizational roles and budgeting. For
example, if DOD were to attempt to develop technologies to higher
readiness levels

before including them in a product development, then organizations
other than weapon system program offices will have to be made
responsible for bridging the gap from the traditional science and
technology role to the redefined program manager's role. Likewise,
the research and

development funding attendant to those bridging responsibilities
may have to be budgeted and accounted for differently. This does
not necessarily mean more research and development money is needed
in the aggregate. Rather, taking the foregoing actions could
actually lower costs in the long run, thereby freeing funding for
other needs. The best practices we have described and the changes
needed to adopt them are not concepts that are foreign to DOD. One
of the recommendations from a 1996 Defense Science Board study
called for DOD to aggressively pursue high- risk technology before
inclusion in a weapon research and development program. 1 We found
several instances such as the photonics mast for the Navy's New

Submarine, propulsion and related technologies for the Marine
Corps Advanced Amphibious Assault Vehicle, and the Air Force's
Integrated High Performance Turbine Engine Technology Program
where DOD has put the organizations and funding in place to bring
technologies to a high level of readiness before they were
included in programs. Another program of interest is the Army's
Future Scout and Cavalry System, which is being managed as an
Advanced Technology Demonstration, a DOD initiative to demonstrate
immature technologies so they can be more easily incorporated into
a product development. Several other DOD initiatives,

1 A Streamlined Approach to Weapon Systems Research, Development,
and Acquisition: The Application of Commercial Practices, a Report
from the Defense Science Board Task Force on Defense Acquisition
Reform, May 1996.

Page 13 GAO/T-NSIAD-99-116 Best Practices

like cost as an independent variable and integrated product teams,
are attempting to draw lessons from commercial practices.

Ongoing Work of Interest to the Subcommittee

The Subcommittee also asked for information on several assignments
we have underway. Most of this work deals with DOD initiatives
related to acquisitions. Summaries follow. Acquisition Workforce
Training in Best Practices

DOD has a number of reform initiatives that are related to the
best practices that we have reported are possible to adopt in the
DOD environment. At the request of this Subcommittee, we are
evaluating the role of DOD's training in getting best acquisition
practices applied to weapon system programs and whether training
could be improved.

We have focused our work on weapon system program offices because
they are where the practices are applied. We selected six program
offices cited as having been successful in applying one or more of
the following practices: cost as an independent variable, past
performance, performance specifications, integrated product teams,
and supplier relations. We are assessing how DOD's training
supported their use of the practices. For comparative purposes, we
are meeting with four leading commercial firms

noted for their excellence in workforce training. We expect to
issue a final report in August 1999.

Best Practices for Test and Evaluation of Weapon Systems

At the request of this Subcommittee, we have recently begun
reviewing best practices as they can be applied to the test and
evaluation of weapon systems. The objectives of this work are to
determine whether (1) the testing practices of leading commercial
firms offer improvements to DOD's testing practices on weapon
acquisition programs, (2) a particular area of

best testing practices stands out as a leverage point that could
offer a significant improvement for weapon acquisitions, (3) the
role or purpose of testing in best commercial product developments
shape actual testing

practices and to what extent testing plays a different role in
major weapon acquisitions, and (4) obstacles hinder the
implementation of best practices on DOD weapon acquisition
programs. We plan to complete the design

phase of this assignment in June 1999.

Page 14 GAO/T-NSIAD-99-116 Best Practices

Pricing of Sole- Source Commercial Items

DOD, with the support of the Congress, is increasing its use of
commercially available products and services. While the current
level of commercial purchasing is relatively small and sole-
source purchases even smaller, we expect commercial purchases to
increase in the future. We are

currently reviewing the price analysis tools DOD contracting
personnel use to arrive at fair and reasonable prices for
commercial sole- source items and the guidance and training
available to assist them in determining price reasonableness. In
March 1998, we reported our preliminary observations on the
pricing of commercial sole- source spare parts during testimony
before the former Subcommittee on Acquisition and Technology. 2
Currently, we are nearing completion of that work and plan to
issue our

report to you in June. Our tentative findings are summarized
below. The current contracting environment for sole- source
commercial items presents negotiating challenges for DOD
contracting personnel. While the Federal Acquisition Regulation
grants DOD contracting officers wide latitude on the type and
extent of price analysis techniques they can use, they are,
nevertheless, required to perform sufficient price analyses to
determine whether offered prices are fair and reasonable. Our
review of

the price analyses they perform found a number of weaknesses. For
example, some contracting personnel accepted initially offered
prices because they had a misperception that if the offered prices
were the same as the catalog or list price, the offered prices
were fair and reasonable. Some contracting personnel did not seem
to use pertinent contract file information on historical pricing
in their price analysis. Others did not

appear to understand the makeup of catalog prices and paid prices
that were based on rapid delivery service when it was not needed
because the purchases were for restocking inventories. DOD
continues to provide training and guidance to assist contracting
personnel in understanding the requirements of a sound price
analysis in a commercial contracting environment. In addition,
recent legislation requires increased guidance for contracting
personnel on price analysis tools, the appropriate use of
information other than certified cost or pricing

data, and the role of support agencies. In time, as more
contracting personnel are trained and additional guidance is
provided, we expect to see improvement in the quality of price
analyses for sole- source commercial items.

2 Defense Acquisition: Improved Program Outcomes Are Possible
(GAO/T-NSIAD-98-123, Mar. 18, 1998).

Page 15 GAO/T-NSIAD-99-116 Best Practices

Spare Parts Price Trends Recently, the services have been
concerned that the prices they are paying for critical spare parts
have been increasing above the rate of inflation. As

a result, we have begun a review of the prices DOD end- users are
paying for consumable spare parts and repairable items and the
change in these prices over time relative to inflation. Where
there are significant increases over

and above the rate of inflation, we will attempt to ascertain the
factors that most contribute to those increases. Cost Accounting
Standards Board

The Congress asked us to establish a panel of experts to review
and make recommendations regarding the Cost Accounting Standards
(CAS) Board and the CAS system in light of recent procurement
reforms. More than 25 years ago, the Congress established the CAS
Board to protect the government from certain risks inherent in
cost- based contracts and to improve communications between the
government and contractors with regard to those contracts. Cost-
based contracts continue to represent the

majority of all federal contracting dollars. The CAS Board Review
Panel is expected to recommend that there is a continuing need for
the CAS Board. The Review Panel is also expected to recommend
reforms to encourage the participation of new commercial companies
in government procurement and to reduce the burden of government
unique accounting system

requirements on smaller companies. The Review Panel's report is
expected to be released within the next few weeks with its
recommendations aimed at significantly reducing the burdens and
costs of the CAS system without

diminishing its benefits. Other Transactions In 1993, the Congress
passed section 845 of the National Defense

Authorization Act for fiscal year 1994, which authorized the
Defense Advanced Research Projects Agency to conduct prototype
projects of weapons or weapon systems under the authority of 10 U.
S. C. 2371. This is commonly referred to as other transaction
authority. An other transaction is distinct from other
instruments, such as contracts, grants,

and cooperative agreements, and is generally not subject to the
laws and regulations that apply to these instruments.
Subsequently, section 804 of the fiscal year 1997 National Defense
Authorization Act extended the authority to enter into other
transaction agreements to the military departments and other DOD
components. Other transactions can be used

even if a contract would be feasible or appropriate. Further,

Page 16 GAO/T-NSIAD-99-116 Best Practices

are not required to share in the costs of the agreements. DOD's
authority under section 845 is to expire on September 30, 2001.

In response to a Subcommittee request, we are determining (1) the
extent to which DOD has used its section 845 authority, (2) why
other transactions were selected as the procurement instrument,
and (3) the extent to which other transactions address key areas
of risk normally covered by Federal Acquisition Regulations. The
scope of our review includes all other transactions that DOD
entered into under the authority of section 845, as amended,
between fiscal years 1994 and 1998 and two other transactions that
were awarded in October 1998 for the Evolved Expendable Launch

Vehicle. Our concerns on the potential risks of using other
transactions for that program were discussed in a prior report. 3
We will issue our report this summer. Governmentwide Information
Technology Contracts

We were asked to examine selected governmentwide agency contracts,
typically for various information technology resources, to
determine whether competition requirements were being met. In
1994, the Congress directed that agencies consider awarding these
task and delivery order

contracts to multiple firms rather than a single firm to provide
for competition in ordering. 4 Federal agencies are to provide
each of the multiple contractors a fair opportunity to be
considered for orders placed under the contracts.

In September 1998, we reported that efforts to provide a fair
opportunity and thereby promote competition varied among the six
organizations reviewed. 5 Four agencies had experienced difficulty
obtaining competition, while two had achieved consistent
competition. Because multiple award task and delivery order
contracts are a relatively new

contracting mechanism, and because of the large size of some
orders that had been awarded pursuant to the new authority, we
were requested to continue a review of the implementation of these
provisions. We have 3 Evolved Expendable Launch Vehicle: DOD
Guidance Needed to Protect Government's Interest (GAO/NSIAD-98-
151, June 11, 1998). 4 A task- or delivery- order contract
provides for an indefinite quantity, within stated limits, of
supplies or services to be furnished during a fixed period, with
deliveries or performance to be scheduled by placing orders with
the contractor.

5 Acquisition Reform: Multiple- Award Contracting at Six Federal
Organizations (GAO/NSIAD-98-215, Sept. 30, 1998)

Page 17 GAO/T-NSIAD-99-116 Best Practices

selected four organizations that have awarded multiple award
contracts for information technology products and services that
are available for governmentwide use. Our work will provide a
perspective on how agencies ensure contractors a fair opportunity
to be considered for interagency orders.

Three of the four organizations were included in our prior work
and have attempted to strengthen their processes for providing
contractors a fair opportunity. The Department of Transportation
which sometimes

accepted unconvincing rationales for placing sole- source orders
now provides customers placing interagency orders specific
guidance about acceptable rationales to justify sole- source
orders. The National Institutes of Health which had identified
preferred contractors when announcing plans to place orders has
eliminated any reference to a preferred contractor from such
announcements. In addition, a proposed change to the
governmentwide procurement regulations would prohibit designating
preferred contractors. Finally, the Defense Information Systems
Agency which had not required that all multiple award contractors
be notified of planned competitive orders modified its procedures
to require that all contractors be notified of orders. Although
our current review is now in its initial phases, our preliminary
work shows that competition for the largest orders under multiple
award contracts has not been routinely achieved. During our work,
we will identify factors that might have deterred contractors from
competing for

these orders and assess how changing agencies' fair opportunity
processes might broaden competition for orders. This concludes our
statement. We appreciate the opportunity to have it placed in the

Page 18 GAO/T-NSIAD-99-116 Best Practices

Appendix I GAO Products on Best Practices Applicable to Weapon
Acquisitions Appendi x I

Best Practices: Commercial Quality Assurance Practices Offer
Improvements for DOD (GAO/NSIAD-96-162, Aug. 26, 1996).

Major Acquisitions: Significant Changes Underway in DOD's Earned
Value Management Process (GAO/NSIAD-97-108, May 5, 1997).

Best Practices: Successful Application to Weapon Acquisitions
Requires Changes in DOD's Environment (GAO/NSIAD-98-56, Feb. 24,

Best Practices: DOD Can Help Suppliers Contribute More to Weapon
System Programs (GAO/NSIAD-98-87, Mar. 17, 1998).

Defense Acquisition: Improved Program Outcomes Are Possible
(GAO/T-NSIAD-98-123, Mar. 18, 1998).

(707408) Let t er

Ordering Information The first copy of each GAO report and
testimony is free. Additional copies are $2 each. Orders should be
sent to the following address, accompanied by a check or money
order made out to the Superintendent of Documents, when necessary,
VISA and MasterCard credit cards are accepted, also.

Orders for 100 or more copies to be mailed to a single address are
discounted 25 percent.

Orders by mail: U. S. General Accounting Office P. O. Box 37050
Washington, DC 20013

or visit: Room 1100 700 4th St. NW (corner of 4th and G Sts. NW)
U. S. General Accounting Office Washington, DC

Orders may also be placed by calling (202) 512- 6000 or by using
fax number (202) 512- 6061, or TDD (202) 512- 2537.

Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any
list from the past 30 days, please call (202) 512- 6000 using a
touchtone phone. A recorded menu will provide information on how
to obtain these lists.

For information on how to access GAO reports on the INTERNET, send
an e- mail message with info in the body to: info@ www. gao. gov
or visit GAO's World Wide Web Home Page at: http:// www. gao. gov

United States General Accounting Office Washington, D. C. 20548-

Official Business Penalty for Private Use $300

Address Correction Requested Bulk Rate

Postage & Fees Paid GAO Permit No. GI00

*** End of document. ***