Index


Best Practices: Better Management of Technology Development Can Improve
Weapon System Outcomes (Chapter Report, 07/30/1999, GAO/NSIAD-99-162).

The Pentagon plans to boost its investment in new weapons to about $60
billion in fiscal year 2001--a 40-percent increase over fiscal year
1997. The military has high expectations for this investment: that new
weapons will be better and less expensive than their predecessors and
will be developed in half the time. However, the Defense Department
(DOD) will not meet these expectations using its traditional management
approach. Leading commercial firms have changed their practices for
developing products and have achieved the kinds of results DOD seeks.
Maturing new technology before it is included in products is one of the
main determinants of these firms' successes. This practice holds promise
for DOD, for immature technologies have been a main source of problems
on weapon systems. This report assesses (1) the impact of technology
maturity on product outcomes, (2) best practices for managing new
technologies and incorporating them into products, and (3) ways DOD can
adapt these practices to get better outcomes on weapon system programs.

--------------------------- Indexing Terms -----------------------------

 REPORTNUM:  NSIAD-99-162
     TITLE:  Best Practices: Better Management of Technology
	     Development Can Improve Weapon System Outcomes
      DATE:  07/30/1999
   SUBJECT:  Private sector practices
	     Defense procurement
	     Comparative analysis
	     Defense capabilities
	     Weapons research and development
	     Weapons systems
	     Testing
IDENTIFIER:  Comanche Helicopter
	     DOD/NASA Integrated High Performance Turbine Engine
	     Technology
	     DOD Defense Acquisition Pilot Program
	     Defense Reform Initiative
	     Brilliant Anti-Armor Submunition
	     DOD Airborne Laser Program
	     DOD Advanced Amphibious Assault Vehicle Program
	     Forward Looking Infrared System
	     HS 702 Satellite
	     Seawolf Attack Submarine
	     Army Future Scout and Cavalry System
	     F-15 Aircraft
	     F-117 Aircraft
	     F-22 Aircraft
	     DOD Advanced Concept Technology Demonstration Program

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO report.  This text was extracted from a PDF file.        **
** Delineations within the text indicating chapter titles,      **
** headings, and bullets have not been preserved, and in some   **
** cases heading text has been incorrectly merged into          **
** body text in the adjacent column.  Graphic images have       **
** not been reproduced, but figure captions are included.       **
** Tables are included, but column deliniations have not been   **
** preserved.                                                   **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
** A printed copy of this report may be obtained from the GAO   **
** Document Distribution Center.  For further details, please   **
** send an e-mail message to:                                   **
**                                                              **
**                    <[email protected]>                        **
**                                                              **
** with the message 'info' in the body.                         **
******************************************************************

    United States General Accounting Office GAO                 Report
    to the Chairman and Ranking Minority Member, Subcommittee on
    Readiness and Management Support, Committee on Armed Services,
    U.S. Senate July 1999           BEST PRACTICES Better Management
    of Technology Development Can Improve Weapon System Outcomes
    GAO/NSIAD-99-162 United States General Accounting Office
    National Security and Washington, D.C. 20548
    International Affairs Division B-280233
    Letter July 30, 1999 The Honorable James Inhofe Chairman The
    Honorable Charles Robb Ranking Minority Member Subcommittee on
    Readiness and Management Support Committee on Armed Services
    United States Senate As you requested, this report assesses how
    best practices offer improvements to the way the Department of
    Defense (DOD) incorporates new technology into weapon system
    programs. It also assesses the factors that can make it difficult
    to mature technologies before they are included on weapon system
    programs and what can be done about them. We make recommendations
    to the Secretary of Defense on how advanced technologies can be
    better managed so they pose less risk when they are included in
    weapon system designs. We are sending copies of this report to the
    Honorable William S. Cohen, Secretary of Defense; the Honorable
    Louis Caldera, Secretary of the Army; the Honorable Richard
    Danzig, Secretary of the Navy; the Honorable F. Whitten Peters,
    Acting Secretary of the Air Force; the Honorable Jacob J. Lew,
    Director, Office of Management and Budget; and to interested
    congressional committees. We will also make copies available to
    others upon request. If you have any questions regarding this
    report, please call me at (202) 512-4841. Other key contacts are
    listed in appendix III. Katherine V. Schinasi Associate Director
    Defense Acquisitions Issues Letter Executive Summary Purpose
    The Department of Defense (DOD) plans to increase its investment
    in new weapons to about $60 billion in fiscal year 2001-a 40-
    percent increase over fiscal year 1997. DOD has high expectations
    from this investment: that new weapons will be better and less
    expensive than their predecessors and will be developed in half
    the time. With its traditional management approach-which has
    produced superior weapons, but at much greater cost and time than
    planned-DOD will not meet these expectations. Leading commercial
    firms have changed their practices for developing products and
    have achieved the kinds of results DOD seeks. Maturing new
    technology before it is included in products is one of the main
    determinants of these firms' successes. This practice holds
    promise for DOD, for immature technologies have been a main source
    of problems on weapon systems. In response to a request from the
    Chairman and the Ranking Minority Member, Subcommittee on
    Readiness and Management Support, Senate Committee on Armed
    Services, GAO assessed (1) the impact of technology maturity on
    product outcomes, (2) best practices for managing new technologies
    and incorporating them into products, and (3) ways DOD can adapt
    these practices to get better outcomes on weapon system programs.
    Background              GAO reviewed commercial and DOD
    experiences in incorporating 23 different technologies into new
    product and weapon system designs. The technologies were drawn
    from (1) six commercial firms recognized for their success in
    developing technically advanced products more quickly than the
    products' predecessors and (2) five DOD weapon system programs
    that incorporated advanced technologies, including some that did
    not encounter problems and some that did. GAO asked the managers
    of these technologies to assess the maturity of the technologies
    at the point they were included in product development by applying
    a tool, referred to as technology readiness levels (TRLs). The
    National Aeronautics and Space Administration and the Air Force
    Research Laboratory use TRLs to determine the readiness of
    technologies to be incorporated into a weapon or another type of
    system. Readiness levels are measured along a scale of one to
    nine, starting with paper studies of the basic concept, proceeding
    with laboratory demonstrations, and ending with a technology that
    has proven itself on the intended product. The Air Force Research
    Laboratory considers TRL 6 an acceptable risk for a weapon system
    entering the program definition stage, the point at which DOD
    launches its weapon programs, and TRL 7 an acceptable risk for the
    Letter    Page 2
    GAO/NSIAD-99-162 Best Practices Executive Summary engineering and
    manufacturing development stage. This is an important distinction
    because leading commercial firms launch a new product later than
    DOD, after technology development is complete. They refer to this
    point as the beginning of product development, the point at which
    they commit to developing and manufacturing the product.
    Typically, technology is still being developed when weapon system
    programs are launched; the point at which a weapon system is far
    enough along to compare to a commercial product development is
    likely to be at or after the start of engineering and
    manufacturing development. Results in Brief        The experiences
    of DOD and commercial technology development cases GAO reviewed
    indicate that demonstrating a high level of maturity before new
    technologies are incorporated into product development programs
    puts those programs in a better position to succeed. The TRLs, as
    applied to the 23 technologies, reconciled the different maturity
    levels with subsequent product development experiences. They also
    revealed when gaps occurred between a technology's maturity and
    the intended product's requirements. For technologies that were
    successfully incorporated into a product, the gap was recognized
    and closed before product development began, improving the chances
    for successful cost and schedule outcomes. The closing of the gap
    was a managed result. It is a rare program that can proceed with a
    gap between product requirements and the maturity of key
    technologies and still be delivered on time and within costs. Two
    conditions were critical to closing the maturity gap. First, the
    right environment for maturing technologies existed. Key to this
    environment was making a science and technology organization,
    rather than the program or product development manager,
    responsible for maturing technologies to a high TRL. When a
    maturity gap persisted, managers were given the flexibility to
    take the time to mature the technology or decrease product
    requirements so that they could use another, already mature
    technology. Second, both technology and product managers were
    supported with the disciplined processes, readily available
    information, readiness standards, and authority to ensure
    technology was ready for products. This support enabled these
    managers to safeguard product development from undue technology
    risks. On the other hand, immature technologies were sometimes
    incorporated into products for reasons such as inflexible
    performance requirements, increasing the likelihood of cost
    overruns and delays in product development. Product managers had
    little choice but to accept the technologies and hope that they
    would mature Letter    Page 3
    GAO/NSIAD-99-162 Best Practices Executive Summary successfully.
    However, the pressures of product development made for an
    environment less conducive to maturing technology. For several
    reasons, DOD is likely to move technologies to product development
    programs before they are mature. Science and technology
    organizations, which traditionally operate within fixed budget
    levels, do not necessarily have the funds to mature technology to
    the higher TRLs. Programs are more able to command the large
    budgets necessary for reaching these levels. The pressures exerted
    on new programs to offer unique performance at low cost encourage
    acceptance of unproven technologies. The technologies GAO reviewed
    indicate these conditions can be overcome on individual cases. DOD
    has several initiatives underway, such as advanced technology
    demonstrations, that could make it more feasible for science and
    technology organizations to mature technology before it is moved
    to product development programs. The challenge will be whether the
    lessons learned from these cases and initiatives offer an approach
    that has a DOD-wide application. GAO makes recommendations to the
    Secretary of Defense on ways to pursue advanced technologies while
    lessening their potential for causing problems on weapon system
    programs. Principal Findings Maturity of Technology at     The 23
    technologies GAO reviewed spanned a wide range of readiness
    Program Start Is an           levels-from a low of TRL 2 to a high
    of TRL 9-when they were included Important Determinant of      in
    product development programs. Programs with key technologies at
    Success                       readiness levels 6 to 8 at the time
    of program launch met or were meeting cost, schedule, and
    performance requirements. All of the commercial technologies and a
    few of the DOD technologies fell into this category. For example,
    Ford managed its voice-activated control technology to TRL 8-a 10-
    year effort-before introducing it on the 1999 Jaguar. Similarly,
    the Defense Advanced Research Projects Agency matured a
    revolutionary periscope technology to TRL 9 before it was included
    on the Virginia class attack submarine. DOD programs that accepted
    technologies at a readiness level of 5 or less experienced
    significant cost and schedule increases due, in part, to problems
    with the technologies. DOD's acceptance of technologies at level 4
    or lower was not unusual. For example, the key technologies for
    the Army's brilliant antiarmor submunition were at levels 2 Page 4
    GAO/NSIAD-99-162 Best Practices Executive Summary and 3 when
    weapon system development began. At these levels, DOD had a
    significant gap in technology maturity at the start of the
    program. The gap was not closed until well into the development
    program, and problems with the technologies were a main
    contributor to the program's 88-percent cost growth and 62-percent
    slip in schedule. Controllable Conditions           Closing the
    technology development gap before beginning product Affect How
    Well a                 development was the result of good
    technology maturation practices and Technology's Inclusion on a
    sound methods for moving technologies to products. The more
    successful Product Can Be Managed            of the 23
    technologies were managed by science and technology organizations
    until they reached at least TRL 6 and more, often TRL 8 or higher.
    This environment was an important condition for successfully
    maturing technologies, as it allowed room for unexpected results
    such as test "failures," which are considered normal events in
    developing technologies. To match technology maturity and product
    requirements, managers also had the option of waiting until
    technologies matured or changing product requirements so that an
    already mature technology could be used. For example, Hughes
    deferred the development of the HS-702 satellite until critical
    solar cell technology had matured-a process that took over 10
    years. Also, Navy managers accepted an existing weapon ejection
    system on the Virginia class attack submarine when technology
    failed to mature as expected. In contrast, performance
    requirements for the Comanche helicopter were inflexible;
    requirements mandated the inclusion of advanced sensors and
    avionics technologies, despite their immaturity. The Comanche
    program has experienced cost growth and schedule delays, partly
    attributable to the inclusion of these technologies. In the more
    successful cases, technology and product managers were given the
    authority and tools to move technology only when it was at high
    readiness levels. Disciplined processes provided managers credible
    information on the status of technologies and high standards for
    assessing readiness. Science and technology managers developed
    technologies to standards acceptable to product managers who could
    reject those technologies that fell short. For example, Ford's
    science and technology managers use agreed-upon standards for
    judging technology readiness, and all new technologies follow the
    same maturation process. Ford's product managers are also
    empowered to say no when technologies are not deemed mature.
    Recently, the Jaguar vehicle team rejected night vision technology
    at TRL 8 because it did not meet cost objectives. DOD program
    managers that had to accept immature technologies had less
    information available to guide them. For example, key technologies
    for the brilliant antiarmor Page 5
    GAO/NSIAD-99-162 Best Practices Executive Summary submunition
    program bypassed Army science and technology organizations,
    forcing the program manager to accept the technologies with little
    information about their readiness. Often, the tools used to assess
    the technologies' status failed to identify high risks; the TRLs
    indicate that risks on the problematic technologies were often
    high. Also, the greater pressures to meet cost and schedule goals
    in product development provided a less forgiving environment for
    fledgling technologies. Impediments to Adopting     Leading
    commercial firms have put the organizations, tools, and other Best
    Practices for          practices in place to foster technology
    development and improve the Technology Inclusion in     outcomes
    of product developments as a matter of necessity. The large DOD
    Are Surmountable        investment required for a new product and
    the risks to that investment if the product does not meet customer
    needs reinforce these practices. The DOD cases that followed a
    similar approach-the Advanced Amphibious Assault Vehicle and the
    Virginia class attack submarine-have so far avoided problems with
    key technologies. Yet these cases are not the norm for DOD
    programs. DOD programs operate under conditions that make it more
    difficult-and less rewarding-to separate technology from product
    development and to allow technology to reach high maturity before
    being included in an acquisition program. It is easier for weapon
    system programs to fund technology development at higher readiness
    levels because they attract much bigger budgets than science and
    technology projects. DOD typically does not fund science and
    technology organizations to take technology past the feasibility
    stage- TRL 5. As a practical matter, it is often necessary to move
    immature technology to a weapon system program to get needed funds
    and management support. New programs are pressured to include
    immature technologies that offer significant performance gains.
    These pressures come from the user's perception of the threat,
    technologists that see the program as an opportunity to apply a
    new technology, and funding competition that rewards weapon
    systems with unique features. DOD and the services have several
    initiatives for improving the technology development process and
    reducing weapon system cycle times. These include defense
    technology objectives, advanced technology demonstrations,
    advanced concept technology demonstrations, the Army's new
    scout/cavalry vehicle, and the Air Force's Integrated High
    Performance Turbine Engine Technology Program. These initiatives
    are aimed at putting the science and technology organizations and
    funding in Page 6
    GAO/NSIAD-99-162 Best Practices Executive Summary place to bring
    technologies to higher readiness levels before they are included
    in weapon system programs. Recommendations    GAO recommends that
    the Secretary of Defense adopt a disciplined and knowledge-based
    approach of assessing technology maturity, such as TRLs, DOD-wide,
    and establish the point at which a match is achieved between key
    technologies and weapon system requirements as the proper point
    for committing to the development and production of a weapon
    system. GAO also recommends that the Secretary (1) require that
    technologies needed to meet a weapon's requirements reach a high
    readiness level (analogous to TRL 7) before making that
    commitment, (2) extract lessons from successful technology
    inclusion cases for application to future technology inclusion
    efforts, and (3) empower program managers to refuse to accept key
    technologies with low levels of maturity by making decisions on
    individual programs that reinforce a best practice approach to
    technology maturation and inclusion. These recommendations appear
    in full in chapter 5. Agency Comments    DOD generally agreed with
    the report and its recommendations. A detailed discussion of DOD's
    comments appear in appendix I. Page 7
    GAO/NSIAD-99-162 Best Practices Contents Letter Executive Summary
    2 Chapter 1                  Separating Technology Development
    From Product Development Introduction                 Is a Best
    Practice                                                       12
    Technology and Product Development Conducted at the Same Time
    Within DOD
    15 Shorter Acquisition Cycle Times Are Needed
    17 Objectives, Scope, and Methodology
    18 Chapter 2                  Technology Maturity Can Be Measured
    and Its Consequences for Maturity of Technology       Products Can
    Be Forecast                                                 22
    Technologies With High Readiness Levels at Launch Were Better at
    Program Start Is an       Able to Meet Product Objectives
    25 Important Determinant of Success Chapter 3
    Providing the Right Environment Is Critical to the Successful
    Controllable                 Maturation of Technology
    35 Good Technology Handoff Decisions Depend on the Tools and
    Conditions Affect How        Authority Given to Managers
    41 Well a Technology's Inclusion on a Product Can Be Managed
    Chapter 4                  Several Factors Make It Difficult to
    Mature Technologies Before Impediments to               They Are
    Included on Weapon Systems                                      50
    Services Encouraged to Use Best Practices
    54 Adopting Best              Two Unique DOD Projects May Provide
    Lessons on How to Enable Practices for                S&T
    Organizations to Manage Technology Further
    56 Technology Inclusion in DOD Are Surmountable Page 8
    GAO/NSIAD-99-162 Best Practices Contents Chapter 5
    Conclusions 61 Conclusions and         Recommendations 63 Agency
    Comments and Our Evaluation
    65 Recommendations Appendixes              Appendix I:
    Technology Readiness Levels and Their Definitions           68
    Appendix II:   Comments From the Department of Defense
    69 Appendix III:   GAO Contacts and Staff Acknowledgments
    73 Related GAO Products
    76 Tables                  Table 2.1: Cost and Schedule
    Experiences on Product Developments
    27 Table 3.1: TRLs of Technologies Managed by S&T Organizations
    36 Page 9                                        GAO/NSIAD-99-162
    Best Practices Contents Figures    Figure 1.1: Cycle for Providing
    Users a Product With Better Capabilities
    13 Figure 1.2: DOD's Weapon System Acquisition Cycle
    16 Figure 1.3: Allocation of DOD's Fiscal Year 1999 Research and
    Development Funds
    17 Figure 2.1: Using TRLs to Match Technology With Product Launch
    Requirements
    24 Figure 2.2: Readiness Levels of Technologies at the Time They
    Were Included in Product Designs
    26 Figure 2.3: Time Line for Ford's Development of Voice Activated
    Controls Technology
    28 Figure 2.4: Jaguar
    29 Figure 2.5: Virginia Class Attack Submarine
    30 Figure 2.6: Brilliant Anti-Armor Submunition
    32 Figure 2.7: Comanche Helicopter
    33 Figure 3.1: Hughes Solar Cell Arrays
    38 Figure 3.2: Integrated Avionics for Comanche Helicopter
    40 Figure 3.3: Process for Closing the Gap Between the Readiness
    of Adaptive Cruise Control Technology and Jaguar Requirements
    43 Figure 3.4: Process for Closing the Gap Between the Readiness
    of Propulsion Technologies and AAAV Requirements
    46 Figure 3.5: AAAV
    47 Figure 3.6: Assimilation of New Technology Into the BAT Program
    48 Figure 4.1: Airborne Laser
    52 Figure 4.2: Comparison of Traditional Technology Development
    Process With the Army's Fast Track Approach
    57 Figure 4.3: Future Scout and Cavalry System
    59 Page 10                                         GAO/NSIAD-99-
    162 Best Practices Abbreviations AAAV       Advanced Amphibious
    Assault Vehicle ABL        Airborne Laser ACTD       Advanced
    Concept Technology Demonstration ATD        Advanced Technology
    Demonstration BAT        Brilliant Anti-Armor Subminition DARPA
    Defense Advanced Research Projects Agency DEAL       Deliverables
    Agreement Log DOD        Department of Defense DTO        defense
    technology objective FLIR       forward-looking infrared NASA
    National Aeronautics and Space Administration S&T        science
    and technology TRL        technology readiness level Page 11
    GAO/NSIAD-99-162 Best Practices Chapter 1 Introduction
    Chapter 1 A central piece of the National Military Strategy is the
    military capability represented by advanced weaponry. The
    Department of Defense (DOD) plans to increase its annual
    investment in new weapons to about $60 billion by fiscal year
    2001-a 40-percent increase over fiscal year 1997. DOD has high
    expectations from this investment: that new weapons will be better
    and less expensive than their predecessors and will be developed
    in half the time. These expectations frame a great challenge for
    managers of programs. The traditional management approach-which
    has produced superior weapons but at much greater cost and time
    than planned-will not meet these expectations. Cycle times-the
    time to develop a new weapon-can be so long that the technology a
    weapon is designed with becomes obsolete before it can be
    produced. Costs of new weapons have reached the point that
    significantly fewer can be bought than planned. These are not new
    issues, but they have become more pressing as the pace and
    sophistication of foreign and commercial technology have
    increased, complicating a national security environment of unknown
    threats. Leading commercial firms have changed the way they
    develop products and have achieved the kinds of results DOD seeks,
    often yielding more sophisticated products in half the time
    formerly needed. Industry experts estimate that resolving
    technology problems before product development begins results in
    10 times the savings compared to correcting problems afterward. In
    this sense, technology maturity breeds product success. The
    practices leading firms use to mature and transition technology to
    products hold promise for DOD, for immature technologies have been
    main sources of problems on weapon systems. We have previously
    reported on the different elements of knowledge firms insist on to
    get better products to market faster. Of these, no element is more
    important than having technology, advanced enough to meet
    requirements but also mature enough to be predictably managed,
    available at the start of the product development cycle. Maturing
    new technology before it is included on a product is perhaps the
    most important determinant of the success of the eventual product-
    or weapon system. It is the topic of this report. Separating
    Technology  The cycle for placing better capabilities in the hands
    of users-both Development From                 military and
    commercial-can be described as consisting of technology Product
    Development Is a Best Practice Page 12
    GAO/NSIAD-99-162 Best Practices Chapter 1 Introduction
    development, product development, and production. In a 1998
    report,1 we characterized the knowledge needed on a new product as
    consisting of three knowledge points: when a match is made between
    a customer's requirements and the available technology; when the
    product's design is determined to be capable of meeting
    performance requirements; and when the product is determined to be
    producible within cost, schedule, and quality targets (see fig.
    1.1). We found that this knowledge, when obtained at the right
    time and in the right sequence-technology, design, and
    manufacturing-was a best practice. This practice lowered product
    development risks, reduced cycle times and costs, and resulted in
    smoother production programs. Figure 1.1:  Cycle for Providing
    Users a Product With Better Capabilities Technology
    Product Development                                 Development
    Production Product launch Knowledge                    Knowledge
    Knowledge Point 1                     Point 2
    Point 3 Technology                      Design
    Control of readiness                    maturity
    manufacturing process Leading commercial firms recognize a
    distinct difference between technology development and product
    development; accordingly, they develop technology before
    introducing it into product development programs. They minimize
    risk, improve cost and schedule outcomes, reduce cycle time, and
    improve quality during product development by 1Best Practices:
    Successful Application to Weapon Acquisitions Requires Changes in
    DOD's Environment (GAO/NSIAD-98-56, Feb. 24, 1998). Page 13
    GAO/NSIAD-99-162 Best Practices Chapter 1 Introduction gaining
    significant knowledge about a technology before launching the
    product development. Scientists and technologists-different people
    than those that manage product developments-manage the development
    of technology until it is ready to be included in the design of a
    product. Program launch is the point at which a firm defines a
    product's performance, cost, and schedule estimates and begins
    making a large investment in human capital, facilities, and
    materials-an investment that increases continuously as the product
    approaches the point of manufacture. It includes a commitment to
    manufacture the product. Therefore, program launch and the start
    of product development are synonymous within commercial firms.
    Protecting this investment provides a strong incentive for firms
    to minimize the potential for technology development problems
    during the product phase and cause delays. Confining delays in
    maturing technology to a time prior to launch-in an environment
    where small teams of technologists work in laboratories and are
    dedicated to perfecting the technology-is critical to saving time
    and money. If delays occur during product development, when a
    large engineering force is in place to design and manufacture the
    product, they would be much more costly. In fact, industry experts
    estimate that identifying and resolving a problem before product
    development can reap a 10-fold savings compared to correcting the
    problem after launch and that correcting the same problem in the
    manufacturing stage would be even more costly. Leading commercial
    firms have found that managing technology development separately
    from and before product development is a major reason they have
    been able to reduce product cycle times. As a whole, 50 to 70
    percent reductions in cycle times are not unrealistic achievements
    by leading commercial firms. For instance, leaders in the
    automobile industry have reduced cycle times from 7 years to 2
    years, or by about 70 percent. The consumer electronics industry
    has recently reduced its cycle time from 2 years to 6 months, and
    the commercial aircraft industry has achieved reductions of 50
    percent. Leading commercial firms have found that reducing the
    product development cycle time brings products to market faster,
    results in an increased market share, and helps to keep products
    from becoming technologically obsolete. Page 14
    GAO/NSIAD-99-162 Best Practices Chapter 1 Introduction Technology
    and                  DOD's process for developing and
    manufacturing weapon systems is Product Development  described as
    a cycle consisting of phases. These phases are concept
    exploration, program definition and risk reduction, engineering
    and Conducted at the Same  manufacturing development, and
    production and fielding. The basic Time Within DOD
    process of gathering knowledge about technology, design, and
    manufacturing is followed, but in practice, the DOD cycle does not
    make a clear distinction between technology development and
    product development. The launch of a program in DOD usually takes
    place several years before the beginning of product development
    does in leading commercial firms. In fact, a new weapon system
    program is normally launched at the start of the program
    definition and risk reduction phase, which is often in the midst
    of technology development, while most product development
    activities do not begin until the engineering and manufacturing
    development phase. Consequently, technology, design, and
    manufacturing knowledge is attained concurrently--in the higher
    cost environment that characterizes product development--
    throughout the weapon system phases. In our February 1998 report,
    we noted that such technology development problems are a major
    cause of cost increases and schedule delays on DOD weapon system
    programs. The phases in DOD's weapon system acquisition cycle and
    the knowledge gathering process, as it is typically followed, are
    shown in figure 1.2. Page 15
    GAO/NSIAD-99-162 Best Practices Chapter 1 Introduction Figure 1.2:
    DOD's Weapon System Acquisition Cycle Concept
    Program                    Engineering and             Production
    exploration                 definition and
    manufacturing risk reduction                 development
    and fielding Program launch                    Begin product
    development Technology maturity Knowledge
    Design maturity attainment Manufacturing processes controlled
    DOD's process also has organizational and budgetary implications.
    Activities accomplished in the first three phases of the
    acquisition cycle use research and development funds whereas
    production programs use procurement funds. Generally, DOD's
    science and technology (S&T) community is responsible for basic
    research, applied research, and advanced technology development to
    produce generic, rather than weapon-specific, technologies. Its
    goal is to conduct research, develop technology, and farm these
    efforts for potential military application, such as a weapon
    system. The S&T community also uses research and development
    funds, but its work generally precedes the acquisition cycle.
    Weapon system program managers, who receive most of DOD's research
    and development budget, apply generic technologies to specific
    weapon systems. However, they often become responsible for
    completing development of generic technologies as well. The
    allocation of DOD's fiscal year 1999 research and development
    funds to these categories is shown in figure 1.3. Page 16
    GAO/NSIAD-99-162 Best Practices Chapter 1 Introduction Figure 1.3:
    Allocation of DOD's Fiscal Year 1999 Research and Development
    Funds 3% Basic research - $1.1 billion 9% Applied research - $3.2
    billion 9% Advanced technology development - $3.5 billion 79%
    Weapon specific development - $29.6 billion Source: DOD S&T
    officials stated their role is to show that technology is feasible
    through laboratory experiments or demonstrations. It is often at
    this point that the technology's military potential will be
    identified and the technology will be harvested for inclusion on a
    weapon system. Because the technology is still not mature, its
    development will be completed as part of the weapon system's
    design and development, under the authority of the weapon system
    manager and apart from the S&T community. Shorter Acquisition
    DOD's weapon acquisition cycle times average between 10 to 15
    years-far Cycle Times Are                               longer
    than the cycle time for commercial products. To an extent, DOD's
    cycle times are longer because they start earlier than commercial
    cycles Needed                                        and often
    entail more complex products. Compounding the length of the weapon
    system development cycle is its unpredictability. Over the years,
    we have issued numerous reports highlighting cost overruns and
    schedule delays during the product development cycle, for which
    technology development problems were a major cause. These problems
    require additional technology development activities to take place
    at a time when Page 17
    GAO/NSIAD-99-162 Best Practices Chapter 1 Introduction the product
    should be undergoing design and manufacturing development. As a
    result, the pace of technology advances outruns the time to
    develop a weapon system and some of the more mature components
    designed into a weapon system become obsolete before the weapon is
    manufactured. For example, the F-22 will have almost 600 obsolete
    components by fiscal year 2000 while the aircraft is still in
    development. The longer a weapon system's development cycle, the
    more prone the program is to management and funding changes.
    According to DOD, an 11-year development program historically
    encounters a 30-percent cost growth over time. Based on historical
    averages, DOD calculates that the typical program will have four
    different program managers, eight defense acquisition executives,
    and seven Secretaries of Defense-all of who are major influences
    and decisionmakers on the program. In addition, the program will
    have gone through 11 annual budget cycles in which funding changes
    could have occurred and affected the program's content. The Under
    Secretary of Defense for Acquisition and Technology has stated
    that cycle time reduction is necessary to meet DOD's goals of
    delivering emerging technologies to warfighters in less time and
    at lower costs. The Under Secretary has set a goal to reduce the
    average acquisition cycle time for all program starts in fiscal
    year 1999 and beyond by 50 percent over historical averages.
    Reductions in cycle times will (1) allow for earlier fielding of
    increased capabilities, (2) reduce costs, (3) free up funds for
    more programs, (4) reduce the potential for components becoming
    obsolete, and (5) take more frequent advantage of technology
    advances found in the commercial world. An emphasis on shorter
    cycle times may also reduce the tendency to add technological
    advances that are unproven and immature into weapon acquisition
    programs. To help achieve this goal, DOD is working on several
    efforts such as Defense Acquisition Pilot Programs, the Defense
    Reform Initiatives, and many acquisition reform projects. The
    Under Secretary has also advocated adopting the practices of
    leading commercial firms and taking a more evolutionary approach
    to developing weapon systems, which would lessen the amount of
    technology development initially attempted within a weapon system
    program. Objectives, Scope, and  The Chairman and the Ranking
    Minority Member, Subcommittee on Methodology
    Readiness and Management Support, Senate Committee on Armed
    Services, requested that we examine various aspects of the
    acquisition process to determine whether the application of best
    practices can improve program outcomes. To date, we have issued
    reports on advanced quality Page 18
    GAO/NSIAD-99-162 Best Practices Chapter 1 Introduction concepts,
    earned value management, management of a product from development
    to production, and management of key suppliers (see related GAO
    products). This report covers the inclusion of technology into
    weapon system programs, and is, in a sense, a prequel to our
    report on product development. Our overall objective was to
    determine whether best practices offer methods to improve the way
    DOD matures new technology so that it can be assimilated into
    weapon system programs with less disruption. Specifically, we
    assessed (1) the impact of technology maturity on product
    outcomes, (2) best practices for managing new technologies and
    incorporating them into products, and (3) ways DOD can adapt best
    practices to achieve better outcomes on weapon system programs.
    Our methodology consisted of analyzing 23 commercial and DOD
    technologies that had transitioned or attempted to transition into
    product development programs. The technologies were drawn from six
    commercial firms recognized for their success in developing
    technically advanced products more quickly than their predecessors
    and five weapon system programs that incorporated advanced
    technologies, including some that did not encounter problems and
    some that did. We asked the managers of these technologies to
    apply a tool, referred to as technology readiness levels (TRLs),
    for our analysis. The managers used TRLs to judge the maturity of
    the technologies at the time they had entered product development
    or were included in programs. The National Aeronautics and Space
    Administration (NASA) originally developed TRLs, and the Air Force
    Research Laboratory uses them to determine when technologies are
    ready to be handed off from S&T managers to product development
    managers. We held discussions with the DOD and NASA users of TRLs
    to better understand their applicability to our review. They
    stated that TRLs can be used as general indicators of a
    technology's readiness level and associated risk of including the
    technology into a product development program, given its TRL at
    that time. TRLs are more fully explained in chapter 2. To
    understand the best practices the commercial sector used to
    include technologies in product development programs, we conducted
    literature searches and focused those searches as the review
    progressed. On the basis of the searches, we identified a number
    of commercial firms with innovative technology development
    processes for including new or advanced technologies into new
    products. We used structured interview questions sent in advance
    of our visits to gather uniform and consistent information about
    each firm's process and practice and the results achieved. In
    addition, we examined four specific technology cases-Ford's night
    vision, adaptive cruise control, and voice activated controls and
    Page 19                                         GAO/NSIAD-99-162
    Best Practices Chapter 1 Introduction Hughes' solar cell array-to
    better understand their processes and practices.   The commercial
    firms we visited were * Ethicon-Endo Surgery (medical device
    manufacturer), Division of Johnson and Johnson, Cincinnati, Ohio;
* Ford Motor Company (automobile manufacturer), Dearborn,
    Michigan; * Harris Semiconductor (semiconductor manufacturer),
    Melbourne, Florida; * Hughes Space and Communications (satellite
    and spacecraft manufacturer), Los Angeles, California; * 3M
    (commercial products manufacturer), St. Paul, Minnesota; and *
    Motorola Corporate Research Headquarters (communications
    technology manufacturer), Schaumburg, Illinois, and Motorola Land
    Mobile Products Sector, Plantation, Florida. We also attended and
    participated in conferences and workshops with recognized leaders
    in the acquisition field to obtain information on how
    organizations are improving their acquisition processes. Finally,
    we interviewed officials from trade organizations concerning the
    application of commercial practices to DOD operations. To better
    understand DOD's technology inclusion process, we selected 19
    advanced technologies that had been included in 5 DOD weapon
    system programs that were in various stages of the acquisition
    process. We collected technical reports, acquisition management,
    and risk management documentation about the technologies. In
    addition, we interviewed S&T and acquisition program management
    officials about each technology's development history, costs, and
    current status. The technologies and programs reviewed were *
    acoustic sensor, infrared seeker, inertial measurement unit,
    tandem shaped charge warhead, and processor technologies from the
    Army's Brilliant Anti-Armor Submunition (referred to as BAT)
    Program; * rotor, engine, integrated avionics, forward looking
    infrared, and helmet mounted display technologies from the Army's
    Comanche Helicopter Program; * nonpenetrating periscope and weapon
    ejection system technologies from the Navy's Virginia class attack
    submarine program; * high speed planing craft, power dense diesel
    engine, lightweight composite armor, high power water jet, moving
    map and advanced navigation technologies from the Marines'
    Advanced Amphibious Assault Vehicle (AAAV) Program; and Page 20
    GAO/NSIAD-99-162 Best Practices Chapter 1 Introduction * laser and
    beam control technologies from the Air Force's Airborne Laser
    (ABL) Program. To determine relevant DOD policy and initiatives,
    we obtained documents and interviewed officials of the Office of
    the Secretary of Defense; the Defense Advanced Research Projects
    Agency (DARPA); and Army, Navy and Air Force Science and
    Technology organizations. We also had discussions with former DOD
    officials and industry experts about DOD acquisition policies and
    practices. Even though we selected firms with product lines of
    varying complexity, we did not concentrate only on firms whose
    products had the most in common with weapon systems. Such an
    approach would have limited our ability to include firms
    recognized as the best at including new, advanced technologies
    into programs. In our analysis, we concentrated on the criteria
    and knowledge used to support technology readiness decisions.
    Although the approach from product to product may vary, the basic
    processes and standards leading commercial firms applied to
    technology inclusion decisions were consistent. We were limited,
    however, in our ability to obtain and present some relevant data
    that commercial companies considered proprietary in nature. This
    information included funding amounts for investing in technology
    development, details on technological innovations, and some
    specific data from recent technology inclusion successes. Our
    report highlights the best commercial practices for including
    technology into product development programs. As such, they are
    not intended to describe all commercial industry practices or to
    suggest that commercial firms do not have any flaws. We conducted
    our review between March 1998 and June 1999 in accordance with
    generally accepted government auditing standards. Page 21
    GAO/NSIAD-99-162 Best Practices Chapter 2 Maturity of Technology
    at Program Start Is an Important Determinant of Success
    Chapter 2 The experiences of the DOD and commercial technology
    development cases we reviewed indicate that demonstrating a high
    level of maturity before allowing new technologies into product
    development programs puts those programs in a better position to
    succeed. Simply put, the more mature technology is at the start of
    the program, the more likely the program will succeed in meeting
    its objectives. Technologies that were included in a product
    development before they were mature later contributed to cost
    increases and schedule delays in those products. We found an
    analytical tool-TRLs-that can assess the maturity level of
    technology as well as the risk that maturity poses if the
    technology is included in a product development. The tool
    associates different TRLs with different levels of demonstrated
    performance, ranging from paper studies to proven performance on
    the intended product. The value of using the tool is that it can
    presage the likely consequences of incorporating a technology at a
    given level of maturity into a product development, enabling
    decisionmakers to make informed choices. TRLs proved to be
    reliable indicators of the relative maturity of the 23
    technologies reviewed, both commercial and military, and their
    eventual success after they were included in product development
    programs. Technology Maturity             Successful technologies
    progress from initial concept to proven Can Be Measured and
    performance, whether they are developed in the laboratory or in
    the factory, by commercial industry or DOD. The Air Force Research
    Its Consequences for            Laboratory has adapted and uses
    TRLs to measure the key steps in this Products Can Be
    progression toward inclusion into weapon systems. TRLs are
    measured Forecast                        along a scale of one to
    nine, starting with paper studies of the basic concept and ending
    with a technology that has proven itself in actual usage on the
    intended product. A detailed description of TRLs is provided in
    appendix II, but the following hypothetical example about an
    airborne communications radio can illustrate the readiness levels.
    First, the idea for a new radio is conceived. The idea reaches TRL
    3 when analytical studies and some tests of the technology's
    elements, such as a circuit, back it up. When initial hand-built
    versions of all of the radio's basic elements are connected and
    tested together, the radio reaches TRL 5. This is sometimes
    referred to as a "breadboard" article; although it may function
    like a radio, it does not look like one because the individual
    parts are attached to plywood and hand-wired together. When the
    technology is built into a generic model, which is well beyond the
    breadboard tested in TRL 5, and demonstrated in a laboratory
    environment, the radio reaches TRL 6. Page 22
    GAO/NSIAD-99-162 Best Practices Chapter 2 Maturity of Technology
    at Program Start Is an Important Determinant of Success This model
    represents the last level of demonstration before the radio
    becomes tailored for application to a specific aircraft. When the
    components are assembled inside a case that resembles the final
    radio design and are demonstrated aboard a surrogate for the
    intended aircraft, the radio reaches TRL 7. TRL 8 is reached when
    the radio is put in its final form, installed in the intended
    aircraft's cockpit, and tested in conjunction with the other
    aircraft equipment with which it must interface. TRL 9 is achieved
    when the radio is successfully operated on the aircraft through
    several test missions. Unexpected problems can arise at every
    level, and effort must be expended to overcome them. This effort
    takes time and can delay the progress to the next readiness level.
    Once a technology's readiness level has been established, the
    risks of including that technology in a product development can be
    assessed. Unlike S&T projects, for which the main objective is to
    develop knowledge, a product development's objective is to deliver
    products that meet strict cost, schedule, and performance targets.
    We found that most leading commercial firms, after they had
    translated their own methods of assessing risk into TRLs,
    determined that a TRL 8 was required before they allowed a new
    technology into a product development.1 DOD launches a program in
    the program definition and risk reduction phase-much earlier than
    the leading commercial firms do. According to the Air Force
    Research Laboratory, a TRL 6 is required for a technology to be an
    acceptable risk for a program in that phase. When weapon system
    development reaches the engineering and manufacturing development
    phase, it more nearly approximates the point at which a commercial
    product development program would start. The Air Force Research
    Laboratory depicts a technology at TRL 7 as an acceptable risk for
    this phase-technologies at lower levels would be considered high
    risks. The lower the level of technology readiness, the more
    ground must be covered to bring the technology to the point at
    which it can meet the intended product's cost, schedule, and
    performance requirements with little risk (see fig. 2.1). 1An
    exception to this is space systems technology. Space-based
    technologies are generally included on a development program once
    they have been prototyped and ground tested-a TRL 6, the highest
    level attainable short of space operation. Page 23
    GAO/NSIAD-99-162 Best Practices Chapter 2 Maturity of Technology
    at Program Start Is an Important Determinant of Success Figure
    2.1:  Using TRLs to Match Technology With Product Launch
    Requirements High risk for                                    Low
    risk for Product                                  product launch
    product launch Requirements
    8      9 Risks or                             6            7
    unknowns 4              5 TR L 1      2      3 The gap between the
    maturity of the technology and the product's requirements
    represents the risks or unknowns about the technology. As each
    succeeding level of readiness is demonstrated, unknowns are
    replaced by knowledge and the gap becomes smaller. Ideally, the
    gap is closed before a new technology is included in a new
    product's design, although the Air Force Research Laboratory
    accepts the amount of risk at TRL 7 for a program entering
    engineering and manufacturing development. Technologies that reach
    TRL 7 or higher at the start of product development allow product
    managers to focus their attention on integrating the technologies
    and proving out the product design. Technologies that are included
    at lower maturity levels require more of the product managers'
    attention and resources, as basic knowledge about those
    technologies must still be gained. Thus, a major purpose served by
    TRLs is to reveal the gap between a technology's maturity and the
    maturity demanded for successful inclusion in the intended
    product. With TRLs as guides, the options available to
    decisionmakers can be framed. Given that a key determinant of
    achieving cost and schedule outcomes for a product development is
    the technology's maturity at product launch, decisionmakers can
    either (1) delay product Page 24
    GAO/NSIAD-99-162 Best Practices Chapter 2 Maturity of Technology
    at Program Start Is an Important Determinant of Success
    development until the technology is matured to a high enough
    readiness level or (2) reduce the product's requirements so that a
    less advanced, but more mature, technology can suffice. If it is
    perceived that the requirements of the product cannot be lowered
    and the product launch cannot be delayed until the requisite
    technology is of a sufficient readiness level, then the remaining
    option is to launch the product development with the immature
    technology. If this option is chosen, then the success of the
    product development will depend heavily on the product manager's
    ability to simultaneously close the technology maturity gap and
    develop the product for manufacture, which is a very challenging
    task. TRLs do not represent strictures that must be adhered to
    without exception. According to the people in DOD who have used
    TRLs, there are occasions when a lower than expected TRL can be
    accepted, such as when the product development's schedule and
    resources are generous enough that the technology will have enough
    time to mature. In other instances, a higher than expected TRL may
    be required, such as if the technology in question is the linchpin
    for the entire product. Nonetheless, we found that TRLs ably
    reconciled the different maturity levels and product experiences
    of the 23 technologies reviewed. Technologies With
    The 23 technologies reviewed spanned a wide range of readiness
    levels at High Readiness Levels  the time they were included in
    product development programs. The least mature reached TRL 2 at
    the time it was included in a product at Launch Were Better
    development, while the most mature had reached TRL 9 at the point
    of Able to Meet Product            inclusion. We observed a
    general relationship between TRLs and the Objectives
    technologies' inclusion on the intended product developments.
    Those products whose technologies reached high TRLs at the time
    they were included were better able to meet cost, schedule, and
    performance requirements. In fact, commercial firms informed us
    that maturing the technology separately from and ahead of the
    product was a main reason they were able to reduce cycle times on
    their products. An official from one of the firms termed the
    approach as "moving discovery to the left." Those technologies
    with low TRLs at inclusion encountered maturation difficulties and
    contributed to problems the products experienced. Other problems,
    such as funding and schedule changes unrelated to the
    technologies, also contributed to problems in the product
    developments. Figure 2.2 shows the TRLs when each of the 23
    technologies was included in a product design, whether at product
    development launch (for commercial technologies) or at program
    launch (for DOD technologies). Page 25
    GAO/NSIAD-99-162 Best Practices Chapter 2 Maturity of Technology
    at Program Start Is an Important Determinant of Success Figure
    2.2:  Readiness Levels of Technologies at the Time They Were
    Included in Product Designs Non-penetrating periscope Adaptive
    cruise control Night vision Voice activated control Solar  cell
    array High-speed planing craft High power water jet Weapon
    ejection system Diesel engine Helicopter rotor Lightweight
    composite armor Helicopter engine Moving map and navigation
    Hemical oxygen iodin laser Beam control system Helmet mounted
    display Helicopter forward linking infrared Integrated avionics
    Data processor Inertial measurement unit Warhead Infrared seeker
    Acousting targeting sensor 1            2                 3
    4       5            6      7          8           9 Technology
    Readiness Levels Commercial technologies DoD technologies Page 26
    GAO/NSIAD-99-162 Best Practices Chapter 2 Maturity of Technology
    at Program Start Is an Important Determinant of Success The cost
    and schedule experiences of some of the products or programs that
    inherited the technologies are shown in table 2.1. Table 2.1:
    Cost and Schedule Experiences on Product Developments Product
    development TRL at Product development and
    program associated technologies                    launch
    Cost growth          Schedule slippage Comanche helicopter
    101 percenta         120 percenta Engine
    5 Rotor                                           5 Forward
    looking infrared                        3 Helmet mounted display
    3 Integrated avionics                             3 BAT
    88 percent           62 percent Acoustic sensor
    2 Infrared seeker                                 3 Warhead
    3 Inertial measurement unit                       3 Data
    processors                                 3 Hughes HS-702
    satellite                                  None
    None Solar cell array                               6 Ford Jaguar
    None                 None Adaptive cruise control
    8 Voice activated controls                       8 aThe Comanche,
    in particular, has experienced a great deal of cost growth and
    schedule slippage for many reasons, of which technology immaturity
    is only one. Other factors, such as changing the scope, funding,
    and pace of the program for affordability reasons, have also
    contributed. Data for three weapon system development programs,
    the Virginia class attack submarine, AAAV, and the ABL, were not
    included in the table because they had not been in the product
    development phase long enough to report actual consequences. To
    date, AAAV and the submarine have stayed within 15 percent of
    their cost and schedule estimates for development. The ABL, for
    which key technologies were much less mature at program launch,
    still faces challenges with these technologies. Ford's night
    vision technology was excluded because the firm decided not to
    include the technology on a product. Details on the Comanche, BAT,
    the Virginia class attack submarine, and Ford technology and
    product experiences follow. Page 27
    GAO/NSIAD-99-162 Best Practices Chapter 2 Maturity of Technology
    at Program Start Is an Important Determinant of Success Technology
    and Product                    The key technologies for the Ford
    Jaguar and the Virginia class attack Experiences on Ford and
    submarine followed the pattern of increasing TRLs until they
    demonstrated Virginia Class Attack                     a low risk
    for transition to the product. Two examples are Ford's voice
    Submarine                                 activated controls
    development and DARPA's nonpenetrating periscope development for
    the submarine. In both cases, the technologies were validated,
    operational prototypes demonstrated, and the technologies had
    demonstrated the form, fit, and function of the final article by
    the beginning of product development. Ford's voice activated
    controls technology, which allows a driver to control certain
    functions such as windows and the radio through verbal commands,
    was under development in the technology base for over 10 years,
    being pushed by the firm's technology leaders. It was not until
    1993 that Ford found that (1) other complementary technologies,
    such as processor speeds and low cost memory, had become available
    and (2) customers wanted more features and functions but less
    distractions from driving. Given this market information, Ford
    decided to pursue voice technology as a strategic technology in
    terms of product differentiation, recognizing the importance of
    being first to market with this enabling technology. Figure 2.3
    shows the time line for developing this technology. Figure 2.3:
    Time Line for Ford's Development of Voice Activated Controls
    Technology 1983                                      1993
    1995                          1999 Ford decides to pursue
    Technology is linked         Technology is ready to
    Technology featured voice activated controls               to a
    specific vehicle.       transition into a product         on model
    year 1999 technology. Technology                 Cost and
    performance         development program.              Jaguar
    designs. under early development                requirements are
    Technology meets all in technology base.
    defined.                     cost and schedule targets for the
    product. TRL 3 - 5                               TRL 6 - 7
    TRL 8                          TRL 9 Between 1993 and 1994, based
    on discussions with customers, Ford developed cost and performance
    requirements for the technology. Ford has Page 28
    GAO/NSIAD-99-162 Best Practices Chapter 2 Maturity of Technology
    at Program Start Is an Important Determinant of Success never
    relaxed them. By September 1995, when Ford allowed the technology
    into the development program for a new Jaguar design, voice
    activated controls had been demonstrated as an integrated system
    in the appropriate form and fit for the Jaguar. Ford officials
    stated that the product has met all cost and cycle time targets
    established at the outset of its development. Figure 2.4 shows the
    Jaguar. Figure 2.4:  Jaguar Ford demonstrated voice activated
    control technology in the appropriate form and fit before
    incorporating it into the Jaguar. Source: Ford Motor Company.
    DARPA began developing the nonpenetrating periscope technology as
    part of its submarine technology development efforts after
    recognizing, in 1988, along with the Navy, that the nonpenetrating
    technology would enhance operator visibility, provide greater
    submarine design flexibility, and be stealthier than conventional
    masts and periscopes. At the time, the Virginia class attack
    submarine program had not been initiated. Once the decision was
    made to include the nonpenetrating periscope, it became a key
    feature of the submarine and was a major design driver for the
    submarine's overall configuration. Nonpenetrating refers to the
    fact that the periscope is essentially a group of sensors that are
    linked to the submarine via fiber optic and other cables. This
    technology uses infrared imaging and advanced sensors to replace
    conventional periscopes and frees up physical space compared with
    a conventional periscope. A conventional periscope relies on a
    series of telescoping shafts and reflecting surfaces to see above
    the water's surface. Page 29
    GAO/NSIAD-99-162 Best Practices Chapter 2 Maturity of Technology
    at Program Start Is an Important Determinant of Success When the
    periscope is retracted, the shafts take up a column of space from
    the top of the submarine to the bottom, through all decks. Its
    location virtually dictates the design and placement of the
    control and other rooms. If the nonpenetrating periscope
    technology did not become available, then the submarine would have
    to be drastically redesigned to accommodate the space required by
    a conventional periscope. The new nonpenetrating periscope and
    photonics mast technology underwent land testing in 1991-a TRL 5.
    The Navy actually tested the new technology at sea on the U.S.S.
    Memphis in 1992 and 1993. According to program officials, these
    sea trials demonstrated the highest level of technology readiness:
    proving the actual system through successful mission operations.
    This readiness equated to a TRL 9. Yet, this technology was not
    included in the Virginia class attack submarine requirements until
    1995. Figure 2.5 shows an artist's concept of the Virginia class
    attack submarine. Figure 2.5:  Virginia Class Attack Submarine The
    Navy demonstrated a key technology at the highest readiness level
    before including it as a requirement for the Virginia class attack
    submarine. Source: DOD. The high readiness level of the
    nonpenetrating periscope afforded the Navy the opportunity to
    develop an improved version of the periscope to a TRL 9. This was
    a relatively low-risk endeavor as the baseline periscope was
    sufficient to meet the submarine's requirements. Program officials
    believe that having knowledge about key technologies, such as the
    nonpenetrating periscope, for the Virginia class attack submarine
    at program launch made a short program definition and risk
    reduction phase possible. This phase for the Virginia class attack
    submarine was about 75 percent shorter than those of previous
    acquisition programs. Based on its demonstrated maturity, we
    anticipate that the nonpenetrating periscope to be less likely to
    impact the cost and schedule of the submarine's development
    program. There are, however, several other technologies that Page
    30
    GAO/NSIAD-99-162 Best Practices Chapter 2 Maturity of Technology
    at Program Start Is an Important Determinant of Success are
    critical to the submarine program. We did not examine these
    technologies and cannot predict their likely outcomes. BAT and
    Comanche Cases     Key technologies for the BAT and Comanche
    programs had much lower readiness levels at the time the product
    developments were launched. Consequently, they did not reduce the
    gap between their demonstrated maturity and the maturity needed to
    meet product requirements until after program launch. For some
    technologies, the gap was not closed until well into the product
    development program. For others, the gap has still not been
    closed. Five key technologies included in the Army's BAT program
    had low TRLs when they were included on the program. The level of
    readiness for most of these technologies at program launch was
    characterized by the program office as experimental in nature but
    with major uncertainty remaining-a TRL 3. The acoustic targeting
    technology was the most important enabling technology needed to
    meet the weapon's performance requirements. This technology
    provides BAT the capability to locate targets from great distances
    based on the sounds generated by the target, such as moving tanks
    and vehicles.   At the time the program was launched, the Army
    knew little about the feasibility of using this technology on this
    program. In fact, the technology was still being defined in paper
    studies-a TRL 2.   The Army did not prototype this technology
    until after the program had entered the engineering and
    manufacturing development phase, more than 6 years after program
    launch. As of December 1998, the BAT had experienced significant
    development cost and schedule increases, which program officials
    attribute at least, in part, to unknowns about the new
    technologies. Figure 2.6 shows the BAT. Page 31
    GAO/NSIAD-99-162 Best Practices Chapter 2 Maturity of Technology
    at Program Start Is an Important Determinant of Success Figure
    2.6:  Brilliant Anti-Armor Submunition At the time the program was
    launched, the Army knew relatively little about the performance of
    several key technologies for the BAT Source: DOD. Two technologies
    key to meeting the Comanche helicopter's requirements integrated
    avionics and forward-looking infrared (FLIR) technologies- were
    included on the program when they were still conceptual in nature.
    The integrated avionics technology replaces individual radios,
    navigation, and other communication equipment with a modular
    system that shares a common processor. The FLIR is a second-
    generation version that uses infrared sensors to improve the
    pilot's ability to see at night and in bad Page 32
    GAO/NSIAD-99-162 Best Practices Chapter 2 Maturity of Technology
    at Program Start Is an Important Determinant of Success weather.
    Program officials stated that both had TRLs of 3 when the
    helicopter program was started. Despite the low readiness levels
    of the technologies, the Army included the technologies on the
    program to meet weight, cost, and performance requirements. The
    development of these technologies has taken longer than the Army
    expected it would. The contractor for the integrated avionics has
    had difficulties in getting the multiple avionics modules to work
    simultaneously within required size and weight parameters, and the
    FLIR technology has undergone several design and performance
    requirement changes. As of September 1998-approximately 10 years
    after program launch-neither the integrated avionics nor the FLIR
    technology had advanced past a TRL 5. Problems with the maturation
    of these technologies have contributed to the program's cost and
    schedule increases. In contrast, the advanced rotor and engine
    technologies, which were the most mature of the Comanche
    technologies we reviewed, have experienced fewer problems in
    maturation and have not contributed significantly to the program's
    cost and schedule increases. Figure 2.7 shows the Comanche. Figure
    2.7:  Comanche Helicopter The Army included two key technologies
    in the Comanche when they were still considered conceptual to meet
    weight, cost, and performance requirements Source: DOD. Page 33
    GAO/NSIAD-99-162 Best Practices Chapter 3 Controllable Conditions
    Affect How Well a Technology's Inclusion on a Product Can Be
    Managed
    Chapter 3 Closing the gap between technology maturity and product
    requirements before a product is launched-and baselines are set-
    distinguished the more successful cases. Notably, closing the gap
    before product launch was a managed result; it put product
    managers in a better position to succeed. Two conditions were
    critical to achieving this kind of result. First was an
    environment that put the primary responsibility for maturing
    technology in the hands of S&T managers and provided them
    considerable flexibility to make decisions. Second was having the
    quality information and standards needed to make good technology
    handoff decisions, coupled with giving the product manager the
    authority to refuse new technology that did not meet product
    requirements. When these conditions were not present, the handoff
    to the product manager was compromised, with negative consequences
    for both technology and product. In each of the successful cases,
    S&T organizations played major roles in bridging the gap between
    technology maturity and product requirements. Flexibility provided
    by requirements communities and resource providers enabled S&T and
    product managers to delay the inclusion of technology if it was
    not ready or to reduce product requirements to match what mature
    technology could deliver. This environment was better suited to
    the unexpected results and delays that accompany technology
    development. Moreover, technology maturation was managed within a
    disciplined process that provided good information to be judged
    against clear and high standards, like TRLs. Armed with the tools
    and the authority to make technology inclusion decisions, both S&T
    and product managers functioned as gatekeepers to safeguard the
    product development. In the more problematic cases, S&T
    organizations disengaged much earlier, and product managers had
    little choice but to accept immature technologies. Accordingly,
    less information about the technologies was available at the point
    of inclusion. Often, the tools used to assess the technologies'
    status failed to identify high risks. In retrospect, TRLs
    indicated that risks were in fact high and perhaps unacceptable
    from a product standpoint. Also, pressures to meet cost and
    schedule estimates in product development provided a less
    forgiving environment for technologies in the discovery process.
    Page 34                                           GAO/NSIAD-99-162
    Best Practices Chapter 3 Controllable Conditions Affect How Well a
    Technology's Inclusion on a Product Can Be Managed Providing the
    Right             While most new technologies-commercial and
    military-are initially Environment Is Critical  managed by the S&T
    community, the more successful cases we reviewed continued to be
    managed by S&T organizations until they reached at least to the
    Successful               TRL 6 and more often TRL 8 or higher.
    These technologies were provided Maturation of
    the environmental advantages an S&T project has over a product
    Technology                      development. This environment
    availed S&T managers and product managers of the less risky
    options of waiting or trading to get the match between technology
    maturity and product requirements-rather than forcing the product
    launch and gambling on the completion of technology maturity. In
    contrast, the more problematic technologies did not have as benign
    an environment. Often, the technologies were handed off early by
    S&T organizations because inflexible performance requirements for
    the product demanded their inclusion. Product development managers
    launched the product development and hoped that the technology
    development would succeed. Once in a product development
    environment, external pressures to keep the program moving become
    dominant, such as preserving cost and schedule estimates to secure
    budget approval. For example, DOD policies require that a program
    be funded in the current year and that funds be made available
    over the next 6 years in the DOD planning cycle. If, during the
    program definition phase, a program manager were to decide that an
    additional year was needed to overcome unexpected technology
    problems to reach the desired level of maturity, the delay could
    push the start of engineering and manufacturing development back.
    This delay could jeopardize the funding for that phase, thus
    risking the funding support for the entire program. Consequently,
    the program manager may be more likely to accept the risk of not
    getting the technology to the desired level of maturity and
    starting the engineering and manufacturing development phase as
    planned, rather than risk the rest of the program. These
    conditions compete with and detract from the needs of technology
    development. One acquisition official stated that these conditions
    cause the weapon system program "to pull double duty," inventing
    new technology while integrating it into a product. In general, he
    believed there is an equal amount of difficulty in both tasks.
    Technologies Matured by         In the most successful cases that
    we reviewed, S&T organizations bridged S&T Organizations Made
    the gap between immature technology and the maturity needed for
    either Smooth Transitions into         program start in DOD (TRL
    6) or product development (TRL 7 or higher). Product Developments
    These cases and the responsible organizations are shown in table
    3.1. Page 35                                          GAO/NSIAD-
    99-162 Best Practices Chapter 3 Controllable Conditions Affect How
    Well a Technology's Inclusion on a Product Can Be Managed Table
    3.1:  TRLs of Technologies Managed by S&T Organizations TRL at
    Responsible                 Receiving product Technology
    handoff S&T organization            development program
    Nonpenetrating                    9 DARPA
    Virginia class attack periscope
    submarine program Adaptive cruise control           8 Ford
    Advanced Vehicle  Jaguar vehicle team Technology Office Voice
    activated controls          8 Ford Advanced Vehicle  Jaguar
    vehicle team Technology Office Solar cell array                  6
    Hughes Laboratories         HS-702 satellite program Weapon
    ejection                   6 Office of Naval             Virginia
    class attack system                               Research
    submarine program Diesel powered engine             6 Office of
    Naval             AAAV program Research High-speed planing
    6 Office of Naval             AAAV program craft
    Research High-power water jet              6 Office of Naval
    AAAV program Research Despite the different circumstances between
    the commercial and DOD sectors and among the DOD cases themselves,
    the results were similar: having S&T organizations bridge the
    maturity gap reduced technology-related problems in the products.
    For the leading commercial firms we visited, it is standard
    practice to have S&T organizations responsible for the bridge. In
    the DOD cases shown in table 3.1, the S&T organizations played
    atypical roles in managing the bridge between technology and
    product by delivering the technology to a TRL 6 or higher.
    Different pressures and incentives that are brought to bear on the
    commercial and DOD product developments explain why DOD product
    managers become responsible for more technology development than
    their commercial counterparts. These influences are discussed in
    chapter 4. Having an S&T organization manage a technology to
    maturation means more than just having a different group of people
    involved than a product development. S&T projects operate in a
    different environment than product developments. The process of
    developing technology culminates in discovery and must, by its
    nature, allow for unexpected results. S&T provides a more
    forgiving environment in which events-such as test "failures," new
    discoveries, and delays in the attainment of knowledge-are
    considered normal. It is also a less costly environment, making
    external pressures to develop knowledge on a schedule less keenly
    felt. On the Page 36
    GAO/NSIAD-99-162 Best Practices Chapter 3 Controllable Conditions
    Affect How Well a Technology's Inclusion on a Product Can Be
    Managed other hand, the process of developing a product culminates
    in delivery, and thus gives great weight to design and production.
    The same events and unexpected results that are considered normal
    for technology development represent problems in the product
    environment; they can jeopardize achievement of cost and schedule
    objectives and draw criticism to the product. The ups and downs
    and the resource changes associated with the technology discovery
    process do not mesh well with a program's need to meet cost,
    schedule, and performance goals. This situation has been described
    as attempting to "schedule inventions." Successful Cases Afforded
    In the early 1980s, Hughes Space and Communications began
    developing Flexibility to                dual junction solar cell
    technology that had the potential of greatly Decisionmakers
    increasing the electrical power on satellites. By 1985, a Hughes
    laboratory had demonstrated the technology by ground testing
    prototypes, a TRL 6, which is considered an acceptable level of
    demonstration for space-based technology.   Nonetheless, Hughes
    was not satisfied that the supporting infrastructure (materials,
    reactors, and test equipment) was mature enough to sustain
    development and production of the new technology on a satellite.
    The infrastructure was seen as critical to meeting the cost and
    schedule requirements of a product. As a result, Hughes did not
    hand off the technology to a product. Instead, the firm kept it in
    a research environment, away from cost and schedule pressures. In
    the early 1990s, Hughes established requirements for a new
    satellite- the HS-702-that would use the solar cell technology to
    leapfrog the competition. After a laboratory demonstration in
    1993, Hughes successfully used the new technology on a high-
    powered version of its existing HS-601 satellite before it began
    product development on the HS-702 satellite. By 1994, it had
    determined that the business base was available to sustain
    development and production of the HS-702 satellite. In all, the
    firm waited 10 years for the demonstrated technology to meet the
    requirements. This experience closely resembled that of Ford's
    voice activated control technology because, in both cases, the new
    technology took 10 years to mature enough for product readiness.
    Thus, the firms' approach was not to accelerate technology
    development but to shorten product development by maturing the
    technology first. Figure 3.1 shows the solar cell arrays installed
    on the HS-702. Page 37
    GAO/NSIAD-99-162 Best Practices Chapter 3 Controllable Conditions
    Affect How Well a Technology's Inclusion on a Product Can Be
    Managed Figure 3.1:  Hughes Solar Cell Arrays Hughes successfully
    proved solar cell array technology on a predecessor satellite
    before beginning product development of the HS-702 Source: Hughes
    Space and Communications. Page 38
    GAO/NSIAD-99-162 Best Practices Chapter 3 Controllable Conditions
    Affect How Well a Technology's Inclusion on a Product Can Be
    Managed The Navy made trade-offs in choosing a technology for the
    weapon ejection system, which is used to deploy weapons like
    torpedoes, of the Virginia class attack submarine. Because of
    quietness, weight, and cost requirements, the Navy preferred a new
    elastomeric (rubber-based) technology. However, this technology
    failed endurance testing, and product managers determined that the
    technology was too risky to be included in the first product.
    Product managers could have declined this technology and its
    attendant risk without delaying the submarine's schedule because
    the Navy accepted marginal increases to the cost and weight
    requirements for the system so that the proven Seawolf ejection
    technology could be used as a substitute. Using proven technology
    on the first submarine has allowed the Navy S&T community to
    continue developing the elastomeric technology, which is to be
    incorporated into the new system on the fourth production
    submarine. As discussed earlier, decisionmakers also had the
    flexibility to wait for the nonpenetrating periscope technology to
    reach TRL 9 before including it in the submarine's requirements.
    Problematic Cases Provided  According to Army officials, the FLIR
    and integrated avionics technologies Little Flexibility to
    required for the Comanche helicopter were critical for providing
    an Managers                           increased operational
    capability over existing Army helicopters. The advanced FLIR
    technology was needed to meet the user's requirements for
    increased targeting range and for improved piloting capabilities
    in bad weather and at night. It represented a quantum leap from
    existing capabilities. Integrated avionics technology was expected
    to replace separate radios, navigation systems, and other
    communication equipment on the helicopter with a modular system
    that uses central processors. These technologies were needed to
    meet weight and size requirements for the aircraft as well as
    improve communications. Both were critical elements of a mission
    equipment package that was supposed to reduce the pilot's workload
    while improving capabilities. Requirements were inflexible. Thus,
    requirements managers informed us they were unwilling to accept
    the product manager's request to trade requirements that was
    prompted by his concerns that the technologies could not advance
    in time to meet the program's schedule. They believed the product
    manager to be too risk averse and said they would not take no for
    an answer. Not only did the user consider the technologies
    nontradable, they became even more confined by weight and cost
    restrictions that were placed on the program. For example, a more
    mature FLIR technology that could possibly meet performance
    requirements but also weighed more was rejected. Page 39
    GAO/NSIAD-99-162 Best Practices Chapter 3 Controllable Conditions
    Affect How Well a Technology's Inclusion on a Product Can Be
    Managed The technological solutions that could meet the strict
    requirements were limited. According to Army officials, the only
    viable option was to develop the new technologies, which were in a
    very immature state, to the required performance levels because no
    suitable back-up technologies existed. When the Comanche
    acquisition program was launched, the FLIR and integrated avionics
    technologies had a TRL 3, barely demonstrated in a laboratory.
    This level placed the burden on the Comanche program manager to
    complete their development during the acquisition program. The
    only ways for the program manager was to slip the schedule or
    increase development costs. Figure 3.2 shows an early model of the
    integrated avionics component for the Comanche. Figure 3.2:
    Integrated Avionics for Comanche Helicopter The Army launched the
    Comanche program with immature technologies, placing the burden on
    the program manager to complete technology development Source:
    DOD. Page 40
    GAO/NSIAD-99-162 Best Practices Chapter 3 Controllable Conditions
    Affect How Well a Technology's Inclusion on a Product Can Be
    Managed Similarly, the acoustic sensor technology on the BAT was
    critical to the submunition's performance because it provided
    breakthrough improvements in the capability for precision attack
    of targets at ranges of up to 500 kilometers and in most weather.
    There was no flexibility for the program manager to ease
    requirements to substitute a more mature technology because the
    Army had no existing capability to perform this mission. Thus, the
    technology, which had a TRL 2 at program launch, was the only
    solution for locating and acquiring targets. Its feasibility was
    based on an engineering analysis in the form of studies. Key
    challenges for the acoustic sensor were to reduce noise to an
    acceptable level, develop microphones with sufficient range, and
    reduce the size of the sensor so it would fit into the BAT
    delivery system. The technology development that was necessary to
    have the sensor meet requirements had to be accomplished during
    the schedule-driven, delivery-oriented product development
    program. The development program encountered technical problems
    that left the program manager with no choice but to slip the
    schedule and increase the cost. By the start of the engineering
    and manufacturing development phase, program officials stated that
    the acoustic sensor had a TRL 5-still a high risk using the Air
    Force Research Laboratory's criteria. Good Technology
    With the right environment as a precondition, managers on the
    successful Handoff Decisions                cases benefited from
    disciplined technology development processes that linked the
    technologies to products and provided credible information on
    Depend on the Tools              the status of technologies. They
    also had standards that were both clear and Authority Given to
    and high for assessing readiness. Once a technology's feasibility
    and Managers                         usefulness were demonstrated,
    it was linked to a product through an early agreement with the
    product developer to use it if it could be fully developed.
    Ideally, as technologies approached the higher readiness levels
    associated with the bridge, S&T managers and receiving product
    managers agreed to more specific terms for accepting or rejecting
    a technology. These agreements were early links to the product
    that were needed for the technology to succeed. If a product
    manager was not willing to make such an agreement, then the
    investment to bring the technology to higher readiness levels
    might not be made. S&T managers were responsible for ensuring that
    information at key junctures was sufficient and that the
    technology was ready for inclusion on a product. They saw their
    role as to screen and develop technologies to standards acceptable
    to product managers. Product managers were responsible for
    ensuring that the product could be developed and brought to market
    within cost and performance targets. They saw as their role to
    encourage the successful Page 41
    GAO/NSIAD-99-162 Best Practices Chapter 3 Controllable Conditions
    Affect How Well a Technology's Inclusion on a Product Can Be
    Managed development of new technology but to decline the handoff
    if it did not meet product performance, cost, and schedule
    requirements. When an S&T organization disengages from a
    technology at a low TRL, the S&T manager gives up much of the
    ability to be a gatekeeper. In the event that unyielding
    requirements or other pressures force product managers to accept
    technologies before they have matured, they are weakened in their
    ability to safeguard the product development from technology
    risks. For the cases in which technologies had problems
    transitioning to products, decisionmakers were disadvantaged by
    the incomplete information available to them, yet were not
    empowered to say no to the handoff. Their situation was further
    degraded by risk assessments that embodied lower standards for
    accepting undemonstrated technology readiness. In the case of the
    BAT, the S&T community was bypassed altogether, as the weapon
    system and its enabling technologies were proposed by a contractor
    and assigned directly to a program manager. Successful Cases
    Benefited  All new technologies at Ford, regardless of whether
    they are proposed by From Strong Gatekeepers,          inside or
    outside sources, take essentially the same path and gates into
    Disciplined Processes, and        products. Initially, technology
    proposals pass through a process that High Maturity Standards
    prioritizes them according to customer needs. The proposals are
    then passed on to the Advanced Vehicle Technology Office, an S&T
    organization that determines the readiness of the proposed
    technology and fits it into Ford's path of technology
    demonstration. Once approved, the technology follows a structured
    process that includes two development phases: concept ready and
    implementation ready. This process results in a smooth transition
    from the technology development environment into a product, once
    the technology is mature. Ford's adaptive cruise control
    technology went through this process, as shown in figure 3.3. Page
    42                                         GAO/NSIAD-99-162 Best
    Practices Chapter 3 Controllable Conditions Affect How Well a
    Technology's Inclusion on a Product Can Be Managed Figure 3.3:
    Process for Closing the Gap Between the Readiness of Adaptive
    Cruise Control Technology and Jaguar Requirements Technology
    Bridge between technology                     Product feasibility
    and product                      development Advanced vehicle
    technology office                                          Jaguar
    vehicle team Enabling technologies          Interest expressed
    Agreement signed            Product launch                 Model
    introduced pursued in the technology       by vehicle team
    between S&T and base
    vehicle team TRL 5                     TRL 7
    TRL 8 1993                          1995                     1996
    1997                             1999 According to Ford officials,
    technologies for adaptive cruise control existed as separate
    projects in the technology base from about 1993 to 1995, when the
    Jaguar vehicle team identified a strong demand for the capability.
    Ford's S&T community inventoried the ongoing projects and
    demonstrated the technology as a laboratory breadboard-a TRL 5. By
    August 1996, the technologists had built a prototype that could
    demonstrate the technology in a relevant environment-a TRL 7. This
    work comprised the concept ready phase, in which the technology
    was taken from concept to where its feasibility was demonstrated
    to potential users. At the end of this phase, S&T representatives
    proved that it could work, and cost, schedule, and performance
    targets were established. Also, a target product and sponsor were
    identified, linking the technology to a product. The sponsor
    agreed that it would accept this technology if specific cost,
    quality, schedule, and performance targets were met. The medium
    for this acceptance was the Deliverables Agreement Log (DEAL),
    which was signed in September 1996 by Jaguar's chief engineer
    based on the prototype demonstration. Ford uses the DEAL as a tool
    to maintain visibility over a new technology as it progresses
    through the development process and to assess its readiness and
    acceptability for inclusion in a vehicle program before handing it
    to a sponsor, the vehicle center, or team. The DEAL formalizes
    Page 43
    GAO/NSIAD-99-162 Best Practices Chapter 3 Controllable Conditions
    Affect How Well a Technology's Inclusion on a Product Can Be
    Managed the content of the two development phases and establishes
    agreements between the technologists managing the project and
    those with authority to accept the technology into a product.
    According to Ford officials, the DEAL is important to this process
    because it is a contract between the parties that addresses the
    technology's performance, cost, quality, weight, producibility,
    and maintainability targets that must be met before the end of
    each phase. It has been invaluable in getting parties to agree on
    what is expected by the giver and receiver of a technology during
    the process. Once these targets are established, the technology
    moves to the implementation ready phase. For the Jaguar, the
    Advanced Vehicle Technology Office matured the technology to a
    high level of readiness by prototyping it in demonstrator
    vehicles-a TRL 8. The technology passed the implementation ready
    milestone in February 1997. At that point, the vehicle team
    accepted the technology for inclusion on a Jaguar product
    development. Ford used this decision-making process to develop the
    night vision technology, but with a different result. Since 1991,
    Ford has been working on this technology to provide a wide field
    of view and depth perception for the driver at night, similar to
    that provided by a FLIR. By 1998, the Advanced Vehicle Technology
    Office brought the technology to a TRL 8. However, the vehicle
    center did not agree to include the technology on a product
    because the technology did not meet the cost targets established
    in the DEAL. Other companies we visited had similar practices for
    supporting technology inclusion decisions. For example, 3M takes
    technology from its technology base when it believes it has a
    customer need. The gatekeeper responsible for moving technology
    into a concept phase-analogous to TRL 3 or 4-is the S&T
    organization of a business unit. That business unit monitors the
    technology's progress until a new product requirement is
    identified and decides whether there is interest from a product
    center to "pull" it. If an interest exists, it begins a
    feasibility phase that refines requirements through quality
    functional deployment and builds working prototypes of the new
    product-a stage that would be analogous to TRL 7 or 8. This phase
    culminates with an agreement between the technologists and the
    product developers-the receivers-as to the specific cost and
    schedule targets that must be met for the technology to be
    included into a product. To help facilitate the transition, 3M
    establishes a product development team that includes people from
    research and development, Page 44
    GAO/NSIAD-99-162 Best Practices Chapter 3 Controllable Conditions
    Affect How Well a Technology's Inclusion on a Product Can Be
    Managed marketing, manufacturing, and other functions that
    transfer with the new technology and ensure it is integrated into
    the new product. 3M also has high standards for measuring the
    readiness of a technology before the product developer accepts it.
    For example, 3M officials told us that they are developing a fuel
    cell technology for which they have built 15 prototypes for
    testing purposes-a TRL 7 or higher. However, because the
    technology has not yet met all of the cost, schedule, and
    performance targets for product development, they have not allowed
    it to be included on a new product, despite demand from the
    marketplace. Among the DOD cases, the process followed and the
    roles played on the AAAV program had several features that enabled
    good technology inclusion decisions. For almost 3 decades, the
    Marine Corps has stated a need for an amphibious vehicle with far
    greater capabilities than the current vehicle. Specifically, the
    requirement to achieve a speed of 20 to 25 knots in the open ocean
    made advances in propulsion technology key enablers for the AAAV
    program. For a vehicle of the planned size and weight of the AAAV,
    this requirement meant achieving 2,700 horsepower with a
    relatively compact engine that must operate on land and in water.
    The Corps had been exploring propulsion technologies for such a
    vehicle in its technology base for many years. Despite this, the
    Office of Naval Research, an S&T organization, assessed the
    propulsion technology and advised that it was not mature enough to
    warrant inclusion on a program. Based on this assessment, Marine
    Corps and Navy decisionmakers delayed program launch from 1991 to
    1995, until the technology could be brought to higher readiness
    levels. Figure 3.4 illustrates the process used to transition this
    technology. Page 45
    GAO/NSIAD-99-162 Best Practices Chapter 3 Controllable Conditions
    Affect How Well a Technology's Inclusion on a Product Can Be
    Managed Figure 3.4:  Process for Closing the Gap Between the
    Readiness of Propulsion Technologies and AAAV Requirements
    Technology                           Bridge between
    Product feasibility                 technology and program
    development Office of Naval Research
    AAAV program office Engine proof               Agreements reached
    between               Program launched in
    Prototype of concept                      S&T and program office
    program definition                          engine demonstrated
    phase                              demonstrated TRL 3
    TRL 6                                 TRL 7 1988
    1991                               1995
    1999 The S&T community and the product managers agreed on what had
    to be done before the program could be launched. The S&T community
    then took the lead in maturing the engine to a TRL 6-a level the
    Air Force Research Laboratory considers acceptable for starting
    the program definition and risk reduction phase. Thus, the
    assessment by the Office of Naval Research provided both the
    information and the criteria that enabled decisionmakers to say no
    to launching the program given the low readiness of the propulsion
    technology. This was coupled with the flexibility to wait for the
    technology to mature and the decision to give an S&T organization
    responsibility for managing the bridge to product readiness.
    Figure 3.5 shows the AAAV. Page 46
    GAO/NSIAD-99-162 Best Practices Chapter 3 Controllable Conditions
    Affect How Well a Technology's Inclusion on a Product Can Be
    Managed Figure 3.5:  AAAV The Marine Corps and Navy delayed
    program launch by 4 years to develop key technologies to a higher
    readiness level Source: DOD. Even with an urgent need for the
    AAAV, the Marine Corps remained disciplined in its development
    approach, allowing the technology to mature to the level of the
    requirement. Two years before program launch, a Navy S&T
    organization demonstrated the technology in a full-scale prototype
    engine. By program launch in 1995, the required 2,700 horsepower
    was demonstrated by a near prototype engine-a TRL 6. The remaining
    risk was limited to marginal weight and size reductions, although
    the demonstrator engine could be used as a backup if the size and
    weight reductions could not be obtained. In early 1999, the AAAV
    program office demonstrated a Page 47
    GAO/NSIAD-99-162 Best Practices Chapter 3 Controllable Conditions
    Affect How Well a Technology's Inclusion on a Product Can Be
    Managed prototype engine at 2,700 horsepower that met size and
    weight requirements-a TRL 7. Technology Handoffs Were
    In the BAT program, neither the S&T community nor the product
    manager Compromised When                          had the
    opportunity to act as a gatekeeper between product requirements
    Managers Had Limited                      and the maturity of
    enabling technologies. All of the technologies for the Information
    and Authority                 BAT came to the program after the
    contractor, in 1985, had proposed a weapon concept for carrying
    out unmanned, deep strike missions to attack enemy armored
    vehicles. Army leadership accepted the concept and drafted
    requirements for the BAT, and the acquisition program was launched
    after the proposal was accepted. Thus, the technology for the
    weapon came directly from the contractor's technology base into
    the acquisition program, with little or no review by the Army's
    S&T organization. The process, information, and standards that
    were critical to successful technology inclusion decisions in
    other cases were not employed on the BAT. The process followed is
    shown in figure 3.6. Figure 3.6:  Assimilation of New Technology
    Into the BAT Program Technology maturation Technology concept
    and product development Contractor technology base
    BAT program office Program launched in
    Start engineering, program definition phase
    manufacturing and development phase TRL 2-3
    TRL 5 1985                                              1991 The
    program office accepted the acoustic sensor, infrared seeker, and
    navigation technologies included on the BAT program. In
    retrospect, the levels of demonstration at the time posed high
    risks to the product development because the acoustic sensor
    technology had a TRL of 2 and Page 48
    GAO/NSIAD-99-162 Best Practices Chapter 3 Controllable Conditions
    Affect How Well a Technology's Inclusion on a Product Can Be
    Managed the infrared seeker and navigation technologies had TRLs
    of 3. Program officials stated that a significant amount of
    technology development was required during product development due
    to the lack of visibility over technology readiness before program
    launch. As a result, the development program's cost and schedule
    significantly increased over original estimates. An interesting
    sidelight to the BAT experience concerns the inertial measurement
    unit, a navigation component of the submunition. When the
    contractor first proposed the BAT concept, the design included a
    mature inertial measurement unit in production on other systems.
    However, after the program was launched, the contractor
    substituted a new quartz rate technology. At the request of the
    BAT program manager, the Army's Missile Research and Development
    Engineering Center, an S&T organization, assessed the maturity of
    the quartz rate technology. The Center concluded that the new
    technology had not demonstrated a high enough level of readiness
    and recommended that a more proven existing technology be used in
    the program. Eventually, the new technology was dropped, and an
    existing technology that was at a higher readiness level was used.
    We observed additional cases in which decisionmakers relied on
    comparatively low standards for including technologies. The Army
    assessed the FLIR, integrated avionics, and helmet mounted display
    technologies as having moderate risk when they were included in
    the Comanche program. Army officials stated that they required
    only the existence of an ongoing S&T technology project as
    acceptable, as long as the technology was projected to be ready by
    the engineering and manufacturing development phase. According to
    program officials, demonstrated maturity was considered but not
    required; proof that the projects were progressing as scheduled
    was enough. These technologies, however, had TRLs of 3 at the time
    of launch-a high risk for the program definition and risk
    reduction phase. This risk assessment is more consistent with the
    actual experience of the technologies' maturation in the program.
    The standards used for accepting the laser technology into the ABL
    program also appeared low when compared with the standards used on
    the more successful cases. While the Air Force had established
    demonstration standards for the laser to meet prior to program
    launch, these standards were met if scale models of the laser
    technology in a laboratory demonstrated they had the potential to
    produce the energy needed for an operational system. This level of
    technical demonstration equated to a TRL of 4, representing a high
    risk for inclusion into an acquisition program. Page 49
    GAO/NSIAD-99-162 Best Practices Chapter 4 Impediments to Adopting
    Best Practices for Technology Inclusion in DOD Are Surmountable
    Chapter 4 Although product developments-commercial or defense-fare
    better when key technologies are matured before they are included
    in the product design, the more traditional approach within DOD is
    to mature technology during a product's development. Rational
    explanations are behind this tradition. S&T organizations,
    operating within fixed budget levels, are not necessarily
    accustomed or equipped to manage the bridge between technology
    feasibility and product readiness. Programs are more able to
    command the large budgets necessary for reaching higher levels of
    technology readiness than S&T projects. Also, pressures are
    exerted on new programs to offer unique performance and acceptable
    cost and schedule projections, which encourage premature
    acceptance of unproven technologies. The Under Secretary of
    Defense for Acquisition and Technology not only supports shorter
    cycle times and a more aggressive pursuit of technology outside of
    programs, but also use of commercial best practices to get these
    results. DOD has several initiatives underway that could make
    conditions more favorable for S&T organizations to mature a
    technology further before it is included in a product development.
    One Army project calls for an S&T organization to manage all
    technology maturation and integration tasks for a new combat
    vehicle up to the engineering and manufacturing development phase.
    Other initiatives may make the S&T community a more integral
    participant in matching user requirements with technology and
    tying S&T projects more closely to product development paths.
    Whether these efforts are effective and can be applied on a
    broader scale remains to be seen. Several Factors Make  Budgetary,
    organizational, and other factors within DOD make it difficult to
    It Difficult to Mature           bring technologies to high
    readiness levels before being included in weapon systems. These
    factors encourage S&T organizations to disengage Technologies
    Before              from technology development too soon and
    weapon system program They Are Included on  managers to accept
    immature technology. Factors other than these Weapon Systems
    encourage leading commercial firms to keep technology development
    out of the product developers' hands and in those of S&T
    organizations. The differences in these factors and in the
    management of technology development stem from differences in what
    helps commercial and DOD programs to succeed. They do not stem
    from capabilities commercial firms possess that DOD does not. Page
    50                                         GAO/NSIAD-99-162 Best
    Practices Chapter 4 Impediments to Adopting Best Practices for
    Technology Inclusion in DOD Are Surmountable Budget and
    Organizational     Budget realities within DOD-the fact that
    weapon system programs Factors                       attract
    higher levels of funding than S&T projects-make these programs a
    more advantageous setting for funding technology development to
    the higher readiness levels. As a practical matter, it is often
    necessary to move immature technology to a weapon system program
    to get needed funds and management support for maturation.
    Normally, DOD S&T organizations do not see their role as going
    beyond demonstrating the feasibility of a technology for generic-
    versus product specific-application (a TRL 5). However, as seen in
    several of the cases we reviewed, even this level often is not
    reached before a product development organization takes over. The
    S&T organizations that helped to bridge the gap from technology
    feasibility to product readiness on the more successful cases had
    gone beyond their typical role. One of the reasons that S&T
    organizations disengage relatively early is that S&T work is
    traditionally funded as a percentage of the overall DOD research
    and development budget. S&T organizations receive about $8 billion
    annually, or about 20 percent, of DOD's research and development
    budget. This money funds several thousand projects, providing less
    than $1 million per project on average. As a result, a project
    needing $100 million or more to mature technology to higher
    readiness levels than normal-not unreasonable sums-would command a
    fairly large share of an S&T organization's budget, thereby
    reducing funds available for other projects. Under the current
    scenario, the remaining 80 percent of DOD's research and
    development funds, approximately $30 billion, is spread out over a
    much smaller number of specific weapon programs. A typical weapon
    system program can receive several hundred million dollars
    annually and occasionally over $1 billion to fund development. A
    major program, such as the F-22, can command $15 billion or more
    in total for product development, receiving sometimes more than $2
    billion in a year. Events on the Air Force's ABL program
    illustrate these realities. Originally, the Air Force had planned
    the ABL as a technology development project to be managed to high
    readiness levels by an S&T organization. The project was started
    in 1992 as an advanced technology transition demonstration to
    design, fabricate, and test a single demonstrator weapon system
    and was to take 8 years to complete. The pacing technologies, the
    laser and the beam control, were to be matured to a high level-
    equivalent to TRL 6 or 7- before being included in a product
    development program. Requirements had not been fixed. In other
    words, the planned approach resembled what Page 51
    GAO/NSIAD-99-162 Best Practices Chapter 4 Impediments to Adopting
    Best Practices for Technology Inclusion in DOD Are Surmountable we
    have described as the more successful cases in our review. Figure
    4.1 shows the ABL. Figure 4.1:  Airborne Laser The Air Force used
    relatively low readiness level standards to include a key
    technology into ABL Source: DOD. In 1996, the Air Force abandoned
    this approach and decided to launch ABL as a weapon system
    development program, not because technologies were sufficiently
    mature but because of funding and sponsorship concerns. At this
    time, the two key technologies were at TRLs 3 and 4. According to
    the retired manager of the S&T project, a product development
    program was deemed necessary to make the technology development
    effort appear real to the users and not a scientific curiosity.
    Within the Air Force, the perceived lack of support by the users
    placed the project in a constant state of funding jeopardy. This
    perception was important because the S&T project was costly, with
    a total estimated cost of $800 million, with some annual funding
    requirements approaching $200 million. The annual funding
    requirements would encompass a large percentage of the Air Force's
    S&T budget unless additional funds were made available from weapon
    system budgets or elsewhere. By transitioning to a weapon system
    program linked to user requirements, the ABL was more likely to
    get these funding levels. This approach was successful-the program
    won user support and the desired funding. However, sacrifices were
    made in technology development. According to the former project
    manager, the new program focused less on the elemental technology
    hurdles and more on meeting all Page 52
    GAO/NSIAD-99-162 Best Practices Chapter 4 Impediments to Adopting
    Best Practices for Technology Inclusion in DOD Are Surmountable
    user requirements. More expensive demonstrations were necessary to
    meet these broader requirements without necessarily doing more to
    demonstrate basic technology readiness. It became a more
    traditional program with technology and product development
    proceeding at the same time, with attendant higher risks. In March
    1999, we reported that, while the ABL has made progress in
    developing these technologies, it still faced technical
    challenges.1 Other Incentives    Pressures exerted on weapon
    system programs can make it advantageous to include in their
    design immature technologies that offer significant performance
    gains. One traditional source has been the perceived threat. Users
    can demand performance improvements that necessitate the
    application of unproven technologies, particularly when a fielding
    date is mandated, to stay ahead of the threat. Another source is
    technologists, whether from S&T organizations or contractors, who
    see a new weapon system as an opportunity to apply a new
    technology. Also, the competition for funds can encourage
    performance features-and requisite technologies-that distinguish
    the new weapon system from competitors. The F-22 was justified as
    being faster, stealthier, and more lethal than other fighters,
    such as the F-15 and F-117, were. As a result, the F-22 is being
    designed with several advanced technologies, including a very
    sophisticated suite of avionics that is critical to its
    performance features that distinguish it from the other fighters.
    However, at the time the F-22 program was launched in 1986, the
    avionics technologies were immature; they have since been a source
    of problems on the program. We recently reported that the
    development of the F-22's integrated avionics systems continues to
    experience cost growth and schedule delays, more than 12 years
    into the program.2 A different set of incentives causes leading
    commercial firms to make their S&T organizations responsible for
    maturing technologies to higher readiness levels. Commercial firms
    are aware of the risks associated with the high investment that
    product development requires. They have a strong incentive in the
    realization that if a product is late, costs more, or performs
    1Defense Acquisitions: DOD Efforts to Develop Laser Weapons for
    Theater Defense (GAO/NSIAD-99-50, Mar. 31, 1999). 2F-22 Aircraft:
    Issues in Achieving Engineering and Manufacturing Development
    Costs (GAO/NSIAD-99-55, Mar. 15, 1999). Page 53
    GAO/NSIAD-99-162 Best Practices Chapter 4 Impediments to Adopting
    Best Practices for Technology Inclusion in DOD Are Surmountable
    less than expected, the customer could walk away from the product
    and the investment would be lost. Minimizing the possibility of
    technology being the cause of such problems is thus a top
    priority. Having their S&T organizations reduce those risks is
    essential to putting product developments in the best position to
    succeed. DOD does not have the same incentives. DOD programs are
    not penalized if a product is late, costs more, or performs less
    than expected, because the customer does not walk away. Services
    Encouraged      Over the past several years, DOD has encouraged
    the services to use best to Use Best Practices    practices to
    streamline the current process for acquiring new weapon systems in
    order to make them faster, cheaper, and better. Shorter
    acquisition cycle times are seen as critical to making the best
    use of advances in technology. To encourage change, DOD has set a
    goal to reduce the average acquisition cycle time for all program
    starts in fiscal year 1999 and beyond by 50 percent over
    historical averages. DOD has several initiatives to improve its
    technology development process and to move technologies to the
    warfighter faster and less expensively than the traditional means.
    The initiatives also attempt to put the organizations and funding
    in place to bring technologies to higher readiness levels before
    they are included in programs. These initiatives-defense
    technology objectives, advanced technology demonstrations, and
    advanced concept technology demonstrations-call for S&T
    organizations to play a bigger role in managing technologies
    closer to the point of product readiness, matching requirements to
    technology projects, and making better use of demonstration
    standards. Defense Technology       Defense technology objectives
    (DTO) are used to bring more discipline to Objectives
    S&T projects and to link them more closely with weapon system
    development programs. A DTO typically involves a particular
    technology advance, such as high temperature materials for turbine
    engines and high fidelity infrared sensors. It can also group
    several technologies into a larger demonstration. Each DTO
    identifies a specific technology advancement that will be
    developed or demonstrated, the anticipated date of the technology
    availability, the ultimate customer, and the specific benefits
    resulting from the technology. It places a corporate attention and
    commitment on the technology project by having the technologists,
    product developer, and customer involved in the project. According
    to DOD, the focus of its S&T investment is enhanced and guided
    through DTOs. Each DTO must go through a formal review and
    approval Page 54
    GAO/NSIAD-99-162 Best Practices Chapter 4 Impediments to Adopting
    Best Practices for Technology Inclusion in DOD Are Surmountable
    process within DOD and must be directly related to advancing the
    operational concepts depicted in DOD's "Joint Vision 2010"
    planning document. According to DOD officials, those requirements
    have helped to eliminate instances in which technologists work on
    projects of particular interest to them, but with no military
    application, because the projects should be linked to a specific
    warfighter need. For fiscal year 1999, DOD established
    approximately 350 DTOs, which accounted for $3 billion, or less
    than 50 percent, of the funds DOD had allocated to S&T projects.
    The remaining funds were allocated to projects under the
    jurisdiction of each military service or other defense agencies
    and did not go through the same review and approval process.
    Advanced Technology               Advanced technology
    demonstrations (ATD) are intended to more rapidly Demonstrations
    evolve and demonstrate new technologies so they can be
    incorporated into a product, if warranted. An ATD has four
    characteristics that distinguish it from a conventional S&T
    project. They (1) require large-scale resources; (2) involve the
    user; (3) use specific cost, schedule, and performance metrics;
    and (4) identify a target product for inclusion. An ATD is managed
    by an S&T organization and should conclude with an operational
    demonstration of the potential capabilities of the technology,
    equating to a TRL 5 or 6. The original approach to the ABL was
    essentially an ATD approach. Most ATDs use laboratory hardware to
    demonstrate the potential capability of nonproduct specific
    technologies and not prototype hardware. If the technology is
    determined to be feasible and provides some military use, then it
    may proceed to the program definition and risk reduction phase of
    an acquisition program. From that point, the product developer
    completes the technology development for a specific product.
    Advanced Concept                  In 1994, DOD initiated Advanced
    Concept Technology Demonstrations Technology Demonstrations
    (ACTD) to help expedite the transition of mature technologies from
    the developers to the warfighters. ACTDs are intended to help the
    DOD acquisition process adapt to budget constraints while
    developing technology more rapidly. The purpose of an ACTD is to
    assess the military use of a capability, such as a weapon,
    comprised of mature technologies. Typically, ACTDs last 2 to 4
    years and consist of building and demonstrating a prototype to
    provide a warfighter the opportunity to assess a prototype's
    capability in realistic operational scenarios. From this
    demonstration, the warfighter can refine operational requirements,
    develop an initial concept of operation, and determine the
    military use of the technology before it proceeds to the product
    development process. Page 55
    GAO/NSIAD-99-162 Best Practices Chapter 4 Impediments to Adopting
    Best Practices for Technology Inclusion in DOD Are Surmountable
    According to DOD, ACTDs, which are managed by S&T organizations,
    will be a key mechanism to ensure technology development is
    separated from product development. In related work on unmanned
    air vehicles, we found that ACTDs provided decisionmakers credible
    data that they used to terminate efforts or transition the
    demonstrator to an acquisition program. In these cases, ACTDs put
    decisionmakers in a better position to be gatekeepers. However, we
    have reported that the ACTD program needs to be improved.3 We
    found that DOD's process for selecting program candidates does not
    include adequate criteria for assessing the maturity of proposed
    technology and has resulted in the approval of projects that
    included immature technologies. We found that the use of specific
    criteria for determining maturity was a best practice in the most
    successful technology development cases we examined. Two Unique
    DOD               Two DOD projects are using S&T organizations to
    manage technology Projects May Provide         development to
    higher readiness levels. One, the Army's Future Scout and Cavalry
    System, is using a modified ATD to mature technologies and make
    Lessons on How to            performance trade-offs in the more
    flexible environment provided by Enable S&T                   S&T.
    The other is a joint government and industry program, which the
    Air Organizations to             Force Research Laboratory is
    managing to reduce the risks associated with new jet engine
    technologies. These projects may provide insights on how Manage
    Technology            S&T organizations could routinely play a
    bigger role in maturing Further                      technologies
    enough for safe inclusion on weapon system programs. They may also
    clarify the concern that playing a bigger role in technology
    maturation could cause S&T organizations to do less basic research
    and technology development. Future Scout and Cavalry     In fiscal
    year 1997, the Army began piloting a variation of an ATD that is
    System                       designed to help bridge the gap
    between technology development and product development by
    expanding the S&T community's role in managing technologies
    further into the development cycle. The Army's initiative, called
    Fast Track, is intended to reduce cost and cycle time by bypassing
    the program definition and risk reduction phase of the DOD
    acquisition process. The Army is testing this concept with its
    Future Scout and Cavalry System project. In this project, the Army
    will design, develop, and build a demonstrator vehicle to show the
    technical feasibility of the weapon. All of 3Defense Acquisition:
    Advanced Concept Technology Demonstration Program Can Be Improved
    (GAO/NSIAD-99-4, Oct. 15, 1998). Page 56
    GAO/NSIAD-99-162 Best Practices Chapter 4 Impediments to Adopting
    Best Practices for Technology Inclusion in DOD Are Surmountable
    these tasks will be done under the management of the Army's S&T
    organization. The Army believes that a more extensive S&T project
    will make the program definition phase unnecessary and estimates
    that this concept will reduce the development process by as much
    as 4 years and save about $400 million. We did not review the
    Future Scout project in terms of its affordability, feasibility,
    or any impacts it may have on the Army's S&T budget. Figure 4.2
    compares the Fast Track development process with the traditional
    approach. Figure 4.2:  Comparison of Traditional Technology
    Development Process With the Army's Fast Track Approach Technology
    maturation Technology feasibility                              and
    product development Traditional        S&T organization
    Weapon system program office approach
    9 years Program launched in
    Production program definition phase TRL 3-5 Bridge built to
    Technology feasibility
    Technology maturation increase technology               and
    product development maturity Future Scout        Army S&T
    organization                                  Weapon system
    program office and Cavalry System
    4.5  years Program office            Program launch
    Production staff joins S&T            in engineering, project
    office          manufacturing and development phase. TRL 6 While
    we do not necessarily agree that the first phase of the
    acquisition cycle can be omitted, so far the Future Scout project
    is emulating technology development practices like those we
    observed in the successful cases. First, it has established
    demonstration criteria that must be met before the technology
    enters product development. Second, it has also Page 57
    GAO/NSIAD-99-162 Best Practices Chapter 4 Impediments to Adopting
    Best Practices for Technology Inclusion in DOD Are Surmountable
    established forums that involve key players on the technology path
    to keep them informed of the technology's development progress.
    For example, the acquisition program manager will be integrated
    into the development project during the final 1.5 years of the S&T
    program. This should provide a good link between the technology
    development and product development, allowing the program manager
    to fully understand the technology before product development
    begins. Finally, by allowing an S&T organization the flexibility
    to manage technologies further into the development cycle, Army
    officials believe they will be able to make trade-offs among cost,
    schedule, and performance requirements before program launch,
    without raising concerns about the state of the project or
    breaching baselines that had been set without enough knowledge.
    While this concept comes closer to the most successful technology
    development cases we reviewed, it still embodies greater technical
    risk. The Army expects to demonstrate some performance
    capabilities of the vehicle before the product development phase
    begins. However, the demonstrator vehicle will only be about 75
    percent of a complete prototype, which means some key technologies
    will not be demonstrated to high readiness levels before that
    phase begins. Nonetheless, the project manager equated the
    expected overall technical maturity of the vehicle at transition
    to a TRL 6. The Army considers this a medium or acceptable level
    of risk, and it is willing to enter product development with some
    immature technologies. If, however, product development begins at
    engineering and manufacturing development, this risk could be
    assessed as high, based on TRLs. Figure 4.3 shows an artist's
    concept of the Future Scout and Cavalry System. Page 58
    GAO/NSIAD-99-162 Best Practices Chapter 4 Impediments to Adopting
    Best Practices for Technology Inclusion in DOD Are Surmountable
    Figure 4.3:  Future Scout and Cavalry System An Army S&T
    organization is maturing technologies and proposing performance
    trade-offs for the Future Scout and Cavalry System before program
    launch Source: DOD. Integrated High                  The
    Integrated High Performance Turbine Engine Technology program-a
    Performance Turbine              joint government and industry
    effort-is focused on developing Engine Technology Program
    technologies for more affordable and higher performance turbine
    engines for both missiles and aircraft. It is a technology
    validation program and is managed by an S&T organization to
    perform demonstrations of various engine technologies to higher
    readiness levels than most S&T projects. After the demonstrations,
    the technologies enter a product development program. The program
    takes the technology through a series of tests that range from
    individual component tests to full-scale engine demonstrations.
    The program has established strong links with the acquisition
    programs for which the technologies are intended. For example, Air
    Force Research Laboratory officials informed us that they
    established formal technology transition plans with the F-22 and
    Joint Strike Fighter programs that document agreements on what
    technology development activities will be performed to support the
    programs. Representatives from each program office are invited to
    all technology demonstrations and are kept informed about
    demonstrated progress. Page 59
    GAO/NSIAD-99-162 Best Practices Chapter 4 Impediments to Adopting
    Best Practices for Technology Inclusion in DOD Are Surmountable As
    part of the program, the Air Force developed a set of standards to
    assess the readiness levels of technologies similar to NASA's
    TRLs. According to the Air Force, the S&T organization uses these
    standards to determine when the project has been completed.
    These standards were the first application of readiness levels by
    the Air Force Research Laboratory. There are five technology
    readiness levels ranging from component-level tests in a
    laboratory to the highest level involving actual flight tests of
    engines. The program typically does not take technologies to the
    highest readiness level (flight test) because of the high cost.
    The program stops when it has been determined the technology is
    well defined within acceptable boundaries and a good correlation
    exists between test results and engineering predictions. This
    readiness level would translate to a TRL of 5 or 6, as used in
    this report. The final step of the technology development is left
    to the product developer who determines if the technology can be
    packaged and integrated into the final product. Page 60
    GAO/NSIAD-99-162 Best Practices Chapter 5 Conclusions and
    Recommendations
    Chapter 5 Conclusions    Clearly, DOD's continued advancement of
    new technologies is essential to the continued superiority of its
    weaponry. The leading edge military capabilities the United States
    possesses today, such as stealth aircraft, precision munitions,
    and intelligence-gathering satellites, bear witness to the effects
    of such technical advances. At the same time, the incorporation of
    advanced technologies before they are mature has been a major
    source of cost increases, schedule delays, and performance
    problems on weapon systems. As DOD contemplates increasing its
    annual investment in new weapons to $60 billion, the expectations
    on program managers are great: they must develop and field weapons
    of superior capability more quickly and less expensively than in
    the past. The way advanced technologies are matured and included
    in weapon systems will play a central role in meeting these
    expectations. Although different ways to better assimilate new
    technologies into weapons are legitimate topics for debate, that
    it has to be done better is not. The leading commercial firms'
    practices have produced results that resemble those sought by DOD:
    more technically advanced, higher quality products, developed in
    significantly less time, and less expensively than their
    predecessors. Managing the development of advanced technology
    differently--and separately--from the development of a product has
    been key to these results. The firms insist that advanced
    technology reach a high level of maturity, the point at which the
    knowledge about that technology is essentially complete, before
    allowing it into a product development. By separating the two, the
    firms lessen the product manager's burden and place that person in
    a better position to succeed in delivering the product. These
    practices may not necessarily accelerate the pace at which
    technology matures. In fact, several of the commercial
    technologies we reviewed took 10 years or more to get to market.
    The clear beneficiaries of the practices are the product
    developments, for which the investments are much larger, and time
    translates into significantly more resources than in a technology
    project. Adapting these practices on its weapon system programs
    can help DOD to reduce costs and the time from product launch to
    fielding, and use technology advances as they become available
    more frequently. Separating technology development from product
    development calls for a new approach to managing technology
    development. Two conditions are essential to such an approach.
    First, the right environment for maturing technologies must exist.
    A practice that is instrumental in providing this environment is
    maturing technology to achieve product readiness before it Page 61
    GAO/NSIAD-99-162 Best Practices Chapter 5 Conclusions and
    Recommendations is constrained by the rules of an acquisition
    program. In the successful DOD cases we reviewed, this environment
    was provided by S&T organizations or a team of S&T and product
    developers who managed technologies to high readiness levels
    before they were included in an acquisition program. These
    organizations provided an environment more conducive to the ups
    and downs normally associated with the discovery process. A
    corollary practice is agreeing on what level of knowledge is
    needed about a new technology before it is considered for
    inclusion in a product design. When that knowledge level does not
    exist, the flexibility for S&T organizations and product managers
    to either take the time to mature the technology or trade off
    product requirements until they can be met with mature technology
    is essential. It is a rare program that can proceed with a gap
    between product requirements and maturity of key technologies and
    still be delivered on time and within costs. Second, S&T and
    product managers must be provided with the disciplined processes,
    information, standards, and authority to make good handoffs of
    technology to product. Prepared with the tools and authority to
    make sound handoff decisions, both S&T and product managers can
    function as gatekeepers to safeguard the product development from
    undue technology risks. Leading commercial firms have adopted this
    approach as a matter of necessity and have used the organizations,
    tools, and other practices to foster technology development and
    improve the outcomes of product developments. The high stakes
    stemming from the large investment required for a new product and
    the risks if the product does not meet customer needs reinforce
    this approach in leading commercial firms. The DOD cases that
    followed a similar approach were realizing better program
    outcomes, at least in the sense that the programs avoided key
    technology development problems. Yet, these cases are not the norm
    for DOD programs for several reasons. * More typically, the
    commitment to develop and produce a weapon system is made before a
    match between technology and weapon system requirements exists. *
    DOD programs operate under different conditions that make it more
    difficult-and less rewarding-to separate technology development
    from product development. * Budget realities make it more
    difficult for S&T organizations to carry technologies to the high
    readiness levels needed to meet product requirements; such
    resources are more available within product developments. Page 62
    GAO/NSIAD-99-162 Best Practices Chapter 5 Conclusions and
    Recommendations * The pressures to show the unequalled performance
    necessary to win funding encourage including promising, but
    immature, technologies in weapon system designs. It will take
    procedural, organizational, and cultural changes within DOD's
    acquisition process to foster an environment in which (1)
    technologies can be successfully matured outside the purview of
    weapon system programs, (2) programs can be relieved of the
    pressures to include immature technologies and the unrealistic
    expectations that the technologies will conform to tight cost and
    schedule projections, and (3) technology advances will not stall
    due to inadequate funding or lack of identification with a product
    in the later, more expensive stages of demonstration. Experience
    has shown that such an approach can work within DOD on individual
    cases. DARPA played a primary role in managing the transition of
    the nonpenetrating photonics mast technology to the Virginia class
    attack submarine. The Integrated High Performance Turbine Engine
    Technology program has carried advanced jet engine technologies to
    TRLs of between 5 and 6 for successful inclusion into programs. In
    the Future Scout program, an Army S&T organization, augmented by
    product development staff, is managing an ATD to lower the risk of
    key technologies before a product development program is launched.
    However, it remains to be seen whether the Army will be successful
    in using large and expensive S&T projects, such as the Future
    Scout program, without affecting other Army S&T projects. A
    challenge for DOD will be whether the lessons learned from these
    individual cases offer an approach that has DOD-wide application.
    Meeting this challenge is essential to fielding technologically
    superior weapons more quickly and within predicted costs.
    Recommendations    We have previously recommended that DOD
    separate technology development from weapon system programs. That
    recommendation was made without prejudice toward the necessity of
    technology development but rather with the intent that programs
    could be better managed if such development was conducted outside
    of a program manager's purview. Similarly, the recommendations
    that follow are made without prejudice toward-or the intention of
    compromising-the basic research and other activities that S&T
    organizations perform. We recognize that implementation of these
    recommendations will have organizational, funding, and process
    implications and will require the cooperation of the Congress.
    Page 63                                         GAO/NSIAD-99-162
    Best Practices Chapter 5 Conclusions and Recommendations To help
    ensure that new technologies are vigorously pursued and
    successfully moved into weapon system programs, we recommend that
    the Secretary of Defense adopt a disciplined and knowledge-based
    method for assessing technology maturity, such as TRLs, DOD-wide.
    This practice should employ standards for assessing risks of
    handoff to program managers that are based on a technology's level
    of demonstration and its criticality to meeting the weapon
    system's requirements. With these tools in hand, we recommend that
    the Secretary (1) establish the place at which a match is achieved
    between key technologies and weapon system requirements as the
    proper time for committing to the cost, schedule, and performance
    baseline for developing and producing that weapon system and (2)
    require that key technologies reach a high maturity level-
    analogous to TRL 7-before making that commitment. This would
    approximate the launch point for product development as practiced
    by leading commercial firms. We recommend that the Secretary find
    ways to ensure that the managers responsible for maturing the
    technologies and designing weapon systems before product
    development are provided the more flexible environment that is
    suitable for the discovery of knowledge, as distinct from the
    delivery of a product. Providing more flexibility will require the
    cooperation of requirements managers and resource managers so that
    rigid requirements or the threat of jeopardizing the funding
    planned to start product development will not put pressure on
    program managers to accept immature technologies. Such an
    environment may not be feasible if the program definition and risk
    reduction phase remains the effective launch point for an entire
    weapon system program. An implication of these recommendations is
    that S&T organizations will have to play a greater role in
    maturing technologies to higher levels and should be funded
    accordingly. Therefore, we recommend that the Secretary of Defense
    evaluate the different ways S&T organizations can play a greater
    role in helping technologies reach high levels of maturity before
    product development begins. For example, given that a technology
    has sufficient potential for application to a weapon system, at a
    minimum, an S&T organization should be responsible for taking a
    technology to TRL 6 before it is handed off to a program office at
    the program definition and risk reduction phase. During this
    phase, the program manager would be responsible for maturing the
    technology to TRL 7 before it is included in an engineering and
    manufacturing development program. In a situation where a single,
    design-pacing technology is to be developed for a known Page 64
    GAO/NSIAD-99-162 Best Practices Chapter 5 Conclusions and
    Recommendations application-like the nonpenetrating periscope-an
    S&T organization should be required to mature that technology to
    TRL 7 before it is turned over to a product development manager.
    S&T organizations could play a similar role when a significant new
    technology is being prepared for insertion into an existing weapon
    system. Finally, when multiple new technologies are to be merged
    to create a weapon system, S&T organizations should be required to
    bring key technologies to TRL 6 and then become part of a hybrid
    organization with product developers to integrate the technologies
    and bring them to TRL 7 before handing full responsibility to a
    product development manager. To help guard against the possibility
    that the more basic research and technology development activities
    would be compromised by having S&T organizations routinely take
    key technologies to TRL 6 or higher, we recommend that the
    Secretary extract lessons from the nonpenetrating periscope, the
    AAAV, and the Army's Future Scout programs, and other ATD and ACTD
    programs. Specifically, the Secretary should assess whether the
    resources needed to enable S&T organizations to play a leading
    role in the development of technologies and, in some cases,
    preliminary system design, detracted from or displaced more basic
    research and technology development programs. Finally, we
    recommend that the Secretary empower managers of product
    development programs to refuse to accept key technologies with low
    levels of demonstrated maturity. The Secretary can encourage this
    behavior through supportive decisions on individual programs, such
    as by denying proposals to defer the development of key
    technologies and by favoring proposals to lengthen schedules or
    lessen requirements to reduce technological risk early. Agency
    Comments and  DOD generally concurred with a draft of this report
    and its Our Evaluation                  recommendations, noting
    that the traditional path to new weapon system development is no
    longer affordable or necessary (see app. I). DOD stated that it
    has embarked upon a "Revolution in Business Affairs" that will
    enable new technologies to be developed more efficiently and
    effectively. It believes that the first steps in this direction
    have already been taken but agrees that more progress needs to be
    made. DOD agreed that TRLs are necessary in assisting
    decisionmakers in deciding on when and where to insert new
    technologies into weapon system programs and that weapon system
    managers should ensure that technology is matured to a TRL 7
    before insertion occurs. DOD concurred that S&T organizations
    should be Page 65
    GAO/NSIAD-99-162 Best Practices Chapter 5 Conclusions and
    Recommendations involved in maturing technologies to high levels,
    such as TRL 6, before transitioning to the engineering and
    manufacturing development phase and agreed to assess the impact of
    this involvement on other S&T resources. We note that the best
    practice is to mature technology to at least a TRL 7 before
    starting the engineering and manufacturing development phase,
    whether the technology is managed by an S&T organization, a weapon
    system program manager, or a hybrid of the two organizations. DOD
    noted that while TRLs are important and necessary, the increasing
    projected life for new weapon systems, total ownership costs, and
    urgency based upon threat assessments are also important
    considerations for system development decisions. We agree and note
    that our recommendations are not intended to cover all aspects of
    weapon system development decisions or to suggest that technology
    maturity is the only factor in such decisions. Rather, the
    recommendations are in keeping with the purpose of the report, "to
    determine whether best practices offer methods to improve the way
    DOD matures new technology so that it can be assimilated into
    weapon system programs with less disruption." We believe that a
    knowledge-based approach to maturing technology, such as TRLs, can
    benefit other considerations as well. For example, decisions on
    what technologies to include in a weapon system and when to
    include them can have a significant bearing on its total ownership
    costs. DOD stated that there should be an established point for
    the transition of technologies and that it plans to supplement its
    milestone review process with additional guidance in the next
    revisions to DOD Directive 5000.2R. It also stated that its policy
    on the evolutionary approach to weapon acquisitions should be
    developed in consonance with the technology transition strategy.
    We cannot comment on the revisions to the directive or the
    evolutionary acquisition policy because they have yet to be
    published. However, under the current milestone review process,
    the pressures placed on a program during the program definition
    and risk reduction phase- when much technology development occurs-
    can operate against the flexibility and judgments that are needed
    to mature technologies. If the revisions to the directive
    supplement the current milestones without relieving the pressures
    brought to bear on programs as they are launched in the program
    definition and risk reduction phase, it will remain difficult to
    discourage the acceptance of immature technologies in the design
    of new weapon systems. To relieve these pressures, we encourage
    DOD, as it develops the directive and the evolutionary acquisition
    policy, to separate technology development from product
    development and to redefine the launch point for a program as the
    point at which enough knowledge has Page 66
    GAO/NSIAD-99-162 Best Practices Chapter 5 Conclusions and
    Recommendations been gained to ensure that a match is reached
    between the maturity of key technologies and weapon system
    requirements. DOD also stated that program managers already have
    the ability to reject inappropriately mature technologies, and to
    the extent technology immaturity affects acquisition baselines, to
    advise acquisition executives of feasible alternatives. We did not
    find this to be the case in our review. Rather, we found that the
    program managers' ability to reject immature technologies is
    hampered by (1) untradable requirements that force acceptance of
    technologies despite their immaturity and (2) reliance on tools
    for judging technology maturity that fail to alert the managers of
    the high risks that would prompt such a rejection. As noted in the
    report, once a weapon system program begins, the environment
    becomes inflexible and deviations to program baselines can attract
    unwanted attention. This reality limits the program managers'
    ability to reject immature technologies. Page 67
    GAO/NSIAD-99-162 Best Practices Appendix I Technology Readiness
    Levels and Their Definitions
    Appendix I Technology readiness level
    Description 1. Basic principles observed and reported.
    Lowest level of technology readiness. Scientific research begins
    to be translated into applied research and development. Examples
    might include paper studies of a technology's basic properties. 2.
    Technology concept and/or application                Invention
    begins. Once basic principles are observed, practical applications
    can be formulated.
    invented. The application is speculative and there is no proof or
    detailed analysis to support the assumption. Examples are still
    limited to paper studies. 3. Analytical and experimental critical
    function        Active research and development is initiated. This
    includes analytical studies and and/or characteristic proof of
    concept.                 laboratory studies to physically validate
    analytical predictions of separate elements of the technology.
    Examples include components that are not yet integrated or
    representative. 4. Component and/or breadboard validation in
    Basic technological components are integrated to establish that
    the pieces will work laboratory environment.
    together. This is relatively "low fidelity" compared to the
    eventual system. Examples include integration of "ad hoc" hardware
    in a laboratory. 5. Component and/or breadboard validation in
    Fidelity of breadboard technology increases significantly. The
    basic technological relevant environment.
    components are integrated with reasonably realistic supporting
    elements so that the technology can be tested in a simulated
    environment. Examples include "high fidelity" laboratory
    integration of components. 6. System/subsystem model or prototype
    Representative model or prototype system, which is well beyond the
    breadboard demonstration in a relevant environment.
    tested for TRL 5, is tested in a relevant environment. Represents
    a major step up in a technology's demonstrated readiness. Examples
    include testing a prototype in a high fidelity laboratory
    environment or in simulated operational environment. 7. System
    prototype demonstration in an                 Prototype near or at
    planned operational system. Represents a major step up from
    operational environment.                                TRL 6,
    requiring the demonstration of an actual system prototype in an
    operational environment, such as in an aircraft, vehicle or space.
    Examples include testing the prototype in a test bed aircraft. 8.
    Actual system completed and "flight qualified"  Technology has
    been proven to work in its final form and under expected
    conditions. through test and demonstration.
    In almost all cases, this TRL represents the end of true system
    development. Examples include developmental test and evaluation of
    the system in its intended weapon system to determine if it meets
    design specifications. 9. Actual system "flight proven" through
    Actual application of the technology in its final form and under
    mission conditions, successful mission operations.
    such as those encountered in operational test and evaluation. In
    almost all cases, this is the end of the last "bug fixing" aspects
    of true system development. Examples include using the system
    under operational mission conditions. Page 68
    GAO/NSIAD-99-162 Best Practices Appendix II Comments From the
    Department of Defense Appendix II Page 69              GAO/NSIAD-
    99-162 Best Practices Appendix II Comments From the Department of
    Defense Now on pp. 7 and 63-64. Now on pp. 7 and 63-64. Now on pp.
    7 and 63-64. Page 70                                    GAO/NSIAD-
    99-162 Best Practices Appendix II Comments From the Department of
    Defense Now on pp. 7 and 63-64. Page 71
    GAO/NSIAD-99-162 Best Practices Appendix II Comments From the
    Department of Defense Now on pp. 7 and 65. Now on pp. 7 and 65.
    Page 72                                    GAO/NSIAD-99-162 Best
    Practices Appendix III GAO Contacts and Staff Acknowledgments
    Appendix I II GAO Contacts       Louis Rodrigues, (202) 512-4841
    Paul Francis, (202) 512-2811 Acknowledgments    In addition to
    those named above, Michael Sullivan, Jeffrey Hunter, Matthew Lea,
    Maria Santos, Rae Ann Sapp, and Katrina Taylor made key
    contributions to this report. Page 73
    GAO/NSIAD-99-162 Best Practices Page 74    GAO/NSIAD-99-162 Best
    Practices Page 75    GAO/NSIAD-99-162 Best Practices Related GAO
    Products Best Practices: Commercial Quality Assurance Practices
    Offer Improvements for DOD (GAO/NSIAD-96-162, Aug. 26,1996). Major
    Acquisitions: Significant Changes Underway in DOD's Earned Value
    Management Process (GAO/NSIAD-97-108, May 5, 1997). Best
    Practices: Successful Application to Weapon Acquisitions Requires
    Changes in DOD's Environment (GAO/NSIAD-98-56, Feb. 24, 1998).
    Best Practices: DOD Can Help Suppliers Contribute More to Weapon
    System Programs (GAO/NSIAD-98-87, Mar. 17, 1998). Defense
    Acquisition: Improved Program Outcomes Are Possible (GAO/T-NSIAD-
    98-123, Mar. 18, 1998). Defense Acquisitions: Best Commercial
    Practices Can Improve Program Outcomes (GAO/T-NSIAD-99-116, Mar.
    17, 1999). (707336)    Letter    Page 76
    GAO/NSIAD-99-162 Best Practices Ordering Information The first
    copy of each GAO report and testimony is free. Additional copies
    are $2 each.  Orders should be sent to the following address,
    accompanied by a check or money order made out to the
    Superintendent of Documents, when necessary, VISA and MasterCard
    credit cards are accepted, also. Orders for 100 or more copies to
    be mailed to a single address are discounted 25 percent. Orders by
    mail: U.S. General Accounting Office P.O. Box 37050 Washington, DC
    20013 or visit: Room 1100 700 4th St. NW (corner of 4th and G Sts.
    NW) U.S. General Accounting Office Washington, DC Orders may also
    be placed by calling (202) 512-6000 or by using fax number (202)
    512-6061, or TDD (202) 512-2537. Each day, GAO issues a list of
    newly available reports and testimony.  To receive facsimile
    copies of the daily list or any list from the past 30 days, please
    call (202) 512-6000 using a touchtone phone.  A recorded menu will
    provide information on how to obtain these lists. For information
    on how to access GAO reports on the INTERNET, send an e-mail
    message with "info" in the body to: [email protected] or visit
    GAO's World Wide Web Home Page at: http://www.gao.gov United
    States                       Bulk Rate General Accounting Office
    Postage & Fees Paid Washington, D.C. 20548-0001            GAO
    Permit No. GI00 Official Business Penalty for Private Use $300
    Address Correction Requested

*** End of document. ***