Mr. Chairman and Members of the Subcommittee:
I am pleased to be here today to discuss the application of best commercial practices to Department of Defense (DOD) weapon acquisitions in general and to the Joint Strike Fighter in particular.
After having done hundreds of reviews of major weapon acquisitions over the last 20 years, we have seen many of the same problems recur—cost increases, schedule delays, and performance problems. Over the last 4 years, we have undertaken a body of work to examine weapon acquisition issues from a different, more cross-cutting perspective. Specifically, we have examined the best product development practices of leading commercial firms. Collectively, our reviews have included the practices of over 20 leading commercial firms that represent a variety of industries, including electronics, satellite communications, automotive, medical, and aircraft. Leading commercial firms are getting the kind of results that DOD seeks: more sophisticated products developed in less time and cost than their predecessors. Our work shows that DOD can learn valuable lessons from the commercial sector to get better and more predictable outcomes from weapon system development programs. A listing of the reports we have issued on best practices that can be applied to weapon acquisitions is included in the appendix.
DOD has taken steps to reflect best commercial practices in its acquisition policies. However, the real test of these policies is in how they influence individual decisions, such as the upcoming engineering and manufacturing development decision on the Joint Strike Fighter program. This program is to produce three fighter variants to meet multiservice requirements: conventional flight for the Air Force, short take-off and landing for the Marine Corps, and carrier operations for the Navy. The program will also provide aircraft to the British royal Navy and Air Force. As currently planned, the program will cost about $200 billion to develop and procure over 3,000 aircraft and related support equipment.
My testimony focuses on the best commercial practices for developing new products, the reasons why DOD does not follow these practices, and the opportunity that Joint Strike Fighter represents to strengthen—or weaken—the effect of best practices and acquisition reform on major weapons.
DOD programs, with some exceptions, proceed with lower levels of knowledge about key factors of product development and allow technology development to take place during product development. DOD’s variances from best practices stem from strong incentives for starting programs too early; overpromising performance capabilities; and understating expected costs, schedules, and technical risks. While these incentives evolved over time to help build support for programs, they put program managers in a very difficult position to deliver better weapons on time and within budget. Moreover, there is little risk that the DOD customer will be dismayed by a program not being delivered as promised. DOD accepts the need to get better outcomes from its weapon system programs and accepts best commercial practices as a way to get those outcomes. In fact, it is currently incorporating such practices in a major revision of its acquisition policy. However, new policies will not produce better program outcomes unless they influence the decisions made on individual weapon systems.
DOD has designated the Joint Strike Fighter as a flagship program for acquisition reform. Funding requests are now before the Congress to support the Joint Strike Fighter’s April 2001 entry into engineering and manufacturing development. By best practices standards, none of the fighter’s critical technology areas identified by the program office are expected to be at readiness levels considered an acceptable risk for entry into engineering and manufacturing development. Delaying this phase of the program until these technologies are mature would improve the chances that the Joint Strike Fighter will be fielded as planned. However, despite not having the requisite knowledge for the eight technologies, DOD has deemed the risks manageable and proposes to proceed with the program as planned. Such a decision reinforces traditional incentives and increases the likelihood for future cost, schedule, and performance problems. DOD’s plans to move the Joint Strike Fighter into engineering and manufacturing development with immature technology illustrates a lack of commitment to following best commercial practices as part of its acquisition reform efforts.
The characteristics of best commercial practices suggest a process for developing new capabilities—whether they are commercial or defense products—that is based on knowledge. It is a process in which technology development and product development are treated differently and managed separately. The process of developing technology culminates in discovery and must, by its nature, allow room for unexpected results and delays. The process of developing a product culminates in delivery, and therefore, gives great weight to design and production. Discipline is inherent because criteria exist, tools are used, and a program does not go forward unless a strong business case on which the program was originally justified continues to hold true.
We have learned that a knowledge-based process is essential to getting better cost, schedule, and performance outcomes. This means that decision makers must have virtual certainty about critical facets of the product under development when needed. Such knowledge is the inverse of risk. Commercial and military programs do not all follow the same processes in their development cycles. However, at some point, full knowledge is attained about a completed product, regardless of the development approach taken. This knowledge can be measured at three key junctures that we refer to as knowledge points:
We have identified metrics that indicate these knowledge levels and can thus help forecast outcomes as a development program progresses. A best practices model for technology development and product development is depicted in figure 1.
Figure 1: Best Practices for Developing Technology and Products
Commercial firms gain more knowledge about a product’s technology, design, and producibility much earlier than DOD acquisition programs we reviewed. Two features of leading commercial products stand out for making a manageable product development. First, there is a clear break between technology development and product development. The launch of a new product development in commercial ventures is a clearly defined undertaking and before beginning, firms insist on having the technology in hand that is needed to meet customer requirements. Second, leading firms limit the length of time it takes to develop the product. This limit is key to getting the product to market and focuses attention on the design and production knowledge points. It also provides discipline to the technology development process to ensure product development will not be launched until the technology match is made.
The leading commercial firms we have visited consciously limited their product developments from 18 months to just over 4 years. They understand that this keeps product development within a time frame that can keep people focused on delivering a product. In fact, one commercial executive observed that it is unreasonable to expect people to focus on a goal like production start that is more than 4 years away. The limited time frames provide strong incentives for a commercial manager to keep immature technologies out of the product design. In fact, these time frames give product managers clout in fostering cost and performance trade-offs before the program begins, ultimately limiting a product’s requirements to what can be achieved with demonstrated technologies within the specified time period. To live within these time constraints and keep innovation alive, commercial firms have adopted an evolutionary approach; they save requirements that cannot be met with proven technologies for the next iteration of the product. Commercial firms also found that limiting product development time frames
Once a product development is under way, the firms demand—and receive—specific knowledge about the design and producibility of the product before production begins. The process of discovery—the accumulation of knowledge and elimination of unknowns—is completed well ahead of production. There is a synergy in this process, as the attainment of each successive knowledge point builds on the preceding one.
In contrast, DOD programs are started earlier and allow technology development to continue into product development and even into production. Consequently, the programs proceed with much less knowledge available—and thus more risk—about required technologies, design capability, and producibility. This approach to technology and product development is shown in figure 2.
Figure 2: Levels of Knowledge Attained in DOD Weapon System Developments
Proceeding with lower levels of knowledge available means that during product development, maturity of technology, the design, and production methods must all be pursued at the same time. The rippling effect of discovering and overcoming problems in product development explains much of the turbulence in DOD program outcomes. Metrics, such as technology readiness levels and percent of engineering drawings complete, can be used to predict these consequences. Product development times, long to begin with, stretch even further in reaction to problems. We calculate that they can take 3 to 10 times as long as commercial products.
Without going into the details of each level, a level four equates to a laboratory demonstration of a technology that is still not in a usable form. Imagine an advanced radio technology that can be demonstrated with components that take up a table top. When initial hand-built versions of all of the radio’s basic elements are hand-wired and tested together in a laboratory, the radio reaches a readiness level of five. A technology readiness level of seven is the demonstration of a technology that approximates its final form and occurs in an environment outside the laboratory. The same radio at level seven would be installed and demonstrated in a platform, such as a fighter aircraft.
The lower the level of the technology at the time it is included in a product development program, the higher the risk that it will cause problems in the product development. According to the Air Force Research Laboratory, level seven enables a technology to be included on a product development with acceptable risk. When we asked leading commercial firms to apply these standards to their own methods for assessing technology maturity, we found that most insisted on even higher levels of readiness before they allowed a new technology into product development. When we examined weapon systems that experienced cost and schedule problems, we found that they started with key technologies at levels three and four. By the time the programs reached a point DOD considers analogous to beginning product development, key technologies were still at level five or lower. Conversely, DOD programs for which key technologies were at significantly higher levels of maturity at the start, had not experienced such problems.
Both DOD and commercial companies consider the design to be essentially complete when about 90 percent of the engineering drawings are completed. Officials from commercial companies such as Boeing and Hughes told us that they typically had over 90 percent of these drawings available for the critical design review. The DOD programs we reviewed had less than 60 percent—one had less than one-third—of the drawings done at the time their critical design reviews were held. Thus, these programs had significantly less knowledge about their designs. The programs did not get or were not expected to get to the 90-percent level of completion on the drawings until late in development or in production. Nonetheless, at the time of the critical design reviews, the risks of proceeding with the rest of development on these programs as planned were deemed acceptable. The programs however encountered significant design problems in testing that occurred after the critical design review.
DOD programs did not have nearly this level of knowledge at production. One weapon system program that had been in production for nearly 9 years at the time of our 1998 review still had less than 13 percent of its key manufacturing processes in control. Another program had 40 percent of its key manufacturing processes in control 2 years before production was scheduled to begin but was not scheduled to have all key processes in control until 4 years into production. Both programs experienced basic producibility problems that were not discovered until late in development or early in production. These risks went unrecognized even though DOD had established criteria for determining whether risks were acceptable and whether enough knowledge had been gained to enter the next development phase.
Production is a dominant concern throughout the product development process and forces discipline and trade-offs in the design process. This environment encourages realistic assessments of risks and costs; doing otherwise would threaten the business case and invite failure. For the same reasons, the environment places a high value on knowledge for making decisions. Program managers have good reasons to identify risks early, be intolerant of unknowns, and not rely on testing late in the process as the main vehicle for discovering the performance characteristics of the product. By protecting the business case as the key to success, program managers in leading commercial firms are conservative in their estimates and aggressive in risk reduction. Ultimately, preserving the business case strengthens the ability of managers to say "no" to pressures to accept high risks or unknowns. Practices such as maturing technologies to high readiness levels before inclusion in a program, having 90 percent of engineering drawings done by the critical design review, and achieving statistical process control before production are adopted because they help ensure success.
Even though less information about a new product development is available at the time DOD programs are launched, the competition for funding forces detailed projections to be made from that information. A product development cannot be launched unless the program’s development and production cost, as well as timing, falls within available funding. Because DOD relies largely on forecasts of cost, schedule, and performance that are comparatively soft at the time, success in competing for funding encourages managers to squeeze cost and schedule estimates into profiles of available funding. Additional requirements, such as high reliability and maintainability, serve to make the fit even tighter. As competition for funding will continue throughout the program’s development, success is measured in terms of ability to secure the next installment.
The risks associated with developing new technologies together with the product—within tight estimates—are deemed acceptable. Production realities, critical to matching technological capabilities with customer requirements on commercial programs, are too far away from the DOD launch decision to have the same curbing effect on technology decisions. The environment for managing weapon system programs is particularly difficult for managing technology development. The ups and downs and resource changes associated with the technology discovery process do not mesh well with a program’s need to meet cost, schedule, and performance goals. Problems with developing technologies, which are to be expected, can actually threaten the support for a program if they become known.
These pressures and incentives explain why the behavior of weapon system managers differs from managers of commercial product developments. Problems or indications that the estimates are being breached do not help sustain funding support for the program in later years, and thus their admission is implicitly discouraged. An optimistic cost estimate makes it easier to launch a product development and sustain annual approval; admission that costs are likely to be higher could invite failure. Rewards for discovering and recognizing potential problems early in a DOD product development are few. Less available knowledge makes it harder for program managers to say "no." In contrast with leading commercial firms, not having attained knowledge—on the full performance of a key technology or the true risks facing manufacture, for example,—can be perceived as better than knowing that problems exist. For these reasons, the practices associated with managing to knowledge standards—such as for technology, design, and production maturity—are not readily adopted in DOD.
These observations about the differences between the commercial and DOD environments should not be interpreted to mean that commercial managers are somehow more skilled or knowledgeable than their DOD counterparts. DOD program managers act in response to the pressures they face. All of the numerous participants in the acquisition process play a part in creating these pressures. Commercial program managers are put in a better position to succeed; they have to worry only about product design and production within the cost, schedule, and performance demands of the business case.
Second, once a program is under way, the participants in the acquisition process must make it acceptable for managers to identify unknowns as high risks so that they can be aggressively worked on earlier in development. If the Congress and DOD weighed program launch decisions and subsequent progress on weapon systems by applying a common set of knowledge standards, like those gleaned from leading commercial firms, they could create a better business case for starting a weapon system program. By developing technology separately to high readiness levels before including it in a program and by adhering to standards such as knowledge points in product development, DOD program managers can be put in a better position to succeed in the timely design and production of weapon systems. The shorter cycle times associated with these practices could make it possible to better align the tenures of program managers with the product development phase, making them more accountable for program outcomes.
The Office of the Under Secretary of Defense (Acquisition, Technology, and Logistics) supports shorter weapon system development times and more aggressive pursuit of technology outside of programs. It also supports the use of best commercial practices, such as taking an evolutionary approach to developing new products. DOD is capturing these and other practices in a substantial revision of the regulations that guide the management of weapon system programs. These regulations are currently in draft form.
The real test of the participants’ resolve to get better outcomes by applying best practices will be the decisions made on individual weapon systems, such as for launch and funding. These decisions define what success means in DOD and what practices contribute to success. Decisions made by DOD or the Congress to advance or fund programs that do not have enough knowledge to meet agreed-upon standards signal to managers that not having the necessary level of knowledge is acceptable. On the other hand, participants should support decisions to not start new programs that need technology advances to meet unforgiving requirements or to recognize early that a change in a program is necessary to attain desired knowledge levels.
The best practice for such a decision is to have a match between mature technologies and weapon requirements. It represents the first knowledge point. The Joint Strike Fighter does not meet this standard; several technologies that are critical to meeting requirements will not be sufficiently mature. Consequently, the Joint Strike Fighter will not enter the engineering and manufacturing development phase with low technical risk. However, DOD would like to go forward with the program anyway. Doing so would have two major consequences. First, it would put the program on a path that has yielded cost growth and schedule slippage on many previous programs. Second, as Joint Strike Fighter is the largest acquisition in the foreseeable future, it will send signals to other programs that best practices and acquisition reform need not be heeded when it comes to major weapon systems.
In our recently issued report, we evaluated the maturity of key technologies on the Joint Strike Fighter program. At our request, the program office identified eight technology areas that are considered critical to meeting the fighter’s cost, schedule, and performance objectives. In conjunction with the program office and the two competing contractors, we determined the readiness levels of these technologies needed to meet Joint Strike Fighter performance requirements at three points in time: when the Joint Strike Fighter program was started in 1996, when we conducted our review in December 1999, and when the program is scheduled to enter engineering and manufacturing development. Those assessments showed that when the Joint Strike Fighter program was started, most of the critical technologies were well below the readiness levels considered acceptable risk to begin a program. The technology readiness levels of the eight critical Joint Strike Fighter technology areas are shown in figure 3.
Figure 3: Joint Strike Fighter Critical Technology Readiness Levels at Program Start and as Projected for Entry into Engineering and Manufacturing Development
As figure 3 shows, all of the critical technology areas are expected to be at readiness levels lower than the level seven considered acceptable risk for entry into engineering and manufacturing development. Six of the technologies will still be below the readiness level that is considered acceptable risk for program start, which occurred over 3 years ago for the Joint Strike Fighter program. Many of these will only be demonstrated in laboratories or in ground tests when the engineering and manufacturing development phase starts. They have a considerable amount of development remaining before they are considered mature. Moreover, as a result of cost growth and schedule concerns, in May 1999, DOD delayed some technology demonstrations until after the engineering and manufacturing development phase begins.
Should any of these technologies be delayed or, worse, not available for incorporation into the final Joint Strike Fighter design, the impact on the program would be dramatic. For example, if one of the above critical technologies needed to be replaced with its planned backup, DOD could expect an increase of several billion dollars in production and operation and support costs. The backup technology would also significantly increase aircraft weight, which can negatively impact aircraft performance. This technology is expected to be at unacceptable readiness levels at the beginning of the engineering and manufacturing development phase, which indicates that substantial technology development must still occur during this phase.
As noted above, at the policy level, DOD officials have agreed that technology development should be kept separate from product development and that technology readiness levels are a valid way to assess technology maturity. However, in its response to our report on the Joint Strike Fighter—an individual program decision—DOD balked at the use of technology readiness levels and their implications for keeping technology development out of the Joint Strike Fighter’s engineering and manufacturing development phase. One of the reasons DOD cited for its unwillingness to accept the technology readiness levels assessed for the eight Joint Strike Fighter technologies was that the levels were based on integration in the Joint Strike Fighter aircraft—too high a standard. On the contrary, the technology readiness levels assessed by the program office and the contractors were based on a clear understanding that a level seven could be reached by demonstrating a technology in a relevant environment. It was further made clear that a relevant environment would include demonstrating a technology in an existing aircraft like an F-16, not a Joint Strike Fighter.
A second reason DOD disagreed with the readiness levels assessed for the eight technologies was that its own risk mitigation plans and judgment were more meaningful and that they showed the technology risk to be acceptable. Risk mitigation plans and judgment are necessary to managing any major development effort like the Joint Strike Fighter. However, without an underpinning such as technology readiness levels that allows transparency into program decisions, these methods allow significant technical unknowns to be judged acceptable risks because a plan exists for resolving the unknowns in the future. Experience on previous programs has shown that such methods have rarely assessed technical unknowns as a high or unacceptable risk; consequently, they failed to guide programs to meet promised outcomes. Technology readiness levels are based on actual demonstrations of how well technologies actually perform. Their strength lies in the fact that they characterize knowledge that exists rather than plans to gain knowledge in the future; they are thus less susceptible to optimism. A clear picture of knowledge—or its absence—may be more likely to prompt action than a favorable risk assessment.
Traditionally, a weapon’s final performance requirements are developed early in a program, or in many cases before the program begins. In the case of Joint Strike Fighter program, requirements were finalized much later in the acquisition cycle. Program officials stated that this provided the program flexibility to conduct cost and performance trade-offs before requirement and design decisions became final. While this approach is consistent with best practices, it has not adequately taken into account the readiness of the critical technologies. Many of the trade-offs that were made involved decisions to bring technologies that were not yet demonstrated into the engineering and manufacturing phase of the program. Thus, the program does not have a baseline design based on demonstrated technologies that could be developed in cycle times commensurate with best practices.
As the Joint Strike Fighter program approaches the engineering and manufacturing development decision point, it is not in a good position to succeed, if success means delivering the aircraft on time and within budget. The design requires significant technological invention in order to satisfy all the user’s requirements. According to Joint Strike Fighter officials, an objective of the engineering and manufacturing development phase is bringing the technologies up to maturity levels that will allow them to be incorporated onto a Joint Strike Fighter. Therefore, at a time when the program should be focused on designing and building the aircraft, the Joint Strike Fighter program will have to put significant effort and resources into demonstrating that key technologies are ready for inclusion onto the product. As a result, the program has planned almost 8 years for its engineering and manufacturing development phase. The length and scope of the effort operate against the ability to estimate cost and completion schedules.
Commercial firms have established practices to limit product development cycle times, thereby increasing the possibility that program managers will remain on programs until they are complete. Holding one program manager accountable for the content of a product at the time the launch decision is made encourages that person to raise issues and problems early and not overpromise the capabilities of a new product by relying on immature technologies. This puts the manager in a good position to deliver a high quality product, on time, and on budget.
Since the next phase of the Joint Strike Fighter program is estimated to last about 8 years, program managers currently involved in key decisions about the development plan will likely not be responsible for its implementation. It has already had three program managers since its beginning about 3 ½ years ago. As a result, conditions to be accepted at engineering and manufacturing development, such as the acceptance of low technology readiness levels, will more than likely become the responsibility of another program manager.
In our report, we recommend that the Joint Strike Fighter program continue in its current program definition and risk reduction phase, delaying the decision to move into engineering and manufacturing until technologies are demonstrated to acceptable levels. Taking the additional time to mature the technologies will then allow the program manager to focus on design and manufacturing risks during engineering and manufacturing development. It also increases the possibility of completing product development in a more timely and predictable manner. Such a delay does not necessarily lengthen the total product development cycle. In fact, the knowledge gained from time spent developing technologies in the beginning can often shorten the time it takes to get the product to market.
Similarly, a delay should not be misinterpreted as a lessening of support for the Joint Strike Fighter program. Rather, it would demonstrate decisionmakers’ willingness to make the up-front investment necessary to mature key technologies before committing the Joint Strike Fighter team to deliver a product. Such a commitment is more likely to put the program on a better footing to succeed than placing the burden on the engineering and manufacturing development phase.
Defense Acquisition: Employing Best Practices Can Shape Better Weapon System Decisions (GAO/T-NSIAD-00-137, Apr. 26, 2000).
Best Practices: DOD Training Can Do More to Help Weapon System Program Implement Best Practices (GAO/NSIAD-99-206, Aug. 16, 1999).
Best Practices: Better Management of Technology Development Can Improve Weapon System Outcomes (GAO/NSIAD-99-162, July 30, 1999).
Defense Acquisitions: Best Commercial Practices Can Improve Program Outcomes (GAO/T-NSIAD-99-116, Mar. 17, 1999).
Defense Acquisition: Improved Program Outcomes Are Possible (GAO/T-NSIAD-98-123, Mar. 17, 1998).
Best Practices: DOD Can Help Suppliers Contribute More to Weapon System Programs (GAO/NSIAD-98-87, Mar. 17, 1998).
Best Practices: Successful Application to Weapon Acquisition Requires Changes in DOD’s Environment (GAO/NSIAD-98-56, Feb. 24, 1998).
Major Acquisitions: Significant Changes Underway in DOD’s Earned Value Management Process (GAO/NSIAD-97-108, May 5, 1997).
Best Practices: Commercial Quality Assurance Practices Offer Improvements for DOD (GAO/NSIAD-96-162, Aug. 26, 1996).
Orders by Internet
For information on how to access GAO reports on the Internet, send an e-mail message with "info" in the body to:
or visit GAO’s World Wide Web home page at:
Web site: http://www.gao.gov/fraudnet/fraudnet.htm
E-mail: [email protected]
1-800-424-5454 (automated answering system)