Programs to develop unmanned aerial vehicles for the U.S. military have frequently run into problems. In the mid-1990s, the Defense Airborne Reconnaissance Office, which is responsible for developing UAVs, tried a new approach: designing unmanned aerial vehicles as Advanced Concept Technology Demonstrations. The Predator medium-altitude endurance UAV was developed that way and is now being procured by the Air Force. Today, three other UAVs, the tactical Outrider and the high-altitude endurance Global Hawk and Darkstar, are being developed as ACTDs. Those programs prompt two questions: why has the Department of Defense had so much trouble developing UAVs? And has the use of the ACTD process improved that situation?

The second question is the more difficult of the two to answer. ACTDs are designed to develop new military systems and demonstrate them in the field faster and more cheaply than the traditional acquisition process. Developers in ACTD programs can use flexible management processes, "mature" technologies (ones that have already been proved in other systems), and close involvement by operational users and joint commanders to pursue those goals. Because of the complexities of UAVs, however, ACTD programs for those vehicles have had mixed success. Their rocky progress suggests that some of the causes of growing costs and delayed schedules are beyond the ability of the ACTD process to reform. But despite those problems, the ACTD approach appears to have had some success in areas where past UAV programs struggled, such as avoiding growth in operational requirements, improving cooperation among services and military commands, and providing commanders with the opportunity to try a new system in the field.
 

WHY DEVELOPING UAVs HAS BEEN DIFFICULT

Some Members of Congress and segments of the defense community have criticized DoD for its seeming inability to develop and field a tactical UAV. Recently, several Congressional committees nearly terminated Outrider in the budget authorization and appropriation process for fiscal year 1998. The problems that UAV programs have experienced stem from a number of different, but interrelated, factors.

First, many people, including defense contractors, appear to underestimate the difficulty of building an unmanned aerial vehicle system. For example, the Army Chief of Staff, General Dennis Reimer, was apparently mystified at the trouble Outrider has experienced, saying "It's not laser brain surgery."⁽¹⁾ That may be true, but UAVs are still much more complex than the radio-controlled model airplanes they somewhat resemble. Building and operating a tactical UAV effectively involves integrating complex components into a small air frame. Those components include the engine, sensors, software, communications link, data link, avionics, ground control equipment, and so on. Making any one of those items work is not too difficult, but making all of them work together and still be much less expensive than a manned tactical aircraft is a major engineering challenge. One of the main problems afflicting the Outrider program is that the principal contractor drastically underestimated the technical challenges and level of effort required to produce a workable system that met both the cost requirement and most of the technical goals.

A second major problem among UAV programs is the tendency for "requirements creep." That is the phenomenon in which, once development has begun, the services impose more and more requirements on the UAV until meeting all of them becomes technically unworkable. In a broad overview of UAV programs, the General Accounting Office observed that requirements creep was a frequent occurrence. The Aquila UAV, for example, was supposed to have been a relatively simple, propeller-driven aircraft that could see over the next hill and relay imagery back to tactical commanders. But before its cancellation, the requirements had grown so much that it was expected "to fly by autopilot, carry sensors to locate and identify enemy point targets in day or night, use a laser to designate the targets for the Copperhead artillery projectile, provide conventional artillery adjustment, and survive against Soviet air defenses."⁽²⁾ Outrider is suffering some of those same problems today; immediately before the start of the ACTD, its technical goals were expanded to cover the requirements that Hunter had been supposed to fill before it was canceled.

A third problem is that until recently, unmanned aerial vehicles have been the orphans of DoD, in that they have not had strong backing from the services compared with other priorities. That has been particularly apparent when problems have arisen in UAV programs, as they inevitably do in almost any development program. For example, near the end of its development, Hunter suffered three crashes in 45 days. The program was apparently terminated in part because the Army had a funding shortage in other areas that it considered higher priorities. Therefore, it helped cancel Hunter in the expectation that some of the funds would be redirected to fill those gaps (although that does not appear to have happened). In contrast, Pioneer--a Navy program--had the strong personal support of Secretary of the Navy John Lehman. He pushed and protected the program and ensured that it received adequate funding. Until Predator, Pioneer was the only UAV in the past decade to complete its development cycle and be fully deployed.
 

HOW ADVANCED CONCEPT TECHNOLOGY DEMONSTRATIONS WORK

DoD launched the Advanced Concept Technology Demonstration program in 1994 as an acquisition reform initiative.⁽³⁾ The program has two main goals: to develop innovative military systems more quickly, and to involve the people who will actually use those systems on the battlefield in their development. The initiative is led by the Office of the Deputy Under Secretary of Defense for Advanced Technology (hereafter referred to as the Advanced Technology Office), which oversees several areas of emerging technology.

The ABCs of the ACTD Process

The ACTD concept occupies a unique niche among DoD's acquisition policies because of the close collaboration that it promotes between developers and operational users (or "warfighters") and because of its focus on mature technologies. Including operational users in the development process allows the military to assess the value of a new technology before acquiring it and develop new doctrine and tactics for that technology. By using the ACTD process, DoD hopes to reduce the time, and hopefully the cost, of producing a new weapon system and proving its military worth.

ACTDs also differ from traditional acquisition programs in schedule and scope. They are intended to be short (lasting no more than four years), small projects, costing anywhere from a few million dollars to a few hundred million dollars to complete. At their conclusion, ACTDs follow one of four paths: termination of the system, continued operation of only a few models, return of the system to the laboratory for continued development, or transition to procurement (see Figure 1).
 

FIGURE 1. POSSIBLE OUTCOMES FOR AN ACTD PROJECT

SOURCE: Congressional Budget Office based on information from Department of Defense, Defense Airborne Reconnaissance Office, UAV Annual Report: FY 1997 (November 6, 1997).

NOTE: ACTD = Advanced Concept Technology Demonstration.

The agency or service leading an ACTD is typically responsible for funding and executing the project, though the Advanced Technology Office occasionally provides some funds. Much of the money for the UAV programs has come from the Office of the Secretary of Defense through the Defense Airborne Reconnaissance Office. That funding arrangement reflects the interest that high-level DoD officials take in those systems. It also means that reviews of the UAV programs include the Office of the Secretary of Defense as well as the service or agency participants.

An ACTD also has a user/sponsor, which is typically one of the joint unified commands. The U.S. Atlantic Command has been the user/sponsor for the high-altitude and Predator UAVs. The user is responsible for providing the military forces for the demonstration exercises in an ACTD. Those exercises are crucial to the outcome of an Advanced Concept Technology Demonstration, since a favorable review by a user can send a system to acquisition.

An explicit goal of the ACTD process is the development of joint systems. That process encourages the services to cooperate in managing projects and actively involves joint commanders in selecting ACTDs. Involving multiple services lets DoD use development funds efficiently in an era of tight defense budgets and also aids the development of joint warfighting strategies and tactics. Joint development of UAVs, however, precedes the advent of the ACTD approach. The Congress ordered DoD to establish the Unmanned Aerial Vehicles Joint Program Office in 1988 to manage the services' development efforts for all UAVs. (The Global Hawk/Darkstar program, which was started much later, is an exception; it is being run by the Defense Advanced Research Projects Agency, or DARPA.)

DoD and the Congress had hoped that a joint approach would end redundant UAV development programs by the different services and improve management. The jointly managed programs have struggled to succeed, however, with Predator being the only fielded system that was developed by the Joint Program Office.

Recent government actions suggest that, at least temporarily, the era of joint management of UAV programs may be waning. Frustrated again with problems in those programs, the Congress slashed funding for Outrider by over half in fiscal year 1998 and gave control of what remained to the Army. It also ordered DoD to transfer most of the Joint Program Office's responsibilities for managing UAV programs back to the services. In another move that may give more authority to the services, DoD has disbanded the Defense Airborne Reconnaissance Office, which had been serving as a central office for managing all of DoD's airborne reconnaissance systems. Many of the same functions will now be performed by the Office of the Deputy Under Secretary of Defense for Command, Control, Communications and Intelligence.

ACTDs as a Form of Prototyping

Although the ACTD process represents a new approach, it is a variation on an existing acquisition strategy, known as prototyping. The uses and kinds of prototypes vary widely, but a study by RAND defined the prototype acquisition strategy as building test models with the intention of learning more about a technology and, should the technology prove useful, reducing the future risks of development.⁽⁴⁾ Essentially, an ACTD is a program that builds a technology demonstration prototype for users to evaluate. UAV prototypes are used for many of the same purposes as prototypes in other programs, such as to provide decisionmakers with better information for developing and acquiring systems or to be a hedge against uncertainty and risk in such development.

DoD has used prototypes to develop fighter aircraft, ships' combat systems, and missiles. How an acquisition program employs a prototype depends on the program's size and goals. DoD's interest in prototypes for the purpose of concept demonstration has varied over the years, but their use in ACTDs represents a new application.⁽⁵⁾ The dominant practice in the past has been to use highly integrated prototypes during the full-scale development of a system. In the ACTD program, the prototype is a means to a goal: the field demonstration.

Comparing examples of prototyping is difficult. Past studies have noted that the uses of prototypes vary so widely that virtually any comparison begs some caution. Besides variations in the prototyping strategy, changes imposed externally or internally to a system--such as changes in planned procurement quantity or performance goals--can affect the progress of development and the outcome of any system. Much of what ACTDs do has been tried in other programs. It is not so much that ACTDs have unique features, but that they are a blend of different methods for running a prototype program.

The unmanned aerial vehicle ACTDs share similar features with past prototyping programs, such as operational testing by service users and a flexible program-management philosophy. The prototype programs for the A-10 and F-16 aircraft, for example, both involved Air Force pilots early in the development phase of the aircraft.⁽⁶⁾ The F-16 prototype program emphasized minimal documentation and left many of the program's details, such as performance objectives, up to the contractors. ACTDs also employ other acquisition reforms, such as making cost an independent variable and using integrated process teams to design products.

Three key features, however, make the ACTD approach different from past prototyping programs. First, the realistic nature of the experimental demonstration and the close cooperation between users and developers are unique to the ACTD process. The testing in an ACTD does not focus on technical testing to validate a system's performance, but instead emphasizes operational testing and field exercises to gather information on a system's performance from a user's point of view.⁽⁷⁾ Second, the ACTD process allows the joint commander to be closely involved in developing a system, an opportunity that was not available in the past. Third, the degree of involvement by users is much greater than in previous prototype programs. The ACTDs, like other acquisition reform initiatives, seek to improve users' input into the development process and involve users in all aspects of the program.

Past studies have drawn no clear conclusions about whether prototyping helps meet cost, schedule, and performance goals. That should not be surprising, since the application of the strategy varies widely among acquisition programs, and a variety of external factors not directly related to prototyping frequently interfere in the management of a program.⁽⁸⁾ Still, many of the ways that DoD plans to use prototypes in ACTDs have been useful in the past, and it hopes that they will help field new systems faster and more cheaply.
 

APPLYING THE ACTD APPROACH TO UNMANNED AERIAL VEHICLES

Are ACTDs appropriate for developing unmanned aerial vehicles? By employing that development approach, the Department of Defense is trying to steer the UAV acquisition strategy toward systems that support joint warfare, are focused on users' requirements, avoid development complications stemming from excessive requirements, and can be developed quickly. By avoiding lengthy acquisition programs and involving operators in demonstrations and development, the ACTD process is a way to educate users about the merits of a system and introduce realism into their expectations of it. Problems in the UAV programs, however, raise questions about the level of technical risk they really entail.⁽⁹⁾

How Risky Are the UAV ACTDs?

For ACTDs to meet their ambitious cost and schedule goals, the technologies they use must be mature enough to be developed quickly into an operating prototype. UAV program managers recognized that when they laid out their program goals. For example, the program manager for Predator wanted a first flight in six months, and the Outrider program set the same goal. Predator met its goal, but Outrider did not come close, a reflection of the programs' different level of maturity before the ACTD. Despite the ambitious goals of the ACTD approach, the myriad technical problems experienced by the current development programs seem all too similar to the problems of failed UAV programs in the past.

Until recently, the staff of the Advanced Technology Office did not adopt a set definition of technical maturity. They argued that case-by-case evaluation by a review board was more effective than a strict definition.⁽¹⁰⁾ Perhaps seeking to reduce some of the controversy that the ACTD process has attracted, the Advanced Technology Office finally provided a definition in the fall of 1997: any technology that is not in DoD's Basic or Applied Research Program or that has been demonstrated before the start of the utility assessment is considered mature. (The office grants exceptions if a technology that fails that test is not in a critical component of the system being developed.)

Although the requirement that ACTDs use only mature technologies is intended to minimize technical risk, critics could argue that this focus underestimates the risks of integrating new technologies into a working system. The highly capable UAVs desired by the services require using many leading technologies, such as advanced radar, composite materials, and stealth design features. In addition, the risks of integrating smaller subcomponents into a new system can be as great as developing a new technology. UAVs are a web of complex subsystems, and integrating the various payloads, software, air vehicles, and ground stations into a single system requires substantial expertise on the part of developers. The delays in the Outrider and Global Hawk/Darkstar programs can be traced in large part to such problems of integration.

In the context of ACTDs, the Advanced Technology Office views technical risk and integration risk as separate issues. That distinction is critical. Whereas the use of mature technologies is required for an ACTD, the acceptable level of integration risk is more flexible. According to the Advanced Technology Office, integration risks are addressed in the assessment of the overall risks of a program. But solving the integration problems of the Outrider and Global Hawk/Darkstar systems has proved more costly and taken longer than the Joint Program Office and DARPA had originally hoped.

Although most of the technologies that UAV programs use may meet the Advanced Technology Office's standard for a mature technology, each program's technical approach to developing a demonstration system is different. The two troubled programs, Global Hawk/Darkstar and Outrider, are new systems, with no predecessor to give their developers experience. In contrast, Predator was based on another system that was already in use, Gnat-750. The Predator program involved some amount of development, but Gnat-750 gave developers a working system on which to base their engineering efforts. Perhaps more important, by selecting a proposal from the same contractor that developed Gnat-750 (General Atomics), DoD was working with a firm that had experience with UAV systems. Thus, although the smooth progress of Predator from development to deployment seems impressive, it probably reflects the maturity of the system that preceded it.

In developing the tactical and high-altitude endurance UAVs, DoD selected models that had no operational predecessors. Outrider was based on a paper system originally designed to meet a different requirement, and Global Hawk and Darkstar were entirely new designs. Difficulties with the design of the air vehicle and subcomponent integration have delayed the Outrider program by at least four months. Global Hawk has been delayed by about a year by software and development problems. Darkstar's flight-testing was postponed for over two years while designers improved the system. Those delays come as a disappointment to people who expected ACTDs to focus on demonstrating proven technologies.

It is not unusual for acquisition programs to encounter the sort of problems that Outrider and Global Hawk/Darkstar have run into. The experiences of those ACTDs suggests that, although the ACTD approach may have some practices that can improve the development process, many of the factors that cause costs to increase and schedules to slip may be beyond the scope of the reforms that DoD is pursuing as a part of the ACTD approach.

Is the ACTD Process Speeding the Delivery of UAV Systems?

The intent of all three UAV ACTDs was to put those systems in the hands of users as quickly as possible. Past UAV programs have languished in development, and few systems have ever made it to users. The amount of time spent in development is critical, because the Advanced Technology Office tries to limit ACTDs to only four years. Moreover, a protracted development schedule dilutes an ACTD's focus on operational evaluation. Because of differences in the maturity of the UAV systems, DoD has had mixed success in speeding deployment to the field. Predator moved relatively quickly to the field, but Outrider, Global Hawk, and Darkstar were soon overwhelmed by technical problems, delaying early demonstrations for their users.

Developing Criteria for Operational Assessment

In an ACTD, the lead service develops a draft concept of operation for a weapon system and the user/sponsor develops standards for evaluating that system in the field. The concept of operation is continuously refined as the ACTD progresses, reflecting changes as the system matures and as users refine their approach to operating it. Once a system is ready, users operate it in field exercises to assess its utility. That demonstration of military utility is at the heart of what makes an ACTD unique.

How much time is necessary for evaluating a system? That depends on how users plan to employ it. Predator participated in nine exercises over fifteen months to demonstrate aspects of the system's capabilities.⁽¹¹⁾ But in the Outrider and Global Hawk/Darkstar programs, the schedule for operational assessment has been shortened by the extra time required for development. Outrider completed a shortened 2½-month military-utility assessment. The high-altitude endurance program has reduced its scheduled utility assessment from about two years to one year.

Supporting the Joint Development of UAVs

The ACTD process has encouraged the joint development of UAV systems, but building such joint systems has proved more difficult than DoD expected. The Hunter and Medium Range UAVs were joint programs (although not ACTDs); both were hobbled by technical problems and management difficulties and were canceled before reaching full-rate procurement. For example, the payload that the Air Force developed for the Medium Range UAV would not fit into the air vehicle built by the Navy. That problem illustrated the importance of creating a means for effective project management among the many parties involved in developing a UAV. The ACTD framework provides just such a means for the current UAV programs and appears to be successful in coordinating the disparate demands of users, sponsors, and the services. But joint development can still complicate the task of creating a UAV system.

Developers can generally follow one of two approaches for joint systems. Either they can build a system that is operated by a single service but is interoperable with the data-dissemination systems of other services, or they can design a common system to be operated by two or more services. Experience with joint development suggests that the second option is the more challenging. Because the services have varying requirements for UAVs, both approaches have their advantages and disadvantages.

In an interoperable UAV system, one service takes charge of development, but other services make sure that their reconnaissance systems and computer networks can receive the UAV's data or imagery. Involving more than one service in a program allows different users to determine whether the system meets their requirements. For example, both the Navy and the Army participated in developing Predator, which gave them an understanding of the system's capabilities and allowed them to evaluate it for their own needs. Developers made sure that Predator's sensors would perform well enough to meet operational needs and that its imagery could be distributed by joint dissemination systems. Similarly, the Global Hawk/Darkstar program is designing its data-dissemination system to be interoperable with those of other services and joint commanders. The Air Force is the lead service on that program, but the Army and Navy are participating as well.

Interoperability has some drawbacks, however. When the demand for UAVs outstrips their availability--as has often been the case--the needs of tactical commanders may be sacrificed to those of higher echelons. That would probably not happen if the tactical commanders had their own, exclusive UAV systems. Another problem with interoperability is that one service can get stuck with the cost of providing UAV support to other services' commanders. If that leads the bill-paying service to monopolize the system's use or somehow diminish its availability, other commanders might have an incentive to demand their own systems or to forgo using UAVs altogether.

The other alternative, building a single UAV system for operation by several services, has the advantage of preventing redundant development efforts and giving each service's commanders their own system. However, to the extent that the services' requirements for a UAV differ, they must be willing to compromise to build a common system. If those requirements are too different, they may be unfulfilled or the system itself may be unworkable.

Problems with Outrider show that developing one system for many services can sometimes increase the technical challenges. That program is led by the Army, with the Navy and Marine Corps also participating. Outrider is in fact the only ACTD unmanned aerial vehicle that has pursued compatibility with naval operations as a system goal. Originally, the developers hoped to build one system for all of those services' tactical commanders to evaluate. However, technical delays have reduced the UAV's ability to meet the needs of some commanders and have postponed the delivery of demonstration systems to the field. For example, the technical goals for Outrider originally called for it to use a diesel engine to make it more suitable for shipboard operations. They also called for it to be able to land and take off from the Navy's large-deck amphibious ships. The diesel engine proved infeasible, however, and had to be dropped from the program. Those problems--and the decision not to test Outrider at sea--raise questions about how committed the Navy remains to the program.
 

THE EFFECTS OF THE ACTD APPROACH ON THE COST AND CAPABILITY OF UAVs

Controlling the costs of acquiring and operating UAVs is a major concern for the agencies and services developing them. In the past, the services were unwilling to continue UAV programs when their costs grew beyond original estimates. (As a result, tactical commanders operating overseas now have only a few aging Pioneer systems to use, since the Aquila and Hunter programs were canceled.) In addition, some analysts have noted, the services were too ambitious in the capabilities that they demanded of past programs.

Balancing Affordability with Capability

Proponents believe that the ACTD approach makes development less expensive by reducing the time it takes. For example, the UAVs being developed as ACTDs forgo extensive flight testing and logistics analysis that normal acquisition programs carry out. During development, the lead service will conduct a logistics and maintenance analysis before assuming responsibility for the system, but that analysis is not supposed to be as significant an effort as would be carried out in a conventional acquisition program. That approach saves money during development but shifts the costs of a complete analysis to the post-ACTD period. If a system moves to acquisition, someone will have to go back and make those investments--as the Air Force is doing now with Predator.

Setting Flexible Goals for Capability

By not setting strict performance goals for UAV systems, and by allowing users to evaluate the systems early in the development cycle, ACTDs hope to avoid the requirements creep that has plagued other UAV programs. The short length of ACTDs and the policy of establishing early agreement about a system's concepts of operation limit opportunities for adding new requirements to a program. Program managers for Predator believe the ACTD approach helped keep the requirements for that UAV simple. The services have avoided adding other requirements to the Global Hawk, Darkstar, and Outrider programs during the ACTD, even though additional missions (beyond those the ACTDs will demonstrate) have been studied.

Moving Demonstrations to Acquisition

Because ACTDs are intended to be development efforts, which could have any one of a number of outcomes, they do not focus on many of the concerns that the acquisition process typically requires. For example, the acquisition process considers the production needs of the system and addresses its reliability and maintenance costs, whereas ACTDs generally do not. The UAV ACTDs are an exception, however, because the services are hoping to procure those systems. As a result, the developers need to worry about the transition process--moving the system that emerges from the ACTD to acquisition with minimal impact on the program's cost and schedule. The Global Hawk/Darkstar program is studying the costs of supporting and operating those systems as a part of the ACTD.

Problems with Predator's transition revealed weaknesses in the original ACTD's plans for preparing the system for acquisition. The Predator program office had achieved many of its demonstration goals, but several steps necessary for acquisition had not been required in the demonstration phase. For example, the Air Force had no technical manuals and insufficient data about Predator's reliability when it inherited responsibility for the system. Those had to be developed later, thus interrupting the program's progress.

Because of the difficulties with Predator's transition, the Advanced Technology Office has developed a more formal management approach to transition issues in ACTDs. The office now requires that an ACTD initially specify a lead service, which is responsible for operating and supporting the system after the demonstration is completed. In addition, managers in an ACTD now establish a special team to coordinate the transition. That increased emphasis on support and acquisition issues in an ACTD is not intended to imply a commitment to acquisition. Rather, by including concerns such as the life-cycle costs of a system in the ACTD, it is meant to keep developers from designing a system that costs too much to support once the demonstration is over.

A critical transition issue now confronting the Global Hawk/Darkstar program is how to support the manufacturing facilities. Cost overruns have forced DARPA to cut the number of air vehicles in that program, so a two-year gap now exists between the last planned air vehicle delivery (in 1999) and the first year any production deliveries could begin. According to the developer, Teledyne Ryan, that two-year gap will cause the loss of production workers and force the company to close the manufacturing facilities for Global Hawk. As a result, any production efforts that follow would have to pick up the additional costs of rebuilding the manufacturing base.⁽¹²⁾ Similar transition issues are confronting other ACTDs, and the Advanced Technology Office is trying to provide them with funding.

Conclusions

Advanced Concept Technology Demonstrations have encouraged a different approach to developing unmanned aerial vehicles. Most of the advantages and disadvantages of that approach revolve around the merits of trying to develop new systems rapidly for demonstration. Demonstration is useful for learning about a system's combat effectiveness, but the progress of the UAVs shows that the ACTD process is prone to the same cost and schedule problems that can occur in the traditional acquisition cycle. Aspects of the ACTD process have prevented some of the pitfalls of past UAV programs. But the extensive development efforts those UAVs have required leave questions about the suitability of using that process to develop unmanned aerial vehicles.

1. ". . . Unless Pilotless," Washington Outlook section, Aviation Week and Space Technology, November 10, 1997, p. 31.

2. Statement of Louis J. Rodrigues, Director, Defense Acquisition Issues, National Security and International Affairs Division, General Accounting Office, before the Subcommittees on Military Research and Development and Military Procurement, House Committee on National Security, published as General Accounting Office, Unmanned Aerial Vehicles: DOD's Acquisition Efforts, GAO/T-NSIAD-97-138 (April 9, 1997), p. 2.

3. See Congressional Budget Office, The Department of Defense's Advanced Concept Technology Demonstrations, CBO Memorandum (September 1998).

4. Jeff Drezner, The Nature and Role of Prototyping in Weapon System Development, R-4161-ACQ (Santa Monica, Calif.: RAND, 1992), p. 9.

5. Ibid., p. 56.

6. See Giles Smith and others, The Use of Prototypes in Weapon System Development, R-2345-AF (Santa Monica, Calif.: RAND, 1981).

7. Because of the complexities of developing UAV systems, the UAV programs have involved the test and evaluation commands of the services and the Office of the Secretary of Defense.

8. Drezner, The Nature and Role of Prototyping in Weapon System Development, p. 67.

9. For a discussion of the technical problems with Darkstar, for example, see Department of Defense, Office of the Inspector General, Audit Report on Advanced Concept Technology Demonstration, Report No. 97-120 (April 7, 1997). For a discussion of technical issues in the development of Outrider, see General Accounting Office, Unmanned Aerial Vehicles: Outrider Demonstrations Will Be Inadequate to Justify Further Production, NSIAD-97-153 (September 1997).

10. That review is done by the "Breakfast Club," a group of senior officials from the Advanced Technology Office, the Office of the Director for Defense Research and Engineering, and the Deputy Assistant Secretary of Defense for Command, Control, Communications, and Intelligence. The club also includes representatives from the Office of the Joint Chiefs of Staff, the Defense Advanced Research Projects Agency, the Ballistic Missile Defense Organization, and the science and technology and operational requirements offices of each military service.

11. Michael Thirtle, Robert Johnson, and John Birkler, The Predator ACTD: A Case Study for Transition Planning to the Formal Acquisition Process, MR-899-OSD (Santa Monica, Calif.: RAND, 1997), p. 37.

12. Michael A. Dornheim, "Global Hawk Begins Flight Test Program," Aviation Week & Space Technology, March 9, 1998, pp. 22-23.

---