FAS | Military Analysis | GAO |||| Index | Search |


Electronic Combat: Consolidation Master Plan Does Not Appear To Be Cost-Effective (Letter Report, 07/10/97, GAO/NSIAD-97-10).

Pursuant to a congressional request, GAO reviewed the Department of
Defense's (DOD) Electronic Combat Consolidation Master Plan, focusing on
the cost and benefits of DOD'S consolidation plans for open air ranges,
hardware-in-the-loop facilities, and installed system test facilities
used in electronic combat testing.

GAO noted that: (1) implementation of the Electronic Combat
Consolidation Master Plan will result in less effective electronic
combat testing capabilities; (2) the planned relocation of the
Electro-Magnetic Test Environment will eliminate DOD's current
capability to test electronic combat systems in conditions that typify
many potential threat locations; (3) DOD will be left with two open air
ranges with very similar environmental characteristics and will no
longer have the ability to test in diverse conditions needed to
understand environmental effects on electronic combat systems; (4) the
planned Real-time Electronic Digitally Controlled Analyzer Processor
relocation will mean replacing existing hardware simulation capability
with digital computer models, thus reducing DOD'S current capability to
simulate realistic aircraft strike scenarios with high confidence and
fidelity; (5) the Master Plan did not contain any cost analysis and did
not identify any savings expected from the consolidations; (6) estimates
used to support 1995 Base Closure and Realignment Commission
deliberations, as well as data provided by users indicate that the
consolidation may increase DOD's electronic combat testing costs; (7) in
addition, the Master Plan does not contain any analysis or
recommendations regarding consolidation of installed system test
facility workloads across the services although the Navy and the Air
Force are spending $512 million for construction of another anechoic
chamber to provide a controlled electromagnetic environment, and other
upgrades to their current primary installed system test facilities; (8)
consequently, the Master Plan, if implemented, may not achieve the most
cost-effective DOD-wide infrastructure; (9) the root cause of this was
DOD officials' inability to overcome service parochialism during the
Master Plan's development; (10) this parochialism resulted in a
gentleman's agreement between the Air Force and the Navy to focus on
intraservice rather than interservice consolidations; (11) prior joint
service studies performed on an interservice basis had identified
alternatives for more cost-effective consolidations; (12) however, the
recommendations of these studies were never implemented; and (13) if
this continues, service rivalry could adversely affect DOD's ongoing,
congressionally mandated Section 277/Vision 21 consolidation effort, wh*

--------------------------- Indexing Terms -----------------------------

 REPORTNUM:  NSIAD-97-10
     TITLE:  Electronic Combat: Consolidation Master Plan Does Not 
             Appear To Be Cost-Effective
      DATE:  07/10/97
   SUBJECT:  Test facilities
             Electronic warfare
             Military downsizing
             Military cost control
             Combat readiness
             Advanced weapons systems
             Computer modeling
             Weapons research
             Base closures
             Military aircraft
IDENTIFIER:  Air Force Electronic Warfare Evaluation Simulator
             Air Force Real-Time Electromagnetic Digitally Controlled 
             Analyzer and Processor
             Air Force Electro-Magnetic Test Environment
             DOD Electronic Combat Consolidation Master Plan
             P-3 Aircraft
             C-5 Aircraft
             F-22 Aircraft
             E-6 Aircraft
             Orion Aircraft
             Galaxy Aircraft
             DOD Vision 21 Plan
             
******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO report.  Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved.  Major          **
** divisions and subdivisions of the text, such as Chapters,    **
** Sections, and Appendixes, are identified by double and       **
** single lines.  The numbers on the right end of these lines   **
** indicate the position of each of the subsections in the      **
** document outline.  These numbers do NOT correspond with the  **
** page numbers of the printed product.                         **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
** A printed copy of this report may be obtained from the GAO   **
** Document Distribution Center.  For further details, please   **
** send an e-mail message to:                                   **
**                                                              **
**                    <[email protected]>                        **
**                                                              **
** with the message 'info' in the body.                         **
******************************************************************


Cover
================================================================ COVER


Report to Congressional Requesters

July 1997

ELECTRONIC COMBAT - CONSOLIDATION
MASTER PLAN DOES NOT APPEAR TO BE
COST-EFFECTIVE

GAO/NSIAD-97-10

Electronic Combat Test Consolidation

(707149)


Abbreviations
=============================================================== ABBREV

  AFEWES - Air Force Electronic Warfare Evaluation Simulator
  BRAC - Base Closure and Realignment Commission
  DOD - Department of Defense
  EMTE - Electro-Magnetic Test Environment
  REDCAP - Real-time Electronic Digitally Controlled Analyzer
     Processor

Letter
=============================================================== LETTER


B-272629

July 10, 1997

The Honorable Connie Mack
The Honorable Bob Graham
United States Senate

The Honorable Joe Scarborough
House of Representatives

In response to your request, we have reviewed the Department of
Defense's (DOD) Electronic Combat Consolidation Master Plan.  As
agreed with your office, our objective was to assess the costs and
benefits of DOD's consolidation plans for open air ranges,
hardware-in-the-loop facilities, and installed system test facilities
used in electronic combat testing. 


   BACKGROUND
------------------------------------------------------------ Letter :1

In its report on the Fiscal Year 1996 National Defense Authorization
Act, the Senate Armed Services Committee criticized DOD for not
having a clear approach to consolidating test infrastructure and
recommended reductions in DOD's Test and Evaluation support accounts. 
The Senate Appropriations Committee agreed with the authorizing
committee, recommended reductions to the fiscal year 1996 Test and
Evaluation support accounts, and acknowledged the need to constrain
spending in this area.  Subsequently, in the Fiscal Year 1996
National Defense Appropriations Act, the Congress limited the
obligation of specified funds until DOD provided the defense
authorizing and appropriating committees with an Electronic Combat
Consolidation Master Plan to establish a DOD-wide infrastructure for
electronic combat testing.  In March 1996, DOD published its Master
Plan. 

In transmitting the Master Plan to the Congress, the Under Secretary
of Defense for Acquisition and Technology stated that DOD would
revisit the Plan in the broader context of section 277 of the
National Defense Authorization Act for Fiscal Year 1996, and adjust
the Plan as appropriate.  Section 277 directs DOD to develop a
consolidation and restructure plan for its laboratories and test and
evaluation centers for the 21st century.  This effort is not yet
complete. 

According to the Master Plan, DOD considered 17 of the services'
electronic combat test facilities for consolidation.  The Army
controls 4 of the
17 facilities, the Navy controls 6, and the Air Force controls 7. 
The conclusion of the Master Plan is that the assets of three of the
seven facilities managed by the Air Force will be moved to other Air
Force locations.  No interservice consolidations and no intraservice
consolidation of the four Army or six Navy facilities are proposed in
the Plan.  The three facilities to be relocated are

  -- the Air Force Electronic Warfare Evaluation Simulator (AFEWES)
     in Fort Worth, Texas;

  -- the Real-time Electronic Digitally Controlled Analyzer Processor
     (REDCAP) in Buffalo, New York; and

  -- the Electro-Magnetic Test Environment (EMTE) at Eglin Air Force
     Base, Florida. 

AFEWES is a specialized hardware-in-the-loop facility that simulates
individual radar and missile threats to aircraft and electronic
combat hardware.  REDCAP is a specialized hardware-in-the-loop
facility that simulates an integrated air defense system with
command, control, and communications networks.  EMTE is an open air
range providing radar and simulated missile threats to aircraft in
flight; it is collocated at Eglin Air Force Base with the Air Force's
development and test and evaluation activities for armaments. 
Installed system test facility consolidation was not proposed in the
Master Plan.  For purposes of this review, we focused on three open
air ranges, two hardware-in-the-loop facilities, and two installed
system test facilities.  The remaining 10 are other kinds of
electronic combat test facilities, such as research laboratories or
radar cross-section measurement facilities or are service unique
capabilities.  DOD's electronic combat test process and the role the
various kinds of facilities play in that process are explained
briefly in appendix I. 


   RESULTS IN BRIEF
------------------------------------------------------------ Letter :2

Implementation of the Electronic Combat Consolidation Master Plan
will result in less effective electronic combat testing capabilities. 

  -- The planned relocation of EMTE will eliminate DOD's current
     capability to test electronic combat systems in conditions that
     typify many potential threat locations.  DOD will be left with
     two open air ranges with very similar environmental
     characteristics and will no longer have the ability to test in
     diverse conditions needed to understand environmental effects on
     electronic combat systems. 

  -- The planned REDCAP relocation will mean replacing existing
     hardware simulation capability with digital computer models,
     thus reducing DOD's current capability to simulate realistic
     aircraft strike scenarios with high confidence and fidelity. 

The Master Plan did not contain any cost analysis and did not
identify any savings expected from the consolidations.  Estimates
used to support 1995 Base Closure and Realignment Commission (BRAC)
deliberations, as well as data provided by users indicate that the
consolidation may increase DOD's electronic combat testing costs.  In
addition, the Master Plan does not contain any analysis or
recommendations regarding consolidation of installed system test
facility workloads across the services although the Navy and the Air
Force are spending $512 million for construction of another anechoic
chamber to provide a controlled electromagnetic environment at
Patuxent River, Maryland, and other upgrades to their current primary
installed system test facilities at Patuxent River and Edwards Air
Force Base, California. 

Consequently, the Master Plan, if implemented, may not achieve the
most cost-effective DOD-wide infrastructure.  The root cause of this
was DOD officials' inability to overcome service parochialism during
the Master Plan's development.  This parochialism resulted in a
"gentlemen's agreement" between the Air Force and the Navy to focus
on intraservice rather than interservice consolidations.  Prior joint
service studies performed on an interservice basis had identified
alternatives for more cost-effective consolidations.  However, the
recommendations of these studies were never implemented.  If this
continues, service rivalry could adversely affect DOD's ongoing,
congressionally mandated
section 277/vision 21 consolidation effort, which is considering the
broader issue of DOD's testing and laboratory facilities. 


   PRINCIPAL FINDINGS
------------------------------------------------------------ Letter :3


      PLANNED CONSOLIDATION OF
      OPEN AIR RANGES WILL REDUCE
      EFFECTIVENESS
---------------------------------------------------------- Letter :3.1

The proposal in the Master Plan to relocate EMTE would eliminate a
test facility that provides unique advantages and keep two testing
facilities with overlapping capabilities.  DOD's acquisition
regulations require systems to be evaluated in operationally
realistic environments, including the expected range of natural
environmental conditions.  Currently, its electronic combat open air
ranges replicate diverse threat environments where the services must
be prepared to conduct operations. 


      TESTING EQUIPMENT IN DIVERSE
      ENVIRONMENTS IS CRITICAL
---------------------------------------------------------- Letter :3.2

DOD's 5000.2R acquisition regulations require testing in natural
environmental conditions representative of intended areas of
operations (e.g.  temperature, pressure, humidity, fog,
precipitation, clouds, blowing dust and sand, steep terrain, storm
surge and tides, etc.).  Testing in diverse conditions provides
performance data needed to understand environmental effects on
electronic combat systems.  This information is critical to making
informed acquisition and mission planning decisions, thereby reducing
the risk of buying ineffective equipment and the potential for
casualties during wartime. 

DOD studies also document the importance of testing electronic combat
equipment in diverse environments.  For example, a 1994 joint service
study of electronic combat open air ranges expressed the need for
electronic combat testing in the correct natural environment.  Test
results for electronic combat systems demonstrate that performance
can differ significantly in differing environments. 

Testing in diverse environments is also important for collecting data
to support development of realistic computer models.  DOD believes
modeling and simulation can be used to reduce the cost of live tests,
but to improve levels of confidence in models they must be built on
high fidelity data collected from diverse environments. 


      PLAN WOULD ELIMINATE
      DIVERSITY FOUND IN CURRENT
      OPEN AIR RANGES
---------------------------------------------------------- Letter :3.3

DOD's proposed open air range consolidation as described in the
Master Plan would eliminate diversity by keeping only desert ranges
and thereby reduce electronic combat open air range testing
effectiveness.  The Air Force and the Navy control three primary open
air ranges for testing electronic combat systems.  These include two
western ranges, one at China Lake, California, and one managed by
Edwards Air Force Base, California.  Both feature dry, desert
climates with steep, rocky terrain.  The third range, EMTE, at Eglin
Air Force Base on the Florida panhandle, features a land/sea
interface, high humidity, and a subtropical, forested environment,
and an over water test range. 

The Master Plan states that preservation of militarily unique
electronic combat test facilities was an important criterion for
deciding which facilities to close.  However, EMTE is unique among
DOD's open air ranges, and the 1994 joint service study noted that
one of the primary disadvantages of closing EMTE would be the loss of
terrain and geographical diversity, since both remaining ranges would
be located in the desert. 


      CURRENT OPEN AIR RANGES
      REPRESENT POTENTIAL THREAT
      ENVIRONMENTS
---------------------------------------------------------- Letter :3.4

Both western ranges provide a capability for conducting essential
electronic combat testing over terrain representative of projected
middle eastern threat environments.  Conversely, EMTE provides DOD
with an environment more typical of most of the other projected U.S. 
threat locations, including North Korea and the Balkans.  Table 1
identifies the terrain of countries that are representative of
possible locations for future conflicts that are of concern to the
United States.  In comparison, table 2 demonstrates that the unique
environmental characteristics of EMTE--over water, land/sea
interface, and foliage--are prevalent in most of the potential threat
locations identified in table 1. 



                                Table 1
                
                 Potential Threat Locations and Terrain
                              Correlation

                                Sea/
                                land
                      Over      interfac
Location              water     e         Desert    Foliage   Mountain
--------------------  --------  --------  --------  --------  --------
Iraq                                      X                   X

Iran                  X         X         X         X         X

N. Korea              X         X                   X         X

China                 X         X         X         X         X

Libya                 X         X                   X         X

Cuba                  X         X                   X         X

Balkans               X         X                   X         X
----------------------------------------------------------------------


                                Table 2
                
                Open Air Ranges and Terrain Correlation

                                Sea/
                                land
                      Over      interfac
Location              water     e         Desert    Foliage   Mountain
--------------------  --------  --------  --------  --------  --------
EMTE                  X         X                   X

China Lake                                X                   X

Air Force Western                         X                   X
Test Range
----------------------------------------------------------------------

   REDCAP AT NEW LOCATION WILL BE
   LESS CAPABLE
------------------------------------------------------------ Letter :4

The Master Plan proposal to move the REDCAP facility from Buffalo and
colocate it with the Air Force's installed system test facility at
Edwards Air Force Base will reduce electronic combat testing
effectiveness.  The intent is to reestablish what the Air Force calls
a "core" REDCAP capability at the new location by developing a
computer model to simulate REDCAP hardware.  However, the model will
not simulate all of the current REDCAP testing features. 

Establishing a core REDCAP capability means not utilizing much of the
REDCAP hardware, and its associated functions, even though the Air
Force completed upgrading this hardware in 1996 at a cost of $75
million over the past 8 years.  The core REDCAP at the proposed new
location will be less capable than the complete REDCAP at its current
location. 

Some of the REDCAP hardware functions that the Air Force does not
plan to make available in core REDCAP do not exist anywhere else in
DOD.  According to DOD and Air Force officials, the REDCAP facility
in Buffalo is unique.  For instance, REDCAP can currently simulate a
realistic scenario of a strike package of multiple aircraft
approaching targets protected by multiple threat radars and threat
aircraft incorporated into an integrated air defense system.  The
proposed core REDCAP will not be able to simulate this scenario. 
Simulating many aircraft versus many threat systems is important
because integrated air defense systems exist in a number of potential
threat locations and integrated defenses are projected by DOD to be a
growth area among potential threat nations. 


   PLANNED CONSOLIDATIONS MAY
   INCREASE COSTS
------------------------------------------------------------ Letter :5

The Master Plan did not contain any cost analysis or identify the
savings expected from the consolidations.  Our analysis of prior
estimates used to support the 1995 BRAC deliberations and other data
provided by users indicates the consolidations may increase DOD's
testing costs.  More specifically (1) BRAC-related data indicates
that a complete EMTE relocation would not be cost- effective, (2)
cost estimates provided to BRAC regarding the relocation of REDCAP
and AFEWES were understated, and (3) increased costs that will be
incurred by user organizations were not considered in Air Force cost
estimates. 


      MASTER PLAN INCLUDES NO
      EVIDENCE OF SAVINGS
---------------------------------------------------------- Letter :5.1

Senior Air Force test officials told us that the Air Force selected
EMTE, REDCAP, and AFEWES for consolidation because they believed they
would ultimately save money by relocating them.  The Electronic
Combat Consolidation Master Plan, however, includes no evidence that
any savings will result and, in fact, contains no cost data at all. 

The Secretary of Defense recommended the relocation of REDCAP and
AFEWES and the partial relocation of EMTE to the 1995 BRAC.  BRAC
approved the REDCAP relocation, rejected the AFEWES proposal, and
significantly scaled back the partial relocation of EMTE.  The Master
Plan, however, incorrectly states that selecting EMTE for relocation
reflects decisions of the 1995 BRAC. 


      BRAC FOUND NO SAVINGS IN
      RELOCATING EMTE IN TOTAL
---------------------------------------------------------- Letter :5.2

The 1995 BRAC scaled back the Secretary's recommendation to realign
the EMTE open air range at Eglin Air Force Base.  DOD proposed
transferring
17 systems designed to simulate various threat radars and missiles,
but BRAC determined that was too costly and would "never net a return
on investment." Ultimately, however, BRAC did approve the movement of
10 systems (for which the BRAC account was eventually charged $6.1
million), but required DOD to leave limited capability systems at
Eglin to support the Air Force's Special Operations Forces, Armaments
Division, and Air Warfare Center, which are also at Eglin. 
Nevertheless, the 1996 Master Plan says the Air Force plans to
"relocate" EMTE, not move just 10 systems. 

According to Air Force officials, "relocate" means 17 systems will be
moved.  Ten will be operated at the new location and 7 will be
cannibalized for parts.  Air Force test officials maintain that the
Special Operations Forces, Air Warfare Center and Armaments Division
do not need these
17 systems at Eglin, and they will leave behind some systems to meet
the customers' needs.  EMTE users, such as the Special Operations
Forces and the 53rd Test Wing and the Army Aviation Test Directorate,
told us that the systems the Air Force plans to leave will not meet
their needs for accomplishing realistic testing because they do not
have the capability to receive and process testing data for
subsequent analysis.  Air Force test officials told us users can
travel to the Air Force's western test range to meet their test
requirements. 


      REDCAP RELOCATION COSTS NOT
      FULLY DISCLOSED
---------------------------------------------------------- Letter :5.3

To mitigate the impact of the reduction in REDCAP effectiveness
described earlier in this report, the Air Force has awarded a
$6.2-million contract to design and build a digital computer model of
REDCAP that it intends to use instead of the REDCAP hardware that
will be stored.  This additional cost, however, was not included in
the Air Force cost estimate that BRAC used in deciding to relocate
REDCAP. 

The Air Force had recommended to the 1995 BRAC that the REDCAP
facility be relocated to Edwards Air Force Base.  The 1995 BRAC found
that Air Force cost estimates to relocate were understated, but
decided to accept the recommendation as they believed it would still
result in overall savings.  As a result, the BRAC account makes
available to the Air Force $3.7 million to relocate REDCAP.  Using
Air Force cost figures, BRAC projected the operating cost to the
government of REDCAP at the new location will be $100,000 compared to
$1 million annually at the current location, BRAC anticipated a
4-year return on investment (4 x $0.9 million).  (The remainder of
REDCAP's operations are funded by customer receipts.)

Since the cost of the new computer model was not taken into account,
the Air Force will not achieve a relatively quick return on
investment.  The additional $6.2 million means it will take an
additional 7 years to recoup costs based on Air Force projected
savings of $0.9 million per year.  This 11-year (4 + 7) return is
well beyond the 1995 BRAC norm of seeking a 6-year or less return on
investment. 


      AFEWES MOVE DELAYED
---------------------------------------------------------- Letter :5.4

The Air Force recommended to the 1995 BRAC that the AFEWES facility
in Fort Worth be relocated to Edwards Air Force Base.  The Air Force
had estimated a cost of $8.9 million to close AFEWES and move it. 
BRAC did not accept the recommendation though because BRAC estimated
it would cost $34.9 million to close the facility and would be over
100 years before a return on investment was realized.  Nevertheless,
the Air Force included the AFEWES relocation in the 1996 Master Plan. 
Air Force officials told us they are now attempting to modify their
outyear budgets so they can move the AFEWES facility sometime in the
year 2000 time frame. 


      USER COSTS WILL INCREASE
      WITH EMTE CLOSURE
---------------------------------------------------------- Letter :5.5

Special Operations Forces based at Hurlburt Field, Florida, adjacent
to Eglin Air Force Base, are users of EMTE.  After the EMTE
relocation, however, Special Operations Forces' electronic combat
testing will be conducted at the Air Force's western test range.  As
a result, Special Operations Forces officials estimate that their
electronic combat testing will cost $23 million over the next 5
years, whereas they have spent only $4 million for electronic combat
testing over the last 4 years. 

We reviewed the analysis supporting this estimate and found it to be
realistic.  The $19 million in additional cost results from sending
aircraft, their crews, and support personnel temporarily to the
western test range more often than in the past.  In contrast, there
are no temporary duty costs associated with testing Special
Operations Forces aircraft at EMTE. 

In addition to the Special Operations Forces, another user
organization based at Eglin, the 53rd Test Wing, estimates that the
proposed EMTE relocation may cost them as much as an additional $1
million per year.  This additional cost would provide for an
estimated 20 additional trips to the Air Force's western test range
to perform electronic combat testing that in the past has been
performed at Eglin Air Force Base. 


   INSTALLED SYSTEM TEST FACILITY
   CONSOLIDATION NOT PRACTICAL
------------------------------------------------------------ Letter :6

DOD's Master Plan does not contain any analysis or recommendations
regarding consolidation of installed system test facility workloads
across the services.  The Navy and the Air Force are spending $512
million for construction of a new anechoic chamber to provide a
controlled electromagnetic environment at Patuxent River, Maryland,
and other upgrades to their current primary installed system test
facilities at Patuxent River and Edwards Air Force Base, California. 
These projects have progressed too far to make any interservice
consolidation practical at this time, however. 

The Navy has a fighter-sized anechoic chamber, has already spent $227
million, and has plans to spend an additional $101 million, to (1)
add a new, medium- sized anechoic chamber and (2) upgrade the
electronic combat test laboratory shared by both the fighter and
medium-sized chambers.  The Navy is planning to have the medium-sized
chamber completed in fiscal year 1999. 

Completion of this work is timed to conduct testing on the Navy's E-6
and P-3 aircraft.  (These specialized aircraft are too large to fit
into the fighter sized facility.) Meanwhile, the Air Force has plans
to spend over $184 million through fiscal year 2002 to make the same
electronic combat test upgrades to its Edwards Air Force Base
installed system test facility as the Navy is making at Patuxent
River. 

The Edwards Air Force Base facility is large enough to accommodate
any military aircraft except a C-5 transport.  Navy officials agreed
that the Edwards facility is large enough to accommodate their
medium-sized E-6 and P-3 aircraft; however, they maintain that the
Edwards facility is not advanced enough right now to conduct the
testing on these aircraft.  Navy officials also insist they cannot
postpone their testing until fiscal year 2002 when the Edwards
facility upgrade is scheduled to be completed.  Furthermore, they
say, the Air Force has blocked out most of the available test time at
the Edwards facility for its future F-22 fighter, an aircraft that
would fit in the Patuxent River chamber. 


   MORE COST-EFFECTIVE
   ALTERNATIVES TO PLANNED
   RELOCATIONS IGNORED
------------------------------------------------------------ Letter :7

In the past 3 years, DOD has conducted two joint service studies of
possible consolidation of electronic combat test facilities.  One
study done in 1994 is referred to as the Board of Directors study and
is cited as justification for the conclusions in the Master Plan.\1
The other study is known as the 1995 Joint Cross Service Group study,
which was done in support of the 1995 BRAC process.\2 These studies
identified a more cost- effective interservice electronic combat
consolidation as compared to the intraservice approach reflected in
the Master Plan.  However, the lack of interservice cooperation
undermined the more cost-effective efforts. 


--------------------
\1 The Board of Directors is made up of the Service Vice Chiefs in
their role as the Test and Evaluation Executive Agent.  Board of
Directors study team members were drawn from each of the services. 

\2 The Joint Cross Service Group was led by representatives of the
Office of the Secretary of Defense and included team members from
each of the services.  The group examined potential consolidations
for airframe and armaments testing, as well as electronic combat
testing. 


      OPEN AIR RANGE CONSOLIDATION
      DOES NOT REFLECT A MORE
      EFFECTIVE ALTERNATIVE
---------------------------------------------------------- Letter :7.1

To reduce excess capacity, the Master Plan recommends relocating test
assets from EMTE to the western test range managed by Edwards Air
Force Base and cites the 1994 Board of Directors Study as
justification.  According to the study, DOD's open air range workload
capacity is 6,000 test hours per year, while actual workload in
fiscal year 1993 was 4,867 test hours, and actual workload is
projected to decline to 4,000 hours per year.  Based on this workload
data, DOD determined it will only need two of the three current open
air range facilities in the future. 

However, that 1994 study, as well as the 1995 Joint Cross Service
Group study done in support of the BRAC process, ranked EMTE as a
more valuable electronic combat test capability than the Navy's China
Lake open air range.  The 1994 study also projected that relocating
test assets from China Lake to EMTE and the Air Force's western test
range would produce about $47 million more in savings over 5 years
than relocating EMTE. 

DOD and Air Force officials with knowledge of the studies told us
that the Navy participated fully in both studies, but once it became
apparent that EMTE would rank higher than China Lake, the Navy would
not cooperate in implementing the study's conclusions. 


      ELECTRONIC LINKING OF REDCAP
      AND AFEWES A MORE
      COST-EFFECTIVE ALTERNATIVE
---------------------------------------------------------- Letter :7.2

In addition to comparing the EMTE and China Lake open air ranges, the
1994 Board of Directors Study considered the possibility of achieving
"synergy" between hardware-in-the-loop facilities, like AFEWES or
REDCAP, by colocating them with installed system test facilities,
like those maintained by the Air Force at Edwards, or the Navy at
Patuxent River, Maryland.  However, the Board of Directors study
concluded that relocation would require 200 years to net a return on
investment.  Instead, according to a 1995 study conducted for the Air
Force, electronic linking of REDCAP and AFEWES to an installed system
test facility was far more cost-effective than relocating them. 

Despite the findings of these studies, the Air Force plans to
relocate AFEWES and REDCAP.  At the same time, the Office of the
Secretary of Defense and the Navy are undertaking the High Level
Architecture Project to electronically link REDCAP and AFEWES'
hardware with the Navy's installed system test facility at Patuxent
River.  This link will allow DOD to test electronic combat systems on
an aircraft in an installed system test facility and do
hardware-in-the-loop testing without having to physically move the
systems to REDCAP or AFEWES.  This approach is consistent with the
1995 study commissioned by the Air Force. 


   MASTER PLAN PROCESS STIFLED BY
   INTRASERVICE FOCUS
------------------------------------------------------------ Letter :8

The failure of the Master Plan effort to achieve any DOD-wide
electronic combat testing consolidations despite direction from the
Congress to do so is due to service parochialism.  This resulted in
focusing on intraservice rather than interservice consolidations. 


      "GENTLEMEN'S AGREEMENT"
      PREVENTED INTERSERVICE OPEN
      AIR RANGE CONSOLIDATION
      EFFORT
---------------------------------------------------------- Letter :8.1

According to officials involved in the development of the Master
Plan, because no DOD-wide consolidations could be agreed upon, Air
Force and Navy representatives responsible for writing the Master
Plan reached a "gentlemen's agreement." The agreement was that there
would be no interservice consolidation until all intraservice
consolidations were complete.  The impact of this agreement was that
the Master Plan consolidation effort for open air ranges focused only
on whether to relocate EMTE or the western test range since they are
both Air Force facilities, instead of focusing on all three open air
ranges to ensure that the two kept would represent what was in the
best interest of all of DOD. 


   INTRASERVICE FOCUS COULD
   INTERFERE WITH BROADER
   CONSOLIDATION EFFORT
------------------------------------------------------------ Letter :9

In a memorandum transmitting the Master Plan to the Congress in March
1996, the Under Secretary of Defense for Acquisition and Technology
stated that DOD would revisit the Master Plan in the broader context
of section 277 of the National Defense Authorization Act for Fiscal
Year 1996, and adjust the Plan as appropriate.  Section 277 directs
DOD to develop a consolidation and restructure plan for its
laboratories and test and evaluation centers for the 21st century. 

This plan, which DOD calls vision 21, will be based on the
requirements to support the test and evaluation of future weapon
systems and identify the critical test facilities needed to support
them.  DOD maintains that vision 21 will include both intraservice
and interservice restructuring.  However, based on the inability of
DOD to implement proposed interservice consolidations originating
from its prior studies of electronic combat test consolidation, we
are concerned that the intraservice focus that interfered with
development of a DOD-wide Electronic Combat Master Plan will
undermine the vision 21 effort. 


   RECOMMENDATION
----------------------------------------------------------- Letter :10

Because (1) the loss of electronic combat effectiveness was not given
adequate consideration in the development of DOD's Electronic Combat
Consolidation Master Plan, (2) the Master Plan contained no costs or
evidence of savings, and (3) service parochialism was allowed to
interfere with development of the Master Plan, we recommend that the
Secretary of Defense take steps to make sure that the methodology for
the ongoing section 277/vision 21 effort include the following
criteria:  (1) accurate, comparable, and reliable data on the true
cost of operating the services' test and evaluation infrastructure;
(2) the needs of and costs to test facility customers; (3) the
maintenance of geographical and topographical diversity in the test
facility base; (4) the requirement that proposed consolidations be
cost-effective for DOD as a whole; and (5) measures to ensure that
implementation of cost-effective decisions cannot be constrained or
avoided. 


   MATTER FOR CONGRESSIONAL
   CONSIDERATION
----------------------------------------------------------- Letter :11

Because DOD's Electronic Combat Consolidation Master Plan may not
provide for the most cost-effective DOD-wide infrastructure for
electronic combat testing as directed by the Congress, the Congress
may wish to consider directing the Secretary of Defense to defer the
transferring of electronic combat test assets until DOD completes its
vision 21 plan for restructuring its laboratories and test and
evaluation centers. 


   AGENCY COMMENTS AND OUR
   EVALUATION
----------------------------------------------------------- Letter :12

In commenting on a draft of this report, DOD indicated that it did
not agree with our findings, recommendation, or matter for
congressional consideration.  According to DOD's response, the
consolidations proposed in the Electronic Combat Consolidation Master
Plan and addressed in our report are in keeping with the intent of
the Congress to reduce the test infrastructure.  We disagree.  The
Congress directed DOD to develop a DOD-wide infrastructure for
electronic combat testing.  DOD's Master Plan did not consider any of
the Army and the Navy electronic combat test facilities as
possibilities for consolidation and merely transfers Air Force test
functions to other Air Force locations. 

DOD's response indicated that the services made decisions to
consolidate in areas that would have the least impact on DOD's
ability to perform effective test and evaluation.  This response is
not supported by the facts.  For instance, the plan to close the EMTE
electronic combat open air range at Eglin Air Force Base will leave
DOD with no non-desert electronic combat test range for tactical
fighters and two desert test ranges--one each for the Navy and the
Air Force.  This is not consistent with DOD's testing policy that
calls for testing to be conducted in a range of natural environments. 

DOD commented that its planned consolidations reflect the 1995 BRAC
legislation and the services' plans to implement congressional
direction.  Our review showed that the planned actions will go
beyond, not "reflect," the 1995 BRAC legislation as the Air Force
intends to relocate the entire EMTE function from Eglin Air Force
Base, not limit itself to the BRAC-directed realignment of 10 systems
(8 threat and 2 podded systems.) The Air Force intends to move
AFEWES, as well.  This planned move is inconsistent with direction
from the 1995 BRAC. 

DOD believes diversity in the testing environments is desirable, but
inconsequential, so long as DOD maintains the capability to replicate
geographical and topographical characteristics through modeling and
simulation and other work arounds.  Our review indicated that DOD
does not need to rely in large measure on computer models and work
arounds.  Instead, DOD could have considered keeping its non-desert
range at Eglin, and could have considered consolidating the Air
Force's and the Navy's desert ranges into one to keep the diverse
test environments required by its regulations and still reduce from
three ranges to two.  We have modified the language from our draft
report concerning our matter for congressional consideration to
ensure that it is not misconstrued and to help focus attention on the
desirability of considering a more cost-effective alternative.  DOD's
comments are reprinted as appendix II, along with our detailed
evaluation of them. 


   SCOPE AND METHODOLOGY
----------------------------------------------------------- Letter :13

To accomplish our objective, we examined DOD's March 1996 Electronic
Combat Consolidation Master Plan and DOD studies of potential
electronic combat test facility consolidations.  Because the
Electronic Combat Consolidation Master Plan did not include any cost
data, we gathered cost data from affected sites, as well as the Air
Force Materiel Command, and other DOD studies of electronic combat
test consolidation.  We interviewed officials from the Office of the
Secretary of Defense, the Army, the Navy, and the Air Force
responsible for or involved in the electronic combat test process. 
We also interviewed contractor personnel involved in the electronic
combat test process.  We visited open air ranges,
hardware-in-the-loop facilities, installed system test facilities,
and observed electronic combat tests in progress.  We reviewed DOD
policy and guidance on testing and evaluation, as well. 

We performed our work at the Offices of the Secretaries of Defense,
the Navy, and the Air Force; the Offices of the Chief of Naval
Operations and the Air Force Chief of Staff; the Air Force Materiel
Command, Wright-Patterson Air Force Base, Ohio; Edwards Air Force
Base, California; Nellis Air Force Base, Nevada; Eglin Air Force
Base, Florida; Hurlburt Field, Florida; Army Aviation and Technical
Test Center, Fort Rucker, Alabama; Army Missile Command, Redstone
Arsenal, Huntsville, Alabama; Naval Air Warfare Centers at Patuxent
River, Maryland, China Lake, California, and Point Mugu, California;
and REDCAP at Buffalo, New York. 

We performed our review from March 1996 to March 1997 in accordance
with generally accepted government auditing standards. 


--------------------------------------------------------- Letter :13.1

We are sending copies of this report to interested congressional
committees; the Secretaries of Defense, the Army, the Navy, and the
Air Force; the Director, Office of Management and Budget; and other
interested parties.  We will make copies available to others upon
request. 

If you have any questions about this report, I may be reached at
(202) 512-4841.  Major contributors to this report are listed in
appendix III

Louis J.  Rodrigues
Director, Defense Acquisitions Issues


THE DEPARTMENT OF DEFENSE'S
ELECTRONIC COMBAT TEST PROCESS
=========================================================== Appendix I


   PREDICT-TEST-COMPARE REPLACES
   FLY-FIX-FLY
--------------------------------------------------------- Appendix I:1

Electronic combat systems, such as radar jammers and warning
receivers, are most often associated with tactical fighter aircraft
because of the threat posed to them by modern surface-to-air
missiles.  However, electronic combat systems are found today on all
types of platforms.  These include ground vehicles, surface and
subsurface naval vessels, missiles, helicopters, and other fixed-wing
aircraft besides tactical fighters.  Hence, wherever the services and
their contractors develop or test platforms and major subsystems for
those platforms, electronic combat test facilities have been
established as necessary support functions. 

In the past 10 years, the Department of Defense (DOD) has spent more
than $300 million to build and upgrade electronic combat test
capabilities.  The vast majority of this new investment has gone into
hardware-in-the-loop and installed system test facilities, which are
highly scientific, laboratory type facilities, and open air ranges
that try to replicate real world environments.  These new and
upgraded facilities were designed and built to accommodate DOD's
revised electronic combat test process. 

DOD's revised electronic combat test process utilizing
hardware-in-the-loop, installed system test facilities, and finally,
open air ranges fits into a broader test philosophy referred to as
"Predict-Test-Compare." According to a former test official,
Predict-Test-Compare was implemented to ensure more rigorous testing
was done before fielding because of a general belief in DOD that its
electronic combat systems did not work very well.  According to the
Air Force, past electronic warfare programs have displayed a pattern
of latent deficiencies manifesting themselves in operational test and
evalution, necessitating expensive fixes and retesting. 
Predict-Test-Compare replaced DOD's "fly-fix-fly" model that
emphasized open air range testing as the primary test method. 

Fly-fix-fly relied too much on trial and error at open air ranges to
find and correct problems.  Often the systems were concurrently built
and tested and already fielded before successful fixes were
identified.  Typical outcomes of a fly-fix-fly philosophy are the
costly, repeated, and continuing attempts to fix the ALQ-161
electronic warfare suite on the Air Force's B-1 Bombers, and the
SLQ-32 electronic warfare suite on the Navy's surface combatants. 

In contrast to trial and error, Predict-Test-Compare is based on the
scientific method of interplay between inductive and deductive
reasoning.  After subjecting systems to testing on the ground under
tightly controlled conditions, testers compare the test outcomes to
their predictions to induce hypotheses that explain the outcomes. 
The inductive hypotheses, in turn, are analyzed by developers and
testers to deduce what hypothetical fixes are necessary to produce
more desirable outcomes in subsequent tests.  Thus, Predict-Test-
Compare is an iterative process in which understanding why a system
behaves as it does is essential to successfully predicting how the
system will behave when it is modified. 


   HARDWARE-IN-THE-LOOP FACILITIES
   PROVIDE CONTROLLED CONDITIONS
   FOR TEST
--------------------------------------------------------- Appendix I:2

Controlling for the conditions of a test is the number one
requirement for ensuring that test outcomes are explainable. 
Hardware-in-the-loop facilities provide this capability in the
electronic combat test process.  In their laboratory type
environments, testers can control for external variables found in
realistic environments such as terrain effects and background noise
that might influence test outcomes.  Hardware-in-the-loop testing
provides the capability to provide repeatable measurements and
verification of protection techniques and system effectiveness. 

The hardware-in-the-loop facility is the first place a new or
modified piece of electronic combat equipment faces an actual or
simulated threat radar.  Prior to hardware-in-the-loop testing, a
developer begins with a concept for electronic combat equipment to
fill a requirement, say an ability to deceive a new threat radar. 
The developer typically will design a computer model representative
of the concept.  The electronic combat tester will then subject the
conceptual model to an increasingly rigorous test against validated
computer models of threat radars.  Once a computer model that works
against the threat models is developed, real electronic combat
hardware that tries to replicate the model's behaviour is built.  The
electronic combat hardware is then subjected to the
hardware-in-the-loop testing, that is, it is tested against actual or
simulated threat radar hardware. 

If testers cannot demonstrate that the hardware will work as
predicted within the controlled conditions of the
hardware-in-the-loop facility, a system should not proceed to the
next phases of the test process.  Success at installed system test
facilities or open air ranges after failure in the
hardware-in-the-loop facility might be evidence of a positive effect
from environmental influences, for example, electronic signals
bouncing uncontrollably off of terrain features to confuse a threat
radar, a factor that will not always be present in every wartime
environment. 

In addition, systems that have failed in the real world can be
brought back to the hardware-in-the-loop facility to evaluate and
improve their performance.  According to test officials, serious
problems with the ALQ-99 system used on the EA-6B and EF-111
stand-off jamming aircraft were unraveled and solutions identified in
the Real-time Electronic Digitally Controlled Analyzer Processor
(REDCAP) hardware-in-the-loop facility before the ALQ-99 went on to
successful testing at the open air range.  In a more recent example,
the Air Force Electronic Warfare Evaluation Simulator (AFEWES)
hardware-in-the-loop facility was able to recreate and simulate the
conditions that led to the shootdown of Captain Scott O'Grady's F-16
over Bosnia in 1995.  The AFEWES results were subsequently proven in
real aircraft testing at the Electro-Magnetic Test Environment (EMTE)
Open Air Range at Eglin Air Force Base. 


   EFFECTS OF ELECTRONIC COMBAT
   SYSTEM ON PLATFORM DETERMINED
   IN INSTALLED SYSTEM TEST
   FACILITY
--------------------------------------------------------- Appendix I:3

After the hardware is tested in the hardware-in-the-loop facility, it
is then placed on the platform intended to eventually carry the
hardware for installed system testing.  Installed system test
facilities consist of anechoic chambers in which simultaneous
operation of electronic warfare systems and host platform avionics
and munitions can be conducted.  It is in the installed system test
facility that systems and subsystems are tested together for
electromagnetic interference and electromagnetic compatibility, both
of which have been major problems in the past.  For instance, a
number of U.S.  aircraft have had radar jammers, radars, and radar
warning receivers in the past that conflicted with each other.  By
identifying the conflicts before flying at the open air range,
testers can more quickly isolate and solve problems.  Once the Air
Force and the Navy complete their ongoing upgrades to their installed
system test facilities, they will be able to test systems for
effectiveness under a wide range of realistic threat and operational
conditions while still on the ground. 


   OPEN AIR RANGE PROVIDES
   REAL-WORLD TEST SCENARIOS
--------------------------------------------------------- Appendix I:4

Finally, when the hardware has been proven successful in each of the
earlier steps, the electronic combat test process ends with open air
testing against actual or simulated threat radars in real-world
environments.  Real-world phenomena encountered during open air
testing can include terrain effects, multi-path propagation,
electromagnetic interference from commercial systems, and other
conditions that affect the atmospheric propagation of electronic
signals.  While often thought of as the place for a "final exam,"
probably because of the association open air ranges have with
operational testing, open air ranges also can have a developmental
role.  According to DOD officials, a properly managed and operated
open air range can provide the proper mix of scientific accuracy and
real-world effects to allow electronic combat system developers to
know if what they have observed in the hardware-in-the-loop facility
and installed system test facility will hold true in the real world. 
The example cited above, in which the AFEWES hardware-in-the-loop and
EMTE open air range facilities together unraveled, recreated, and
demonstrated how the F-16 was shot down in 1995 over Bosnia provides
evidence of this. 




(See figure in printed edition.)Appendix II
COMMENTS FROM THE OFFICE OF THE
SECRETARY OF DEFENSE
=========================================================== Appendix I



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)


The following are GAO's comments on DOD's letter dated March 10,
1997. 

GAO COMMENTS

1.  The Congress directed DOD to develop a plan "to establish a
DOD-wide infrastructure for electronic combat testing." DOD's
proposed plan fails to establish a DOD-wide infrastructure.  Instead,
DOD's plan did not consider any of the 10 Army and Navy electronic
combat test facilities as possibilities for consolidation or the
results of DOD studies that identified consolidations that would
result in a more cost-effective DOD-wide infrastructure. 

Our report does not conflict with the report entitled Defense
Infrastructure (GAO/HR-97-7, Feb.  1997).  In fact, this report
substantiates its conclusions.  The prior report stated that: 

     ".  .  .  breaking down cultural resistance to change,
     overcoming service parochialism, and setting forth a clear
     framework for a reduced defense infrastructure are key to
     avoiding waste and inefficiency.  To do this, the Secretary of
     Defense and the Service Secretaries need to give greater
     structure to their efforts by developing an overall strategic
     plan."

In this report, we point out that the process used by the services in
developing the Electronic Combat Consolidation Master Plan did not
overcome parochialism, as evidenced by the lack of effort to
consolidate across service lines.  The Master Plan does not reflect a
DOD-wide strategic plan, but rather merely an Air Force plan to move
Air Force functions to other Air Force locations. 

DOD's comment that ".  .  .  the Services made decisions to
consolidate in areas where they would have the least impact on the
Department to perform effective T&E" is not supported by the facts. 
For instance, the plan to close the EMTE electronic combat open air
range at Eglin Air Force Base will leave DOD with no non-desert
electronic combat test range for tactical fighters, and two desert
test ranges--one for the Navy and one for the Air Force.  This is
contrary to DOD's testing policy that requires testing to be
conducted in a range of natural environments.  As an alternative, DOD
could have considered, but decided to forego, the option of
consolidating the test assets of the two desert ranges into one, and
keep its only non-desert electronic combat open air range. 

As our report shows, the Air Force intends to "relocate" the EMTE
function from Eglin Air Force Base, not limit itself to the Base
Closure and Realignment Commission (BRAC) directed realignment.  If
the Air Force transfers more than eight threat systems and two podded
threat systems out of Eglin, its actions will go beyond, not
"reflect," the 1995 BRAC recommendation.  The 1995 BRAC
recommendation involves the movement of only 8 threat systems and 2
podded threat systems, but DOD's Master Plan states that EMTE
consists of 65 highly instrumented threat systems and high fidelity
validated simulators. 

2.  How funding for upgrades was authorized and appropriated is not
relevant to the issue of whether a facility should have been
considered for consolidation or whether more cost-effective
consolidation alternatives exist. 

3.  According to Air Force test policy, modeling and simulation is
not an adequate replacement for actual hardware testing because it
cannot predict absolute performance and effectiveness with high
confidence or achieve the same degree of fidelity for complex
functions as testing of the hardware itself. 

The ongoing vision 21 consolidation effort gives DOD the opportunity
to consider how it will maintain geographical and topographical
diversity, among other things, and still achieve "as few [facilities]
as is practicable and possible." For instance, DOD could consider
keeping its non-desert range at Eglin, and consolidate the Air
Force's and the Navy's desert ranges into one to keep the diverse
test environments required by its regulations and still reduce from
three ranges to two. 

4.  There may be no significant loss of capabilities if the Air Force
limits the movement from EMTE to the eight systems and two pods that
are described in the BRAC decision and keeps the other residual test
assets available for testing at Eglin.  However, if the Air Force
carries out the Master Plan proposal to "relocate" the EMTE function
to accomplish a reduction from three to two electronic combat ranges,
there will be a loss in DOD's current ability to test with high
fidelity and confidence.  Testing only in dry, desert air over rocky,
mountainous terrain will limit DOD's real-world testing to one
environment and one set of operating conditions.  Moreover, the
desert ranges are not representative of most places in which DOD must
be prepared to fight. 

5.  Although the Air Force maintains there is no question of
affordability in the proposed move of AFEWES, the 1995 BRAC found
that such a move would cost $34.9 million and take over 100 years to
achieve a return on that investment.  The Air Force's refusal to
consider electronic linking, despite an independent Air Force
contractor's conclusion that linking would be far more
cost-effective, demonstrates that the Air Force is not in step with
the rest of DOD, which is demonstrating electronic linking of AFEWES,
REDCAP, and the Navy's anechoic facilities at Patuxent River,
Maryland. 

6.  We agree that the Air Force should keep REDCAP and AFEWES test
capabilities for which there are test requirements.  These test
requirements are outlined in Air Force Manual 99-112, Electronic
Warfare Test and Evaluation Process--Direction and Methodology for EW
Testing.  According to the manual, hardware-in-the-loop facilities
(such as AFEWES and REDCAP) are an important test category because
they represent the first opportunity to test components against
simulations of hostile weapon system hardware or actual hostile
weapon system hardware.  That is why we question the Air Force's plan
to put REDCAP hardware in storage in favor of an unproven digital
computer model. 

7.  We continue to believe that the transfer of test assets should be
deferred until the ongoing vision 21 consolidation effort is complete
because this would provide DOD with an opportunity to create a plan
for a future DOD-wide infrastructure for its testing, instead of an
infrastructure that preserves each service's ability to maintain its
own set of separate facilities across the test spectrum.  The 1995
BRAC decisions have a 6-year implementation period.  The planned
transfers do not have to be made immediately to satisfy BRAC.  We
have modified the words in the matter for congressional consideration
to more clearly articulate our position.  Also see comments 1, 2, and
3. 

8.  Open air ranges are used to evaluate electronic combat systems in
background, clutter, noise, and dynamic environments.  Dynamic
environments contain numerous important variables besides those
mentioned in DOD's comments.  According to the Air Force's electronic
combat test manual, an operationally realistic open air test
environment includes real-world phenomena such as terrain effects,
multi-path propagation, electromagnetic interference from commercial
sources, and effects caused by atmospheric propagation factors (i.e.,
the tendency of atmospheric conditions to enhance or inhibit signal
transmission). 

Providing realistic and diverse representations of threat radar
systems in the numbers ("density") and dispersion ("laydown") that
the system under test would be expected to defeat in actual
electronic combat does not negate the requirement to test in
operationally realistic environments.  Also see comment 3. 

9.  The disadvantage of climatic predictability at the desert test
ranges is that the effects of various meteorological conditions
cannot be observed. 

10.  The cost of testing at the western test range, the specifics of
which the Air Force has classified, far exceed those at EMTE at
Eglin.  In fact, eliminating EMTE eliminates the Air Force's lower
cost range.  In addition, allowing foreign customers to utilize the
Eglin range generates revenue, but for classified reasons most
foreign customers are precluded from using the western range. 

11.  While it seems clear that moving EMTE's threat systems to the
Air Force's western test range could improve the western test range's
technical capability, it does not automatically follow that this is
the most cost-effective solution for DOD as a whole to pursue. 

12.  If environmental effects were as well understood and accounted
for in electronic combat testing as DOD's response claims, real-world
testing at open air ranges would not be required; testing indoors at
hardware-in-the-loop and installed system test facilities would be an
adequate substitute.  Environmental effects on electronic combat
system performance can be more accurately determined on open air
ranges where the system is exposed to the complexities of different
real-world environments. 

Furthermore, without the ability to test in at least two distinct
representative environments (e.g.  wet and flat versus dry and
mountainous), DOD will be unable to predict with significant
assurance how an electronic combat system will perform in any
environment other than the one in which it was tested.  Hence,
because the electronic combat test environment provided by the Eglin
range provides DOD with its only alternative to the desert test
environment, DOD's response that "the specific electronic combat
environment offered at EMTE is not critical to RF [radio frequency]
testing .  .  ." is not supportable.  In addition, DOD regulations
and the Air Force electronic combat test process require testing
under real-world representative environment and operating conditions
whether or not DOD believes that a given specific test environment is
not critical for a given type of testing. 

13.  We did not assert that testing conducted in the environment at
EMTE is scientifically "of a higher value" than testing done in a
desert environment.  What we stated was that DOD must prepare to
fight in diverse environments; testing conducted in diverse
environments is of a higher value than testing limited to a single
environment. 

An operationally realistic test environment allows testers to gain
insight into understanding how a system will perform in that
environment.  Testers cannot assume that the system will perform the
same way in different environments.  If DOD reduces its testing
capability to only a desert environment, it will not be able to prove
its systems work in anything other than a desert environment.  This
is contrary to DOD testing policy that requires testing to be
conducted in a range of natural environments.  In addition, testing
indoors in a contractor's laboratory is not considered an acceptable
substitute for real-world testing on the aircraft according to the
Air Force's electronic combat test process guide. 

Neither Point Mugu nor Vandenberg Air Force Base have the necessary
threat system test assets to create realistic threat environments for
electronic combat testing for tactical aircraft systems.  To utilize
Point Mugu or Vandenburg for this purpose, DOD would essentially be
recreating EMTE on the west coast.  Moreover, no naval battle group
currently has the capability to create a realistic open air threat
density and laydown of hostile land-based surface-to-air missile and
anti-aircraft artillery systems.  Also see comments 2 and 10. 

14.  If these sites mentioned by DOD "easily support" electronic
combat testing of tactical aircraft, they should have been considered
for consolidation along with EMTE in the Master Plan process. 
However, the reality is that none of the places mentioned by DOD has
the test assets to create the realistic threat densities and laydowns
that DOD earlier in its response said were the most important factors
in developing "operational realism and diversity."

DOD's statement that once an electronic combat system is operating at
airspeeds and altitudes normal for tactical aircraft the
environmental conditions at the surface have little or no effect on
performance unrealistically assumes no aircraft will ever be called
upon to fly at low altitudes (such as flying low to avoid radar
detection).  Moreover, DOD's statement is counter to its policy
statement on the need to operationally test in different
environments. 

15.  In addition to electronic combat testing, Eglin conducts other
kinds of testing, including bombing and live missile firings. 
Moreover, the main civilian air corridor between Los Angeles and
destinations further east, including Las Vegas, one of the nation's
fastest growing cities, buttresses against the restricted air space
available to the Air Force and the Navy at their desert test ranges. 
Also see comment 1. 

16.  DOD's comment seems to assume that keeping EMTE would mean that
the strengths of the Air Force's western test range, which it
delineates here, would have to be sacrificed.  We do not suggest that
the western test range be closed instead of EMTE.  Also see comment
1. 

17.  Operational testers have been using and continue to use
operationally relevant scenarios at EMTE.  Test aircraft at EMTE can
also fly with live ordnance through simulated hostile airspace and
live ordnance can be delivered on a real target.  Also see comment 2. 

18.  Our point is that the body of potential hostile nations contains
a variety of environments, not just desert.  Testing at EMTE and in
the desert allows the operational tester to gain insight into
electronic combat system performance in multiple environments. 

The threat dispersion at EMTE can be changed if necessary, as it has
been in the past.  In fact, the threat dispersion at all of the
ranges should be changed regularly to ensure that testing includes
operationally relevant scenarios since many modern threat systems are
designed to be mobile. 

China Lake is a facility with naval surface-to-air missiles located
deep in a desert ringed by mountains.  Placing naval surface-to-air
missiles at EMTE with its flat terrain, humid environment and
littoral location could provide a more realistic and operationally
relevant scenario for naval aircraft. 

Despite DOD's assertion that severe operational limitations exist at
EMTE, EMTE's annual workload historically has been significantly
greater than the two desert test ranges.  The Air Force and the Navy
both use EMTE for testing despite the presence of the desert ranges. 
Thus, it appears their past testing behavior indicates they believe
the benefits of EMTE outweigh any such limitations. 

19.  REDCAP at Edwards will be less capable as a hardware-in-the-loop
facility because the Air Force intends to put the hardware in
storage, replacing it with a digital computer model to simulate
actual hardware testing. 

According to Air Force Manual 99-112, Electronic Warfare Test and
Evaluation Process--Direction and Methodology for EW Testing, the Air
Force's electronic combat testing policy requires
hardware-in-the-loop testing.  Also, REDCAP currently has paying
customers who do want to use it.  Furthermore, hardware-in-the-loop
facilities such as REDCAP and AFEWES use "real equipment." It is in
digital modeling, such as DOD's comment proposes as a substitute for
REDCAP, where actual electronic combat systems are replaced by
software representations instead of real equipment. 

The software-based computer model of REDCAP being developed may cost
less to operate than the actual REDCAP hardware-in-the-loop facility,
just as flight simulators cost less to operate than actual aircraft. 
However, modeling and simulation is not hardware-in-the-loop testing. 
Because they are different kinds of testing with different purposes,
they are not directly comparable for purposes of determining which is
more cost-effective. 

DOD's statement that "Currently, REDCAP goes practically unused" is
not supported by recent usage data.  Reimbursable costs from test
customers are up significantly over the past 3 years.  Recent
customers include a major U.S.  Air Force aircraft program that used
the REDCAP Mission Level Assessment Tool for several months, as well
as a foreign customer having some of its electronic combat hardware
tested.  See also comment 5. 

20.  DOD's Master Plan included no cost estimates.  We reported (1)
the cost estimates that were independently arrived at by BRAC, which
do not support relocating AFEWES or all of EMTE; (2) known additional
costs that the Air Force will incur by replacing REDCAP with a
digital model, which will in turn allow the Air Force to keep down
the cost-estimate for the REDCAP move; and (3) additional costs that
current EMTE customers report they will incur as a result of the EMTE
closing.  DOD's comments provide no evidence to suggest that these
are wrong. 

21.  According to the BRAC language, some EMTE assets were
specifically directed to be left at Eglin "to support" several
customers, including the Special Operations Forces, as well as the
Air Force Materiel Command Armaments/Weapons Test and Evaluation
activities, and other users.  DOD's position that the BRAC
legislation prohibits testing and limits customer support to
providing training capability is not adressed in the BRAC direction. 

22.  We agree the cost analysis to support any test facility closure
should include additional costs to users associated with the
relocation. 

23.  According to the BRAC recommendation regarding Eglin, BRAC
expected DOD to use the Master Plan process to come up with the
"optimal" consolidation plan.  Closing EMTE (not just relocating
those 10 test assets recommended to be moved by BRAC), relocating
AFEWES despite BRAC's determination that this would not be
cost-effective, and ignoring Army and Navy test facilities completely
as possibilities for consolidation, does not support DOD's claim that
the Master Plan is "the result of BRAC decisions." Moreover, previous
DOD cost-effectiveness studies concluded that the three relocations
planned to be relocated by the Air Force will not be cost-effective. 

24.  The Navy and the Air Force authors of the Master Plan told us
they did not consider costs in the Master Plan because there was no
requirement to do so. 

25.  It is not clear why DOD raises the issue of the REDCAP
contractor's estimate of the cost of moving REDCAP.  We do not use
that figure in our report.  Our report shows that the Air Force
intends to replace REDCAP hardware being moved from its current
location with a digital computer model that will simulate REDCAP. 
The Air Force's contracted cost for the model is $6.2 million.  If
the Air Force was not replacing the REDCAP hardware with the digital
model, it would have to reestablish the REDCAP hardware at some
unknown additional cost.  Hence, the cost to make REDCAP operational
at the new location is either (1) the cost of the move plus the
digital model (with current hardware going into storage) or (2) the
cost of the move plus set-up costs for the current hardware (with no
digital model).  Since DOD has selected option number (1), $6.2
million should be added to the cost of the REDCAP move.  See also
comments
2 and 5. 

26.  REDCAP does have some outdated systems.  But as our report
shows, REDCAP also just completed a $75 million upgrade.  Also,
customer usage and receipts over the past 3 years have increased. 

27.  We have changed the title of this finding. 

28.  According to Special Operations Forces test officials, EMTE
provides a more cost-effective test capability to meet their needs
compared to traveling to the western test range.  Also see comment
21. 

29.  Air Force officials reported to BRAC that they anticipated
saving $3.7 million per year after spending $6.1 million to move the
threat systems out of EMTE.  Even if this savings materializes, it
will not offset the additional costs anticipated by the current users
of EMTE.  Special Operations Forces officials told us they must use
operational aircraft from Hurlburt Field, Florida, adjacent to Eglin,
to accomplish their testing because they have no dedicated test
aircraft at either Edwards or Eglin Air Force Base. 

30.  DOD has no studies to show that the relocations delineated in
the Master Plan are cost-effective, and now claims that its 1994 and
1995 joint studies, which do not support the Master Plan moves, were
incomplete and flawed.  We spoke with Air Force, Army, and DOD
Inspector General officials involved in preparation or oversight of
the 1994 and 1995 studies and they do not agree the studies were
flawed.  They told us what made the recommendations of these studies
"unrealistic" was not the content, but the refusal of the Navy to
consider closing China Lake while the Air Force retained two open air
ranges.  Navy officials associated with China Lake do maintain the
studies were incomplete and flawed. 

The specific examples provided to us on 15 August 1996 represent the
dissenting position that China Lake's open air range was not given
adequate consideration in studies that compared it to the Eglin open
air range.  This data does not support the alternative position that
the Master Plan proposal to relocate EMTE is cost-effective. 

31.  We agree that the Air Force's cuts in funding for investment at
EMTE over the past several years, coupled with the Navy's increased
investment funding at the China Lake range, could affect the outcome
of a comparison of the two if the 1994 study was redone today.  DOD,
however, has not done such a study to demonstrate that the outcome
would be different. 

32.  The claim that Eglin has a capacity disadvantage does not appear
to be accurate.  During the run up to the 1990-91 Gulf War, the Eglin
Range conducted the largest share of electronic combat testing of the
three open air ranges.  During fiscal year 1993, 2,133 hours of
testing were conducted at EMTE, while China Lake and the western test
range conducted 1,649 and 1,085 hours, respectively. 

33.  The referenced DOD Inspector General's report compared EMTE with
the western test range.  We do not assert that EMTE should be kept in
lieu of the western test range.  The Inspector General's report did
not consider the Navy's open air range at China Lake compared to
EMTE. 

34.  The 1995 study conducted for the Air Force by Georgia Tech
Research Institute concluded that electronic linking would be far
more cost-effective than relocating AFEWES and REDCAP.  In addition,
a 1994 Board of Directors synergy study concluded that moving the
hardware-in-the-loop facilities would not be cost-effective.  We know
of no study that concludes it is less expensive to relocate and
reassemble AFEWES or REDCAP hardware at a new location. 

DOD's position that successful electronic linking will be impossible
due to the laws of physics has not yet materialized.  DOD's project
to link REDCAP and AFEWES with Patuxent is well underway, and as DOD
states, the REDCAP link "shows potential." Additional support for the
linking project comes from the Georgia Tech study concluding that
linking will be more cost-effective, and the 1994 DOD synergy study
concluding that moving the hardware-in-the-loop facilities is less
cost-effective.  Hence, DOD could have advocated pursuing electronic
linking instead of relocation of REDCAP and AFEWES in the Master
Plan. 

35.  The 1994 synergy study conducted for DOD's Test and Evaluation
Board of Directors concluded that it would take 200 years to recover
the investment to relocate and reassemble the hardware-in-the-loop
facilities at the Edwards Air Force Base installed system test
facility for "one stop shopping." As a result, the Navy shows no
inclination to relocate its hardware-in-the-loop facility from Point
Mugu, California, to its installed system test facility at Patuxent
River, Maryland. 

36.  Even taking into account the continued operations and
maintenance costs at AFEWES and REDCAP, the 1994 DOD synergy study
and the 1995 Georgia Tech Research Institute study concluded that
these moves would not be cost-effective. 

37.  The memoranda cited by DOD were all written in 1992 and referred
to another study that concluded that keeping China Lake's open air
range was less cost-effective than EMTE.  The DOD joint service
studies cited in our report were conducted in 1994 and 1995. 
Although DOD asserts that it is not service parochialism that
prevents interservice consolidation from occurring, we note that DOD
has now produced three studies with a conclusion that China Lake is
less cost-effective to keep, yet the Master Plan calls for assets to
be relocated from one Air Force location to another Air Force
location.  The Director of Air Force Test and Evaluation told us that
this is because the Navy would not consider relocating China Lake's
test assets. 

38.  As with the Electronic Combat Consolidation Master Plan, we
believe that service parochialism may interfere with the ongoing
vision 21 effort.  There have been no DOD-wide electronic combat test
consolidations in the Major Range Test Facility Base despite a number
of studies that have recommended such consolidations. 

39.  We have modified the language from our draft matter for
congressional consideration to ensure that our focus is not
misconstrued by others. 


MAJOR CONTRIBUTORS TO THIS REPORT
========================================================= Appendix III

NATIONAL SECURITY AND
INTERNATIONAL AFFAIRS DIVISION,
WASHINGTON, D.C. 

Delores Cohen
Charles Ward

ATLANTA REGIONAL OFFICE

Mark Lambert

CHICAGO REGIONAL OFFICE

Don Springman

*** End of document. ***





FAS | Military Analysis | GAO |||| Index | Search |


Maintained by Webmaster