McDonnell Douglas Aerospace (MDA), Intelligence Systems and Services (ISS) submits this white paper as deliverable DI-A-7089/T, Scientific and Technical Report, required by Task Requirements Notice (TRN) 48, Client/Server Topic under contract F19628-90-D-0029, Automated Message Handling System (AMHS) Deployment Contract.
"The Implementation of Client/Server TOPICreg." presents the findings and approaches to integrating and utilizing new Topic products by Verity, Incorporated. The new products are a radical departure from the Topic 3.1.5 product family currently implemented in the AMHS baseline. The new products introduce a true client/server architecture for Topic and are designed to provide Topic search and retrieval capabilities in Local Area Network (LAN), Wide Area Network (WAN), and World Wide Web (WWW) environments.
This white paper describes the new Topic products and proposes an approach to integrating these new products with AMHS. MDA proposes a series of phased baseline roll-outs aimed at providing new functionality sooner rather than requiring the AMHS community to wait for a comprehensive release. The phases have been structured to provide the most needed benefits first, followed by additional releases that will migrate AMHS to a platform independent, Web accessible application, and then on into full Defense Messaging System (DMS) compliance. The four phased approach is as follows:
1. Integrate Topic Internet Server for Remote User Access
2. Replace Topic 3.1.5 Database Engine with Topic Enterprise Server
3. Migrate AMHS Client Processes to Web Based Platform Independent Applications
4. Add DMS Gateways and Support for User Agents
Internet Web technology is moving quickly, almost exponentially with companies introducing new products every day. Collaborative efforts are becoming the norm. Basing a design only on existing products and capabilities means the design will most likely be outdated once it is implemented and fielded. The design for the third phase discussed in this white paper is not yet solidified as COTS products and development environments that will be available 3 to 6 months from now are unknown. MDA continues to keep abreast of current Web technologies. MDA has made several assumptions in designing the third phase. Any of these assumptions, should they change, will impact its implementation and schedule. For this reason, MDA proposes the third phase be viewed only as a preliminary design. As existing products mature and new COTS products are introduced into the marketplace, our design will solidify with the promise of being more reliable and robust, and having lower integration costs.
No detailed designed is presented for the fourth phase, DMS migration, as it is outside the scope of this white paper. The information presented is to complete a migration path picture that will ultimately transition AMHS into the DMS world.
ACSP - AMHS Client Server Protocol
AIS - AMHS Internet Server
AIWC - AMHS Integrated Web Client
AMHS - Automated Message Handling System
C/S - Client/Server
CGI - Common Gateway Interface
COTS - Commercial Off The Shelf
CSC - Computer Software Component
DM - Deliver Message
DMS - Defense Messaging System
GUI - Graphical User Interface
HTML - Hyper-Text Markup Language
HTTP - Hyper-Text Transfer Protocol
HTTPd - HTTP daemon
IM - Inbound Message Distribution CSC
IMCB - Inbound Message Collection Builder LLCSC
IMDM - Inbound Message Deliver Message LLCSC
IMTP - Inbound Message Topic LLCSC
IP - Internet Protocol
ISS - Intelligence Systems and Services
LAN - Local Area Network
LLCSC - Lower Level CSC
MDA - McDonnell Douglas Aerospace
MDB - Message Data Base
MF - Mail Forwarder
NCSA - National Center for Supercomputing Applications
NFS - Network File System
OM - Outbound Message Distribution CSC
RPC - Remote Procedure Call
SM - Session Management CSC
TAP - Topic Agent Protocol
TAS - Topic Agent Server
TCP - Transfer Control Protocol
TDK - Topic Developer Kit
TES - Topic Enterprise Server
TIS - Topic Internet Server
TOR - Time of Receipt
TRN - Task Requirements Notice
URL - Universal Resource Locator
TIMPC - Topic Integrated Multi-Protocol Client
WAN - Wide Area Network
WWW - World Wide Web
3.1. AMHS v2.0 and TOPIC v3.1.5
The current Topic product line operates in a distributed processing environment. The retrieval engine runs on the workstation requiring Topic indices (Partitions) to be shipped to the workstation during initialization of each Topic client. As a rule of thumb, a user can expect a data transfer to the workstation equal to 30% of the size of the MDB he is interested in. For a 3-day query on systems that average 8500 messages per day, the result is a 25MB data transfer to the workstation each time a Topic client is invoked (assuming no caching on the workstation). As new messages arrive at the server, Topic Partitions are consolidated into new larger Partitions while older, smaller Partitions are removed. Changes to Topic Partitions must be communicated to each active Topic client.
All Topic clients poll the Topic server using a common mailbox for notification of database changes. Only one Topic client can access this mailbox at any given time. As the number of users increases the amount of time it takes to gain access to the mailbox and obtain the current state of the Topic indices increases. Once the client becomes aware of a change in the Topic database, Network File System (NFS) protocols are used to transfer new Partitions to the client. NFS reads are at a minimum 8KB. Using our example of a site receiving 8500 messages per day, an average 36KB per second bandwidth is needed to keep the Topic indices current on each active workstation. At this volume, an ethernet LAN would be saturated by 140 idle users running Topic!
While the network load is relatively high for users connected to the AMHS by ethernet, the load is prohibitive for "remote" users whose access is across networks operating at less than ethernet speeds (10Mb/sec). Because of the network overhead, a graphical interface to these remote users can only be provided to a very few users at a time.
3.2. Topic Products and Web Technology
The answer to reducing the network load incurred by Topic v3.1.5 and the AMHS is to integrate the new set of Topic World Wide Web based products and implement AMHS custom applications as client/server processes. Through the use of protocols implemented to support the Web, a true client/server architecture can be realized along with a huge reduction in network overhead. Further, this client/server implementation can be designed to achieve platform independence for the AMHS client, eliminating client ports and multiplatform client baselines.
The integration of the new Topic products, and Internet and web technologies, introduces a new set of terms and products. The following paragraphs describe these terms and products.
Web browsers and servers: Web browsers are applications that communicate with Web servers (sometimes called HTTP servers or daemons) that display document pages formatted with HTML. Some examples of Web browser applications are NCSA Mosaic, Netscape Navigator, and Microsoft Internet Explorer. Examples of Web servers are CERN Server, NCSA HTTPd, and Netscape Communications Server.
Hypertext Transfer Protocol (http): HTTP is an application level protocol that is generic, stateless, and object-oriented. Messages are passed in a format similar to that used by Internet Mail and the Multipurpose Internet Mail Extensions (MIME). The HTTP protocol is based on a request/response paradigm. A client establishes a connection with the server and sends a request to the server. The server responds with a status line followed by a MIME like message. For most implementations, the connection is establish by the client subsequent to each request and closed by the server after sending the response.
Hyper Text Markup Language (HTML): HTML is a simple data format used to create hypertext documents that are portable from one platform to another. HTML is an application of ISO Standard 8879:1986 Information Processing Text and Office Systems; Standard Generalized Markup Language. HTML has been in use by the World Wide Web global information initiative since 1990. Several enhancements have been made by a variety of vendors to the specification since its implementation. As a defacto standard, Netscape has maintained control and direction of the HTML specification up to this point. Recently, Microsoft has proposed their own extensions to the HTML specifications, creating some interoperability (display) problems between document and browser. Typically, a user will see "best viewed with ... browser" when accessing HTML documents that employ HTML extensions not yet part of the ISO standard.
Topic Collections: A Collection represents metadata for a set of documents, and its architecture is optimized for searching. Document metadata includes indexes (optimized for searching document locations), required fields, and custom fields to related documents. These features allow clients to search over multiple collections' documents without interruption.
Topic Developer Kit (TDK): The TDK is the core Verity product, encompassing the retrieval engine and the application programmer interfaces (APIs) which facilitate integration of the TDK with custom applications. The TDK is a bundled component of the Topic Internet Server and the Topic Enterprise Server.
Topic Internet Server (TIS): Verity's first ported client/server product to the Alpha OSF/1 platform is the Topic Internet Server (TIS). The TIS is a server-based module which is integrated with the Topic Developer's Kit (TDK). The TDK provides the retrieval engine and support to message collections. The TIS communicates Hyper-Text Markup Language (HTML) displays via Hyper-Text Transfer Protocol (HTTP) with most Web browsers.
Watcher and Searcher Agents: An agent is a software object that is empowered to represent someone and carry out some action. An agent can send a message, retrieve information, or interact with other agents. Agents are transportable across operating systems and environments. Watcher agents are submitted by a user to monitor real-time traffic; searcher agents are submitted by a user to perform a retrospective search of the Topic database.
Common Gateway Interface (CGI): CGI is a specification for parsing data between a Web Server and an external program such as a database. Within AMHS, the Web Server is NCSA HTTPd 1.4.2 and the external program is the Topic Internet Server. The CGI programs, along with supporting HTML documents, comprise the AMHS Internet Server.
Topic Enterprise Server (TES): The TES is Verity's latest offering to provide a complete client/server information retrieval solution. The TES employs a multi-process, multi-threaded architecture that scales easily to support thousands of simultaneous searches and millions of documents. The TES automatically indexes and serves documents on a real-time basis, enabling immediate retrieval of newly-arriving documents. Users can actively search Topic data stores while new documents are being index into a collection. The TES optimizes network bandwidth and performance by transmitting only the data required to refresh the screen. Use of client/server protocols eliminates the additional overhead created by file sharing. The Enterprise Server supports full discretionary access controls. The TES communicates with the Topic client via watcher and searcher agents.
Topic Agent Server (TAS): The Topic Agent Server is a powerful tool based on the real-time filtering capability of the Topic database engine. It can host watcher agents for persistently watching and delivering information. It is analogous to the central profiler that is part of the Topic 3.1.5 product line. The TAS is optimized for the processing of a single message against hundreds of thousands interest profiles. The TAS does not provide discretionary access capabilities. For uncontrolled environments, users can submit watcher agents to the TAS as a means of real-time dissemination. Within the AMHS environment however, only the system administrator will be allowed to submit watcher agents to the TAS.
Java Applets: Java is a new programming language developed by Sun Microsystems that is based on C++ and designed for secure two-way, real-time interaction. Using Java, developers can write custom mini-applications called Java Applets. When integrated into Web Pages, Java Applets can enable expert graphic rendering, real-time interaction with users, live information updating, and instant interaction with servers over the network. Applets are downloadable from any server and run safely on any platform. Java is designed to provide maximum security on public networks, with multiple safeguards against virus, tampering, and other threats.
Remote/Local Users: For the purposes of this paper, and the AMHS program in general, a local user is defined as a user who gains access to the AMHS server across networks supporting data rates of 10Mb/sec or greater. This implies the user's workstation communicates with the AMHS server across an ethernet network or something faster. Conversely, a remote user communicates at speeds less than that of ethernet or below 10Mb/sec. The terms remote and local have no bearing on the proximity of the user to the AMHS server.
MDA has planned a series of phased releases aimed at providing new functionality sooner rather than requiring the AMHS community to wait for a comprehensive version release. The phases detailed below are based on commercial product availability, integration and development time, and user demand for any given functionality.
4.1 Client/Server AMHS Phase I
Client/Server AMHS Phase I is aimed at providing a first and rudimentary capability to access the AMHS using http. Specifically, Phase I focuses on providing only inbound message retrieval and review functions. Using http as the protocol vehicle for retrieving and reviewing incoming message traffic eliminates the high network bandwidth obstacle to providing remote AMHS users with a graphical interface to the AMHS. Phase I is designed solely to provide inbound message support and provide it as soon as possible.
4.1.1 Technical Discussion
The C/S AMHS Phase I architecture introduces several new components to the AMHS:
* Netscape 2.0 Web Browser
* HTTP Daemon (HTTPd) Server
* AMHS Internet Server (AIS)
* Topic Internet Server (TIS)
* Collection Builder
The Web Browser, HTTPd and TIS are COTS products; the AIS and Collection Builder are MDA developed. Figure 4.1-1 illustrates the Phase 1 architecture.
Verity's first ported client/server product to the Alpha OSF/1 platform is the Topic Internet Server (TIS). The TIS communicates Hyper-Text Markup Language (HTML) displays via Hyper-Text Transfer Protocol (HTTP) with most Web browsers. The only interface available to these generic Web browsers, however, is an uncontrolled retrospective search capability. To provide missing AMHS functionality, such as real-time message delivery, the TIS is being enhanced with a MDA developed module termed the AMHS Internet Server (AIS).
A remote user interfaces with AMHS inbound messages through a web browser, in this case, the Netscape Navigator v2.0. The browser displays HTML forms on the user's workstation and provides the interface for the user to specify the search criteria for queries and for message selection from the results list.
Outbound message functions remain exactly the same as for local AMHS users, implementing the same AMHS client modules and communicating using NFS. But since outbound functions are used less often than inbound message functions, the NFS load on the remote user's network is acceptable.
Figure 4.1-1 -- C/S AMHS Phase I Architecture.
The Phase I C/S AMHS for local users does not change at all. Local AMHS users, implementing Topic 3.1.5 clients, can continue to search against the existing and new message partitions with full retrieval functionality. Remote users will use a Web browser as their message review client. Through the browser, remote users will issue requests to the TIS which will run queries against both the legacy Topic v3.1.5 Partitions and the new method of managing the message indices, Topic Collections.
Collections are best described as encapsulated partitions. Through encapsulation, Verity has eliminated the maintenance of partitions by implementing partition maintenance features within their products which recognize and repair partition faults. Topic can support as many as 128 collections and users can select the appropriate collections for any query.
The TIS is responsible for creating and maintaining the new Collections database structures. A feature of the TIS is its ability to execute searches across the new Collections and, for retrospective searches, across legacy information contained in partitions. Therefore, Phase I does not require a reformating of the message data base into collections in order to allow access by remote users.
Message feeds into the AMHS will remain the same. All messages types are supported by both V3.1.5 and TIS. However, IMDM_DeliverMsg (designated DM on the figure), the existing message parser in the AMHS, will distribute incoming messages to both Topic data base engines (V3.15 and the TIS). Mail Forwarder (MF) will continue to guarantee message delivery into Topic 3.1.5, but for the TIS, MDA will develop a new module called the Collection Builder (CB). CB will provide a message clustering function to enhance performance of the Topic Internet Server.
Although the messages still will only be stored once on the AMHS server, there will now be two sets of indices for each message; one for V3.1.5 partitions and one for Collections. Each set of indexes is equal to approximatly 80% of the message data base size. Therefore, a significant amount of disk space is going to be used for storing indexes. This additional storage reduces the amount of messages that can be stored within the AMHS MDB. MDA asserts that this condition is acceptable considering Phase I is anticipated to have a life span of only several months and in Phase II, the space used to support partitions will be recovered when all message indices are converted to collections.
AMHS Web based technology uses a stateless protocol (HTTP) to communicate between client and server. Connections are established only for the duration of an HTTP transaction such as submitting a form and displaying a resulting HTML document. Because of its stateless nature, no login or logout functions are directly associated with the browser and its connection to the HTTP server. HTTP does provide authentication controls, but they are handled by the HTTP server and are not associated with standard UNIX password controls. The HTTP server maintains its own access control tables and transaction audits.
Under Phase I, Web transaction audits and authentication events will be collected by the HTTP server, but will not be merged with AMHS audit events. SMValSession, currently responsible for user session validation, will only be able to validate AMHS login and logout actions. Should a user circumvent the AMHS main menu and access the AMHS Web server directly from a Web browser, no login or logout information will be recorded in the AMHS audit logs. Authentication and HTTP transactions will always be audited by the Web server.
In an effort to bring Phase I to the field as quickly as possible, some features of the current AMHS v2.0 baseline will not be implemented for remote users. These features include:
* Topic editor. The Topic Internet Server does not provide support for the Topic editor which allows users to edit topics;
* Comeback copies. Comeback copies are returned to an outbound message drafter and any other designated addressees after the outbound message has been transmitted to the AUTODIN interface. For remote users under Phase I, comeback copies will not be supported.
* Launch menu items. The Launch menu supports several features within the AMHS such as View Full AUTODIN message format, Reply to AUTODIN, inbound message annotations, and section message statistics. These launch items are not available to browser users.
* Delete messages from MDB. Messages cannot be deleted from collections like they can be from Topic partitions. The greatest impact of this as seen by the user will be the presence of individual message sections after the message has been merged. For local users, these individual sections are deleted from the database, as currently supported.
188.8.131.52 New Modules
The following paragraphs describe the new Client/Server AMHS Phase I modules:
Netscape 2.0 Web Browser
Web browsers will pose as the Topic client for the remote user. The Web browser has the ability to provide displays and forms to the user and to accept form input and submit forms to an HTTP Server (HTTPd), or daemon, to process requests for information. Using the formatting extensions of HTML, forms can be designed and presented which emulate the function, look and feel of a Topic client.
To review inbound messages, remote AMHS users will be presented with a query form that will contain fields for DTG, Message Type, and other message defining information. Once all information is entered on the query form, it is submitted to the HTTP server.
Message hits are returned in a results list formatted as a HTML document. The results list contains hypertext links to messages that comprise the hit list. To display a message, a user clicks on its particular hyperlink. The Web browser passes the document title through the HTTPd to the AIS which builds a HTML document dynamically containing the text of the message. The document is then returned to the Web Browser for display.
To construct the query results list, the AIS formats a results list using HTML 3.0 formatting extensions. HTML 3.0 extensions provide features such as tables which are not supported by previous versions of HTML. Tables are required in the MDA implementation in order to design a results list which resembles that of the Topic v3.1.5 client with which users are familiar. As such, the web browser responsible for displaying the HTML forms must be capable of interpreting the 3.0 extensions.
Netscape Navigator v2.0 meets this requirement. Future releases of Mosaic and other COTS Web browsers are expected to support HTML 3.0 formatting extensions, but currently do not. In Phase II, the AMHS will implement Verity's Motif based client, which is an integrated Web browser product that is dependent on other HTML extensions and capabilities that are only supported by Netscape's Navigator 2.0 and Spyglass 2.1. But while the Netscape browser appears to be the obvious choice, Navigator v2.0 is presently available only as a ß release.
Netscape Communications has not committed to a final release date of their v2.0 Web browser. The v2.0 product is in its fifth ß release, including an extensive "Bugs Bounty" program where users were rewarded for finding problems in the browser. The expiration date of the ß5 licenses, just released, has extended the ß license expiration date to 1 March from the original 15 February.
MDA proposes the Netscape product as the choice for implementation in Phase I despite its current ß status. MDA has worked with Windows, Macintosh, and UNIX versions of the Netscape 2.0 beta products for several months and has experienced no major problems. Further, it appears Netscape Communications is the leader in web browser technology and can be counted on to continue providing advanced features first.
The problem with the Netscape product is the availability of the formal v2.0 release. The recent slip of the ß expiration license date means the formal product may be after the intended ß test date for AMHS on 19 February.
The HTTP Server (HTTPd in Figure 4.1-1), or daemon as it is referred to in UNIX, is responsible for all communications with the client based Web browser. It is responsible for access controls (not discretionary access), fielding requests for HTML documents, delivering HTML documents to the browser, and passing query field data from the browser to Common Gateway Interface (CGI) scripts.
The connection between HTTPD and the browser is stateless. The browser establishes a connection with the server and sends a request along with authentication information. Once the request is processed by the HTTP server and a resulting HTML document is downloaded to the Web browser, the connection is terminated. User authentication information is kept in the Web browser's memory space and is passed to the server with every request.
When a HTML form is submitted to the HTTP server, it contains embedded information telling the HTTP server what action to perform. The HTTPD retrieves the appropriate HTML form to return to the web browser or calls a CGI program to build a dynamic HTML form. The HTML forms and corresponding CGI scripts constitute the AMHS Internet Server (AIS).
MDA will deliver the NCSA Web server as part of the C/S AMHS Phase I architecture. The NCSA HTTPD is a proven product with source available and no licensing requirements.
AMHS Internet Server
The AMHS Internet Server (AIS) augments the functions provided by the Topic Internet Server. The AIS module consists of HTML documents and Common Gateway Interface (CGI) programs. HTML documents are often static, being displayed identically every time requested. An example of such a form might be a login screen. HTML documents can be created dynamically, however, by CGI programs, building a specific document with potentially unique attributes each time the CGI is called by the HTTPD. The CGI is a standard for interfacing external applications with information servers.
MDA is implementing the AIS to replace capabilities that would be lost in the transition from the Topic 3.1.5 technology to the client/server version of Topic if Phase I were to rely on only those inbound message features supported by the TIS. These key features are:
* Real time message processing
* User ReadList
* Discretionary Access Control
* Saved Query Management
Real-time message processing is part and parcel to the AMHS application. The TIS, however, is designed for retrospective searches only, collecting message hits for every submission of a query as a snapshot of the database. To emulate real-time delivery, the AIS will present to the user a field identifying how often to re-post a query once it has been submitted. The field will be set with a default value of "never", but the user can select values of 1, 5, 10, 30, and 60 minutes. For remote users, it is anticipated that this periodic refreshing of the query will provide sufficient delivery of new message traffic. The Verity Topic Enterprise Server (fielded in Phase II) uses watcher agents to provide real-time message processing capabilities.
The current version of Topic provides a readlist capability on a per user basis. This is the capability to track which messages in the AMHS MDB have been read by each user. During Phase I, a stubbed Time of Receipt (TOR) field will be provided on each query form allowing the user to control how much of the MDB will be included in his/her results list. The resulting hit list can be sorted by TOR to provide an interim mechanism for the user to identify which messages he/she has read. Phase II of the Client Server AMHS will implement the Verity Topic Enterprise Server which supports full read list capabilities.
The third feature is Discretionary Access Control (DAC) which restricts a user's access to certain messages within the Topic database (collections or partitions). The AIS will add user discretionary access terms to all submitted queries, providing the "NOT" filter supported by Topic v3.1.5. Verity's Topic Enterprise Server (fielded in Phase II) has full discretionary access capabilities.
The fourth feature is saved query management. Users will have the capability to store, retrieve, and delete completed query forms for future query submissions. The Verity Topic Enterprise Server (fielded in Phase II) has full query management functions.
The functional flow of the AIS can be traced from a submittal of a retrospective search query to the receipt of the results. A search query is submitted to the AMHS system via the Web browser. This in reality is a HTML form request sent to the HTTP daemon. The HTML form request will instruct the HTTP server to execute a AMHS CGI program (within the AIS) to process the form's field data. The AMHS CGI program checks the user's credentials and DAC information. A modified query request, modified to include DAC information, is then submitted to the TIS to search the Topic databases. The TIS returns a results list that meets the criteria of the query to the AMHS CGI program. The AIS reformats the results list as a HTML document that looks very much like Topic 3.1.5's results list.
Topic Internet Server
Client/Server Topic is a merging of Verity's text retrieval technology with current advances in Internet technology. Unlike its previous versions of Topic, Client/Server Topic utilizes TCP/IP to communicate with its clients, thereby decreasing the amount of NFS traffic on the network. Further, the Client/Server Topic organizes information in what is known as collections instead of partitions. For our purposes, the Phase I Client/Server Topic can be considered as two components, collections and the Topic Internet Server (TIS).
Verity's first ported client/server product to the Alpha OSF/1 platform is the Topic Internet Server (TIS). The TIS is a server-based module which incorporates Topic's document search engine and is integrated with the Topic Developer's Kit (TDK) to provide application programmer interfaces (API) for the integration of Topic with custom applications. The TIS communicates Hyper-Text Markup Language (HTML) displays via Hyper-Text Transfer Protocol (HTTP) with most Web browsers. The TIS is a multi-threaded/multi-process application. It has the ability to scale efficiently across multiple CPUs to support simultaneous searches and serve documents. The TIS serves two main purposes: 1] maintain the Collections and 2] process the queries submitted by users.
The Topic collection architecture supports the following features:
* Collections can be updated continuously while clients are searching over them;
* Document indexing can occur continuously and in concert with client operations since a collection maintains revision data about documents;
* Collections are automatically repaired when error conditions occur; and,
* Collections are automatically serviced to optimize collection files for searching to monitor the state of the collection and clean up collection files when no longer needed.
The collection builder is analogous to the Mail Forwarder in grouping messages for insertion into the Topic Database. The Collection Builder receives messages from Deliver Message, holds and groups them based on quantity, type, and polling interval, and finally passes them to the TIS-integrated TDK engine for insertion into collections. Unlike Mail Forwarder, the Collection Builder does not throttle delivery of messages to Topic.
184.108.40.206 Modifications and Effects on Existing CSCs
To successfully integrate the proposed C/S AMHS Phase I into the current AMHS system, the following CSCs will be modified:
* Inbound Message Services (IM)
* Session Management (SM)
Deliver Message CSC
Changes will be made to the IMDM_DeliverMsg Lower Level CSC (LLCSC) and a newly developed Collection Builder (IMCB) will be added.
* The IMDM_DeliverMsg LLCSC with be altered so that each time a message is processed it will create not only a file that contains information to notify Topic 3.1.5 Database engine but also a file that contains the same information but in a format that is compatible with the TIS.
* MDA will develop a Collection Builder (IMCB) for notifying the TIS of new messages based on site specified maximum number of messages and/or time-out periods.
Session Management CSC
Changes will be made to AMHS Main Menu to create a pull down menu option that will launch a Web browser. Through the development of another session startup script, a system administrator will select whether users at a particular workstation will access the AMHS as a local user or a remote user. By selecting the remote user startup script, the AMHS session presented to the user will show the Topic v3.1.5 menu options disabled (grayed out) and the web selection available. For local users, the web selection will be unavailable.
It is necessary to restrict users to a particular method of access because inappropriate use of either could cause undesirable system impacts. For example, remote users employing Topic v3.1.5 clients would quickly bog down the slower speed networks over which they access the AMHS. Local users accessing AMHS using the web browser would create additional CPU load that might overwhelm the AMHS.
SMValSession will be modified to operate in a low bandwidth environment. In the current AMHS baseline, SMValSession uses NFS to access the CURRENTUSERS file. When a user accesses the file, the file is locked and other users attempting to login are delayed until the lock is removed. For remote users, the slower networks would cause longer locks and could cause timeouts and error messages to the users.
This problem was addressed during the character based remote effort. The NFS connection was replaced by a remote procedure call (RPC). MDA will transfer this modification to the Phase I effort to facilitate rapid integration of SMValSession changes.
4.1.2 Cost and Schedule
220.127.116.11 Phase I Implementation
Key to the decisions made in the design of Phase I is the immediate need for a graphical AMHS capability for remote users. This need has been magnified by the actions in Bosnia.
To accommodate this need, the implementation schedule drove design decisions. Certain features that could have been implemented were omitted in order to bring the Phase I product to the field sooner.
Figure 4.1-2, Schedule for Client/Server AMHS: Phase I, illustrates the implementation schedule for AMHS Client/Server Phase I. The schedule describes a completion of development and code freeze on 19 January and completion of in plant testing by 19 February.
There is no figure here
Figure 4.1-3, AMHS Client/Server: Phase I Cost and Labor, describes the labor hours and costs estimated to develop the Phase I capability.
Figure 4.1-4, AMHS Client/Server: Phase I Materiel Costs, summarizes the costs for PNDI licenses required for the Phase I development. Perl, a scripting language interpreter, and the NCSA HTTP Server have no license costs associated with them. Because Netscape Navigator v2.0 is available in ß release only, no license fee is being charged. Upon formal release of Navigator v2.0, a license fee of about $50 is anticipated.
The second ß test of Phase I is scheduled at Air Intelligence Squadron in Ramstein on 19 March.
There is no figure hereFigure 4.1-2. Schedule for Client/Server AMHS: Phase I.
There is no figure hereFigure 4.1-3. AMHS Client/Server: Phase I Cost and Labor
There is no figure hereFigure 4.1-4. AMHS Client/Server: Phase I Materiel Costs.
18.104.22.168 Phase I ß Testing
The ß test of Phase I is scheduled to occur at USEUCOM after completion of the IPAT (Figure 4.1-5). The USEUCOM ß test is anticipated to consist of an initial ß test occurring at Patch Barracks and will simulate remote environments. After the initial testing, the Phase I software will be installed at one of USEUCOM's key remote sites for true remote user access testing.
There is no figure hereFigure 4.1-5. TA&C Phase 1 ß Test Schedule.
Figures 4.1-6 and 4.1-7 identify the costs of supporting the two ß deployments of Phase I, the first to USEUCOM and the second to the Air Intelligence Squadron. For the purposes of this estimate, at EUCOM, it is assumed MDA will install OSF/1 v3.2 and the Phase I associated software, execute security accreditation tests, and execute specific ad hoc testing appropriate to the new Phase I capability.
Three days are also planned to execute tests of the Phase I capability from a remote site (Aviano for costing purposes). For the remote testing, one MDA engineer will travel to the remote site and the other will support the EUCOM site.
The costs include labor, travel, and other direct charges. Figure 4.1-6 estimates the labor costs to support the ß tests and figure 4.1-7 estimates the other direct costs associated with the tests.
There is no figure hereFigure 4.1-6. Phase I ß Test Labor Costs.
The main disadvantage of a Phase I solution is CPU performance. Although network load will greatly decrease for Web based AMHS clients, CPU utilization will increase dramatically with the TIS because of the necessity to re-issue the remote users' queries to emulate real-time message delivery. This CPU load limits the number of remote users the Phase I implementation can reasonably support, but the actual number of users is, today, unknown. The CPU load is why MDA proposes to implement the restrictions which prevent local AMHS users from accessing the AMHS via a web browser. Load tests are planned after the development code freeze and should give some insight to the number of users that can be supported. The Topic Enterprise Server (Phase II) will handle real-time messaging in a more CPU friendly manner.
The key schedule risk is the availability of Netscape 2.0. As explained in the Technical Discussion, Netscape Communications has not committed to a final release date of their Web browser but MDA anticipates the release to be around 1 March 1996. If the user community does not object to installing the currently available ß5 Netscape release, the risk is eliminated. If deployment of Phase I with a ß5 release is not possible, Phase I may have to be delayed until Netscape releases the formal v2.0 product.
There is no figure hereFigure 4.1-7. Phase I ß Test Other Direct Costs.
4.2 Client/Server AMHS Phase II
MDA's integration of the Verity's Motif based client and the Topic Enterprise Server (TES) will provide AMHS with a low network bandwidth capability for all users. Phase II will field an AMHS capable of providing support to several hundred concurrent users and a common user interface in local and wide area networks, deployed and dial-up environments.
4.2.1 Technical Overview
In C/S AMHS Phase II, Verity's Topic Integrated Multi-Protocol Client (TIMPC) and Topic Enterprise Server (TES) will be employed for AMHS MDB search and retrievals. Verity has chosen to use an integrated browser solution, specifically, Netscape v2.0 or Spyglass v2.1, for the Motif based clients. The TIMPC begins the transition of Verity's client product line, and, in Phase III, the AMHS client, to a machine independent architecture. Figure 4.2-1, C/S AMHS Phase II Architecture, illustrates the Phase II configuration.
Figure 4.2-1. C/S AMHS Phase II Architecture.
Under C/S AMHS Phase I, a generic Web browser, via the AIS/TIS, was the only means to query the Collections database. In Phase II, the TIMPC, like the generic Web browser, will establish a connection with the AMHS Web Server when information is to be retrieved. The TES also has the capability to push data toward the TIMPC via Topic Agent Protocol (TAP). The user through the TIMPC will be able to set up watcher agents that run on the AMHS Server and send results back to the browser in real-time.
Performance and functional characteristics of the TES are vital in determining how message traffic will be delivered to the end user or system. The TES must first index an incoming message before TES watcher agents can profile and distribute it (update a results list). If the TES cannot deliver high precedence messages in time frames specified in the System Specification, the Topic Agent Server (TAS) will need to be used for high precedence delivery. The TAS is optimized for high speed dissemination of real-time message traffic, but does not provide for discretionary access. TAS watcher agents will be available only to the profile administrator, or some method of applying discretionary access to the TAS delivery process will need to be added. The TAS may also be required if the TES has no delivery mechanism suitable to support backside subscriber systems.
Some loss in AMHS functionality is anticipated with the new Topic product line. Verity's marketing strategy has focused their products toward search and retrieval functions and away from group-ware functions. Specifically, Topic will no longer support hard queues that AMHS uses for message comeback, message forward, and high precedence delivery mechanisms. To circumvent the loss of hard queues within Topic, MDA is proposing the following:
* Comeback copies. The Outbound Message Distribution (OM) CSC will send documents to users' electronic mail accounts. The OM CSC discussion in section 22.214.171.124 provides additional detail on our proposed handling of comeback copies.
* Message forward. Message forward capabilities are again accomplished through the site's native electronic mail. Most sites prefer electronic mail over message forward when exchanging message traffic. In AMHS v2.0, a launch was added by user request to the Topic v3.1.5 pull-down for electronic mailing of a document. This launch or a built-in TIMPC mail launch will be used to forward messages to other AMHS users.
* The high precedence queue is rarely used in today's AMHS. In addition to the high precedence queue, a high precedence message is stored in the normal Topic data store and is displayed in a dynamically created X-window for users that are identified as high precedence recipients. MDA believes that high precedence traffic will be delivered in a much more efficient manner with the TES/TAS and that high precedence queues will no longer be necessary.
A generic Web browser will not be able to communicate with the TES directly since the TAP is a proprietary protocol. The AIS and TIS modules will remain, as a result, enabling users who do not have access to a TIMPC to continue to access their message traffic. Re-posting of queries for pseudo real-time message processing will be disabled in the AIS during Phase II. This is CPU intensive; watcher agents and a TIMPC are the preferred method to handle receipt of real-time message traffic.
126.96.36.199 New Modules
The Topic COTS products are the most affected. The Topic Enterprise Server must be integrated to support http access to the inbound AMHS for all users through the TIMPC. Integrating the TES allows the Topic v3.1.5 server to be eliminated from the baseline.
This change in Topic server will require that all message indices be maintained within collections and that the existing partitions will be eliminated. Eliminating the partitions recovers the disk space lost in Phase I due to the need to support both partitions and collections. Existing partitions, however, must be converted to collections to avoid losing access to historical traffic. A conversion program will be developed to assist in this conversion.
MDA will also implement support for message delete from the MDB and collections. This delete capability was one of the missing functions in Phase I. The implementation of message delete will support both the system administrator and the Message Assembly (MA) CSC which uses the function to delete individual sectioned messages once a message's sections have been merged.
Modifications to the IMTP Topic CSC will focus on integrating existing launches not supported by the TIMPC itself. These include:
* Annotate Message
* Build AUTODIN Reply
* View full AUTODIN
* Get Section Statistics
* Print w/ Annotations
In general, these launches will continue to create new X-window displays. For the most part, the underlying code to support these functions remains unchanged. Schedule permitting, some launches may be redesigned to operate in a HTML environment.
188.8.131.52. Modifications of Existing CSCs
To successfully integrate the proposed C/S AMHS Phase II into the AMHS system, the following modules/CSCs must be modified.
* Inbound Message Distribution
* Session Management
* Security Management
* Outbound Message Distribution
* Message Assembly
* AMHS Internet Server (AIS)
Inbound Message Distribution (IM) CSC
Within the Inbound Message Distribution CSC, several changes will be implemented. To support the delivery of high precedence messages, IMDM_Highprec will be modified to deliver high precedence messages using the HTTPD. Until recently, data could only be pulled from the http server, but recent innovations have introduced a push mechanism.
IMDM_DeliverMsg, modified in Phase I to support the delivery of messages to the Topic Internet Server as well as the Topic v3.1.5 server, will have its feed to Topic v3.1.5 removed. Further, the Mail Forwarder implemented in the Topic v3.1.5 inbound process will be eliminated. These changes will remove all inbound message delivery to the current Topic product.
Session Management CSC
The Topic associated launches on the Messages menu of the AMHS Main Menu will be deleted. These launches include:
* Inbound Messages (3 DAYS)
* Inbound Messages (WEEK)
* Inbound Messages (MONTH)
* Inbound Messages (OTHER)
* Inbound Message (ALL)
* Delete Queries
* Change Results Layout
The database defining launches (Inbound Messages) are no longer required as they were created to reduce network load for the Topic v3.1.5 product. Delete Queries and Change Results Layout will be supported by the TIMPC.
The Special Messages launch will remain as it supports messages which are not introduced to the Topic products and the AMHS MDB.
Security Management CSC
A new Topic ISSO GUI will be developed to manage user's ability to access certain AMHS Topic launches, Topic Agents, and C/S Collections.
Outbound Message Distribution (OM) CSC
The support for comeback copies for all users will be reinstated in C/S AMHS Phase II. Under Phase II, comeback copies will be delivered to users through the native electronic mail capability. As such, automatic introduction of the comeback copy into the inbound message traffic stream can be controlled by the user. Within OM, an additional radio button will be implemented on the Local Addressee window to allow each user to signify his/her comeback copy preference, introduction of the comeback message into the AMHS message stream or simply return the copy solely to the composer.
Message Assembly CSC
A method for message delete must be developed for collections. This delete mechanism will be integrated with Message Assembly for the deletion of individual message sections after the AMHS has merge all received sections.
AMHS Internet Server CSC
The AIS will be modified to eliminate automatic re-posting of queries used for pseudo real-time message processing. Under Phase II, the AIS could be eliminated since the TIMPC does not require the AIS to support its functions. MDA proposes to keep the AIS and the TIS, however, to maintain support to remote users which might not have the TIMPC installed on their workstations. Maintenance of the AIS and the TIS product maintains support to generic browser users.
4.2.2 Cost and Schedule
Figure 4.2-2, Schedule for Client/Server AMHS: Phase II, illustrates the implementation schedule for AMHS Client/Server Phase II. The schedule is based on the development of about 4000 source lines of code and the need to integrate two centerpiece COTS products, the Topic Enterprise Server and the Topic Integrated MultiProtocol Client. The schedule show a start of design in mid-February, a code freeze in mid-July, and a completion of in-plant testing at the end of August.
There is no figure hereFigure 4.2-2. Schedule for Client/Server AMHS: Phase II.
Figure 4.2-3, AMHS Client/Server: Phase II Cost and Labor, describes the labor hours and costs estimated to develop the Phase I capability.
There is no figure hereFigure 4.2-3. AMHS Client/Server: Phase II Cost and Labor.
Figure 4.2-4, AMHS Client/Server: Phase II Materiel Costs, presents the estimated costs for software licenses and other materials required. MDA is hoping to implement Phase II without requiring the Topic Agent Server or the TAS Run-time User products. These products are only included in case where high precedence delivery and backside delivery cannot be effectively achieved without them.
There is no figure hereFigure 4-2-4. AMHS Client/Server: Phase II Materiel Costs.
The largest risk is the availability of the Topic TIMPC and TES and how their release affects the schedule for Phase II. There is no doubt the products exist and will be released, but Verity is withholding release of these products pending the release of Netscape v2.0. The Verity products are said to be "ready to go", but, upon release of Netscape, the actual degree of the products readiness is yet to be seen. Further, first releases of new products can be somewhat buggy, causing delays in integration schedules as vendor bugs are worked out in patched releases.
The integration of the TES on the AMHS server is another risk. MDA has worked hard to suggest to Verity that certain TES features are required for the TES's integration with the AMHS, but until the product is delivered, the support of these features can only be expected, not promised.
4.3 Client/Server AMHS Phase III
This phase completes the conversion of AMHS client applications to a platform independent product. The implementation of web technology across the AMHS client will eliminate any future client porting efforts to specific workstation platforms and cut site integration and installation time to a minimum. AMHS will truly be plug and play!
By eliminating the AMHS workstation binaries, dependencies on desktop environments such as WSS, EISS, CSE will no longer exist. In addition, training time is reduced as users already familiar with Web technology through Intelink or from their home personal computers will quickly grasp the AMHS user interface.
4.3.1 Technical Discussion
Phase III focuses on the MDA-developed client side of AMHS. The majority of AMHS server code remains unchanged as only those server based CSCs and LLCSCs that communicate with the client via NFS will be affected.
JAVA is a recently invented, platform independent language developed by SUN Microsystems. The JAVA language is an object oriented language, and coupled with a platform specific JAVA interpreter, allows applications to be developed on any workstation and run on any workstation without the usual porting expenses. JAVA interpreters are currently available for SUNOS and Solaris operating systems, but ports of the interpreters are underway for Digital Unix (OSF/1) and other popular operating systems.
Since JAVA is so new, the availability of development tools is only now growing. Even the JAVA language itself is maturing quickly as new features, such as a scripting language, are being announced as this paper is published. As a part of Phase III, MDA will do a market survey of JAVA development environments and Web authoring packages to assist in the design and development of JAVA applets.
MDA will provide a clean client/server architecture for Outbound Message coordination and release by infusing technology from its X.400 work. The AIS and TIS modules will remain, enabling users who do not have access to a AIWC to continue to access their message traffic.
Figure 4.3-1. C/S AMHS Phase III Architecture.
The greatest advantage of Phase III is the elimination of the MDA developed client applications. Significant reductions in life cycle costs can be realized since the AMHS is truly platform independent. This independence eliminates the need to port the AMHS client to new workstation platforms and eliminates the need to design, implement, and test user applications on multiple workstations. Instantly, the AMHS is accessible through any workstation capable of running a browser, be it Unix, Windows, or Macintosh. Enhancements or fixes are implemented once and the changes are applicable to all clients. The platform independence also eliminates workstation specific DPRs.
Another advantage of the Phase III platform independence is the reduction in installation time and site impact. Under Phase III, MDA developed AMHS client applications are no longer installed on site workstations or application servers. Client installation is limited to the installation of the TIMPC, a process easily accomplished by a system administrator.
A disadvantage of Phase III is the large amount of effort required to retool the AMHS applications into object oriented applets. Development productivity tools are only now becoming available and there is an inherent learning curve in their implementation. But as the technology grows in its maturity, the rigor and capability of the tools will improve and bring increases in development productivity and efficiency.
Another consideration is the training impact on existing sites. The browser approach to AMHS is different from the current client look and feel and will initially require retraining. As browser use grows throughout the community, however, the overall training effort may ultimately reduce as users knowledge of browsers makes learning the AMHS more intuitive.
4.3.3 Modifications and Effects on Existing CSCs
To successfully integrate the proposed C/S AMHS Phase III into the AMHS system, the following modules/CSCs must be modified. All AMHS client applications will be redesigned to run in a true client/server environment.
* Audit Trail Service (AT)
* Information Labels (IL)
* Session Management (SM)
* Security Management (SC)
* Profile Management (PM)
* System Management (SY)
* Notices Services (NS)
* Outbound Message Distribution (OM)
* Message Composition (MC)
Audit Trail Services CSC
AMHS client audit trails are currently handled through the UNIX workstation's Syslogger. A platform independent audit trail mechanism will need to be designed, preferably using browser APIs to isolate AMHS audit trail services from the operating system and the desktop environment.
Information Labels (IL) CSC
The IL editor displays and supporting software will be ported to the Client/Server HTML/JAVA architecture.
Session Management CSC
The AMHS main menu display and underlying code will be ported to Client/Server HTML/JAVA architecture.
Security Management CSC
All ISSO displays and underlying code will be ported to Client/Server HTML/JAVA architecture.
Profile Management CSC
All Profile Administrator displays and underlying code will be ported to Client/Server HTML/JAVA architecture.
System Management CSC
All System Administrator displays and underlying code will be ported to Client/Server HTML/JAVA architecture.
Notices Services CSC
All Notices displays and underlying code will be ported to Client/Server HTML/JAVA architecture.
Outbound Message Distribution CSC
Message Coordination client applications will be ported to Client/Server HTML/JAVA applet architecture. MDA will use DMS TRN developed Message Coordination as a basis for its client/server design. The DMS developed code is still X-Window based; transition to a platform independent application is still needed.
Message Composition CSC
Client applications will be ported to Client/Server HTML/JAVA architecture.
4.3.2 Cost and Schedule
Figure 4.3-2, Schedule for Client/Server AMHS: Phase III, illustrates the implementation schedule for AMHS Client/Server Phase III. The schedule is based on the porting/conversion of 35.7K source lines of code from their current NFS based implementation to the object oriented, HTTP JAVA applets and scripts. The schedule shows a start date of June 1 and completion about one year later.
Figure 4.3-3, AMHS Client/Server: Phase III Cost and Labor, describes the labor hours and costs estimated to develop the Phase III capability.
There is no figure
Figure 4.3-3. AMHS Client/Server: Phase III Cost and Labor. Figure 4.3-4, AMHS Client/Server: Phase III Materiel Costs, summarizes the software licenses and equipment needed to implement Phase III. As the schedule for Phase III shows, MDA will determine the best productivity tool to support Phase III implementation as the result of an early task. The item identified in the table is an rough estimate and a placeholder.
There is no figureFigure 4.3-4. AMHS Client/Server: Phase III Materiel Costs
The risks of Phase III are those that are inherited in any new technology. Cost and schedule are difficult to estimate with certainty when the experience base with the new technology is low. A learning curve is obviously included in the implementation.
As the technology is new, there is a current lack of productivity tools and authoring tools available to facilitate implementation. The marketplace is just seeing the introduction of these tools and it is anticipated that the rigor of these tools will increase as the market grows for web technology. This explains, in part, why MDA has proposed a June start for Phase III. As the technology is moving so fast, even a few months will yield better selection and improved capability in the tool sets.
4.4 Client/Server AMHS Phase IV
Although out of scope of this white paper, AMHS, a DODIIS migration system, must define a path to achieve full DMS functionality, compliance and interoperability. MDA envisions an AMHS that can support concurrent operations in both the traditional AUTODIN environment and future DMS environments. As illustrationed in Figure 4.4-1, MDA will develop and integrate a DMS gateway that will feed X.400 traffic into the AMHS data stores and will interface with X.400 User Agents, providing DMS users with search and retrieval capabilities as well as real-time message delivery. The risks are many as the role of organizational traffic in the DMS environment is not yet clear. On the positive side, we are a development partner with Netscape Communications Corporation. Netscape has already announced support for Fortezza cryptographic capabilities, a key component in the DMS architecture. This is why MDA selected Netscape 2.0 as the browser of choice for AMHS.
Figure 4.4-1. Notional C/S AMHS Phase IV Architecture
Integration of Client/Server Topic promises a much lower bandwidth requirement and will provide a solution for WAN and deployed users. A Web accessible AMHS will keep AMHS a viable message handling system for the DODIIS community for years to come. Web browser based architectures provide client platform independence that will reduce life cycle and installation costs. A phased approach and infusion of existing MDA technologies will provide needed functionality to the end user soon after COTS products come to market.
With any new technology, there is risk. Our phased approach attempts to mitigate these risks by fielding enhancements in increments and not waiting for an "all encompassing" product that may be technologically outdated once it is fielded. We are very excited about Web technologies and believe they are ultimate solutions for the Intelligence Community. MDA proposed evolution of the DoDIIS AMHS, to include Web based clients, e-mail based clients, and X.400 user agents, is a powerful, affordable, and attainable approach to meeting DoDIIS messaging needs well into the next century.
13 September 1999
Technical POC email