A battlefield environment presents a highly challenging situational scenario to the project of gathering, processing and depicting time-critical information. Present age battlefields are especially harrowing to military personnel due to the insidiously stealth nature of such modern threats as landmines and/or chemical/biological weapons. In both cases, there is initially a need for preemptive reconnaisance, and then, if the threat is activated, for rapid and effective response(s) to minimize injury and loss of life. Presenting critical information to key personnel who are simultaneously enabled/empowered to respond effectively has long been a problem. Inadequate capacity to cope with various threats necessitates great resource and personnel allocation just to insure that higher level priorities can go forward. It is our understanding that DARPA aims to remove certain threats from the "planning and conduct of U.S. military operations;" this proposal responds proactively to this aim by specifying an "interventional informatic" methodology for comprehensively addressing the problem of connecting strategic personnel to emerging information and enabling their interaction with that information. We propose a single, multi-modally integrated interface system for the perceptibility and expressablity of massive quantities of critical information.

The research, prototyping, development and demonstration of technologies to support gathering, interpretation and use of data is a major challenge/obstacle. Such complex phenomena as battlefield dynamics easily overwhelm the best available information delivery systems. Presently, the large number of relevant gathering and controlling systems and techniques creates an interface problem of vast proportions. Funneling the greatest amount of high quality (prepared/processed) information to key personnel would be the ideal solution. It is therefore critical that technologies be researched and demonstrated for the representation and interpretation of multiple, diverse data sources, for human interpretation of those data sources, and for powerful, deliberate/calculated and rapid interaction with that information. Our offering, then, is a integrated interactive perceptualization interface system: a sensorial combinetric integrator for enhanced expressivity.

We propose to research and develop a realistic plan for the development and implementation of an interactive perceptualization system for the purposes of enhancing the dynamic interaction between the human user and a set of complex and diverse data types. The end result of our work will be an interface imbued suit worn by military field personnel; the suit itself is the "interactive perceptualization system." The plan shall include defining the systems, architecture, and design engineering required to research, prototype, and demonstrate capabilities in the following areas:

• Integration and fusion of information from multi-modality human to computer input devices and their application to problems such as locating and neutralizing land mines or herding and manipulating mission specific micro-robotic entities.
• Innovative mechanisms to support interactive informatic environments.
• Development of computer to human multi-sensory data rendering and analysis.
• Refinement of experimental protocols for optimal utilization of the system.

We will design and develop a highly advanced, novel and application rich interface system for the Distributed Robotics Program. Successful interfacing with external sensors and actuators is grounded on a high degree of context/need specific malleability within the interface itself. Ability to transmit any information presented to the sensory technology back to expert personnel is primarily an issue of how intelligent in its reconfigurability, processing and rendering options the interface is.

We propose a system which takes advantage of the entire body's neural capacities for sensation, processing and yielding to consciousness information. Our system's multisensory interfaces will thus make maximal use of the 'feature extraction' properties of the human senses. That is, a neurological margin of maximally meaningful input from the outside can be rendered to each sensory system. The system will accordingly map to this margin for each of the involved senses; system resident artificial intelligence will allow the reconfigurability based on context need. 3-D visual displays, spatialized audiomorphic and tactile body surface significations and other methods will present information from the sensors to the expert. EMG-like sensors across muscle surfaces, foot activated pressure sensors, voice recognition, and specially signified body movements will all serve as methods for controlling actuators in real time as they perform work and send back information

The contractor shall develop project plans to provide insight into the various phases of the research.

The overall objective will be to develop one single ergonomic 'interactive perceptualization interface system' for simultaneously perceiving and expressing information.

The following specifications of the system, then, will be heavily structured by the dual requirements for

1. perception: perceptually enhanced experience through generation of a pansensory rendering of relevant information so as to provide an enhanced capability to discriminate between classes of complex dynamic interactions of information; and
2. expression: resultant tactical actions taken based on that information.

Interface is a two-fold project: an integrative interface system, based on our research, is one whose very structure and function weaves together these two requirements into one process which radically augments perception and expression in the context of a given need.

4.1. Perceptual: having gathered raw data from remote sensors and/or micro-robots, processes which bring the information from "out there" into the mind are computer to human elements of the interface.

1. The contractor shall provide the systems, architecture, and design engineering required to research, prototype, and demonstrate advanced interfacing technology for the rendering of computer information onto multiple human sensory systems to achieve and demonstrate a sustained perceptual effect (i.e., a sensation with a context).
2. The contractor shall provide the systems, architecture, and design engineering required to research, prototype, and demonstrate how the implementation of vision, hearing, and touch technologies can allow for simultaneous sensation of multiple independent and dynamic data sets that can be integrated physiologically into a single perceptual state. Incidentally, from an evolutionary neuro-information processing perspective this technology creates a new potentiality for response to perceptual awareness: it canalizes not a single response to a single stimulus, but rather multiple responses to multiple stimuli born of a single though multi-dimensional sensorial perceptual state.
3. The contractor shall provide the systems, architecture, and design engineering required to research, prototype, and demonstrate the combination of these different rendering modalities with somatotopic placement, in order to achieve and demonstrate spatial coding of the rendered information.
4. The contractor shall provide the systems, architecture, and design engineering required to research, prototype, and demonstrate enhancements to the interface system to match the human nervous system's ability to transduce, transmit, and render to consciousness the necessary information to interact intelligently with information.

Below is some of the thinking which grounds this (i.e., 4.1.4.) our fundamental operational design principle:

• Research in human sensory physiology, specifically sensory transduction mechanisms, shows us that there are designs in our nervous systems optimized for feature extraction of spatially rendered data, temporally rendered data, and textures.
• We will develop these interface techniques and technologies consistent with the basic neuroscience issues of modality, duration, intensity, distribution, frequency, spatial

• We can then identify the optimal perceptual state space parameters in which information can best be rendered. That is, what types of information are best rendered to each specific sense modality, a sense specific optimization of rendered information.

Computational Preparation of Rendered Data: after merely rendering then there is the honing to the nervous system phase (i.e., how to best render for the highest quality information).

Possible Sensory Input:

• visualization techniques.
• 3-D sound and graphics.
• sonoification of data: in these spatialized environments we can shift our attentional focus from source to source for real time comparison of multiple sets of data.
• Devices now exist which we will exploit so we can stimulate the sensation of pressure, vibration, texture and temperature.

4.2. Expressional: having cognized the information gathered, processes which bring the deliberate response of the human out to the world are human to computer elements of the interface

Notes:

• Physiologically Oriented Interface Design: the next paradigm of human-computer interface will optimize the technology to the physiology -- a biologically responsive interactive interface.
• paradigm of interface technology is based on new theories of human-computer interaction which are physiologically and cognitively oriented.

Abstracting function for the sake of machine resident intelligence and user characterization and quantitative/qualitative recording.

In short, the integrated perceptual/expressional suit is the interactive interface environment. It is essentially a "wearable computer." We believe that the optimal interface for the expert military personnel in the field implementing the capabilities of remote sensors and actuators is a fully "wired" body suit. A helmet complete with visual, auditory input and motion output as well as controller fitted gloves and boots are included. This is a primary 'enabling technology' critical to the success of militarily relevant operations in the field. The suit is itself a perceptual and expressional environment; information gathering and sending respectively. Militarily relevant information capability strives for the greatest volume, most diverse types and the most quickly ascertained information. Limitations inherent to traditional visual display and radio feedback renderers stifle the potential of a program as ideally robust as Distributed Robotics and Deminers.

Viewing the entire body as a perceptual and expressional technology opens up possibilities for exploiting the heretofore untapped richness and greater volumetric potential of its informatic capacities. Hence, we propose to develop an interactive environment incorporating new ways to render complex information to the user by optimizing the interface system to match the human nervous system’s ability to transduce, transmit, and render to consciousness the necessary information. Such a system will be based on the human user’s neural information processing that directly supports perception. A perceptualization environment such as we are proposing optimizes the human’s ability to discriminate and iteratively refine emergent patterns from any variety of sensor data, proactively.

The perceptual interface systems of the suit will be designed so as to optimize the salience and content of data sets. Some data which conventionally would be displayed visually might be processed so as to be perceived (in the suit) in a tactile or auditory manner. So the human may feel the pain of an injured war fighter, feel the sound, see the pressure and ultimately be able to reconfigure the rendering parameters of the interface based on the specific elements of a situation. Seeing colors may be more appropriate in one context whereas hearing them may be more suitable for another; many factors will determine the tailoring of rendered data: which data will be shunted to which renderer? Novel interface controllers are essential here.

Our success will have been to provide a completely integrated perceptual and expressional system which becomes the sense and agency of the expert controller.

Proposals