Proposal abstract
BAA# 97-41
Technical area: Distributed robotics
Lead organization -MindTel
Type of business: Other Small busness
contractors ref # N/A
Other team members:
Proposal Title: Bot-Masters
Technical POC. Dave Warner MD
Dr. Warner , Dave
2-212 Center for Science and Technology
111 College Place
Syracuse NY 13244-4100
315-443-5654 voice
314 443 1973 fax
800 950 0849 pager w/voice mail
davew@npac.syr.ed
Innovative claims
Deliverables
Key Personnel
We will design and develop a highly advanced, novel and application rich interface system for the Distributed Robotics Program: Bot-Masters. Work will be pursued along three axes: 1) Human-Bot Interface (HBI); 2) Communication; 3) Expert Training. Number 1 is the primary structure we are proposing to develop while 2 & 3 are its essential content functionalities. Successful interfacing with the bots is grounded on a high degree of context/need specific malleability in the interface itself. Ability to communicate any information presented to the bots back to expert personnel is primarily an issue of how intelligent in its reconfigurability, processesing and rendering options the interface is. Further, using this interface and already existing synthetic environment technology, experts may be well trained in the use of bots.
We propose a system which takes advantage of the entire body's neural capacities for sensation, processing and yielding to consciousness information. Bot-Masters' multisensory interfaces will thus make maximal use of the 'feature extraction' properties of the human senses. That is, a neurological margin of maximally meaningful input from the outside can be rendered to each sensory system. The system will accordingly map to this margin for each of the involved senses; system resident artificial intelligence will allow the reconfigurability based on context need. 3-D visual displays, spatialized audiomorphic and tactile body surface significations and other methods will present information from the bots to the expert. EMG-like sensors across muscle surfaces, foot activated pressure sensors, voice recognition, and specially signified body movements will all serve as methods for controlling the bots in real time as they perform work and send back information. Perception and expression--surveillance and control--of bot behaviors is thus the outcome of the work we here propose as a complete interface system for distributed centi-botics.
BOT-MASTERS
1) Interface
We believe that the optimal interface for the expert milititary personnel in the field implementing the capabilities of centi-bots is a fully "wired" body suit. A helmet complete with visual, auditory input and motion output as well as controller fitted gloves and boots are included. This is a primary 'enabling technology' critical to the succes of militarily relevant operations. The suit is itself a perceptual and expressional environment; information gathering and sending respectively. Militarily relevant information capability strives for the greatest volume, most diverse types and the most quickly ascertained information. Limitations inherent to traditional visual display and radio feedback renderers stifle the potential of a program as ideally robust as Distributed Robotics. Bot-Masters lifts these limitations.
Viewing the entire body as a perceptual and expressional technology opens up possibilities for exploiting the heretofore untapped richness and greater volumetric potential of its informatic capacities. Hence, we propose to develop an interactive environment incorporating new ways to render complex information to the user by optimizing the interface system to match the human nervous system's ability to transduce, transmit, and render to consciousness the necessary information. Such a system will be based on the human user's neural information processing that directly supports perception. A perceptualization environment like Bot-Masters optimizes the human's ability to discriminate and iteratively refine emergent patterns from any variety of sensor data.
The perceptual interface systems of the suit will be designed so as to optimize the salience and content of data sets. Some data which conventionally would be displayed visually might be processed so as to be perceived (in the suit) in a tactile or auditory manner. So the human may feel the pain of an injured war fighther, feel the sound, see the pressure and ultimately be able to reconfigure the rendering parameters of the interface based on the specific elements of a situation. Seeing colors may be more appropriate in one context whereas hearing them may be more suitable for another; many factors will determine the tailoring of rendered data: which data will be shunted to which renderer? Novel interface controllers are essential here.
Our goal is to provide a completely integrated perceptual and expressional system which becomes the sense and agency of the expert controller. Bot groupings/clusters will essentially become a tight mapping of human perception and expression at x distance. Reducing the perceptual field of a human in the suit interface to that of the bot sensors is a critical first step ("microfied telepresence"). The control systems would then need to be as "tight" as possible; that is, what the bots sense the human would percieve in real time and what the bots do would be a real time doing of what the human did. Essentially, through hardware and software systems the bots become extensions of human perception and control/expression. Sensory and control interfaces would thus need to be closely linked with expert behaviors; bot activity is a mechanizing of expert intention.
1.1 Possible Scenario
Controlling the bots at a distance is the essence of the project. Bots could be launched in an artillery or heavy artillery like fashion, or else deployed using tactical missile technology. A single bot-pod containing multiple bots would land at a specific place, decay, open, or explode releasing the bots and instantly creating a sensitized locality with functionality. Medibots we will call our little life saving friends. Once on site, employing their many and diverse microsensor technologies, they would gravitate to the site of need and go to work, each class of bots performing its own unique functions. For example, let's suppose a germ bomb has been detonated among a battallion of war fighters. Live contagion makes it very difficult for rescue personnel to move in with conventional anti germ methods. Protecting the rescue personnel is in itself an entire dimension of technological and expert necessity which needs to be layed out before rescuing can even occur. Bots would cut out this entire step and allow direct response to the actual problem: treating infected war fighters. Perhaps by specific movements of some fingers some bots would release a neutralizing chemical onto the infected personnel. By specific movements of other fingers, other bots would inject medicines into the bodies of infected personnel, and so on. Or maybe the entire group of different classes of bots could all be combined by expert control to form a powerful tugboat-like device which could pull surviving war fighters out of the dangerous area of exposure.
In both cases, the expert controller has within the Bot-Masters all the necessary interface commands to control either individual or grouped bots. For example, let's say the interface command to aggregate all the bots is a bodily movement: the expert controller interlocking her two hands and forming a double fist and then straightening the plane of her arms. Performing this command movement instantly turns the myriad bots into a single tugboat-like vehicle with some bots being the engine, others the wheels and still others the hooks.
To better comprehend the supererior facility and power of Bot-Masters, we may use an analogy from classic virtual reality interfacing systems to express the relationship between the Bot-Masters' interface and the bots in the field. Virtual reality glove drivers are the dynamic avatars in synthetic environments. By opening, closing or moving the hand specific kinds and degrees of movement of the users total perceptual field is simulated. In the same way, Bot-Masters' interface commands (e.g., movements, muscle flexings, vocalizations, etc.) induce behaviors of the bot avatars at a distance from within their own perception. The difference, of course, is that the bots are real and not virtual (though in training they will be virtual).
2) Information Communication
Communication between bots and humans is the content of information gathering and consequent control and adjustment along the axes of mission tasks. In order to connect inputs and outputs between human and bots so as to make them reciprocally meaningful (i.e., rendering and controlling) we maintain extremely powerful and cutting edge software to actuate the interface system. Neatools is the name of this exceptional and unrivalled software (www.pulsar.org/neattools/neattools.html). Neatools is a input/output program which essentially takes any input and is able to filter, process and transform it into any output (e.g., kind of pathogen rendered as a tactile pattern across the inner thigh, level of infection rendered as sound pattern throughout helmet, etc.) Hence, bot sensor datum are transduced, filtered through Neatools and then rendered to the human in novel and optimized ways. Compound bots (i.e., command induced aggregations of single bots) are also completely comprehendible by the software as number and variability of inputs is practically unlimited with Neattools.
Novel processing to render sensor information will be accomplished by innovative methods for data fusion enabling meaningful correlations across various input modalities. We will implement a data analysis subsystem designed to enhance the ways that relevant data may then be rendered optimally to the human controller's sensory modalities. Linear and nonlinear multivariate analysis tools will be utilized for the processing of multiple data sets in a variety of ways, including graphical anyalysis (e.g., phase portraits, compressed arrays, recurrence maps, etc.; see www.sci.sdsu.edu/People/Jeff/isosurfaces/isosurfaces.html) and sound editing (mixing, filtering). Automated trends and correlations using fuzzy logic may be performed in the background or in a post processsing mode.
Communication we aim to foster within a larger web-based collaboratory type structure.
Using collaboratory information technology such as TANGOsim (http://trurl.npac.syr.edu/tango/papers/tangowp.html) is our aim here. The collaborative, then, is a networked aggregate of expert knowledge and skill resources deployable as an entity, as needed. "TANGO is an integration platform which enables building useful and deployable Web-based collaborative environments. The system provides the means of fast integration of Web and non Web applications into one multi-user collaborative environment" (Walczak et al., 1996). Minimal numbers of humans deployed in a mission is made possible with bots. Collaboratory systems for on-line interaction, such as TANGO boost this project up several notches. Our interface solutions are always grounded on collaboratory friendly models of informatic interaction. Our end goal will be a single and highly integrated interface with diverse multisensory rendering and expressional devices integrated into several control environments. Thus, the expert in the field can at the same time co-control a set of bots with a distant, on line, collaborating medical expert (e.g., a rare specialist in retrovirus vaccine assembly).
Central to the concept of a highly effective human to bot interface communication will be the making of generalized behavior controllers which are generically designed so as to allow interaction with a task heterogenous set of bots. Human to bot communication systems allow the human to simultaneously control functionality at many levels:
Communication then is massively augmented within Bot-Masters because of our Neatools software where any input can be "morphed" into any output/rendering.
3) Training
Fewer and more powerfully trained experts is a possible outcome of Bot-Masters. Deploying only a few, or even one, experts imparts greater precision and efficiency into a task. Thus, information exchange between personnel and systems is focused.
Beginning with bot release, a highly specialized trained individual or group of individuals (kind of like K9 cops) would begin interfacing/controlling the bots from within extremely advanced and powerful interface mechanisms. Suits embedded with the many and diverse interface devices would be ideal as they would allow typical battlefield behaviors on the part of a military personnel (e.g., hiking, running, crawling, hiding, etc.)
Proper preparation in the use of bots, we believe, will require the training of both expert and bot together in very tight correspondence employing the powers of full simulation environments (e.g., VRML, Meme, Muse). Again, similar to the K9 police officer model (coproach), those selected individuals who will be the bot lords will receive advanced specialized training beyond the conventional cirricula of their respective military discipline. Developing the sensory and expressional agilities needed to be effective in a live time critical mission will be massively augmented by high quality, real time simulation systems. Additionally, simulation exercises will be extraordinary indicators for the design and functionality of the end systems themselves. Bots may be simulated with artifical life algoryhthms etc.
Summary of the System
The perceptualization environment; the Bot-Masters will integrate several vital components of an interactive information environment. Key elements include multisensory rendering systems, advanced human input devices, and an array of
computational techniques that transform the diverse data types into perceptible patterns which enhance human capacity to perceive meaningful signals in a "sea of noise." A comprehensive set of visual, aural, tactile, proprioceptive, somatosensory, and olfactory rendering devices will be integrated into the system to give the user an integrative experiential interaction with the complex data types. The system will also integrate several unique input systems that allow the user to have a multiplicity of
interaction options; in this way, the user will be able to feed back the perceived significance to the system for further enhancement. The Bot-Masters will be a tool to interactively experience a wide variety of natural and unnatural perceptualization techniques.
Historically verified, our research has centered precisely on how to capture novel and multi-modal signalings from the human body and use these intention driven signals to control an informatically based system of end functionalities. Our approaches have been unique and innovative in the contexts of interface for both disabled user access, and professional expert knowledge system access (e.g.,www.pulsar.org/springwebb/i3/i3page.htm,www.pulsar.org/springwebb/grokbox/gb.htm respectively).
Statement of Work
TASK 1
A plan of implementation and demonstration will be determined in the initial
period of Phase 1.
TASK 2
We propose to develop and integrate a set of advanced human-to-computer input
devices into a single interface system. This integration of data input devices
into a single system will include EEG, EMG, EOG, (bioelectric signals from
brain, muscle and eye), dynamic bend sensors, pressure sensors, audio and
video digitizers, and other devices as they are relevant. We will develop
methods for data fusion to enable meaningful correlations across various
input modalities. The devices will be connected to an external module which
will route the data both to a central multitasking server and to the rendering
subsystem for immediate feedback. The server will be intelligent enough to
automatically implement a custom configuration of input-device parameters,
interface functionality, and relevant records based on the device(s) connected
and the identity of the operator(s) currently on the system.
TASK 3
We will integrate a comprehensive set of visual, aural, tactile, proprioceptive,
and somatosensory rendering devices into the Bot-Masters system to give the
user an integrative experiential interaction with the complex data types.
We will utilize interface technology that renders computer information onto
multiple human sensory systems to give a sustained perceptual effect (i.e.,
a sensation with a context). We propose to combine these different rendering
modalities with somatotopic placement, thus providing for spatial coding
of the rendered information. We will demonstrate how the implementation of
vision, hearing, and touch technologies can allow for simultaneous sensation
of multiple independent and dynamic data sets that can be integrated
physiologically into a single perceptual state.
TASK 4
We will develop an interactive environment that combines new ways to render
complex information with the advanced human-to-computer input devices.
Interactive interface technology will be implemented such that it renders
content specific information onto multiple human sensory systems giving a
sustained perceptual effect, while monitoring human response in the form
of physiometric gestures, speech, eye movements, and various other inputs.
We will refine quantitative measurement of activity during purposeful tasks.
TASK 5
We will refine the interface system to match the human nervous system's ability
to transduce, transmit, and render to consciousness the necessary information
to interact intelligently with information. We will implement a data analysis
subsystem designed to enhance the ways that relevant data may then be rendered
optimally to the operator's sensory modalities. Linear and nonlinear multivariate
analysis tools will be utilized for the processing of multiple data sets
in a variety of ways, including graphical analysis (phase portraits, compressed
arrays, recurrence maps, etc.) and sound editing (mixing, filtering). Automated
detection of trends and correlations using fuzzy logic may be performed in
the background or in a post-processing mode. The system will be designed
so that the user may then be alerted by the system if it detects areas worthy
of further investigation
TASK 6
We will design an experimental protocol to evaluate the Bot-Masters system
for providing mission critical support. We will interact with the appropriate
agencies to ensure the specific requirements of this task to meet mission
critical objectives are met.
TASK 7
We will generate an ongoing reporting and demonstration of the developing
functionality of the system. We will provide seminars and workshops for the
purposes of disseminating and furthering this work.
Integration Into the Real World
The greatest potential of this interactive perceptualization environment will be realized when it is integrated into a Web-based collaboratory. Using the multiplicity of information resources available through the Web and using the latest perceptualization techniques, the Bot-Masters will constitute a powerful vehicle to explore and exploit the riches of "cyberspace."