Integrating TNG, NEAT and SMART DESK technologies, the objective of Grok Box is to render simultaneous multisensory information to the human body. This technology is based on a new paradigm of human computer interaction known as Biocybenetics. Grok Box systems will render information to the visual, auditory, and tactile senses of its user. Based on principles of human sensory physiology, Grok Box' multisensory interfaces will make maximal use of the 'feature extraction' properties of the human senses. That is, a neurological margin of maximally meaningful input from the outside can be rendered to each sensory system. Rendering systems of Grok Box will accordingly map to this margin for respective senses. Variably located 3-D visual displays, spatialized sounds, precision tactile body surface rendering serve as output; EMG-like sensors across muscle surfaces, foot activated pressure sensors, voice recognition systems, and hand held devices allow user input. Perception and expression of greater quantities of complex information will be the end value to expert knowledge users (e.g., battlefield ER physician)

The GROK-BOX

Real world environments are characterized by an ever increasing array of heterogeneous sensors and sensing modalities. This mixture of potentially vital but mostly obscure data, in their native form, exceeds the human limits of integrative sensibility. Attempts to address this information overload problem by an array of methods ranging from new AI "tricks" to graphical techniques and virtual reality interface systems have achieved only limited success. In reviewing projects addressing this problem, it has become apparent that most of those who have tried to solve the problem have focused on the technology. Such projects addressing human factors aspects of the problem appear to be attempts to develop better "Reality Renderers."

Plan

We propose, instead, to adopt an approach with greater potential to resolve this "sensor/sensory" overload. That is, we propose to develop an interactive environment incorporating new ways to render complex information to the user by optimizing the interface system to match the human nervous system’s ability to transduce, transmit, and render to consciousness the necessary information. Such a system will be based on the human user’s neural information processing that directly supports perception. A perceptualization environment could be built that optimizes the human’s ability to discriminate and iteratively refine emergent patterns from any variety of sensor data.

Description of the System

The perceptualization environment, the "GROK-BOX," will integrate several vital components of an interactive information environment. Key elements include multisensory rendering systems, advanced human input devices, and an array of computational techniques that transform the diverse data types into perceptible patterns which enhance human capacity to perceive meaningful signals in a "sea of noise." A comprehensive set of visual, aural, tactile, proprioceptive, somatosensory, and olfactory rendering devices will be integrated into the system to give the user an integrative experiential interaction with the complex data types. The system will also integrate several unique input systems that allow the user to have a multiplicity of interaction options; in this way, the user will be able to feed back the perceived significance to the system for further enhancement. The GROK-BOX will be a tool to interactively experience a wide variety of natural and unnatural perceptualization techniques.

Tasks

• Task 1: To plan and deliver demonstrations of the core technologies at the rate of one every six months.
  Task 2: To develop a set of advanced human-to-computer input devices into a single interface system.
  Task 3: To integrate a comprehensive set of visual, aural, tactile, proprioceptive, and somatosensory rendering devices into the GROK-BOX system to give the user an integrative experiential interaction with the complex data types.
  Task 4: To develop an interactive environment that combines new ways to render complex information with the advanced human-to-computer input devices.
  Task 5: To optimize the interface system to match the human nervous system’s ability to transduce, transmit, and render to consciousness the necessary information to interact intelligently with information.
  Task 6: To evaluate the GROK-BOX system for providing mission critical support.
  Task 7: To generate an ongoing reporting and demonstration of the developing functionality of the system.

Project Phases, Milestones and Deliverables

The proposed GROK-BOX project will take place in three phases: (1 phase per year)

• Tasks 1-3 will be the focus of Phase 1.
  Tasks 4-5 will be the focus of Phase 2.
  Task 6 will be the focus of Phase 3.
  Task 7 will extend throughout the project.

All tasks will be documented on an ongoing basis and delivered as Web pages, papers, and conference presentations. Each phase will consist of 2 demonstrations per year, as described below.

Phase 1:

• Deliverable 1: Demonstration of the functional integration of relevant human-to-computer input devices into the system. (approx. 6 months)
  Deliverable 2: Demonstration of various multisensory rendering systems integrated into the system.

Phase 2:

• Deliverable 3: Demonstration of an integrated system of human to computer input devices with the multisensory rendering systems. Deliverable 4: Demonstration of an interactive, experiential environment optimized for intelligent interaction with information.

Phase 3:

• Deliverable 5: Evaluation of the effectiveness of the GROK-BOX as a system capable of supporting mission critical support.
  Deliverable 6: Final report.

Statement of Work

TASK 1

• A plan of implementation and demonstration will be determined in the initial period of Phase 1.

TASK 2

• We propose to develop and integrate a set of advanced human-to-computer input devices into a single interface system. This integration of data input devices into a single system will include EEG, EMG, EOG, (bioelectric signals from brain, muscle and eye), dynamic bend sensors, pressure sensors, audio and video digitizers, and other devices as they are relevant. We will develop methods for data fusion to enable meaningful correlations across various input modalities. The devices will be connected to an external module which will route the data both to a central multitasking server and to the rendering subsystem for immediate feedback. The server will be intelligent enough to automatically implement a custom configuration of input-device parameters, interface functionality, and relevant records based on the device(s) connected and the identity of the operator(s) currently on the system.

TASK 3

• We will integrate a comprehensive set of visual, aural, tactile, proprioceptive, and somatosensory rendering devices into the GROK-BOX system to give the user an integrative experiential interaction with the complex data types. We will utilize interface technology that renders computer information onto multiple human sensory systems to give a sustained perceptual effect (i.e., a sensation with a context). We propose to combine these different rendering modalities with somatotopic placement, thus providing for spatial coding of the rendered information. We will demonstrate how the implementation of vision, hearing, and touch technologies can allow for simultaneous sensation of multiple independent and dynamic data sets that can be integrated physiologically into a single perceptual state.

TASK 4

• We will develop an interactive environment that combines new ways to render complex information with the advanced human-to-computer input devices. Interactive interface technology will be implemented such that it renders content specific information onto multiple human sensory systems giving a sustained perceptual effect, while monitoring human response in the form of physiometric gestures, speech, eye movements, and various other inputs. We will refine quantitative measurement of activity during purposeful tasks.

TASK 5

• We will refine the interface system to match the human nervous system’s ability to transduce, transmit, and render to consciousness the necessary information to interact intelligently with information. We will implement a data analysis subsystem designed to enhance the ways that relevant data may then be rendered optimally to the operator’s sensory modalities. Linear and nonlinear multivariate analysis tools will be utilized for the processing of multiple data sets in a variety of ways, including graphical analysis (phase portraits, compressed arrays, recurrence maps, etc.) and sound editing (mixing, filtering). Automated detection of trends and correlations using fuzzy logic may be performed in the background or in a post-processing mode. The system will be designed so that the user may then be alerted by the system if it detects areas worthy of further investigation

TASK 6

• We will design an experimental protocol to evaluate the GROK-BOX system for providing mission critical support. We will interact with the appropriate agencies to ensure the specific requirements of this task to meet mission critical objectives are met.

TASK 7

• We will generate an ongoing reporting and demonstration of the developing functionality of the system. We will provide seminars and workshops for the purposes of disseminating and furthering this work.

Integration Into the Real World

The greatest potential of this interactive perceptualization environment will be realized when it is integrated into a Web-based collaboratory. Using the multiplicity of information resources available through the Web and using the latest perceptualization techniques, the GROK-BOX will constitute a powerful vehicle to explore and exploit the riches of "cyberspace."

NORTHEAST PARALLEL ARCHITECTURES CENTER (NPAC)

The Northeast Parallel Architectures Center (NPAC) is an advanced computing center at Syracuse University in Syracuse, New York. Directed by Geoffrey Fox, NPAC specializes in High Performance Computing and Communications (HPCC), parallel processing, distributed computing, computational science, education, and technology transfer through the InfoMall program. The scope of NPAC has broadened to include world class computational science research and education, and most recently research and development in high performance communications with particular focus on support for the National Information Infrastructure (NII). At NPAC there is a strong emphasis on computational science applications where one seeks to obtain "real solutions to real problems." NPAC's InfoMall technology transfer program puts high-performance computing and communications (HPCC) to work in industry. Other major projects include research and development in the areas of parallel languages and parallel compilers, including work on High Performance Fortran (HPF; a standardized parallel version of Fortran); distributed computing; parallel database technology; integration of relational databases with the NII; parallel algorithms; distributed and cluster computing; and networked digital multimedia applications providing Information, Video, Imagery, and Simulation on Demand.

Programs of NPAC

NPAC, with its academic partners at Syracuse University and at institutions nationwide, engages in interdisciplinary research, development, education and technology transfer in high-performance parallel computing. NPAC has also developed an active research and development program in the new arena defined by the convergence of high performance computing with high speed networks that handle digital-analog conversions and with distributed and collaborative networking tools. NPAC Core Technology Projects provide the base enabling technologies upon which NPAC builds its applications and collaborations.

INSTITUTE FOR INTERVENTIONAL INFORMATICS (I³)

Institute for Interventional Informatics (I3) is a non-profit organization. I3’s mission is to utilize information and information technology in socially responsible applications that improve quality of life. I3 has gained international recognition for the primary role it plays is a developing interface systems for severely disabled persons. I3 is also known for being a liaison between the medical community and high-tech development companies. I3’s role is to provide a "reality based" application assessment of new technologies for possible medical and educational uses. Experience in interactive human computer interface systems, combined with knowledge of clinical medicine, allows the researchers at the Institute the unique opportunity to exchange relevant information between the high-tech industry and the medical and educational communities, specifically by implementing off the shelf technologies that were developed for the military, entertainment, aerospace industries in medical and educational applications.

Currently Active Projects of I3

Advanced instrumentation for the acquisition and analysis of medically relevant biological signals. Advanced information systems which augment the general flow of medical information and provide decision support for the health care professional. · Public access to health information databases designed to empower the average citizen to become more involved in their own health care. · Advanced training technologies which will allow the rapid dispersion of newly developed techniques. New interface devices for persons with disabilities. · Educational systems that adapt to the users ability to learn.

Current NPAC–I3 Collaborations

Over the past year, NPAC and I3 have teamed together to develop applications of Web technologies in Health Care, Communication and Education. Applications specifically developed to Improve Quality of Life. The focus of this collaboration has been on the design of intelligent Web-based interface systems. Web technologies provide a unique opportunity to rapidly develop human-computer interface systems. Such systems provide a rich medium for augmented human expression thus enriching human-human communication.

The following Web systems are currently being developed:

• CareWeb: A prototype system developed for healthcare is being developed in the context of Telemedicine and Distributed Medical Intelligence.
  NeatTools: A Web based Expressional interface system being developed to enable disabled persons to express themselves fluently. Pulsar: A free Web service for disabled Web users. A Web-based resource repository providing free software, pointers to inexpensive interface hardware and intelligent directories of resources, support groups and other relevant information for disabled Web users. GROK-BOX: A Web-based collaboratory for interactive perceptualization. An instrumented environment for experiencing and communicating complex information.
  SmartDesk: A instrumented learning environment which allows for dynamic tracking of a "learner’s" navigational trajectory through a Web-based content delivery system.

These projects enable individuals to create new educational opportunities, employment opportunities and increase their socialization through their cultural integration into the information society. Integrating interactive Web based systems empowers disabled users to become more involved in their world.