FROM BRAIN WAVES TO VR:

The Theory and Practice of Physiologically Interfacing a Computer.

David Warner graduated from SDSU in Spring 1988, with a degree in Physical Science. In the Fall he entered the combined MD/Ph.D program at Loma Linda University Medical Center. Dr. Warner's passion as an undergraduate was human expression. Specifically, he sought to learn how a thought becomes an intention for expression and then how the physiology of the body facilitates that expression through some medium. The medium he chose was information systems: informatics. At LLUMC Dr. Warner's doctoral research was in the Department of Neurophysiology. The following synopsis gives the trajectory Dr. Warner has followed in constructing a (target-user) viable physiological foundation for human-computer interaction: "Its a new kind of thinking-you can use your biological system as the controller." The research began in Loma Linda's Department of Clinical Neurology and is continuing as a Nason Fellowship at Syracuse University's Northeastern Parallel Architectures Center (NPAC).

The following outline is understood from within a more basic inquiry into diverse and numerous instances of somatic bioelectric information. It is the body’s processing of such information Dr. Warner sought to learn the mechanisms of. Heading towards more ambitions applications of virtual, or perceptualization technologies, much has already been learned about physiological bioelectrics. For quadreplegics, for example, facial movements are one of the few actions that can be willfully performed with the body. Access this and a bi-directional feedback-a biocybernetic- loop can be established between a person’s syntactic physiological output and the capacities of an informatic system. Then an interface is in place which makes the electrical output meaningful to a machine: the interface increases the throughput from the mind through the body to the computer; the computer processes it back through the body to the mind. Generally, this only enhances communicability and richness of interactions between people and machines.

Related to this is the hope of evolving clinical therapeutics, as the same information may be interacted with by both patients and physicians. There is a measured constant which forms the basis for a much more rigorous and efficient practice of medicine. This outline traces the steps toward the first physiological human-computer interfaces. The context for the application of technology to human need has been primarily neurological and rehabilitative medicine. There is a fourfold commitemt to technological applications in medicine and education.

1. Quantitative Assessment

2. Agumentative Communication

3. Enviornmental Control

4. Education

The specific technologies (Introduced in Bold Face) brought to bear on the problem of developing more powerful and multi-modal interface designs are also mentioned. Dr. Warner’s work forged an altogether different path for how high technology could be brought into the service of medical need. Consequently, the story is also a record of pioneering technologies such as Virtual Reality graphical rendering systems, physiological signal sensors and 3-D sound devices and enviornments themselves being applied in unanticipated and surrogate fashions. Over time Dr. Warner and his collegues would develop cheaper, more accessible and more efficiently powerful tools to substitute the more expensive technologies, not their own.

1988-1990: Exploration of Neurophysiological Bioelectrics and Imaging :

1988

Dave Warner begins medical school: combined M.D./Ph.D program.

1989/90.

Through Loma Linda Chairman of Neurology, Dr. Douglas Will, Dr.Warner gains use of the QSI 9000 Electroencephalograph (EEG). (Also availble was an MEG, Magnitoencephalograph, and a Functional MRI). Experimentation with healthy and nueropathic subjects begins.
Novel application of non-linear dynamical analysis to the rendering and clinical interpretations of EEG and other data from

Parkinson’s Disease (most extensively treated) e.g., Ron Williams’ inter-ictal siezure and seizure data was used to generate values of correlation dimension, theorized as a measure of complexity.
Alzheimer’s Disease
Epilepsy
Acute Psychosis

To create both a structural and functional neurological mapping, both EEG and MRI data was fused together and rendered in several dimensions, as opposed to 2. Non-lineaer dynamics relates to the space-filling properties of systems. Rather than mere spike marks across a page, the data from the EEG and MRI was rendered with Scrivner’s AVS (Application Visualization Systems) Stellar software on a Stardent 9010 graphics workstation.
Spatio-Temporal Isosurfaces are 3-D color-coded surface contours of the dynamically rendered EEG information. This is a novel and highly powerful method of interpretation. A qualitative image based ouput is generated by previously clinically inaccessible information. In its rendering by the computer it becomes highly complex quantitative data. This method simply provides for a greater amount of meaningful information to medical perception in diagnosis and treatment. Jeff Sale was central to conceptualization and technical fleshing out of this manner of rendering physiological data. To see these beautiful images along with technical details of how they are created click here.

1989-91: Motion and Pressure Sensing Devices

1989

Dr. Warner and colleague Jeff Sale met with VPL Labs. Motivated by NASA Technical Briefs dealing with the earliest manual optical attenuation devices, they procured the famous Data Glove. The glove was seen as a potentially powerful tool for [historically novel] quantitative measurement of Parkinson’s disease and movement disorders in general. For the first time, all movement during a standard neurological exam would be quantized and recorded. The effects of physical and chemical therapuetics could be exactly measured. A "taxonomy of tremor" is now possible.
Polhemus 3-Space Tracker with 6 degrees of freedom: x,y,z and yaw, pitch, roll.
Both the Data Glove and Polhemus operated on the Macintosh IIci running the respective company’s proprietary software.

Some American Neurological Association abstract titles demonstrate the powerful applications of the various technologies at the time:

-"The VPL Data Glove as an Instrument for Quantitative Motion Analysis" (A.D. Will, D.J.Warner, G.W. Peterson, S.H. Price, E.J Sale: 1990; Department of Neurology, LLUMC).

-"Quantitative Analysis of Tremor and Chorea Using the VPL Data Glove" (A.D. Will, D.J.Warner, G.W. Peterson, S.H. Price, E.J Sale: 1990; Department of Neurology, LLUMC).

-"The VPL Data Glove as a Tool for Hand Rehabilitation and Communication" ().

-"The Data Glove for Precise Quantitative Measurement of Upper Motor Neuron (UMN) Function in Amyotrophic Lateral Sclerosis (ALS)" (A.D. Will, D.J.Warner, G.W. Peterson, S.H. Price, E.J Sale: 1990; Department of Neurology, LLUMC).

The data glove was also applied in Gesture to Speech systems. This application is very promising for the various Aphagias. By establishing a patient specific set of hand gestures corresponding to phonetic structures a patient could regain some level of verbal communicability.

Several Loma Linda M.D. students did a great amount of data collection with which to establish the foundation for further work.

1990/91

Tekscan Pressure Sensors used for 2 and 3 dimensional color-coded visual renderings of presssure data. To indicate pressure criticality the image would be morphologically isometric in the negative. The degree of peak height is the significant unit. Used for

Patient gait analysis
Prosthetic device fitting
Bed and wheel chair sores of confined patients

1991

Published "The Neuro-Rehabilitation Workstation: A Clinical Application For Machine-Resident Intelligence" (D. Warner, J. Sale, S. Price; Neurology Research Center/NERT, LLUMC).

The paper outlined a plan which "integrates multiple data aquisition devices, interface technologies, advanced analytical techniques, and multi-sensory rendering capabilities." The NRW represents the first serious synthesis of all the research into

1. Captured Physiological Output

2. Quantification of Motion

3. Complex Graphical Rendering Systems

1990-1993: Enviornmental Control Systems for Wellness Response

Convulvitron Chair was a 3-D sound and contour pliant somatic interface. The work was based on possible psychological and emotional effects of sound. If managed and applied through the chair a subject could be brought into subtle altered states of consciousness by various specialized 3-dimensional renderings of sound.
Dream Rider/Dream Wave was another of these analog sound devices for adjusting frequencies and intensities in a full length body chair.
Motion Platforms for designing realistic virtual worlds. Also, the platforms allowed experimentation with vestibular system sense of orientation in simulacrum space. Motion sickness was a problem early on for VR interfaces. Visual system perception of space events and movements conflicted physiologically with the vestibular system’s perception of no movement resulting in naseua. Like the chairs, these platforms were experimented with heavily.

1992-1995: Physiological Signal Processing and the Closing of the Bio-cybernetic Loop.

1991

Dr. Warner was given the opportunity to test an EMG driven interface called Biomuse on a single day in March. Biomuse was the first instance of a technology which allowed bi-directional feedback response between a computer and a human physiological command such as that created by a flexed muslce. In its original form Biomuse transduced a leg/arm EMG into a musical tone. Dr. Warner had another idea:

LLUMC Pediatric Rehab patient Crystal Earwood (18 months) was the youngest person to ever control a computer with voltage output from her eyes. Dr. Warner’s lab purchased Biomuse in Fall of 1992. The physiological platform for human-computer interaction had been initiated in earnest.

First media coverage of the SAMARITAN project.

1992/93

Jo Johansen was writing code for the Next Wave Chair (Enviornmental Control) and was asked by Dr. Warner to write some code allowing one to moniter EMG signals and then connect those signals as gestures which could command a computer. Bio-enviornmental Control (BEC) substituted Biomuse’s regular music output software and allowed the conversion of EMG into gesture recognition.
Neat Software supplanted BEC and continues to be revised as the other evolving systems have dictated.
Biomuse and all its associated rehab devices was run on a 386 PC.

1993-1994

The work reached its most elaborate and successful point with a seven year old C1 quadreplegic, Ashley Hughes. Neat Software was in fact designed right from Ashley’s facial muscle capacities. The following technologies were used in developing a physiological interface for Ashley:

Sega Super Mario Bros. was a popular video game of the time.
MEME was developed by programmer Marc DeGroot, MEME was the first virtual reality software implemented in medicine. It gave Ashley and others the experience of flying through a 3-dimensional space while interacting with various objects in that space.
Biocar was purchased at a local Radio Shack. Biocar began as a basic remote control car. The remote control function of the car was redesigned to fit the EMG signals from Ashley's face. This was accomplished by Wala Wala University engineers Jerry E and Todd Anderson. Together with Cindy Cyberspace, Biocar allowed what is known as Telepresence. Telepresence is the phenomenon wherein perception of remote spaces, or the experience of being in those spaces is facilitated by a camera. Biocar was fitted with a camera whose viewing was channeled back to Ashley and generated in a small screen inside a special pair of glasses.
Cindy Cyberspace is a styrofoam bust with cameras for eyes and microphones for ears Ashley could see and hear any space remotely with special glasses and headphones.

A very significant ‘quality of life’ enhancing feature of the last three elements of the list is their allowing others to interact with Ashley. Mario Bros. allowed Ashley to simultaneously play with her peers. Biocar and Cindy Cyberspace greatly enticed people around Ashley to play with her. With these off the shelf technologies other children would be inclined to pilot Cindy around Ashley's house and yard while she sat stationary in her wheelchair and directed them to climb a tree or swim in the pool. As Ashley controlled Biocar, her friends and family members would follow it around the house and move it from place to place. In fun, they could put it in practically any space they wanted Ashley to "be" in and she would immediately be their becuase of telepresence. Such technologies begin to break down the common barriers normal children run up against when confronted by disabled peers.

1994/95

Development of THNG 1. THNG 1 is a 4-channel EMG signal transducer which supplanted Biomuse. Biomuse is a $20K device. This price would prevent most potential users form having access to it. THNG 1 is more efficient, compact and less than $200. Former chair designer Solomo Maturnes developed THNG based upon how Biomuse was being used for rehab.

1994

Intra-operative data glove/EEG monitering of Parkinson’s patients having a pallidotomy or thaladotomy. As the surgeon went through the operation, awake patients were asked to perform hand movements which were measured. The data clearly showed the positive effects in the post lesioned patient. Neurosurgeon, Dr. Iacona recognizes and praises Dr. Warner's solution to a longstanding depravity of solid quantitative measurements in neurosurgurgical outcome.

M.D./Ph.D student Sandy Kuniyoshi was doing her research on this procedure. Dr. Warner was instrumental to her program as he had done so much work with the physiological and mechanical assessments of biological function.

1995

Completion of M.D. Program

1995-Present: Nason Fellowship, Syracuse University

Consistent with the original goal of giving the disabled access to information systems and ultimately the World Wide Web Dr. Warner has succeeded in his work with

Eyal Sherman: Like Ashley, Eyal is a C1 quadreplegic. With similar technologies as were used at LLUMC and going beyond Eyal is one of the first quadreplegics to navigate the WWW using physiological inputs from his face.
July Jacoby: July is limited cognitively, bahaviorally and linguistically by neural retardation. Smart Desk is a project instigated by Dr. Warner to give July greater power with the limited expressional capacities she has. Smart desk is the state of the art in multi-sensory input technology for interaction with informatic systems.

In terms of rehabilitation, Dr. Warner’s research established a greater degree of patient success. The bio-cybernetic loop of machine feedback engages a patient in a motivational escalation. As one’s efforts get closer and closer to the aim of a task (e.g., playing Mario Bros. against an opponent, controlling Biocar etc) motivation increases and reduces the amount of time required to perform the task. Furthermore, while this "play" is occurring real neural rehabilitation and relearning entrainments are happening. As the work evolves it is certain that the regular pace of rehabilitation medicine will be significantly increased.

At a more fundamental level, this work moves towards a novel and powerful analytic of learning itself. The bio-cybernetic loop is a structure in which a mind may embed itself. If an interface between the mind and an informatic is both sufficiently "tight" and accurrate, learning and performance are massively augmented. Biomuse was the opening of the technological possibility for an Interventional Informatic. The entire continuum of Dr. Warner’s neurophysiological research is punctuated by Biomuse. The bio-cybernetic is engendered with Biomuse technological thinking. However, the foundational research disclosing physiological signal processing was the essential beginning.

PULSAR