Figure 1: A schematic cycle with human interaction
| DESIGN AND IMPLEMENTATION OF AN EXPRESSIONAL INTERFACE WITH SENSOR DEVICES AND WEB-BASED SOFTWARE (August, 1996) |
| Anna Berlin |
|
|
|
| Jiangang Guo |
|
|
|
This project involves the design and implementation of a Java version of the current MS-DOS version of NeatTools for the World Wide Web. NeatTools[1] is a visual programming language that creates programs for data collection, gesture recognition, control of external devices, virtual world control, and perceptual modulation. Data input was collected via various input devices designed for the NeatTools hardware/software interface. These devices were constructed specifically for that of the physically disabled user. This project focused on the case of a quadraplegic user, Eyal Sherman. Applications of the NeatTools technology included ability of the user to control video games and write text, via a Java-based text triangle.
1 Introduction
Today's ever increasing advancements in computer technology have cleared avenues for faster, more efficient and accessible methods of facilitating communication with disabled users. One of these methods is the NeatTools program which was initiated by the Institute for Interventional for Informatics[2]. NeatTools is an interface between hardware and software which opens new channels of communication and commands between the user and the computer.
1.1 NeatTools
There are two main types of hardware designed in association with the Neat software: one measuring voltage specialized for myoelectric signals (Thing1,TNG1) and the other measuring variable resistance from sensors (Thing2,TNG2). This hardware was designed to be versatile so as to accept a wide array of sensory inputs and cost efficient so as to be easy to use and easy to make. The software allows for easy manageability of data coming in from various input devices. The incoming signals are transformed into a visual portrayal of the user responses. This feature is useful insofar that the user can correlate his motions to the changes in his cyber environment.
1.2 Eyal Sherman
Currently, the efforts of the project have been focus on the case of Eyal Sherman[3], a 14 year old quadraplegic who is able to control only his facial muscles. The preliminary work was done with electromyography (EMG) to sense potential differences resulting from various muscle movements. Previously, EMG has been successfully implemented to obtain signals on the computer and to navigate a remote controlled vehicle, simply by making facial gestures. In the course of this project, various other sensors were used including pressure, light and magnetic sensors, all of which will be described and compared in later sections.
1.3 Visual Interpretation
The proceeding first section discusses the biological hardware interface with the NeatTools software. This section will include a discussion of the various input mechanisms that were prototyped and tested during the research program and other future possibilities of more efficient and consistent input devices. The next section focuses in on the computer interface with sensor data, specifically that of Java programming and processing of incoming signals. Finally, the third section involves the rendering of the computer data and possible multisensory display that the user can respond to in real-time.
2 Sensory Interface
A basic problem of certain disabled patients is the limited number of usable channels of communication available. In effect, the number of possible methods of input information and commands independently and willfully into a computer becomes limited to a few manageable motions. Hence, the need for the NeatTools program becomes apparent -- the expansion and increase of input channels from a user of limited capabilities. NeatTools is a Java-based visual program that provides a graphic environment for processing data collected from various input devices that are connected to the user. Currently, up to 16 data channels can be designated to various input and output mechanisms on NeatTools and such advantageous displays include the multichannel oscilloscope-type array which can be used to train the user to activate particular channels[4].
The input devices used with NeatTools are external devices that are modified specifically for the the user so as to employ whatever controllable actions he can make. Such input devices vary from commercial joysticks to the "Thing" devices, which were developed for NeatTools application. The Thing 1 (TNG1) device, for example, is a 4 channel EMG signal detector, developed at the Institute for Interventional Informatics . TNG1 utilizes the mechanisms of muscle contraction by the changes in the potential difference of the muscle tissues. These potential differences can be picked up by the EMG electrodes which are then connected to TNG1 and NeatTools. In turn, up to four different channels can be initiated by NeatTools to give a visual display of the muscle movements in the user. These channels can then be converted by the computer into commands for a set number of rules, a program, and/or communication via text and possible audio generation.
In the case of Eyal Sherman, the task of increasing the number of channels increases in difficulty not so much due to any limitations of the NeatTools program, but rather the limitations of Eyal himself. The workable area of the user is focused on the facial gestures which Eyal has sole control over. The method of exploiting the few accessible biological channels that he has is centered then, to the available input devices that can be made around and on his facial muscles so as to obtain input data. Such devices are discussed below.
2.1 Electromyography
Electromyography (EMG) is the monitoring of muscle contraction and movement via sensing the voltage changes resulting from muscle activity. We used this technology used initially for sensing the facial movements of Eyal. By placing electrodes on areas of maximal movement (e.g.. cheeks, and forehead) and connecting them to TNG1 which measures and processes myoelectric voltages, signals could be generated and input into NeatTools software.
2.1.1 Velostat
Electromyography (EMG) is the monitoring of muscle contraction and movement via sensing the voltage changes resulting from muscle activity. We used this technology used initially for sensing the facial movements of Eyal. By placing electrodes on areas of maximal movement (e.g.. cheeks, and forehead) and connecting them to TNG1 which measures and processes myoelectric voltages, signals could be generated and input into NeatTools software.
2.1.2 Cadmium-Sulfide photocells
Cadmium-sulfide (CdS) photocells (Radio Shack cat.# 276-1657) were used with 22K ohm resistors to make photosensors which decreased resistivity (increased conductance) with an increase of light on the photocell. These sensors were used with TNG2, measuring voltage, by positioning the photocells on Eyals face.
2.1.3 Magnetic
By using a Hall Effect Transducer (HET) which detects changes in a magnetic field, facial motion could be recognized by placeing a rare earth magnet on the appropriate facial area(s) muscle and noting movement with a HET probe[5]. This sensor are very sensitive to minute changes in the magnetic field and were easily adapted to the NeatTools interface, again by TNG2.
2.1.4 Electromyography EMG
EMG was used as an interface with TNG1 before the REU project began. EMG was used with self-adhesive electromyographic electrodes. The electrodes were successful but had several limitations. First, were stable lasted for a limited amount of time. The electrodes would eventually dry out and could not be relied on for a continuous signal without repeated moistening and recalibrating. Also they are very large and one could only get a limited amount of channels using this method.
2.2 First Headpiece
The first helmet used TNG2 with two sets of two copper connectors, one taped to each cheek and the other set attached to stationary extensions over the cheek. Upon movement of the cheek, contact between the copper foil extensions and the copper on the cheek allowed current to pass through, giving a signal on Neat. The variance of connection between the copper pieces allowed for resistance differences that was interfaced with TNG2. The piece of the copper extensions was attached to a flexible, stationary wire, to keep them positioned over the cheek. This model was too flimsy, however, and the connectors did not stay attached to the skin too long. In effect, the signals were inconsistent and/or weak.
2.2.1 Second Headpiece
By combining two microphone headsets, the structure was more sturdy and more manageable connectors for the cheek. It was possible to use, for the first time, cadmium sulfide photoconductive devices, measuring resistance, as an input. However, the setup was not very stable when positioned; whenever the user gave a signal (blocked light to the cell) the headset would move, requiring repositioning, in effect, giving off inconsistent, unreliable signals.
2.2.2 Velostat Chin Strap
By utilizing Velostat, which is an antistatic plastic packaging material that changes resistance with changes in pressure, signals from pressure exerted by the chin were exploited via TNG2. Sandwiching Velostat between two layers of copper gave a resistance differential when the user opened his jaw thereby applying pressure to the chin strap. This model needs further modification in strapping it securely to the head and positioning it firmly onto one location on the chin.
2.2.3 Metallic Chin Strap
A CdS photocell was inserted into the vertex of a metallic chin piece. This prototype worked very well with the photosensor in that it could be attached to the headrest of a paralyzed user with clamps so that the chinstrap did not have to rest on the head of the user which would not always be stationary. When the user would lower his jaw, the bottom of his chin would cover the photosensor and give a signal to NeatTools via TNG2. This model worked very well with the third headpiece.
2.2.4 Third Headpiece
This piece was designed by intergrating a sports face mask along with photosensors for the cheeks, and TNG2. Individual light sources were focused onto each photosensor to increase sensitivity and were powered with disposable batteries. The sensors were attached to solid line wire which was flexible and light yet sturdy. The mask gave stable signals. However, it proved difficult to do fine adjustments with the photosensors since the wire it was attached to could not be bent easily into acute angles without pliers. Using tools on the mask while positioned on the user's face was too dangerous. Also, the light sources distracted the user's peripheral vision. The next models invisioned use of fiber optics which were more accurate in focusing light and better means of supporting the sensors, such as LocLine tubing. LocLine is a rigid but flexible series of ball and socket links that can be positioned and lined with wire and sensors,. providing stationary extensions over the face.
2.2.5 Third Headpiece, modified
There were two major modifications to this headpiece. First, the center area was taken out to allow Eyal to see better. Second, the LocLine mechanisms were used for the cheek sensors. The LocLine allowed us to position the sensor with greater accuracy, stability and consistancy. This was a major improvement from the original third headpiece (HP3).
2.2.6 Fourth Headpiece
This headset was designed using the shell of a construction helmet and the LocLine tubing-photosensor setup. The helmet was constructed to rest on Eyal's headrest and attach to the clamps holding the metallic chin strap. Correspondingly, the sensors were connected to TNG2. This model allowed for the forehead to be utilized again, which was not possible with the sports helmet model. This helmet allowed for a gap between the forehead and the plastic, since it was not resting directly on it, as in the previous model. A different sensor was used to obtain signals from forehead movement and is described in the next section.
2.2.7 HET Sensor
The Hall Effect Transducer (HET) sensor was developed using technology published in the August issue of "Scientific American". Signals were generated by having the HET probe rub against a rare earth magnet. The magnet was taped onto Eyal's forehead. The probe remained in a stationary position while the magnet moved accordingly with Eyal's forehead. The movement generated a very sensitive signal which was fed into NeatTools via TNG2.
3 Future Extensions
This project part of an ongoing journey towards improving the lives of the disabled into a lifestyle closest to that of the physically capable individual. Hence, this project is far from done.
The experimental extensions that will lay out future work for this project should include possible pattern recognition for facial expression of the alphabet using existing and/or advanced sensors. This will allow a user to make patterns, which will be picked up by sensors, to differentiate the letters of the alphabet. These patterns can be comparable to using a keyboard whereas the user has several inputs to command text. Also, cost efficiency of sensors and making them more accessible for the home user to utilize and purchase remains to be improved.
4 Design of Web-Based NeatTools
NeatTools software is a Web-based interface system which augments our ability to create, express, retrieve, analyze, process, communicate, and experience information in an interactive mode via interface devices (see above), it senses bioelectric signals from eye, muscle and brain activity as input and render information in ways that take advantage of psychophysiologic signal processing of the human nervous system[6]. We developed and integrated the NeatText module where the user can interface with the outside world with the use of text.
The NeatText module adds an easy and fast way to communicate through the computer. The model implemented in this module uses a triangular-based geometry of the alphabet as seen in figure 2.
The NeatTools module allows the disabled individual to use facial movements to select letters of the alphabet to form words. The number of times the user has to make a facial gesture to select a letter of is three. This is the reason behind the triangle; we want the minimum number of gesture movements. Within the one large triangle, there are three smaller triangles with another set of three smaller triangles in each of the three larger triangles which consist of letters. To get to a particular letter, the user chooses one of three directions and simplifies the big triangle to the smaller triangle that the user has chosen. The user continues to simplify the triangles until he has reached the letter he desires. This model is very efficient; three selections in any direction by the user can get a specific letter. In the case of Eyal, the sensor frame will give these three signals allowing him to select letters and form complete phrases more quickly than ever before. This idea of text generation is a way to communicate without having to use a physical keyboard input.
Two design algorithms were taken into consideration for this problem. First algorithm was a point value system for each movement. The initial signal to the left had a point value of 1, right had a point value of 10, or top had a point value of 100. Second signal multiply by 10 and the last signal multiply by 100. The letter would be the one with the matching point value, but conflict in similar point values occurred. Second algorithm that was implemented was to divide the triangle up into three distinct levels: TOP = level 1, RIGHT = level 2, and LEFT = level 3. Then we take each of the three levels and break it down into three more levels. From the TOP triangle, we are at level 1. Each triangle in the TOP triangle will be numbered: TOP, RIGHT : 12 ; TOP, LEFT : 13 ; TOP, TOP : 11. The same type of numbering was done for RIGHT and LEFT ( level 2 and level 3 ). The user can to choose a letter now. All the cases of third choice were taken from all the options of the first two choices.
The NeatText module that was created was implemented in Java and integrated into the NeatTools environment. One great feature of Java is platform independence, which means that your program can run on any system from PC to a workstation, but the trade off is speed. Our first notation of how slow Java is in the usage of fourteen breakdowns of the triangular image to display the users selection in the triangular-model. This idea slowed down the process because the user would have made a selection and the image was still trying to download. Our next alternative was to use a Java feature called "crop filtering with MediaTracking" which used one triangle image. Every time the user selected a particular region in the triangle, that region would be highlighted. This idea increased the performance of the algorithm, but not to our satisfaction. The alternative that we implemented and that does not slow the process down as much was to black out the other two triangles that were not chosen. The reason why this alternative was faster is because we did not have to create an extra object.
Another factor that has contributed to the slow down of the algorithm was the continuous incoming signals from the serial port. This continuous stream of input signals causes data congestion between the modules. Three techniques were used: first, we cleared the buffer of the previous input signals. Although minimized, collision and interference still impeded the interpretation of signals. Next, we reduced collisions by periodically dropping redundant values from the signal. Finally, a data delay was introduced to control the number of incoming signals being processed.
The NeatText module opens up numerous possibilities for the future. The simple triangle model can be extended into a pyramid-model to consist of common phrases and reserved key words for URL. Word processing applications like MS-Word and Notepad are future extensions of the text area in the NeatText module. Through text on the screen, it is possible to write URL addresses which extends to Netscape to navigate through the World Wide Web (WWW).
5 Computer Data Rendering
After establishing sensory inputs and a computer interface, the final aspect of the interface cycle is the computer output, or rendering, that the user can recognize as changes in his environment. By correlating these changes with the changes in sensory input as provided by the user, the user can identify what input causes what changes; this completes the interface cycle.
5.1 Computer Interface Results
This part of the project was intended mainly to design a specialized,
user-friendly interface for disabled people. The user interface is based
on a set of frames and Java applets that help a user to communicate easily
with the World Wide Web.
One of the frames provides an option to enter a new URL into a text editor or choose one from the bookmark list provided the context of that Web page will be displayed in the target frame, that is located underneath the first frame.
The NeatTools application is displayed in the frame on the bottom of the Web page. It allows the user to set it up, to load a saved file, and to run it. When NeatTools runs, a pop-up window of the NeatText comes up and the user can type text in the window provided.
On the left side there is a menu frame, which contains the items that can be added. One of them is a "Pointer Icon" and the other one is "Telnet via Nova".
Pointer Icon is a prototype for a software control device over the mouse pointer, which will be using alternative input devices, such as TNG1 and TNG2, form Institute of Interventional Informatics. The time Pointer Icon prototype exists in a Java applet version.
As shown in Figure 4 Pointer Icon consists of a circle and a ball. The five buttons controlling the movement of the Pointer icon are at the top of the applet: Stop, Left, Right, Move, and Click. One click on the Left button makes the ball rotate around the circle in the counterclockwise direction. Clicking on the Right button makes the ball rotate around the circle in the clockwise direction. The Move button allows the whole circle to move in the direction of the ball, as shown in the Figure 5. The movement of the ball and the circle is stopped by clicking the Stop button.
The Click button will be implemented so that the inputs to the buttons will be accessed through TNG1 or TNG2 via the serial port of a PC, in order to control the movements of the circle and the ball. The Pointer Icon will send the x and y coordinates to the mouse pointer which will be moving with the circle in the direction of the ball, as shown in Figure 5. That is where the implementation of the Click button will be implemented.
The second item in the menu frame is "Telnet via Nova". This frame provides easy access to Telnet. Without closing the Web browser it provides a new Telnet window. Actual use of this function for Eyal will be challenging.
5.2 The Pulsar Project
We are also in the process of designing a Web page for the Pulsar project. This page will provide a Web-based interface systems and free Web services for supporting accessible interfaces for disabled people. The goal of the Pulsar project is to develop a Web service which will be a clearing house for tools for disabled individuals. We will provide a database of empowering software which is to be available to download for free and a list of off-the-shelf devices for the disabled. There will be a list of local support groups and resources provided . Pulsar will also develop a forum where users can exchange information on methods and techniques which empower the users to become employed members of the information society.
Appendix
We have utilized a notational system to map the flow of information as it occurs in interactive human-computer interfaces. This notational system enables us to identify and characterize a discrete set of information state spaces involved in human-computer interaction.
PSI = perceptual state of intention
PHX = biologically parameterized state space
SNS = sensor state space
KHI = computational state space
SKY = state of rendered information
For an expressional interface system, it is postulated that there is first a conscious experience of intent to be expressive (PSI). This is a state of perception (mind). A biologically parameterized state (PHX) responding to this intention occurs. Information in the form of modulated patterns of energy is detected by sensor technology (SNS). A computational process (KHI) interprets that information and determines a specific rendering for it. The rendered information (SKY) is biologically (PHX) transduced and processed initiating a new perceptual state (PSI).
6 Acknowledgements
The following people have helped with this project in various ways: Johannes Johannsen, scribe of NeatTools; Alvin Leung from NPAC, Lou Buda and Sam Sampere from the Physics Department, and all the students from the REU program. The Sherman Family for their determination, and especially Eyal Sherman for his courage and dedication. Thanks for making the summer exciting, educational and fun.
References
[1] Johannes Johannsen: Programmer of Neat in mature version and NeatTools
in Jave. jojo@well.com
[2] D. Warner: Institute for Interventional Informatics
(I3). www.pulsar.org
[3] Eyal Sherman: http://www.npac.syr.edu/projects/neattools/Eyal
[4] Sangeeta: http://www.npac.syr.edu/users/sangeeta/neattools/
[5] Shawn Carlson: August 1996, pp. 96-99
[6] Dave Warner; Jeff Sale; Stephen Price; Doug Will: The Neurorehabilitation
Workstation: a clinical application for machine-resident intelligence