A low cost SSVEP-EEG based human-computer-interaction system for completely locked-in patients

Received Feb 27, 2021 Revised May 20, 2021 Accepted Jun 16, 2021 Human computer interaction (HCI) for completely locked-in patients is a very difficult task. Nowadays, information technology (IT) is becoming an essential part of human life. Patients with completely locked-in state are generally unable to facilitate themselves by these useful technological advancements. Hence, they cannot use modern IT gadgets and applications throughout the lifespan after disability. Advancements in brain computer interface (BCI) enable operating IT devices using brain signals specifically when a person is unable to interact with the devices in conventional manner due to cognitive motor disability. However, existing state-of-the-art application specific BCI devices are comparatively too expensive. This paper presents a research and development work that aims to design and develop a low-cost general purpose HCI system that can be used to operate computers and a general purpose control panel through brain signals. The system is based on steady state visual evoked potentials (SSVEP). In proposed system, these electrical signals are obtained in response of a number of different flickering lights of different frequencies through electroencephalogram (EEG) electrodes and an open source BCI hardware. Successful trails conducted on healthy participants suggest that severely paralyzed subjects can operate a computer or control panel as an alternative to conventional HCI device.


INTRODUCTION
Volume of stroke patients is quite high. As per a recent finding by 'global burden of disease (GBD) 2016 Stoke Collaborators' [1], around 116.4 million stroke survivors' cases in a year were reported worldwide who are living with disabilities. Donkor [2] indicated that severe brain injury results in up to 50% of survivors being chronically disabled. Guan described [3] that around 30% of stroke survivors require various forms of rehabilitation therapy especially for upper limb problems and loss of hand movements. Moreover, Jhonson et al. [4] presented that globally, 70% of strokes occur in low and middle-income countries. In the last four decades, the stroke incidence in low and middle-income countries has increased by more than 100% whereas during this period, stroke occurrence has declined by 42% in high-income countries.
These figures indicate that a large number of human populations who are facing disabilities due to stroke belongs to low and middle-income countries. Due to severe motor disability, they are unable to use computer and other IT devices through conventional means. An extensive research and development exercise is essentially needed to support the stroke victims in improving their quality of life and to enable them to continue using computers and IT devices. Advancements in brain computer interface (BCI) technology enabled the researchers to carryout extensive research in design and development of such types of low-cost human computer interface (HCI) systems that can be interfaced with the computers and other IT gadgets as a plug and play HCI device operated through brain waves. Hence, these HCIs can be used by the completely locked-in patients who cannot use keyboard, mouse and touch screen. The targeted HCI system exploits a neural process of steady state visual evoked potential (SSVEP). visual evoked potential (VEP) is the potential-difference on human scalp obtained from the EEG electrodes placed on the surface at visual cortex of the brain during a visual stimulus of flickering light. According to Wu et al. [5], "SSVEP system selects target by eye focus. SSVEP invoke when the retina is excited by a visual stimulus ranging from 3.5 Hz to 70 Hz, the brain generates electrical activity at the same frequency of visual stimulus or some harmonics". Researchers are targeting SSVEP based BCIs in many areas. Hwang et al. have proposed a SSVEP based psychosomatic application [6] for spelling tasks by applying a Q-W-E-R-T-Y style layout keyboard based on 30 light emitting diodes (LEDs).
In an interesting work [7], Volosyak et al. proposed a SSVEP-BCI that were assessed with a number of stimulation frequencies in an assignment related to spelling and concluded that fastest SSVEP response was achieved for the stimulus frequency of 6.67 Hz. Another similar work for spelling assignment by Cecotti suggested a SSVEP based self-paced BCI speller [8] which can work without any training from the user or from the signal processing section. Kim et al. [9] proposed a valuable point-and-click intra-cortical neural interface system (NIS) that permits tetraplegia subjects to operate a 2-dimensional computer cursor. The subjects can move the cursor on computer screen in any required direction, with hold and click on the area on screen through BCI.
Liu et al. presented a SSVEP-BCI based augmented reality system based on HoloLens [10] and achieved significant accuracy and information transfer rate (ITR). Khatri and Farooq proposed a research work [11] that describes the use of SSVEP to develop a sentence speller to aid locked-in syndrome (LIS) patients. In an interesting work, Angrisani et al. [12] utilizes the architecture of typical SSVEP-BCI and proposed and tested a low-cost single channel SSVEP based instrument that can be used in control applications. Lin et al. developed a method [13] to operate a web browser using SSVEP combined with eye tracking through a 64-channel device. Similarly, Saravanakumar and Reddy [14] demonstrated that use of electro-oculogram (EoG) along with SSVEP can improve the performance. Bianchi et al. described a work [15] that studies the concept of providing analog like control using SSVEP through seven flickering stimuli.
Ajami et al. [16] also worked on single channel SSVEP speller application aimed with user comfort. Hasan et al. described a direct SSVEP BCI System [17] that targeting higher information transfer rate to select commands in graphical user interface. Cao et al. recently presented a SSVEP BCI based Speller [18] for numerical inputs based on sliding control. Chiuzbaian et al. [19] discussed an innovative SSVEP based system to control a drone. Phothisonothai and Tantisatirapong [20] presented a SSVEP based virtual Thai keyboard for spelling and control tasks. Ramírez-Quintana et al. described their work [21] for SSVEP embedded processing with higher accuracy on signal channel input. Lin et al. also presented a recent work [22] on SSVEP based spelling system combined with eye tracking. Adams et al. [23] presented the concept of a smart home that can be controlled by SSVEP-BCI. Similarly, Park et al. presented a detailed work [24] on development of augmented reality and SSVEP-BCI based system to control home appliances. Garro et al. presented a SSVEP-BCI based alternative communication system [25] for the people with speech impairments. The concept of using EEG to perform robot motion control was proposed by Dewangga et al. [26]. The EEG data acquisition system can also be used to detect learning disorder in children using brain waves [27] as described by Kamaruddin et al.
The usability and effectiveness of SSVEP-BCI based systems for common people are mainly depended on less number of electrodes to allow wear-ability and home use, low cost implementation through simple hardware and light weight algorithms and minimum training requirements to allow subject independency with less inter-subject variability. Joseph and Govindaraju have verified [28] that even a major reduction of EEG electrodes from 118 to 10 does not significantly affecting the BCI performance. Along with these features maintained with low-cost, more controllable options, and commands that can be triggered by the completely locked-in patient is a very important requirement. Inspired by abovementioned contributions that are based on expensive hardware, we have proposed a low-cost general purpose SSVEP-BCI based HCI system that can be used by locked-in patients in place of conventional HCI to operate computers and other devices through eight different controllable classes.
The effectiveness of proposed system was tested through experiments on eight healthy subjects. The trials were conducted in accordance with concerned declaration. Experimental results demonstrated that the proposed low-cost system is providing sufficient interface with the computer and can be used by the targeted individuals. The rest of the paper is organized is being as: section 2 presents materials and methods followed by experimental results and discussion in section 3. Finally, the conclusion and future recommendations are provided in section 4.

RESEARCH METHODS
This section discusses the research method on the material and techniques used in the testing of proposed system including system design, experimental setup, signal acquisition, processing, and control. The proof-of-principle was demonstrated by implementing the proposed HCI using a notebook computer and open-source low cost hardware module with an open-source software.

System design
The system comprises of four modules: a set of three EEG electrodes, a low-cost open source biosignal amplifier, a notebook computer, and an output module as mentioned in Figure 1. Eight Ag/AgCl electrodes are placed on visual cortex at the back of scalp, one on forehead as ground and one on ear lobe as reference electrode. The low-cost open source amplifier (OpenBCI, U.S.A.) is a small and light-weight bio signal amplifier which is used to send SSVEP signals to the computer via bluetooth. SSVEP-EEG signals were sampled at 250 Hz. A portable battery pack of four AA size batteries is used to power the amplifier. The amplifier along with battery pack is completely head-mounted.
The computer receives SSVEP signals from the amplifier on universal serial bus (USB) port through bluetooth dongle. After processing the signals through modified open source program, command signals elicited from SSVEP are sent to the output module. A customized open source code of OpenBCI software runs on the computer. The software is customized to provide user interface for output mode selection and to operate the output module through visualizing flickering LEDs. The raw EEG signals are also displayed. The output module is selectable by the user either to operate a control panel consists of eight discrete outputs or to work as SSVEP-BCI based computer mouse connected to the computer via USB cable.

Experimental setup and signal acquisition
Eight healthy participants of normal or adjusted to normal eye-vision who were naïve to SSVEP-BCI were selected for trials through convenience sampling method. The participants were taken normal diet without consumption of alcohol or caffeine to avoid possible effect of SSVEP elicitation as described by Pradhan et al. [29]. Procedures and design of trials were in accordance with the principles of the helsinki declaration and experiment protocol was properly described to each participant. The trials were divided in 3 stages in which participants are required to confirm the stimuli detection, to the operate a control panel of 8 discrete outputs and to move the cursor on computer screen respectively through visualizing flickering LEDs.
In each trial, participants were asked to sit on a chair in front of the computer screen at a distance of around 4 foots to get visual stimuli as shown in Figure 2. Flickering LEDs of 8, 10,12,14,17,19,21,23,25,27,29,31,33,35,37, and 40 Hz were used to get the stimuli. All the LEDs are of green color to exploit the  [30], [31]. Electrodes were placed at visual cortex around O1, O2, and OZ positions of 10-20 system of EEG electrode placement whereas reference and ground electrodes were placed at Fp2 and A2 respectively.

Figure 2. Visual stimuli and corresponding amplitude
The electrodes were connected to the head-mounted OpenBCI board that transmits the SSVEP signals to the Notebook computer via Bluetooth. The board samples the signals at 250 Hz via ADS1299 and PIC32MX250F128B micro-controller and powered with 6 V through four AA size batteries. The board size was 2.41" x 2.41" excluding battery pack.
The graphical user interface (GUI) of modified OpenBCI program provided the real time SSVEP-EEG signals, visualized input, mode selection input, and status of 8 output channels. Real time SSVEP-EEG signals are the graphical representation of EEG activity at visual cortex. Information of visualized input gazed by the user is displayed instantly in the screen. Mode selection input allowing users to select the targeted operation as control penal or operating the computer mouse. The control panel is proposed to be of 8 digital transistor-transistor logic (TTL) outputs to control any connected devices for home automation, bed adjustments, and alarms. User can toggle the status (OFF/ON) by gazing the respective LED. Status of these outputs were displayed on GUI in red and green colors with OFF/ON text.

Data processing and control
SSVEP signal obtained from visual cortex was processed in modified open source program of OpenBCI build on processing platform in Java. The signal was first filtered using 5-50 Hz bandpass filter and 50 Hz notch filter. Classification of different SSVEP commands corresponding to the visualized LEDs were carried out using built-in feature of fast fourier transform (FFT). The amplitude of frequencies was directly obtained from the filtered data ready for FFT plot. A command for particular visual stimuli was considered TRUE if amplitude at corresponding frequency is highest. A TRUE command toggles the state of the designated feature of mode selection from 'control panel' to 'computer mouse' or vice versa for mode selection stimuli. The TRUE commands then trigger the cursor to move towards corresponding direction or to perform the click in mouse operation. Similarly, it toggles the output state of corresponding control panel output pin when control panel operation was selected.

EXPERIMENTAL RESULTS AND DISCUSSION
The experiments were conducted in a softly-lit room in three different sessions in which participants were seated in a comfortable chair wearing electrodes in front of experimental setup.

Experimental session 1
In first session, participants were asked to gaze each flickering LED for 3 seconds with 3 seconds interval in response of manual audio cue to confirm the stimuli detection by the experimental setup. The trails were repeated 5 times and SSVEP responses were recorded which were then used to compute the corresponding amplitude level of each visual stimulus obtained in each trial. Corresponding amplitudes values with minimum, maximum, mean, and standard deviation are described in Figure 3. The participants were also filled a questioner to rate the comfort level of gazing each flickering LEDs on a scale of 10 that are described in Figure 4.   (10) The SSVEP amplitudes correspond to each frequency obtained through this low cost setup are above 5.5 µV and reached up to 9.6 µV that are quite enough to detect the corresponding SSVEP frequencies. The results indicating higher amplitudes at frequencies from 21 Hz to 31 Hz, therefore these stimuli are used for more critical inputs like mode selection and control panel operation. Experiments also indicating the tendency of decreasing amplitudes at frequencies lower than 14 Hz and higher than 37 Hz. This is limiting the possibility of addition of more stimuli beyond 8 Hz and 40 Hz. The participants reported very less comfort level at lower frequencies and good comfort at higher frequencies. This is due the eye discomfort by continuously gazing low frequency stimuli which is less at higher frequencies. Lindquist et al. [32] reported that short term disclosure of uncomfortable chromatic stimuli is not creating harmful effect on visual task performance except creating a sharp neural response across cortex. However, to provide comfort to the users, continuous exposure to low frequency stimuli should be avoided. For this reason, higher frequency stimuli are used in this system to trigger mouse control operation that requires continuous gazing of desired stimuli. Further research may consider the use of other visual stimuli to address the eye discomfort like Mouli and Palaniappan [33] presented in a work aiming to develop a hybrid BCI platform for stimuli generation to evoke SSVEP and P300 to minimize fatigue with improved classification performance.

Experimental session 2
Second session was designed to test the accuracy of control panel operation in which participants were first selected the control panel by gazing the mode selection LED. Then they asked to toggle each output by gazing the corresponding stimuli for 2 seconds in response of audio cue with different time intervals of 1 to 5 seconds. Each output was tested 10 times and the output states on control panel were recorded to perform offline analysis for efficiency/accuracy of trials on the basis of success and failure of triggering the output pins each time. Average success rates of each operation are given in Table 1. Experimental results of this trial confirm the efficiency of control panel operation up to 60% or at higher success rate. Since this was a training less trial, subjects may exhibit more efficient success rate after performing repeated trials or after use in day to day tasks. However, further research may be conducted to improve the efficiency by incorporating some new features in processing techniques and stimuli design.

Experimental session 3
Third session examined the mouse control operation in which participants were asked to visualize the corresponding LEDs to move the cursor from center of the screen C to each corner of the screen referred to as E, F, G, and H in clockwise manner. The time to move the cursor from one point to next by each participant were recorded to calculate the time consumed to move the cursor step by step. Individual and average time for each operation are given in Table 2 and Figure 5.  The time to move cursor from center to the first corner of the screen is less due to the less distance. Time consumed to move cursor from E to F, F to G, G to H, and H to E increased respectively due to the use of successive higher frequency stimuli that are of relatively less SSVEP amplitudes and exhibiting more false negative triggers as compared to former corresponding stimuli. At the end of experiments, participants were asked to fill the questionnaire to rate the overall experience in selection of mode, mouse control in each direction and operating the output of control panel. They were also asked to rate the overall comfort level during the trials. The response of the participants is given in Table 3. Mode selection operation was reported most easy as compared to control panel and mouse pointer control as the mode selection is not required to be triggered continuously. Mouse pointer control was reported comparatively less easy due to the eye discomfort in this operation due to continuous gazing of stimuli. Overall cost of the setup is quite less as compared to other commercially available systems as described in Table 4.  [34] but it requires dry electrode consisting of 12 pointy pinches to be placed at the back of the scalp which is limiting its use for locked in patients as they are not able to use the system at sitting position. The proposed system is based on open source hardware with low components cost and providing medium level of efficiency with wear-ability and comfort for locked-in patients. To the best of our knowledge, the proposed system is a cost effective solution with features of comfort and wear-ability particularly for the users of low and middle-income countries.

CONCLUSION
This paper presents a low-cost general purpose HCI system that can be used to operate computers and a control panel through brain signals. The proposed system comprises of a low cost bio-signal amplifier interfaced with eight electrodes to acquire SSVEP-EEG signals through gazing LEDs flickering at different frequencies. These inputs were used to operate a control panel of eight discrete outputs and to operate a computer mouse. The experimental results confirm that this low cost implementation can be beneficial for completely locked-in patients and will enable them to use computer and IT devices in a cost efficient manner with acceptable efficiency and accuracy up to 60% or higher. Further research may target improvements in the efficiency of the system incorporating some new processing techniques and stimuli design in low cost manner to supplement this proposed system.