The main idea behind this project is to build a multi-modal user interface in order to enhance the experience of using a touch-based system for CAD modeling
For most 2D operations, touch-based gestures help provide an intuitive and natural way for users to communicate with an interface. When it comes to 3D applications, however, 2D gestures prove to have a few limitations.
The following examples illustrate a few difficulties associated with 2D gestures and 3D applications. The examples are based on a mock-up 3D CAD modeling interface that was developed on OpenGL for the purpose of this project.
One possible interpretation of the left - right gesture on object (a.) could be that the user wants to move the object from left to right. On the other hand, the user might use a similar gesture to rotate the object in order to see its left face.
Similar conflicts exist with the gestures used on objects (b.) and (c.). In these two cases, there is possibly more ambiguity associated with the gestures, since they could also represent a backward (in case b) or forward (in case c) motion of the object in the 3rd dimension
Multimodal interfaces are versatile interfaces that users can interact with, by using a combination of input techniques such as speech-based input, touch-based gestures, or camera-based 3D gestures.
Each of these input techniques, however, has its own limitations, and trade-offs have be made while incorporating a certain input modality into an interface.
A brain-computer interface was chosen to help boost the performance of a touch-based system.
Experiments were designed and conducted on 7 subjects (4 male and 3 female) in order to gather data for analysis.
A total of 6 experiments were conducted, each involving simple 3D transformations of the given 3D model. Each experiment was designed to be repetitive, comprising a total of 10 consecutive trials. Each trial took 6 seconds to perform, and a 4 second pause was used in between trials. The subjects were timed using a stop-watch and were instructed when to begin and end each trial. The experiments performed by the subjects are listed below:
- Left - Right Motion
- Left - Right Rotation
- Down - Up Motion
- Down - Up Rotation
- Up - Down Motion
- Forward Motion
The interface was designed to respond to the experiments as they were being carried out, and a high resolution, multi-channel, wireless neuroheadset called the "B-Alert X10" was used to record Electroencephalograph (EEG) signals of the subjects as they were performing the experiments.
Data analysis was carried out in three separate phases:
Stage 1: Analysis of Gesture-based Data.
The gestures used by each subject were captured in the form of 2D point coordinates. The following features were then extracted to form gesture-based feature vectors:
- Angle between starting point and end point of gesture
- Mean of angles between each point of gesture
- Square of mean of angles between each point of gesture
Stage 2: Analysis of EEG data.
The brain signals (EEG signals) collected from each subject were analyzed in the frequency domain, and EEG-based feature vectors were constructed using the following two techniques:
- By calculating the power spectral density of 5 different frequency bands
- By using the method of Common Spatial Patterns (CSP) to project data from different classes into separate sub-spaces.
Stage 3: Combination of Gesture-based Data with EEG Data:
The features extracted from each modality were finally merged into a single "master feature vector".
After the construction of the 3 sets of feature vectors, classification algorithms were developed using MATLAB to classify between tasks that were performed using similar gestures. Based on the experiments that were performed, the following classification strategies were used:
(a.) Classification between Left - Right Motion and Left - Right Rotation
(b.) Classification between Down - Up Motion and Down - Up Rotation
(c.) Classification between Up - Down Motion and Forward Motion
From the corresponding bar graphs, it can be seen that the classification based purely on gesture-based data (black bar) is significantly lower than the BCI-based classification (grey bar) and the classification based on the combined data (white bar), especially in cases (b.) and (c.).
This indicates that a Brain-Computer Interface could be used to help resolve the ambiguity associated with 2D gestures and 3D applications
Topoplots to illustrate how brain activity related to Left - Right Motion is different from Left - Right Rotation.
Topoplots to illustrate how brain activity related to Down - Up Motion is different from Down - Up Rotation.
Topoplots to illustrate how brain activity related to Up - Down Motion is different from Forward Motion.
gLike
A Method of Enhancing Touch-Based Gestures Using a Brain-Computer Interface
The aim of this project is to explore how multimodal Computer Aided Design (CAD) interfaces can be developed by integrating brain-computer interfaces (BCI) and touch-based systems. This integration is proposed in order to help reduce the ambiguity associated with 2-dimensional (2D) touch-based gestures on 3-dimensional (3D) touch-based applications.
Gestures on a touch-based system are subject to ambiguity in recognition since they are 2-dimensional in nature and could lead to multiple interpretations, more specifically when used to operate a 3-dimensional application. This project describes a novel method to utilize brain signals of users to help resolve this problem.
The project was carried out as a part of my thesis requirements under the guidance of Dr. Ehsan Tarkesh Esfahani, Assistant Professor, University at Buffalo, the State University of New York
