We began by brainstorming possible applications for integrating the MindFlex, which included ideas such as turning the iPhone on and off, flipping pages on a Kindle, or creating predictive modeling of Pandora tastes.
As we brainstormed for possible applications, we noticed a pattern in our ideas: each project was based on a generic yes or no switching algorithm. In other words, as opposed to designing or building a series of independent demos, we thought it would be more valuable to the community to create a general decision function that could be implemented as a layer between the incoming data stream of a BCI and an application, to know when a user has changed states.
This switching algorithm could be used as an on/off switch, detecting a change in color, direction, mood, or really any binary behavior. Each individual would have to be trained, similar to how the Dragon Naturally Speaking software trains its users until it has enough data to function reliably. However, our goal was to simplify the process, by only having the user wear the MindFlex device, connected to the Arduino + computer and running a processing algorithm in real-time. With this goal in mind, we broke up the problem into logical parts.
First component: Getting the data from the device to the computer
Second component: Translating and operating on that data to detect a change in state
The first component: MindFlex -> Arduino -> Computer
We knew we could get data from the MindFlex by hacking it’s serial port. Since our computer has no serial port, we connect the MindFlex to the Arduino as an intermediary, an interpreter which relays the serial data to our computer. This is all documented extensively in the Frontier Labs walkthrough. We do not need the Processing or Arduino IDE to run while getting data. Once the sketch has been loaded, we can access the data directly from the terminal through a COM Port.
The second component: Data Processing
The programming and math modeling team decided a good route would be best implemented in Python, because of the many open-source modeling and statistics libraries available. With the data coming from the USB port into the computer, a Python script would run from the command line, getting input piped from the terminal command. As noted by our lead developer “don’t even think about trying to run Python without iPython.”
Next, make sure you have the SciKit learn library, along with its dependencies NumPy, and SciPy. To compile all of these, you’ll need a C-based compiler (GCC), fortran. However, before getting a real time algorithm running, we programmed an intermediary step as a starting point, of processing the data of a csv file. After several trials, if verifying that our data was stable and reliable, we’d convert the algorithm into a real-time process that operate on the stream of incoming data in real time
The DRAFT NOTES below is a compilation of incomplete notes as we went along…. loosely documenting our process, concerns and challenges:
Once we got the first component working, we set up a trial, which we called a binary classifier. Here, we asked a team member to wear the MindFlex for a certain amount of time, focused on one idea.
We decided for a simple binary set of options… the first “yes”… for 20 seconds. then 20 seconds of “no”. Generating 40 data points
We passed this data file to our programming team, while others analyzed the data manually looking for trends.
We noticed a few things: First, most of the dimensions were in sync, 7 or 8.. (delta, theta, low b) so when one value rises, the others rise as well. Second, we noticed was that after a switch, the integer values of all of the dimensions changed almost uniformly, however tapered off, after some time returning to the baseline value
our theory for this, is that your brain is wired for change, and inherently can detect a change, but if this change does not persist, it becomes .. insignificant and “normal”. another way of saying this is it “returned to trend”
We continued with 3 other trials, which gave us similar results: the logistic regression was between 60-80%, and SVN was 100% on the 4th trial,
we then tacked on an extra column of data, on top of the 11 original values.
This value was originally a string, “yes” or “no”, however we later changed it to an integer to allow the python script processing it to stay entirely within the same data type of integer values.
In the final minutes of our team’s effort, we realized that some of our data was suspect.
As we investigated, we discovered if the signal strength value for each data point was NOT 0, the data should have really been discarded, as it was unreliable. We were not sure how it was possible, that our first test, which has “reliable” data, closely matched the statistical analysis of the other tests, which was “unreliable”.
In the final presentation, we realized our Arduino was at fault, as after switching boards we once again were getting data values with signal strength of 0, which indicates reliable data.
Nonetheless, we believe coming up with a standard library of basic functions of translating brain activity to recognizable on/off state changes would have value for anyone using BCI interfaces, and are looking forward to continuing our experiments.
Reflecting on the experience – our group comprised of team members from many backgrounds including sound and music, software development and scripting, mathematics and statistical modeling, hardware and circuit design, neuroscience theory. This diversity allowed us to quickly split up into tasks by domain, working towards a common goal to come up with a prototype in one afternoon.