How did you train the system?
The Emotiv SDK comes with a module known as the Cognitiv module. You can train up to four Actions, where you hold a particular thought or feeling for a number of seconds while the system captures your EEG signature. Each Action can be trained multiple times. The primary purpose of Cognitiv is to train it to recognize thoughts with as little facial activity as possible, since muscle movements overwhelm the EEG sensors. I subvert Cognitiv for my purposes and use it specifically to capture facial expressions; the kind that can only come from experiencing real emotions, not faked. In Rosalind Picard’s wonderful book on Affective Computing she makes a vital point:
“Neurological studies indicate that true emotions follow their own special path to the motor system.”
In other words, different parts of the brain control your facial muscles when you try to smile, instead of when you smile involuntarily when something makes you feel happy. It’s why bad acting is so easy to spot and why the best actors don’t act. They feel. They convince themselves that what isn’t real, is real, and then react to it.
Fortunately the Emotiv EPOC system is good at detecting real emotions and not so good at detecting forced emotions (phony). When we truly feel a primary emotion our face really shows it and we are much more consistent in that expression. Unfortunately that makes demonstrations difficult because the demonstrator has to train the system on their emotions before being able to use it, and that training has to include experiencing genuine emotions while watching content that has been tagged for particular emotion. Somebody just walking up to the system and making a few half-hearted faces will be disappointed. This is one of those rare situations where the technology is far better for building a real product than for building a demo.
But how did you train it?
As I said, I misused the Cognitiv module to recognize the four primary emotions instead of non facial gesture related thought patterns. Because they had to be real emotions, I watched YouTube videos that I knew would affect me emotionally in a deep way. Once I felt the emotion I was trying to train I initiated training on the Emotiv Control panel. I repeated this process across all the primary emotions until I got a satisfactory response from the EmoRate system. Here are the videos I used, grouped by primary emotion:
Happy
Happiness was the easiest since it is the emotion that is most socially acceptable to show, even in strong intensity. I simply watched lots of episodes of the hilarious show “Whose Line Is It Anyway?”, smiling and laughing frequently along the way.
Sad
Sadness is a much harder emotion to show than happiness, especially for men. Tragically it was fairly easy in this case due to recent environmental catastrophes. I just queued up a few of the BP Oil spill videos on YouTube. There are some that show oil soaked birds and the pain and anguish of the residents that are just heartbreaking to watch.
Anger
I watched a variety of opinion talk shows, especially those where the participants are intent at going ballistic at each other. That kind of yelling and arguing really gets to me.
Fear
This was the hardest of them all. I believe that the mind has neurological barriers to remembering fear easily, perhaps because we would never get out of the cave if we really thought about the Tiger in the woods. Fear is an amazing ancient emotion that is hard to to trigger consciously. However, once triggered by real stimuli, it takes over like a fast spreading four alarm fire. Here I watched videos of Tornados and Animal Attacks, and concentrated hard on believing that the threat was close by and allowed my mind to imagine the frightening consequences of the threat, like being sucked helplessly up into the sky by a giant vortex or chewed to pieces by something large and hostile with claws.
Usage
Once trained I then watched an amazing movie trailer for the animation movie Sintel and “lowered my shields”, allowing the video to impact me fully without being emotionally prepared in order to avoid trying to force the emotion artificially and subsequently throwing off the Cognitiv detection system.
Experimenting with the system was quite enlightening. When I was in “live” mode where the system updates the EmoMeters while I’m just sitting in my chair, I could never get it to work well by forciing an emotion or facial expression. I had to completely forget about the system and turn my mind’s eye inwards and think of something that made me happy, sad, angry, or frightened.
Is the demo I saw in the video real?
Absolutely.
The EmoRate software is a Delphi 6 Professional application that integrates with the Emotiv EPOC headset via their software development kit (SDK). To control the YouTube video player I used swfobject.js, an open source Javascript library that exposes the functionality of the Flash-based YouTube player. It allowed me to pull out the current video location in seconds while the video played to facilitate the emotion recording. It also gave me the ability to seek to a particular point in the video by time index.
Did I “help” the system in any way during the demonstration?
Absolutely.
But no more than any other demo of a brand new technology. I’ll use speech recognition as an example. When you watch a speech recognition demonstration the presenter almost always speaks slowly and very clearly with careful enunciation of each word. When I used the system I tried to remain as calm as possible until I reacted emotionally to the video, and when I did react, I embraced that emotion fully with complete concentration. In other words, I tried to help the system understand my emotions as best as I could. This means that the current system would perform poorly for anybody just working up to it off the street. (See the notes above on how I trained the system).
Is this a barrier to current adoption of the technology?
Only in the short term. Just like speech recognition, emotion recognition systems will get faster and more robust over time, especially as they are used by more and more people and their algorithms learn from those interactions. The improvement of emotion recognition systems will result from a co-evolution of two distinct tasks. One will be the task of of working with trained actors who by profession learn to express emotions with deep concentration and focus, in order to create a database of universal emotion profiles or signatures. The other will be the improvement of the detection algorithms to adapt better and faster to individual users during user specific system training. Those of you familiar with the path taken by the speech recognition industry can intuit how things will unfold from here. In that industry a co-evolution occurred where speaker independent engines, those that required no user training, and more powerful and accurate speaker dependent engines, those required significant user training, improved rapidly over time. The pinnacle of that parallel evolution is what we have now, powerful hybrid systems that work reasonably well right out of the box, but improve significantly in their accuracy with sufficient and continued training.
|
Future Applications
EmoProfiles
The list of potential applications for Affective Computing aka Computing With Emotions is incredible. As the technology develops people will develop EmoProfiles, profiles of their emotional reactions to videos, songs, books, reviews, anything and everything that sparks an emotional reaction (which basically is everything since we are emotional beings). The definition I have come up with for an EmoProfile:
“An EmoProfile is a profile that contains data cataloguing your emotional reactions to various stimuli, tagged by keywords and other ontological aids and statistics, in a manner that allows one profile to be compared with another or a group of other profiles digitally by any reasonable computing device.”
As adoption of EmoProfiles spreads they will become powerful tools for predicting what someone likes and dislikes, and how compatible people will be with each other and when interacting in a group. This brings up an important security issue that must be addressed up front by both providers and users. That of controlling who gets access to your EmoProfile and making sure you have strict control over that information. Critical data like your medical is history will always be more important and vital than something like your EmoProfile. On the other hand, your EmoProfile may feel more intimate and personal. The other danger here is that the law does not catch up fast enough and put in place important safeguards against nefarious activities like screening people for employment by EmoProfile, or demanding that data when they apply for a job. Note these are the same issues we have always faced with safeguarding personal data.
Here is just a brief list of some of the amazing applications coming our way, made possible by affordable Brain Control Interfaces (BCI):
Emotion Based Recommendation Systems
How much more powerful would a recommendation system like those of Amazon, Netflix, Pandora, YouTube, Facebook and many others be, if the content they recommended to you was based on your EmoProfile? It’s one thing to say a book is funny, or to have statistics on a group of people that typically like the same videos as you. But it is far more powerful give a recommendation based on a group of people that all laughed hysterically at the same key points across a group of movies (or cried at the same parts of various songs, or were frightened by similar news stories, etc.). This extends to services like Twitter too and similar services. Wouldn’t it be great to see a list suggested Tweeters that have the EmoProfiles similar to yours? How much time would that save you sifting out from the firehose what is of interest to you and what isn’t?
And yes I’m betting that we will soon see an emotion enhanced Like/Dislike button coming from Facebook and popping up on web content everywhere.
Emotion Guided Search
Imagine a search engine where what you were feeling at the time played a pivotal role at what search results were returned. For example, a frightened cancer patient looking for help sees a list of hospices and support groups while a curious graduate student is presented with a list of the latest research papers on chemotherapy techniques. Instead of either of them having to guess what keywords are the most relevant, their PC is sensing their emotions and altering the results based on what other searchers with similar feelings found useful during their searches.
Emotion Based Dating Services
Obviously it won’t be long before EmoDating begins to supplant speed dating. How much time could people save, looking for that special someone if you knew in advance what they really liked and disliked, instead of what the entered into a form? Form entered data always suffers from a big problem. Knowing that the information will be shared, we put down what we want others to see and therefore think of us, and what we want to think about ourselves as a person. This is a far cry from sharing with others how we actually react to seeing a puppy playing, a news story about the plight of the homeless, the latest hit song by Lady Gaga, etc.
Emotion Driven Location Based Services
A milder offshoot than EmoDating, the current crop of location based services could be updated to suggest meeting people that are nearby
Your Emotions Are Your Password
Most banks show you a picture when you login after you enter your user name. One that you selected when you set up your online banking account. But it’s only there for confirmation, for you to say it’s the one you selected, like saying to the magician “yes that is the playing card I picked out of the deck”. Wouldn’t it be far more secure to show you pictures (or play a sound, or a video, etc.), and then sense your emotional reaction to it? It will be like the scene in Bladerunner where the replicant is given the emotion-test that separates replicants from real humans, except in this case to safeguard your bank account. CAPTCHA tests that try to prevent spam are getting harder and harder because computers are getting better at “reading” them with optical character recognition, thereby forcing them to create more difficult to read images. What happens when computers get better at recognizing text than us? Simple. They will still be light years from being able to feel and even better, from feeling just like one of us. In the future, a web site will show you an image and sense your emotions. If they aren’t what a human would feel than you’re a spam-bot.
Augmented Memory
Expanding on the tech shown in the demonstration video, Emotion Recording + Cell Phones + GPS will lead to software that not only tracks your location throughout the day, recording the time as you go, but also your emotions. This will allow you to trace your progress throughout the day and remember key events you may have forgotten the details of, by searching through the days log to replay key events. Cell phones will be equipped with miniature high capacity hard drives that will allow you to record the days events via your camera and microphone, automatically “cropping” the video and audio by significance to save disk space. For example, anything with a close-up face, near proximity speech (someone talking to you), significant motion in front of the camera, loud noises, or anything that occurs while you have a strong emotional response will be archived to the disk. At night, people will replay the significant events of their life and upload that data to the “cloud” or to their home PC for persistent archiving, indexing and cataloging, a process similar to what the brain does at night when it converts selected data from short term memory to long term memory. In the future you will be able to ask your PC what you were nervous about last year during a convention in New Orleans and it will playback the audio and video surrounding that event, streamed from your life archive account from “the cloud” or your home PC. People will also be able to, with proper privacy safeguards and permision based sharing facilities, do cross-person archive searches to find mutually emotionally evocative events for fun or even for important reasons like recreating a crime scene.
Live Audience Emo-Participation, Remotely or In-Person
Bands, dancer troupes, you name it will respond to the emotions of their audience, allowing their collective mood in real time to shape the audio, video, choreography, lights, etc., creating a real-time feedback loop before performer and audience that will make every live performance unique and personal. It won’t matter either if the audience or performer(s) are in the same room or geographically diverse. In fact, the role of performer and audience could very easily merge, diverge, and shift as part of an overall group experience.
Games, Software and Web Sites That Adapt to Your Emotions
I’m going to be short here because of time reasons. But just let your mind roam for a bit on how much better of an experience a video game or software program (desktop or web) could provide you if it could tell you were frustrated, or sad, angry, or frightened?
Emotion Enhanced Speech Recognition (and other machine intelligence technologies)
This is one that would take a full article to explain so I will have to put it off for another time. But any machine intelligence technology like speech or vision recognition could benefit fantastically from sensing your emotions when trying to disambiguate one item from another during classification and pattern recogniton, where an item could be a word, phrase, object, picture, etc.
And much, much, much, more...
|
