Saturday, July 4, 2015

Flappy Brain - Cambridge Jam Raspberry Pi project

At the last Egham Raspberry Jam I showed the Mindflex game EEG Brainwave reader (CNET article from when it was launched : http://www.cnet.com/uk/news/moving-objects-with-mattels-brainwave-reading-mindflex/) displaying bar graphs for the different brainwave signals.  It was interesting but served no real purpose other than to show the EEG was doing something. Even got a mention on the Linux Luddites Podcast (https://linuxluddites.com/shows/episode-41/)


So, for the Cambridge Raspberry Jam I wanted to get the EEG reader to be used for something.  I decided it had to be a single signal and immediately Flappy Birds came to mind.
Nice simple one button game.
Decision made I went about coding the game in Python on the Raspberry Pi.  I had already done the work to get the EEG

To hack the Mindflex and get it working with an Arduino I used this great guide and the code from the following article: http://frontiernerds.com/brain-hack.
This articles gives a lot of detailed background on the Mindflex and how it as well as lovely clear pictures showing where the TX and GND wires need to be attached.
For strain relief the article uses hot glue.  I wrapped the wires around the already in place knot so the wires could not be pulled no matter what.

The Processing code provided didn't work for me on the Raspberry Pi and that's the main reason I had to code up my own visualisation program.




The example I used to send the data is BrainSerialTest with one small modification. I commented out the following line as it caused the import and breaking out of the variable in Python to fail.  (Limit of my coding ability showing):
        Serial.println(brain.readErrors());

To read this from Python I had to install pyserial as it wasn't installed by default using the following commands.

apt-get update
apt-get install python-serial

You will have to use sudo for this. I left it out from the commands so that  you have to make the decision to run the commands

With the Mindflex hacked, and hooked to the Arduino and then sending it's data over serial I did the code to visualise the data.  Here is the Python code  I used on the Raspberry Pi for the last Egham Raspberry Jam

import serial
import pygame
# initialise pygame
pygame.init()

# Define the colors we will use in RGB format
black = [ 0, 0, 0]
white = [255,255,255]
red = [255, 0, 0]
blue = [0,0,255]
darkBlue = [0,0,128]
pink = [255,200,255]
green = [0,255,0]
orange = [255,102,0]
brown = [153,102,0]
yellow = [255,255,0]
purple = [128,0,128]

# Set the font for the text. Windows computer so usd Ariel

myfont = pygame.font.SysFont("OpenDyslexic", 18)


# Set the height and width of the screen
size=[800,600]
screen=pygame.display.set_mode(size)
# Fill the screen White
screen.fill(white)
# Put something in the application Bar
pygame.display.set_caption("Serial Mind reader")
pygame.display.update()

done = False
div = 2000
fatt = 20
fmed = 20
fdelta = 20
ftheta = 20
flalpha = 20
fhalpha = 20
flbeta = 20
fhbeta = 20
flgamma = 20
fhgamma = 20

ser = serial.Serial('/dev/ttyUSB0',9600)

# Clear the serial input buffer
ser.flushInput()

pygame.draw.rect(screen, white, (0,0,800,600), 0)
pygame.display.update()


while done == False:
    for event in pygame.event.get():
        if event.type == pygame.QUIT:
            done=True

    data = ser.readline()

    signal,att,med,delta,theta,lalpha,halpha,lbeta,hbeta,lgamma,hgamma = data.split(",")

    print signal
    print att
    print data


#  clear screen before putting new square on it 
    pygame.draw.rect(screen, white, (0,0,800,600), 0)
    pygame.display.update()


    fatt = int(att)
    fmed = int(med)
    fdelta = int(delta)
    ftheta = int(theta)
    flalpha = int(lalpha)
    fhalpha = int(halpha)
    flbeta = int(lbeta)
    fhbeta = int(hbeta)
    flgamma = int(lgamma)
    fhgamma = int(hgamma)

    pygame.draw.rect(screen, red, (1,600-(fatt*4),75,fatt*4), 0)
    pygame.draw.rect(screen, blue, (76,600-(fmed*4),75,fmed*4), 0)
    pygame.draw.rect(screen, pink, (152,600-(fdelta/div/2),75,(fdelta/div/2)), 0)
    pygame.draw.rect(screen, green, (227,600-(ftheta/div),75,(ftheta/div)), 0)
    pygame.draw.rect(screen, orange, (303,600-(flalpha/div),75,(flalpha/div)), 0)
    pygame.draw.rect(screen, darkBlue, (379,600-(fhalpha/div),75,(fhalpha/div)), 0)
    pygame.draw.rect(screen, brown, (455,600-(flbeta/div),75,(flbeta/div)), 0)
    pygame.draw.rect(screen, black, (531,600-(fhbeta/div),75,(fhbeta/div)), 0)
    pygame.draw.rect(screen, yellow, (607,600-(flgamma/div),75,(flgamma/div)), 0)
    pygame.draw.rect(screen, purple, (683,600-(fhgamma/div),75,(fhgamma/div)), 0)

    screen.blit(myfont.render("Attention",1,black), (4, 1))
    screen.blit(myfont.render("Mediation",1,black), (79, 1))
    screen.blit(myfont.render("Delta",1,black), (155, 1))
    screen.blit(myfont.render("Theta",1,black), (230, 1))
    screen.blit(myfont.render("Low Alpha",1,black), (306, 1))
    screen.blit(myfont.render("High Alpha",1,black), (382, 1))
    screen.blit(myfont.render("Low Beta",1,black), (458, 1))
    screen.blit(myfont.render("High Beta",1,black), (534, 1))
    screen.blit(myfont.render("Low Gamma",1,black), (610, 1))
    screen.blit(myfont.render("High Gamma",1,black), (686, 1))
# show the whole thing
    pygame.display.update()

pygame.quit ()


As a side note. I did a D&T day at my kids school where I used this code for a Year 1 demonstration. All the kids got to see their brainwaves bouncing around as they moves, did maths and recited poems.  It even went really flat when the kids closed their eyes and calmed down.   Some of the kids had to wait so long that they missed their morning break for the opportunity to see their brains waves bouncing around,

Next onto getting the Flappy Birds inspired game (for the purest you could say helicopter inspired) game working.
I thought myself to code in the 80's on a Commodore 64 and to be honest I haven't updated my skills since, so doing the same in Python was a new challenge for me.
Fortunately, I read an article in Linux Format Issue 112 from December 2008 that made a top down racer using Python and Pygame in 100 lines of code. http://www.linuxformat.com/includes/download.php?PDF=LXF112.tut_code.pdf
Yes, using a tutorial from over 7 years ago.  I may not be great at coding but I do hoard useful bits of knowledge.  I bought the Mindflex from eBay in September 2014 and did nothing with it until January.  Another bit of my electronics stash.

The Racer game if you think about it is the same but more complex than the Flappy Brain game with a few differences.
Racer scrolls vertically, Flappy  Brain scrolls horizontally.  That's just changing the value of the X variable instead of the Y variable
Then it's just change some of the graphics.

Once I figured out the Racer code I worked out the bits I could throw away and then bits I had to keep as well as worked out how to modify the code to do what I needed.

This took me a while to work through again due to a lack of knowledge or practice.

In the end I created the final game that was shown at the Cambridge Raspberry Jam.

Code Below:

from pygame import *
import random
import time
import serial

# set up the serial port
ser = serial.Serial('/dev/ttyUSB0',9600)

# Clear the serial input buffer
ser.flushInput()

# variables for colours 
black = [ 0, 0, 0]
white = [255,255,255]
red = [255, 0, 0]
blue = [0,0,255]
darkBlue = [0,0,128]
pink = [255,200,255]
green = [0,255,0]
orange = [255,102,0]
brown = [153,102,0]
yellow = [255,255,0]
purple = [128,0,128]

# gap in wall
gap  = 200

# width and height of screen 
width = 1000
height = 600
count_gap = int(height/gap)

# 0 = hard, 1 = easier, 2 is easiest
difficulty = 2

# class to create sprites and render them
class Sprite:
    def __init__(self,xpos,ypos,filename):
        self.x = xpos
        self.y = ypos
        self.bitmap = image.load(filename)
    def render(self):
        screen.blit(self.bitmap,(self.x,self.y))

# screen size
size=[width,height]
# initialise pygame
init()

# create the screen (window)
screen = display.set_mode(size)

# Caption to go at the top of the window
display.set_caption("Flappy EEG 2")

# set the music, its volume and start to play. -1 means infinite loop
mixer.music.load("Pinky_and_the_Brain.ogg")
mixer.music.set_volume(0.5)
mixer.music.play(-1)

# crreate sound for crash
crasheffect = mixer.Sound("ouch.ogg")


# fill the screen with blue, update the screen
# not really required but I like it
screen.fill(blue)
display.update()
#time.sleep(1)

# create the sprites for the brain, columns and the background image

playerbrain = Sprite(20,200,"brain_75.png")
column1 = Sprite(1200,100,"column1.png")
column2 = Sprite(1200,100,"column2.png")
background = Sprite(0,0,"background.png")

# set fonts for different purposes
scorefont = font.Font(None,60)
datafont = font.Font(None,20)
overfont = font.Font(None,100)

# set default values for some variables
score = 0
maxscore = 0
quit = 0
gameover = 0

# master loop for the program. If quit == 0 then exit program
while quit == 0:
# flush the serial port of all data to begin fresh
    ser.flushInput()

    gameover = 0

# set the height of top column
    column1.y = (random.randrange(0,(count_gap))*gap)-800

# set the height of the bottom column
    column2.y = column1.y + 800 + gap

# start of loop (using while) to move the columns

# start off screen to the right
    x = width +50

# x>-100 means still valid (yes there is a minus in there)
# gameover when collision
# quit selected. either pressed q or click x on window
    while x >-100 and gameover == 0 and quit == 0:

# increment the score and if higher than maxscore make maxscore = score
        score = score + 1
        if score > maxscore:
            maxscore = score

# update columns location and set x positions
        x = x - 50
        column1.x = x
        column2.x =x  

        data = ser.readline()
#        print data

        signal,att,med,delta,theta,lalpha,halpha,lbeta,hbeta,lgamma,hgamma = data.split(",")

        print "signal: " + signal
        print "attention: " + att
        print "data: " + data
        intatt = int(att)
        if intatt > 90:
            intatt = 90   
        brainpos = intatt * 6
# set brain location based att (attention)

# is intatt near the gap above 
        if brainpos < column1.y +800 and brainpos > column1.y + 800 - (difficulty * 10):
            playerbrain.y = column1.y +800 +70
            print "brain near top and moved down!"
# is intatt near gap bottom
        elif brainpos > column2.y-75 and brainpos < column2.y + (difficulty * 10):
            playerbrain.y = column1.y +800 +70
            print "brain near bottom and moved up!"

        else:
            playerbrain.y = brainpos
            print "brain where is should be"


#        print playerbrain.y
        background.render()
        playerbrain.render()
        column1.render()
        column2.render() 

# display some information on screen
        screen.blit(scorefont.render("Score: "+ str(score),1,white), (100, 5))
        screen.blit(scorefont.render("High Score: "+ str(maxscore),1,white), (400, 5))

        screen.blit(datafont.render("signal: "+ signal,1,white), (5, 570))
        screen.blit(datafont.render("attention: "+ att,1,white), (150, 570))

        screen.blit(datafont.render("playerbrain.y: "+ str(brainpos),1,white), (250, 570))
        screen.blit(datafont.render("column1.y: "+ str(column1.y+800),1,white), (500, 570))
        screen.blit(datafont.render("difficulty: "+ str(difficulty),1,white), (650, 570))

        display.update()

#        print playerbrain.y

# collision dection
        if ((playerbrain.y < column1.y+801 or playerbrain.y > column2.y-74) and (x <150 and x > 20)):
            mixer.music.stop()
            mixer.Sound.play(crasheffect)
            print "BUMP"
            gameover = 1

# check if clicked x on window to exit 
        for ourevent in event.get():
            if ourevent.type == QUIT:
                quit = 1

# has key been pressed. K_q is to quit 
            if ourevent.type == KEYDOWN:
                if ourevent.key == K_DOWN:
                    playerbrain.y = playerbrain.y+10

                if ourevent.key == K_UP:
                    playerbrain.y = playerbrain.y-10

                if ourevent.key == K_q:
                    quit = 1 

# if game over show message
    while gameover == 1 and quit == 0:
        screen.blit(overfont.render("GAME OVER",1,yellow), (380, 260))
        display.update()

# then wait for a key to pressed  before starting again
        for ourevent in event.get():
            if ourevent.type == KEYDOWN:
                if ourevent.key == K_0:
                    difficulty = 0
                    score = 0
                    gameover = 0
                    mixer.music.play(-1)

                if ourevent.key == K_1:
                    difficulty = 1
                    score = 0
                    gameover = 0
                    mixer.music.play(-1)

                if ourevent.key == K_2:
                    difficulty = 2
                    score = 0
                    gameover = 0
                    mixer.music.play(-1)

                if ourevent.key == K_SPACE:
                    score = 0
                    gameover = 0
                    mixer.music.play(-1)

                if ourevent.key == K_q:
                    quit = 1 
                    score = 0
                    gameover = 0


The sharp eyed amongst you will recognise the bits taken from the Racer game and the bits taken from the original visualisation code I create.

I added the ability to Quit the game by pressing 'q' and also 3 difficulty levels. On the Game Over screen press 1, 2, or 3 with the higher number being easier.
It does this by widening the readings that will get you through the gap.

At the Cambridge Raspberry Jam I had the pleasure of meeting Alex Eames who brought us to HDMIPi and also the RasPi IO Duino which was another successful Kickstarter and is now for general sale.  Alex did a great video which you can see below.



Following this the project was picked up by the Raspberry Pi Foundation and featured on their Blog (https://www.raspberrypi.org/flappy-brain/) and most recently was featured as The MagPi Magazine Issue 35 in their Amazing Pi Projects List (https://www.raspberrypi.org/magpi/)
The MagPi magazine is a a resource for everybody with a Raspberry Pi. It includes articles, project details and tutorials in everything from Scratch, to Python and C++ and has software and hardware projects.  Also, they announced recently that it is going to become a physical magazine soon and be available to buy in shops as well as the current PDF download and acces through the Apple App Store and Google Play stores.

It's been a fun few months and to me is an example of what can be done if you take advantage of the resources out there if you have idea.  For this project I used some great code and guide to hack the Mindflex.  Code from a 2008 Linux Format article and some previous work I did to figure out how to read serial data on the Raspberry Pi.

I will be bringing Flappy Brain to the next Egham Raspberry Jam. It is on the 12th of July 2015 and as always is free to attend.  All that's required is to register beforehand on the Eventbrite page.
http://eghamjam8.eventbrite.com

Maybe I'll see you there.