Pi Down !!!!

Over the past week of beaverish actively I’m seen a couple of my Pi’s fail and even after a decent cool down and reboot they have failed to boot properly, loosing all network connectivity.

I’ve eventually tracked the problem down to what looks to be shoddy SD cards. I purchase 4 Kingston 16GB Class 4 cards from Amazon at about £6 a pop and now regretting scrimping on this important part.

With all the compiling and configuration it looks like they are failing one by one, however having purchase a 16GB Extreme Class 10 Sandisk earlier I’ve popped this into one of the Pi’s and its been running like a dream.

Looks like I’m going to need a few more of these Extremes to keep Colossus running

Parallel Programming in Python

Now that we have MPI installed and working we can start using it with real programming languages, and first language of choice for the Pi is ofcourse Python.

Following on with the instructions from Southampton Uni there is some information at the bottom about the use of mpi4py one of a number of Python libraries for MPI.

MPI4PY can be installed through apt-get

sudo apt-get install python-mpi4py

Once installed we can pull down some demo code

mkdir mpi4py
cd mpi4py
wget http://mpi4py.googlecode.com/files/mpi4py-1.3.tar.gz
tar xfz mpi4py-1.3.tar.gz
cd mpi4py-1.3/demo

And once its installed we can check it works by running one of the basic examples

mpirun.openmpi -np 2 -machinefile /home/pi/mpi_testing/machinefile python helloworld.py

This produces the following output if running on one node, in this instance pislave1

Hello, World! I am process 0 of 2 on pislave1.
Hello, World! I am process 1 of 2 on pislave2.

The above instructions can now be repeated on pimaster, and pislaves2, 3 and 4

Do not order your Pi from RS

Why because basically they are pants, The Register is reporting that customers who ordered in June are still waiting for their Pi’s and the best way is to cancel your order and re-order, my advice is to cancel your order and place a new one with Farnell who I’ve never had a problem with.

Such a shame that a UK company lets down another UK company by just not being able to fulfil its most basic capability that of a component supplier to supply components.

Pi Ram doubled to 512M

Various news boards are reporting that Farnell are now taking orders for PI’s with 512MB of RAM rather than 256MB

This is great news, can’t wait to see the performance increase for the X Windows system now we can allocate more to it

The Register also reports just how crap RS Components are, with customer who ordered in June still waiting for orders. My view is bin RS and go to Farnell who have shipped just about all my Pis ( 7 so far ) in under 48 hours.

Parallel Processing on a Pi … it works !

Shamelessly this is my take on the information provided by Prof Simon Cox from Southampton University in his website Steps to make a RaspberryPi Supercomputer

He first recommends some background reading on some parallel processing and the tools we’ll use for this, these can be found at

Once you have digested these and got a rough handle on what we are about to try, we can get down to configuring one of the PI’s with the required tools and libraries

Update the system

First we update out system

sudo apt-get update

Install Fortran

At this stage I skip over some of Prof Cox’s instructions, I don’t know Fortran, having not used it since 1986 and therefore its pointless even installing it

Download MPI Libraries

First we need to download and compile to MPI libraries, so lets create a directory to store the sources

mkdir ~/mpich2
cd ~/mpich2

Then get the sources from their home on the internet

wget http://www.mcs.anl.gov/research/projects/mpich2/downloads/tarballs/1.4.1p1/mpich2-1.4.1p1.tar.gz

Next steps is to unpack the compressed file we just downloaded

tar xfz mpich2-1.4.1p1.tar.gz

Compile the Libraries

Make yourself a place to put the compiled stuff – this will also make it easier to figure out what you have put in new on your system. Also you may end up building this a few times…

sudo mkdir mpich2_install

Next make a build directory (so we keep the source directory clean of build things) and change to it

mkdir mpich2_build
cd mpich2_build

Now we are going to configure the build, make the code and install it. Note that because we have decided not to install Fortran, we need to add a couple of extra commands ( –disable-f77 –disable-fc ) to configure, which stop it looking for Fortran

The instructions from Southampton University initially create a stand alone MPI package, but I want to use this almost exclusively with Python and the mpi4py library and this requires the MPI library to be built as a shared library, so we also need to add the  –enable-shared flag to the configure options

sudo /home/keith/mpich2/mpich2-1.4.1p1/configure -prefix=/home/keith/mpich2/mpich2_install --disable-f77 --disable-fc --enable-shared
sudo make
sudo make install

The above 3 commands take some time on a Pi, don’t be phased if a couple hours pass and you are still waiting lines and lines of text scrolling past your console.

Once the 3 lines are complete, MPI is compiled and installed, we can do a couple of house keeping tasks to make it easier to access the various MPI tools now available to us. First off is to add the MPI bin to our path

Add the place that you put the install to your path

export PATH=$PATH:/home/keith/mpich2/mpich2_install/bin

We can make this permanent by adding this to our .profile file

vi .profile

and add the following 2 lines to the bottom

# Add MPI to path
PATH="$PATH:/home/keith/mpich2/mpich2_install/bin"

At this stage you need to make action the changes to your path statement, you can either reboot with

sudo reboot

Or while still logged on

source ~/.profile

We can now check whether things did install or not

which mpicc
/home/keith/mpich2/mpich2_install/bin/mpicc
which mpiexec
/home/keith/mpich2/mpich2_install/bin/mpiexec

Testing the installation

Change directory back to home and create somewhere to do your tests

cd ~
mkdir mpi_testing
cd mpi_testing

Before we can run some tests we need to create a file with the IP addresses of the machines we will distribute the load onto. For the first test this will be this single node, so we only need to add its address. Create file called ‘machinefile’ and put a single line containing the ip address of this node

vi machine

And it should look something like

pislave

Now we can test whether MPI works for you on a single node

mpiexec -f machinefile -n 1 hostname
pislave1

So far so good, its all working, now lets run a simple example which calculates Pi, on a Pi, get it, funny eh ?, ok so not so funny, but its a test

cd /home/keith/mpich2/mpich2_testing
mpiexec -f machinefile -n 2 ~/mpich2/mpich2_build/examples/cpi

You will see out put that looks like

Process 0 of 2 is on pislave1
Process 1 of 2 is on pislave1
pi is approximately 3.1415926544231318, Error is 0.0000000008333387
wall clock time = 0.005259

Thats it, thanks to Prof Cox for a faultless description to make a Super Computer.

Branching out on to multiple nodes

The above set of instructions needs to be carried out on all nodes, once you have tested that all nodes work in a single node setting, we now need to set up the master/slave relationship. All the remaining instructions should now be carried out on your master node, in my instance ‘pimaster’, so first ssh into this Pi

First we need to allow master to log into all the slave nodes without needing password. We’ve already come across the use of SSH in a previous blog. First we need to create a RSA key pair on pi master

ssh-keygen -t rsa -C "keith@pimaster"
Generating public/private rsa key pair.
Enter file in which to save the key (/home/keith/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/keith/.ssh/id_rsa.
Your public key has been saved in /home/keith/.ssh/id_rsa.pub.
The key fingerprint is:
aa:8e:86:43:57:e7:09:27:bc:05:ad:1e:be:7b:ca:c1 keith@pimaster
The key's randomart image is:
+--[ RSA 2048]----+
|      .          |
|     . .         |
|    . o          |
|     B +         |
|    + X S        |
| . ..+ +         |
|...  Eo          |
|o ...o..         |
| o..o++          |
+-----------------+

Now that we have public and private keys we need to copy the public key to each slave node

cat ~/.ssh/id_rsa.pub | ssh keith@192.168.1.201 "cat >> .ssh/authorized_keys"
cat ~/.ssh/id_rsa.pub | ssh keith@192.168.1.202 "cat >> .ssh/authorized_keys"
cat ~/.ssh/id_rsa.pub | ssh keith@192.168.1.203 "cat >> .ssh/authorized_keys"
cat ~/.ssh/id_rsa.pub | ssh keith@192.168.1.204 "cat >> .ssh/authorized_keys"

You can check each of these is working by ssh’ing into each one in turn and then ‘exiting’ back out

ssh pislave1
ssh pislave2
ssh pislave3
ssh pislave4

Now back on Pi Master we need to create a machinefile with the ip addresses of all the slaves

vi machinefile

And enter the addresses of all the nodes

192.168.1.200
192.168.1.201
192.168.1.202
192.168.1.203
192.168.1.204

Now we run the same C program as before, but this time with the modified machinefile containing the ip addresses of our 4 slaves nodes. The first time we run this we’ll be prompted to enter the password for RSA key, after that it shouldn’t ask for it until after you reboot

mpiexec -f machinefile -n 5 ~/mpi/mpich2_build/examples/cpi

You should then see out put such as

Process 0 of 2 is on pimaster
Process 1 of 5 is on pislave1
Process 2 of 5 is on pislave2
Process 3 of 5 is on pislave3
Process 4 of 5 is on pislave4
pi is approximately 3.1415926544231318, Error is 0.0000000008333387

Well thats it, we have a super computer running 5 nodes in a master slave configuration, … wow !!!!

Duplicating Master into 4 Slaves

All the work so far on creating a development environment, adding languages, apache, mysql, couchdb and nagios has been on a single CD card which was referred to as PiMaster which will form the coordinating node on my super computer.

Rather that duplicate everything 4 more times for each node, we can take a copy of the PiMaster SD card and burn it onto the 4 slaves.

First we insert the PiMaster SD card into my Macbook and use df to find the number of the disk it has been assigned

sudo diskutil unmount /dev/disk2s1
sudo dd bs=1m if=/dev/rdisk2 of=./PiMasterBackup.img
sudo diskutil eject /dev/rdisk2

We now have an image PiMasterBackup.img on our hard disk, which we can copy onto the 4 slave SD cards, so we repeat the following steps for each CD in turn

sudo diskutil unmount /dev/disk2s1
sudo dd bs=1m if=./PiMasterBackup.img of=/dev/rdisk2 
sudo diskutil eject /dev/rdisk2

However this gives us 5 copies of the same SD card which all have the same ip address, server name and configuration. We now need to personalise each SD card in turn.

Set aside PiMaster SD card as that is configured as we need it. At this stage don’t put power up its PI. For this next section we use one Pi on its own, and insert, configure and then eject each slave SD card.

Repeat these steps for each slave card

  1. Power down Pi ( if not already ), making sure no other Pi’s powered up
  2. Insert Slave SD Card
  3. Power up Pi which will appear on the network as PiMaster, ip address 192.168.1.200
  4. ssh pimaster
  5. sudo vi /etc/hostname
  6. Change the name to pislaveN, where N is the number of card, 1, 2, 3 or 4
  7. sudo vi /etc/network/interfaces
  8. Change the IP address to 192.168.0.20N, where N is the number of the card
  9. sudo vi /etc/hosts
  10. Change the the name pimaster to pislaveN
  11. sudo rcconf
  12. Turn off nagios3, move the cursor down and press space, this will remove the ‘*’ from between the square brackets ‘[ ]’
  13. sudo vi /etc/mysql/my.cnf
  14. Change the value for bind-address to the ip of the slave
  15. sudo vi /etc/couchdb/local.ini
  16. Change the ipaddress for bind_address to ip address of the slave
  17. sudo reboot
  18. You should now be able to ssh into pislaveN

Repeat for cards 2, 3 and 4

Monitoring the super computer with Nagios – Part 1: Installation

Once we have all 5 nodes up and running we’ll want to monitor them, for this I intend to use Nagios, which is a enterprise monitoring tool based on a client/server architecture. We install the server on pimaster and then the client on all nodes. Nagios will then provide a web based interface to view various performance characteristics of our running super computer.

To install use the following command

sudo apt-get install nagios3

During the install a number of config screens appear, the first asks you for the password for ‘nagiosadmin’ user.

Enter a password and the again at the re-confirmation screen which appears next, then wait because the next bit takes a while but it eventually ends. You can check its running with by either running rcconf or

sudo /etc/init.d/nagios3 status
[ ok ] checking /usr/sbin/nagios3...done (running).

To access Nagios, enter the URL http://pimaster/nagios3/ into a browser and when prompted for userid and password enter, nagiosadmin for the userid and the password we entered above. You will then be presented with a screen similar to this

Congratulations you have installed Nagios and confirmed its running, its time to start configuring it to monitor our nodes

To configure each node as a client we need add the Nagios plugs. Most of this next bill is pulled from Platonic

sudo apt-get install nagios-plugins nagios-nrpe-plugin nagios-nrpe-server

You can then view the nagios server via a web browser at http://

Configuring services

We now have a number of applications installed on our Pi, including Apache, Tomcat, MySQL and CouchDB and their default install has them configured to start when the system boots, but we don’t need them all at startup, so we need a way to control their startup options.

For this we use a tool called rcconf, and a good description of its use, along with basic Debian service management can be found at Debian Admin

First we need to install rcconf

sudo apt-get install rcconf

Once installed you can start the application by typing

sudo rcconf

And the following screen will be displayed

You can then select which services to not start at start up, including mysql, tomcat6 and couch. However you can start them at any time with by using /etc/init.d

sudo /etc/init.d mysql start
sudo /etc/init.d tomcat6 start
sudo /etc/init.d couchdb start

To stop them at any time there is a similar command

sudo /etc/init.d mysql stop
sudo /etc/init.d tomcat6 stop
sudo /etc/init.d couchdb stop

To stop and the immediately restart, use the restart option

sudo /etc/init.d mysql restart
sudo /etc/init.d tomcat6 restart
sudo /etc/init.d couchdb restart

And to just get an update on the status, use the status option

sudo /etc/init.d mysql status
sudo /etc/init.d tomcat6 status
sudo /etc/init.d couchdb status

Installing and configuring CouchDB

CouchDB is a database that uses JSON for documents, JavaScript for MapReduce queries, and regular HTTP for an API

UPDATE: See the addendum at the bottom of the page for a full update of how to get couchdb running after reboot. You still need to follow these instructions, then pop down the bottom for final piece of the puzzle

I struggled with this several times, until a colleague of mine called Stephen Lock showed me the light, so shout out to him for showing me how to make it work post install.

sudo apt-get install couchdb

UPDATE: The latest version no longer fails and you end up with the last view lines of you console looking like

[ ok ] Starting database server: couchdb.
Setting up wwwconfig-common (0.2.2) ...
Setting up javascript-common (7) ...
Setting up libsctp1 (1.0.11+dfsg-2) ...
Setting up lksctp-tools (1.0.11+dfsg-2) ...
Processing triggers for menu ...

If you do get an error, then its probably the problem reported in at the bottom of this blog, but now we can just continue by making sure couch is binding to the correct IP address so we can connect to it from another machine

cd /etc/couchdb
sudo vi local.ini

To change the bind_address to the address of the machine you are installing on, look for the following line

;bind_address = 127.0.0.1

Remove the ; from the beginning and then change the ip address to the address of the server on which couch is installed. Finall restart the couchdb service

/etc/init.d/couchdb restart

This time you’ll see the following line which signifies Couch is up and running

[ ok ] Starting database server: couchdb.

To check its all working, open a browser and enter the url http://hostname:5984 and you’ll see a single line of text along the lines of

{"couchdb":"Welcome","version":"1.2.0"}

We can then bring up the admin console by entering in a browser, the url http://hostname:5984/_utils which will show something like this

If install fails

When this is finished it will probably report an error that it couldn’t start couch. Typically the install will finish with the following lines

[....] Starting database server: couchdbApache CouchDB needs write permission on the PID file: /var/run/couchdb/couchdb.pid
 failed!
invoke-rc.d: initscript couchdb, action "start" failed.
dpkg: error processing couchdb (--configure):
 subprocess installed post-installation script returned error exit status 1
Setting up libsctp1 (1.0.11+dfsg-2) ...
Setting up lksctp-tools (1.0.11+dfsg-2) ...
Processing triggers for menu ...
Errors were encountered while processing:
 couchdb
E: Sub-process /usr/bin/dpkg returned an error code (1)

Don’t panic !, we just need to do some jiggery-pokery to set the permissions and ip addresses that couch uses. First we need to make sure that couch owns the right folders which are not set correctly during install

sudo chown couchdb:couchdb -R /var/lib/couchdb /var/log/couchdb /var/run/couchdb
sudo update-rc.d couchdb defaults

Then we need to make sure couch is binding to the correct IP address so we can connect to it from another machine

cd /etc/couchdb
sudo vi local.ini

To change the bind_address to the address of the machine you are installing on, look for the following line

;bind_address = 127.0.0.1

Remove the ; from the beginning and then change the ip address to the address of the server on which couch is installed. Finall restart the couchdb service

/etc/init.d/couchdb start

This time you’ll see the following line which signifies Couch is up and running

[ ok ] Starting database server: couchdb.

To check its all working, open a browser and enter the url http://hostname:5984 and you’ll see a single line of text along the lines of

{"couchdb":"Welcome","version":"1.2.0"}

We can then bring up the admin console by entering in a browser, the url http://hostname:5984/_utils which will show something like this

 

 

Addendum

I’ve had problems with couchdb starting because of permissions and searching the internet shows this to be a common bug with the current version and is because /var/run/couchdb has its permissions set back to root after reboot. The easiet way to fix this for now is to modiy rc.local and make it reset the permissions on startup

sudo vi /etc/init.d/couchdb

And look for the method that starts with the following at about line 84

start_couchdb () {

Add a new line after mkdir -P “$RUN_DIR”
and add the following

start_couchdb () {
    # Start Apache CouchDB as a background process.

    mkdir -P "$RUN_DIR"

    #Add this line
    chown couchdb:couchdb -R "$RUN_DIR"

    command="$COUCHDB -b"

Adding a few more languages

Our core platform is now configured with Python ( by default ), PHP ( installed ) and Java ( installed ), we also get Perl ( by default ), but I want more languages. I spend a lot of more time these days either using Ruby or Groovy and have a passing interest in Scala for its functional programming capabilities.

So lets add each of these in turn, first Ruby, the best source of information is found at RPi Ruby on Rails

sudo apt-get install ruby
sudo apt-get install rails

We can test that all is installed with 3 commands,

ruby -v
ruby 1.9.3p194 (2012-04-20 revision 35410) [arm-linux-eabihf]
gem -v
1.8.23
rails -v
Rails 2.3.14

Once Ruby is installed lets install a couple of key JVM languages, specifically Groovy and Scala, first Groovy

 
sudo apt-get install groovy
sudo apt-get install groovy-doc

We can check Groovy is installed by checking its version

groovy -version
Groovy Version: 1.8.6 JVM: 1.7.0_07 Vendor: Oracle Corporation OS: Linux

And now we install Scala

sudo apt-get install scala
sudo apt-get install scala-doc

Again we can check it is installed by checking its version

scala -version
Scala code runner version 2.9.2 -- Copyright 2002-2011, LAMP/EPFL