Configuring Eclipse to run MPI4PY

If Eclipse is your IDE of choice then you can configure it to run MPI4PY tools from within the editor itself.

The primary app we use for running parallel programms is mpiexec and this needs to be run as an external application from Eclipse, this is asked from the Top Level Run menu, then click External Tools, and then click External Tool Configurations

Run -> External Tools -> External Tool Configurations...

This will bring up the following screen

Select Program in the list on the left, and then click the Add New Icon, which is the left most icons, the square with the + on it

In Location: we put the full path to the app we want to run. You can get this from the command line by typing ‘which mpiexec’ and using the result returned in that field

In Working Directory: use the Eclipse Variable workspace_loc with the addition of the path to where the Python files are stored

${workspace_loc:/Parallel-O-Gram}

Finally in Arguments: use the following information. You can edit this to change the number of processes as and when

-n 4 -f machinefile python ${resource_name}

 

Parallel Programming with Python – The Basics

Parallel Programming with Python – The Basics

Before you begin follow the instructions in Parallel Programming in Python to learn how to install the required Python libraries

And I mean the real basics, this is more of a blog about me learning how to do parallel programming in Python than a detailed tutorial. I’ll probably come back and revisit it as I learn more and more.

Most of this text has been taken from the very good book on parallel programming called Parallel Programming with MPI, available at Amazon.

However this book is almost exclusively C and Fortran based and we are new age Python programmers, so as I’ve worked through the examples I’ve converted them to Python and added my own explanations to some of the topics and problems I’ve come across

I searched high and wide for some tutorials on the web about MPI4PY and they all seem to be the same rehash of an academic presentation so they all demonstrate pretty much the same concepts in the same way using the same language. To be honest I really struggled with what I was reading, and it took me a while to get my head around parallel programming with MPI. There are a few basic constructs that you need to get your head round first before trying to use the MPI libraries in anger

The Basics

MPI can be huge and daunting when you first start with it, especially trying to understand the way the APIs work in 1 of 2 very distinct ways, ( see Basic Mindset below for more details ), but the reason for MPI is in its title, this is a message passing API that allows the same programming running on multiple processes or processors to communicate in realtime with each other.

Rather than get into the nasty details of asynchronous programming which comes later, there are 2 basic ways MPI works,

  • Point To Point – Allowing one process to send a message to another process, each process identified by a unique number
  • Multicast – Allows one process to distribute data ( i.e Load ) to multiple other processes and then recombine the data back into a single piece. The process with id 0, sends the data and all other processes with unique numbers > 0 receive it.

Executing an MPI Program

Hopefully you followed the instructions for install MPI on your PI, ran through a couple of the C examples that came with the install and then went on to install MPI4PI and downloaded and ran one of the examples. To recap we first need to identify all the machines in our network, we do this by creating a file with all the IP addresses of the machine we communicate with. Typically this is called machinefile and for now can look like this

127.0.0.1

To run parallel programmes written in Python we still use mpiexec command, but tell it to run the python interpreter,

mpiexec -n 2 -f machinefile python program.py

Where -n is the number of processes to create with our program, -f points to the name with all the IP addresses on machines to use in our network, and program.py is the name of the Python programme we want to execute.

Basic Send and Receive Example

This is an example of a simple programme which if run twice via the MPI exec will send a message from one of its instances to the other.

from mpi4py import MPI			# Import the MPI library

comm = MPI.COMM_WORLD			# Initialise MPI library ( more on this later )
rank = comm.Get_rank()			# Get the unique number of this process, know as its rank
size = comm.Get_size()			# Get the number of processes available to this task
name = MPI.Get_processor_name();	# Gets the name of the processor, usually its network name

print
print "Rank : ", rank
print "Size : ", size
print "Name : ", name

if rank == 0:				# If the first process, create the data and send it
        data = [1, 2, 3, 4, 5]
        print "Sending..."
        comm.send(data, dest=1)
        print "Sent {0}".format(data)
else:					# Otherwise I am the receiver, so receive it
        print "Receiving..."
        data = comm.recv(source=0)
        print "Recv {0}".format(data)

print "Data on rank {0} : {1}", rank, data

If we run this, then something like the following will be displayed

Rank :  0
Size :  2
Name :  fender
Sending...
Sent [1, 2, 3, 4, 5]
Data on rank {0} : {1} 0 [1, 2, 3, 4, 5]

Rank :  1
Size :  2
Name :  fender
Receiving...
Recv [1, 2, 3, 4, 5]
Data on rank {0} : {1} 1 [1, 2, 3, 4, 5]

Note, as with all parallel programming you cannot determine the order in which the processes will produce output, so for all these examples you’ll get something that looks similar but the lines may be in slightly different order

Basic Mindset

MPI is a unqiue way of coding, generally with procedural and object orientated programming you expect every function call to behave in one way and one way only; you have a defined set of parameters defined as the API, the function normally has an explanation of what it does, and more often than not the function returns a value. If you call the API with the right parameters you expect it always to behave the same. However for alot of MPI functions this is not the case and for many it depends on the context of the caller and whether the function takes data or returns data.

A good example is the method scatter which scatters the elements of an list to all
available processes

return = comm.scatter(self, sendBuffer, recvBuffer=None, root=0)

Basic parallel programs tend to have sequential sections and then parallel sections. When the mpi subsystem runs you program, it sends a copy of the programm to every process it you tell it about ( more on this later ) and executes them all at the same time ( this is parallel programming after all ), The first program running is often called the root and is generally given the unique id number 0, every other programme is then given the number 1, 2, 3 etc in sequence to signify their uniqueness and the fact that they are not root ( i.e = 0 )

At some point in your code you will have created a list, and pass it to scatter

if rank == 0:
	data = [1, 2, 3] 
else
	data = None
return = comm.scatter(data, root=0)

This says, scatter the elements of data to all processes and remember that they all came from root=0, ( the first process ), now you have to imagine all the process running this parallel, each one hits the above lines at approximately the same time.

Process with rank=0, has data set to an array of 3 numbers, all other processes have no array, when each process makes the call, the library hides the magic, and distributes the array to all processes, so that each return ends up with a different number

Process with rank = 0, gets 1
Process with rank = 1, gets 2, but Data is still None
Process with rank = 2, gets 3, but Data is still None

Its quite a mind change to visualise the same code running at the same time and depending upon the value of rank, different behaviour can happen explicitly as in our if statement to set the data only for rank = 0, or implicitly as within the MPI calls

Wierd or what !!!!

Broadcast Example

Point to point doesn’t always have to be one process to another, it can one process to a number of processes. For this we use the broadcast API call

from mpi4py import MPI

comm = MPI.COMM_WORLD
rank = comm.Get_rank()
size = comm.Get_size()
name = MPI.Get_processor_name();

print "Rank : ", rank
print "Size : ", size
print "Name : ", name

if rank == 0:
        data = [1, 2, 3, 4, 5]
else:
        data = None

# bcast sends the same data to all processes
# For rank 0, data is not None and therefore is the one that is issued and return
# For rank > 0, data passed is None, but the value returned is the data sent from rank 0
data = comm.bcast(data, root=0)
print "Data on rank {0} : {1}".format(rank, data)

Run this example and you’ll see something like the following which shows that the array defined by process with rank = 0 has been passed to all other processes

Rank :  0
Size :  3
Name :  fender
Rank :  1
Size :  3
Name :  fender
Rank :  2
Size :  3
Name :  fender
Data on rank 0 : [1, 2, 3, 4, 5]
Data on rank 1 : [1, 2, 3, 4, 5]
Data on rank 2 : [1, 2, 3, 4, 5]

This is useful if you need multiple processes to work on the same data.

Scatter Example

from mpi4py import MPI

comm = MPI.COMM_WORLD
rank = comm.Get_rank()
size = comm.Get_size()
name = MPI.Get_processor_name();

print "Rank :{0}, Size :{1}, Name : {2}".format(rank, size, name)

if rank == 0:
        data = [1,2,3]
else:
        data = None

# Scatters distributes the values of an array out to waiting processes
# If this process is rank 0, then this sends the data to all processes, 
# If its rank 1 or above, it waits for data from scatter
# Each call to scatter returns with one element from the array
data = comm.scatter(data, root=0)
print "Data on rank {0} : {1}".format(rank, data)

# Then we add one to each value
data = data + 1

# Gather, collects all single values of an array back into an array
# A Call to gather with rank = 0, returns with an array, and takes a value to be gathered
# A call to gather with rank 1 or above takes the value to be gathered
data = comm.gather(data,root=0)
if rank==0:
    print "Data on rank {0} : {1}".format(rank, data)
# At this point, all other processes will have complete gather but it will have returned them nothing, 
# only the process with rank 0 gets a return value

If we run this example then we see the something like the following

Rank :0, Size :3, Name : fender
Data on rank 0 : 1
Data on rank 0 : [2, 3, 4]
Rank :1, Size :3, Name : fender
Data on rank 1 : 2
Rank :2, Size :3, Name : fender
Data on rank 2 : 3

Scatter Example 2

In this example we show that we don’t just pass single values, we can pass objects. In this instance the objects are further arrays of values, but can be any valid Python object

comm = MPI.COMM_WORLD
rank = comm.Get_rank()
size = comm.Get_size()
name = MPI.Get_processor_name();

print "Rank :{0}, Size :{1}, Name : {2}".format(rank, size, name)

if rank == 0:
        data = [[1,2,3], [4,5,6], [7,8,9]]
else:
        data = None

# Scatters distributes the values of an array out to waiting processes
# If this process is rank 0, then this sends the data to all processes, 
# If its rank 1 or above, it waits for data from scatter
# Each call to scatter returns with one element from the array
data = comm.scatter(data, root=0)
comm.sca
# Then we do something with the data
for i in range(0,len(data)) :
    data[i] = data[i] + 1

# Gather, collects all single values of an array back into an array
# A Call to gather with rank = 0, returns with an array, and takes a value to be gathered
# A call to gather with rank 1 or above takes the value to be gathered
data = comm.gather(data,root=0)
if rank==0:
    print "Data on rank {0} : {1}".format(rank, data)

Running this program gives us output similar to this

Rank :0, Size :3, Name : fender
Rank :1, Size :3, Name : fender
Rank :2, Size :3, Name : fender
Data on rank 0 : [[2, 3, 4], [5, 6, 7], [8, 9, 10]]

Each array has been manipulated and returned to the original sender

Pi Back !!!

I posted earlier that I was suffering problems from SD cards in Pi Down

Well over the weekend I received 4 16GB Extreme Class 10 Sandisk SD cards from Amazon and set about taking an image of each current SD card using the instructions here and copying the image onto the new card

At the same time I ran rasp-config and switch off all Turbo modes, and I’m happy to say that all nodes and up and running and working a treat and now running MPI libraries and their Python variants MPI4PI

Pi Down !!!!

Over the past week of beaverish actively I’m seen a couple of my Pi’s fail and even after a decent cool down and reboot they have failed to boot properly, loosing all network connectivity.

I’ve eventually tracked the problem down to what looks to be shoddy SD cards. I purchase 4 Kingston 16GB Class 4 cards from Amazon at about £6 a pop and now regretting scrimping on this important part.

With all the compiling and configuration it looks like they are failing one by one, however having purchase a 16GB Extreme Class 10 Sandisk earlier I’ve popped this into one of the Pi’s and its been running like a dream.

Looks like I’m going to need a few more of these Extremes to keep Colossus running

Parallel Programming in Python

Now that we have MPI installed and working we can start using it with real programming languages, and first language of choice for the Pi is ofcourse Python.

Following on with the instructions from Southampton Uni there is some information at the bottom about the use of mpi4py one of a number of Python libraries for MPI.

MPI4PY can be installed through apt-get

sudo apt-get install python-mpi4py

Once installed we can pull down some demo code

mkdir mpi4py
cd mpi4py
wget http://mpi4py.googlecode.com/files/mpi4py-1.3.tar.gz
tar xfz mpi4py-1.3.tar.gz
cd mpi4py-1.3/demo

And once its installed we can check it works by running one of the basic examples

mpirun.openmpi -np 2 -machinefile /home/pi/mpi_testing/machinefile python helloworld.py

This produces the following output if running on one node, in this instance pislave1

Hello, World! I am process 0 of 2 on pislave1.
Hello, World! I am process 1 of 2 on pislave2.

The above instructions can now be repeated on pimaster, and pislaves2, 3 and 4

Do not order your Pi from RS

Why because basically they are pants, The Register is reporting that customers who ordered in June are still waiting for their Pi’s and the best way is to cancel your order and re-order, my advice is to cancel your order and place a new one with Farnell who I’ve never had a problem with.

Such a shame that a UK company lets down another UK company by just not being able to fulfil its most basic capability that of a component supplier to supply components.

Pi Ram doubled to 512M

Various news boards are reporting that Farnell are now taking orders for PI’s with 512MB of RAM rather than 256MB

This is great news, can’t wait to see the performance increase for the X Windows system now we can allocate more to it

The Register also reports just how crap RS Components are, with customer who ordered in June still waiting for orders. My view is bin RS and go to Farnell who have shipped just about all my Pis ( 7 so far ) in under 48 hours.

Parallel Processing on a Pi … it works !

Shamelessly this is my take on the information provided by Prof Simon Cox from Southampton University in his website Steps to make a RaspberryPi Supercomputer

He first recommends some background reading on some parallel processing and the tools we’ll use for this, these can be found at

Once you have digested these and got a rough handle on what we are about to try, we can get down to configuring one of the PI’s with the required tools and libraries

Update the system

First we update out system

sudo apt-get update

Install Fortran

At this stage I skip over some of Prof Cox’s instructions, I don’t know Fortran, having not used it since 1986 and therefore its pointless even installing it

Download MPI Libraries

First we need to download and compile to MPI libraries, so lets create a directory to store the sources

mkdir ~/mpich2
cd ~/mpich2

Then get the sources from their home on the internet

wget http://www.mcs.anl.gov/research/projects/mpich2/downloads/tarballs/1.4.1p1/mpich2-1.4.1p1.tar.gz

Next steps is to unpack the compressed file we just downloaded

tar xfz mpich2-1.4.1p1.tar.gz

Compile the Libraries

Make yourself a place to put the compiled stuff – this will also make it easier to figure out what you have put in new on your system. Also you may end up building this a few times…

sudo mkdir mpich2_install

Next make a build directory (so we keep the source directory clean of build things) and change to it

mkdir mpich2_build
cd mpich2_build

Now we are going to configure the build, make the code and install it. Note that because we have decided not to install Fortran, we need to add a couple of extra commands ( –disable-f77 –disable-fc ) to configure, which stop it looking for Fortran

The instructions from Southampton University initially create a stand alone MPI package, but I want to use this almost exclusively with Python and the mpi4py library and this requires the MPI library to be built as a shared library, so we also need to add the  –enable-shared flag to the configure options

sudo /home/keith/mpich2/mpich2-1.4.1p1/configure -prefix=/home/keith/mpich2/mpich2_install --disable-f77 --disable-fc --enable-shared
sudo make
sudo make install

The above 3 commands take some time on a Pi, don’t be phased if a couple hours pass and you are still waiting lines and lines of text scrolling past your console.

Once the 3 lines are complete, MPI is compiled and installed, we can do a couple of house keeping tasks to make it easier to access the various MPI tools now available to us. First off is to add the MPI bin to our path

Add the place that you put the install to your path

export PATH=$PATH:/home/keith/mpich2/mpich2_install/bin

We can make this permanent by adding this to our .profile file

vi .profile

and add the following 2 lines to the bottom

# Add MPI to path
PATH="$PATH:/home/keith/mpich2/mpich2_install/bin"

At this stage you need to make action the changes to your path statement, you can either reboot with

sudo reboot

Or while still logged on

source ~/.profile

We can now check whether things did install or not

which mpicc
/home/keith/mpich2/mpich2_install/bin/mpicc
which mpiexec
/home/keith/mpich2/mpich2_install/bin/mpiexec

Testing the installation

Change directory back to home and create somewhere to do your tests

cd ~
mkdir mpi_testing
cd mpi_testing

Before we can run some tests we need to create a file with the IP addresses of the machines we will distribute the load onto. For the first test this will be this single node, so we only need to add its address. Create file called ‘machinefile’ and put a single line containing the ip address of this node

vi machine

And it should look something like

pislave

Now we can test whether MPI works for you on a single node

mpiexec -f machinefile -n 1 hostname
pislave1

So far so good, its all working, now lets run a simple example which calculates Pi, on a Pi, get it, funny eh ?, ok so not so funny, but its a test

cd /home/keith/mpich2/mpich2_testing
mpiexec -f machinefile -n 2 ~/mpich2/mpich2_build/examples/cpi

You will see out put that looks like

Process 0 of 2 is on pislave1
Process 1 of 2 is on pislave1
pi is approximately 3.1415926544231318, Error is 0.0000000008333387
wall clock time = 0.005259

Thats it, thanks to Prof Cox for a faultless description to make a Super Computer.

Branching out on to multiple nodes

The above set of instructions needs to be carried out on all nodes, once you have tested that all nodes work in a single node setting, we now need to set up the master/slave relationship. All the remaining instructions should now be carried out on your master node, in my instance ‘pimaster’, so first ssh into this Pi

First we need to allow master to log into all the slave nodes without needing password. We’ve already come across the use of SSH in a previous blog. First we need to create a RSA key pair on pi master

ssh-keygen -t rsa -C "keith@pimaster"
Generating public/private rsa key pair.
Enter file in which to save the key (/home/keith/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/keith/.ssh/id_rsa.
Your public key has been saved in /home/keith/.ssh/id_rsa.pub.
The key fingerprint is:
aa:8e:86:43:57:e7:09:27:bc:05:ad:1e:be:7b:ca:c1 keith@pimaster
The key's randomart image is:
+--[ RSA 2048]----+
|      .          |
|     . .         |
|    . o          |
|     B +         |
|    + X S        |
| . ..+ +         |
|...  Eo          |
|o ...o..         |
| o..o++          |
+-----------------+

Now that we have public and private keys we need to copy the public key to each slave node

cat ~/.ssh/id_rsa.pub | ssh keith@192.168.1.201 "cat >> .ssh/authorized_keys"
cat ~/.ssh/id_rsa.pub | ssh keith@192.168.1.202 "cat >> .ssh/authorized_keys"
cat ~/.ssh/id_rsa.pub | ssh keith@192.168.1.203 "cat >> .ssh/authorized_keys"
cat ~/.ssh/id_rsa.pub | ssh keith@192.168.1.204 "cat >> .ssh/authorized_keys"

You can check each of these is working by ssh’ing into each one in turn and then ‘exiting’ back out

ssh pislave1
ssh pislave2
ssh pislave3
ssh pislave4

Now back on Pi Master we need to create a machinefile with the ip addresses of all the slaves

vi machinefile

And enter the addresses of all the nodes

192.168.1.200
192.168.1.201
192.168.1.202
192.168.1.203
192.168.1.204

Now we run the same C program as before, but this time with the modified machinefile containing the ip addresses of our 4 slaves nodes. The first time we run this we’ll be prompted to enter the password for RSA key, after that it shouldn’t ask for it until after you reboot

mpiexec -f machinefile -n 5 ~/mpi/mpich2_build/examples/cpi

You should then see out put such as

Process 0 of 2 is on pimaster
Process 1 of 5 is on pislave1
Process 2 of 5 is on pislave2
Process 3 of 5 is on pislave3
Process 4 of 5 is on pislave4
pi is approximately 3.1415926544231318, Error is 0.0000000008333387

Well thats it, we have a super computer running 5 nodes in a master slave configuration, … wow !!!!

Duplicating Master into 4 Slaves

All the work so far on creating a development environment, adding languages, apache, mysql, couchdb and nagios has been on a single CD card which was referred to as PiMaster which will form the coordinating node on my super computer.

Rather that duplicate everything 4 more times for each node, we can take a copy of the PiMaster SD card and burn it onto the 4 slaves.

First we insert the PiMaster SD card into my Macbook and use df to find the number of the disk it has been assigned

sudo diskutil unmount /dev/disk2s1
sudo dd bs=1m if=/dev/rdisk2 of=./PiMasterBackup.img
sudo diskutil eject /dev/rdisk2

We now have an image PiMasterBackup.img on our hard disk, which we can copy onto the 4 slave SD cards, so we repeat the following steps for each CD in turn

sudo diskutil unmount /dev/disk2s1
sudo dd bs=1m if=./PiMasterBackup.img of=/dev/rdisk2 
sudo diskutil eject /dev/rdisk2

However this gives us 5 copies of the same SD card which all have the same ip address, server name and configuration. We now need to personalise each SD card in turn.

Set aside PiMaster SD card as that is configured as we need it. At this stage don’t put power up its PI. For this next section we use one Pi on its own, and insert, configure and then eject each slave SD card.

Repeat these steps for each slave card

  1. Power down Pi ( if not already ), making sure no other Pi’s powered up
  2. Insert Slave SD Card
  3. Power up Pi which will appear on the network as PiMaster, ip address 192.168.1.200
  4. ssh pimaster
  5. sudo vi /etc/hostname
  6. Change the name to pislaveN, where N is the number of card, 1, 2, 3 or 4
  7. sudo vi /etc/network/interfaces
  8. Change the IP address to 192.168.0.20N, where N is the number of the card
  9. sudo vi /etc/hosts
  10. Change the the name pimaster to pislaveN
  11. sudo rcconf
  12. Turn off nagios3, move the cursor down and press space, this will remove the ‘*’ from between the square brackets ‘[ ]’
  13. sudo vi /etc/mysql/my.cnf
  14. Change the value for bind-address to the ip of the slave
  15. sudo vi /etc/couchdb/local.ini
  16. Change the ipaddress for bind_address to ip address of the slave
  17. sudo reboot
  18. You should now be able to ssh into pislaveN

Repeat for cards 2, 3 and 4