Monitoring the super computer with Nagios – Part 1: Installation

Once we have all 5 nodes up and running we’ll want to monitor them, for this I intend to use Nagios, which is a enterprise monitoring tool based on a client/server architecture. We install the server on pimaster and then the client on all nodes. Nagios will then provide a web based interface to view various performance characteristics of our running super computer.

To install use the following command

sudo apt-get install nagios3

During the install a number of config screens appear, the first asks you for the password for ‘nagiosadmin’ user.

Enter a password and the again at the re-confirmation screen which appears next, then wait because the next bit takes a while but it eventually ends. You can check its running with by either running rcconf or

sudo /etc/init.d/nagios3 status
[ ok ] checking /usr/sbin/nagios3...done (running).

To access Nagios, enter the URL http://pimaster/nagios3/ into a browser and when prompted for userid and password enter, nagiosadmin for the userid and the password we entered above. You will then be presented with a screen similar to this

Congratulations you have installed Nagios and confirmed its running, its time to start configuring it to monitor our nodes

To configure each node as a client we need add the Nagios plugs. Most of this next bill is pulled from Platonic

sudo apt-get install nagios-plugins nagios-nrpe-plugin nagios-nrpe-server

You can then view the nagios server via a web browser at http://

Configuring services

We now have a number of applications installed on our Pi, including Apache, Tomcat, MySQL and CouchDB and their default install has them configured to start when the system boots, but we don’t need them all at startup, so we need a way to control their startup options.

For this we use a tool called rcconf, and a good description of its use, along with basic Debian service management can be found at Debian Admin

First we need to install rcconf

sudo apt-get install rcconf

Once installed you can start the application by typing

sudo rcconf

And the following screen will be displayed

You can then select which services to not start at start up, including mysql, tomcat6 and couch. However you can start them at any time with by using /etc/init.d

sudo /etc/init.d mysql start
sudo /etc/init.d tomcat6 start
sudo /etc/init.d couchdb start

To stop them at any time there is a similar command

sudo /etc/init.d mysql stop
sudo /etc/init.d tomcat6 stop
sudo /etc/init.d couchdb stop

To stop and the immediately restart, use the restart option

sudo /etc/init.d mysql restart
sudo /etc/init.d tomcat6 restart
sudo /etc/init.d couchdb restart

And to just get an update on the status, use the status option

sudo /etc/init.d mysql status
sudo /etc/init.d tomcat6 status
sudo /etc/init.d couchdb status

Installing and configuring CouchDB

CouchDB is a database that uses JSON for documents, JavaScript for MapReduce queries, and regular HTTP for an API

UPDATE: See the addendum at the bottom of the page for a full update of how to get couchdb running after reboot. You still need to follow these instructions, then pop down the bottom for final piece of the puzzle

I struggled with this several times, until a colleague of mine called Stephen Lock showed me the light, so shout out to him for showing me how to make it work post install.

sudo apt-get install couchdb

UPDATE: The latest version no longer fails and you end up with the last view lines of you console looking like

[ ok ] Starting database server: couchdb.
Setting up wwwconfig-common (0.2.2) ...
Setting up javascript-common (7) ...
Setting up libsctp1 (1.0.11+dfsg-2) ...
Setting up lksctp-tools (1.0.11+dfsg-2) ...
Processing triggers for menu ...

If you do get an error, then its probably the problem reported in at the bottom of this blog, but now we can just continue by making sure couch is binding to the correct IP address so we can connect to it from another machine

cd /etc/couchdb
sudo vi local.ini

To change the bind_address to the address of the machine you are installing on, look for the following line

;bind_address = 127.0.0.1

Remove the ; from the beginning and then change the ip address to the address of the server on which couch is installed. Finall restart the couchdb service

/etc/init.d/couchdb restart

This time you’ll see the following line which signifies Couch is up and running

[ ok ] Starting database server: couchdb.

To check its all working, open a browser and enter the url http://hostname:5984 and you’ll see a single line of text along the lines of

{"couchdb":"Welcome","version":"1.2.0"}

We can then bring up the admin console by entering in a browser, the url http://hostname:5984/_utils which will show something like this

If install fails

When this is finished it will probably report an error that it couldn’t start couch. Typically the install will finish with the following lines

[....] Starting database server: couchdbApache CouchDB needs write permission on the PID file: /var/run/couchdb/couchdb.pid
 failed!
invoke-rc.d: initscript couchdb, action "start" failed.
dpkg: error processing couchdb (--configure):
 subprocess installed post-installation script returned error exit status 1
Setting up libsctp1 (1.0.11+dfsg-2) ...
Setting up lksctp-tools (1.0.11+dfsg-2) ...
Processing triggers for menu ...
Errors were encountered while processing:
 couchdb
E: Sub-process /usr/bin/dpkg returned an error code (1)

Don’t panic !, we just need to do some jiggery-pokery to set the permissions and ip addresses that couch uses. First we need to make sure that couch owns the right folders which are not set correctly during install

sudo chown couchdb:couchdb -R /var/lib/couchdb /var/log/couchdb /var/run/couchdb
sudo update-rc.d couchdb defaults

Then we need to make sure couch is binding to the correct IP address so we can connect to it from another machine

cd /etc/couchdb
sudo vi local.ini

To change the bind_address to the address of the machine you are installing on, look for the following line

;bind_address = 127.0.0.1

Remove the ; from the beginning and then change the ip address to the address of the server on which couch is installed. Finall restart the couchdb service

/etc/init.d/couchdb start

This time you’ll see the following line which signifies Couch is up and running

[ ok ] Starting database server: couchdb.

To check its all working, open a browser and enter the url http://hostname:5984 and you’ll see a single line of text along the lines of

{"couchdb":"Welcome","version":"1.2.0"}

We can then bring up the admin console by entering in a browser, the url http://hostname:5984/_utils which will show something like this

 

 

Addendum

I’ve had problems with couchdb starting because of permissions and searching the internet shows this to be a common bug with the current version and is because /var/run/couchdb has its permissions set back to root after reboot. The easiet way to fix this for now is to modiy rc.local and make it reset the permissions on startup

sudo vi /etc/init.d/couchdb

And look for the method that starts with the following at about line 84

start_couchdb () {

Add a new line after mkdir -P “$RUN_DIR”
and add the following

start_couchdb () {
    # Start Apache CouchDB as a background process.

    mkdir -P "$RUN_DIR"

    #Add this line
    chown couchdb:couchdb -R "$RUN_DIR"

    command="$COUCHDB -b"

Adding a few more languages

Our core platform is now configured with Python ( by default ), PHP ( installed ) and Java ( installed ), we also get Perl ( by default ), but I want more languages. I spend a lot of more time these days either using Ruby or Groovy and have a passing interest in Scala for its functional programming capabilities.

So lets add each of these in turn, first Ruby, the best source of information is found at RPi Ruby on Rails

sudo apt-get install ruby
sudo apt-get install rails

We can test that all is installed with 3 commands,

ruby -v
ruby 1.9.3p194 (2012-04-20 revision 35410) [arm-linux-eabihf]
gem -v
1.8.23
rails -v
Rails 2.3.14

Once Ruby is installed lets install a couple of key JVM languages, specifically Groovy and Scala, first Groovy

 
sudo apt-get install groovy
sudo apt-get install groovy-doc

We can check Groovy is installed by checking its version

groovy -version
Groovy Version: 1.8.6 JVM: 1.7.0_07 Vendor: Oracle Corporation OS: Linux

And now we install Scala

sudo apt-get install scala
sudo apt-get install scala-doc

Again we can check it is installed by checking its version

scala -version
Scala code runner version 2.9.2 -- Copyright 2002-2011, LAMP/EPFL

Adding Java to the Development Environment

Now we have LAMP and Python installed its time to start adding some more of the programming languages we are going to use, first step is Java because it opens up a range of additional languages such as Groovy, Scala, Closure along with Java versions of Python ( JPython ) and Ruby ( JRuby )

The version of Java we will use is Open JDK. There are 2 components to install Java, the Java Runtime Environment ( JRE ), which is used by Java based applications that we will install later such as Tomcat; and the Java Development Kit ( JDK ) which contains the programming environment, libraries and compilers necessary to write, compile, debug and execute Java code.

sudo apt-get install openjdk-7-jre
sudo apt-get install openjdk-7-jdk

You can check its all installed by checking the version

java -version

should return the following

java version "1.7.0_07"
OpenJDK Runtime Environment (IcedTea7 2.3.2) (7u7-2.3.2a-1+rpi1)
OpenJDK Zero VM (build 22.0-b10, mixed mode)

Once we have Java installed we will then add Tomcat, an open source software implementation of the Java Servlet and JavaServer Pages technologies

sudo apt-get install tomcat6
sudo apt-get install tomcat6-docs
sudo apt-get install tomcat6-examples
sudo apt-get install tomcat6-admin

Tomcat starts on port 8080 so to make sure its working point a browser to http://:8080 and you should see something similar to the following

Creating a Basic Development Environment

To allow us to use our super computer as a via development environment we need to install some essential software and then we can begin installing the languages and supporting tools.

Underpinning all the installs we are about to do is ‘build-essential’. The build-essential is a reference for all the packages needed to compile a debian package. It generally includes the gcc/g++ compilers an libraries and some other utils

sudo apt-get install build-essential

Now we have the core tools lets install the 2 most common version control systems, subversion and git

sudo apt-get install subversion
sudo apt-get install git-core

As Python is the defacto language on the Pi lets install some python package management tools. This section is scraped from SaltyCrane Blog

Pip is a tool for installing and managing python packages

sudo apt-get install python-pip python-dev 
sudo pip install --upgrade pip

Next we are going to turn out Pi into a LAMP Server. LAMP stands for Linux, Apache, MySQL and PHP, a common framework for web development

This section was helped from Chris Potters Blog

Note, that during the below Lamp install you will be asked for certain information, specifically userid/password for MySQL and some supporting tools. The installers typically run their own application to configure.

First we install Apache, our web server

sudo apt-get install apache2

You can now test that Apache is installed correctly by opening your browser and entering the url http:// where is the name of the server you are installing into, this should show the following screen

Next we install PHP which is one of the easiest languages to write dynamic web pages

sudo apt-get install php5 libapache2-mod-php5

We can now check that PHP is installed correctly. First create a file in /var/www called info.php and include the following information

Open a browser and then open the url http:///info.php, again where server name is the name of the server you are installing on. You should get the following screen

Next we add MySQL a relational database that integrates well with Apache and PHP

sudo apt-get install mysql-server

Now we hook up Apache, PHP and MySQL by installing the PHP libraries for Apache and the libraries that allow you to make SQL calls to MySQL from PHP code

sudo apt-get install libapache2-mod-auth-mysql php5-mysql

And then we add in PHPMyAdmin a great web based tool for managing your MySQL databases which integrates with Apache

sudo apt-get install phpmyadmin

We can check that PHPMyAdmin is installed by loading it up into our browser. Enter the url http:///phpmyadmin and the following screen should be displayed

Login and you should see

Finally we can add some additional PHP libraries which are useful including PEAR which is a PHP package manager

sudo apt-get install php5-cli php5-common php5-curl php5-dev php-pear

Thats it, you have a decent LAMP server that you can write applications in Python and PHP, create websites and hook them up to a back end database, cool or what ?

Connecting via hostnames

One of the problems with IP addresses is they are hard to reconcile to the machine they relate to. Its always much easier to talk about pimaster or pislave1 than 192.168.1.200 or 192.168.1.201. If you have a crappy DHCP server like I have with the BT Home Hub, then the easier way around this is to modify the hosts file on each of the machines, which is used by the OS to lookup the mapping between hostname and ip before it goes off to name servers.

sudo vi /etc/hosts

and you’ll see something like this

Add lines ( highlighted in blue ) for each device in your network

I’ve replaced the name for 127.0.0.1 with pislave1, the name of the device I am logged into, and then added the ip addresses for pimaster, pislave1-4 and readynas

Repeat this for each machine, machine sure you change the name for the device 127.0.0.1 to the actual name of the device you are logged into

It lives !!!!!!!!!!!!!!!!!!!

It finally works, all the cards are formatted and the devices networked, I now have a 5 node Super Computer just itching for world domination.

Below you can see PiSlaves1-4 and PiMaster sat on top, connected through their own 100Mbs hub to my main 1GB hub

And there are all 5 machines opened at the terminal

Its time to create dastardly deeds with all this computing power now

 

Mounting an NFS Folder

Most of this info was pulled from the website Peter Mount’s Blog

First we need to tell NFS to use statd, to do this

sudo /etc/init.d/rpcbind start 
sudo vi /etc/default/nfs-common
add the line "NEED_STATD=yes"
reboot PI or /etc/init.d/nfs-common restart

Next we need to create a local folder that will be the mount point of the remount NFS folder. In this instance I have a folder called ‘raspberry’ on my Netgear RasdyNAS NV+ device

sudo mkdir -p /nas/raspberry
sudo chmod 777 /nas/raspberry

Now we can mount the remote folder to local folder

sudo mount 192.168.1.199:/raspberry /nas/raspberry

Change the IP address 192.168.1.199 to whatever you NFS server has.

If you subsequently want to remove the remote folder, or unmount it, then using the following command

sudo umount /nas/raspberry

If you want the mount to be available everytime you boot the pi, then you need an entry in /etc/fstab

sudo vi /etc/fstab

And add the following entry at the end of the file:

192.168.1.199:/raspberry /nas/raspberry nfs defaults 0 0

Thats it. The next time you reboot the pi it will automatically mount that drive.

Duplicating an SD Card Image

Once we have created an SD Card image of our base system, updated the OS and configured for SSH access we don’t want to have to do this over and over again, so this section will show you how to make a copy of your original SD Card and then burn it onto all the others

Without plugging in your SD card, from the command line run the command df

df

Filesystem                        512-blocks       Used Available Capacity  iused     ifree %iused  Mounted on
/dev/disk1                        1247961912  377253544 870196368    31% 47220691 108774546   30%   /
devfs                                    378        378         0   100%      656         0  100%   /dev
map -hosts                                 0          0         0   100%        0         0  100%   /net
map auto_home                              0          0         0   100%        0         0  100%   /home
localhost:/FubuLodeAG-ddMZnEma0co 1247961912 1247961912         0   100%        0         0  100%   /Volumes/MobileBackups

Then insert the card, wait for it to appear on your desktop and then df it again. You will notice the only difference is the new line with your card name on it

df

Filesystem                        512-blocks       Used Available Capacity  iused     ifree %iused  Mounted on
/dev/disk1                        1247961912  377253544 870196368    31% 47220691 108774546   30%   /
devfs                                    378        378         0   100%      656         0  100%   /dev
map -hosts                                 0          0         0   100%        0         0  100%   /net
map auto_home                              0          0         0   100%        0         0  100%   /home
localhost:/FubuLodeAG-ddMZnEma0co 1247961912 1247961912         0   100%        0         0  100%   /Volumes/MobileBackups
/dev/disk2s1                          114576      68000     46576    60%      512         0  100%   /Volumes/Untitled

In this instance my card has been mounted as /dev/disk2s1 and its the number 2 that we are interested in so remember it.

To be able to copy the image we first need to unmount it with the unmount command

sudo diskutil unmount /dev/disk2s1

You will notice the card disappears from your desktop

Next we want to make a copy of the image using the dd command

sudo dd bs=1m if=/dev/rdisk2 of=RPi.img

And finally eject the SD card so we can insert another one and copy the new image to that

sudo diskutil eject /dev/rdisk2

Once one card is done you can repeat this for every card for the super computer. As each card is done, you can boot them individually and change the static ip and hostname using the information documented in Setup Stage 2 – Network Configuration