Simple Client/Server RESTful Services with Python

In this article we look at creating the basics of a RESTful service and associated client, that we’ll build on later to create our home automation server

First we need to make sure we have a couple of libraries installed, specifically httplib2 and Flask. Install them using pip

sudo pip install httplib2
sudo pip install Flask

We can check they are installed with

pip list

which should display a list which includes the following

$ pip list
Flask (0.10.1)
httplib2 (0.8)

RESTFul Server

First the server, which we build using Flask

from flask import Flask, request, json
from pymongo import MongoClient

app = Flask(__name__)
mongo_client = MongoClient()
mongo_client['HomeControl'].temperature.drop ()

@app.route("/homecontrol/api/v1.0/temperature", methods=['POST'])
def api_temperature():
    if request.headers['Content-Type'] == 'text/plain':
        print (request.data)
        return 'OK', 200
    elif request.headers['Content-Type'] == 'application/json':
        print (json.dumps(request.json))
        mongo_client['HomeControl']['temperature'].insert(request.json)
        print (mongo_client['HomeControl']['temperature'].count ())
        return "OK", 200
    else:
        return "Unsupported Media Type", 415

if __name__ == '__main__':
    app.run(debug = False)

For a description of how to create a RESTFul server in Flask, check out the excellent blog by Miguel Grinberg

RESTFul Client

Then a basic client that emulates our temp server

import httplib2
import json
import time
import datetime

if __name__ == '__main__':

    httplib2.debuglevel     = 0
    http                    = httplib2.Http()
    content_type_header     = "application/json"

    url = "http://127.0.0.1:5000/homecontrol/api/v1.0/temperature"

    data = {    'room':         "Living Room",
                'temp':         23.45,
                'humidity':     50.00,
                'timestamp':    str(datetime.datetime.now())
           }

    headers = {'Content-Type': content_type_header}
    print ("Posting %s" % data)

    while True:
        response, content = http.request( url,
                                          'POST',
                                          json.dumps(data),
                                          headers=headers)
        print (response)
        print (content)
        time.sleep(3)

This simple client sits in a loop posting a JSON object to our server every 3 seconds
We’ll fill out this client as time goes by, hooking it up to real temp sensors on the pi

Room Temperature Sensor With Raspberry Pi

This article is taken from the following web pages, with a few modifications by me.

The intention is to create a Raspberry monitoring station for each room in the house, each broadcasting temperature and other information back to a central server in my office.

To start with I am using DS18B20 temperature sensor, which you can get from Amazon and eBay for about 50p-£1 each.

Along with the sensor you are going to need a breadboard, ribbon cable, some wires and 4.7k resistor

breadboard-ic.png

Once you have this wired up, you need to activate the sensor

$ sudo modprobe w1-gpio
$ sudo modprobe w1-therm
$ cd /sys/bus/w1/devices
$ ls

This will give you the id of the device starting 28- and looking something like

28-xxxxxxxxxxxxxxx

Now create a shell script we can query the sensor

nano ~/temp.sh

And add the following lines

#! /bin/bash
sudo modprobe w1-gpio
sudo modprobe w1-therm
roomtemp=$(cat /sys/bus/w1/devices/28-xxxxxxxxxxxx/w1_slave | grep  -E -o ".{0,0}t=.{0,5}" | cut -c 3-)
echo "Temperature: $roomtemp"

To make it executable, give it +x permissions

chmod +x ~/temp.sh

And it will print out the temp, device / 1000 for ‘C

$ ~./temp.sh
186903

Which is 18.69’C

Now lets create a little Python script that parses the data and loops printing out the value every 10 secs

nano ~/temp.py

And add the following lines

import time
while True:
        # Open the file that we viewed earlier so that python 
        # can see what is in it. Replace the serial number as before.
        tfile = open("/sys/bus/w1/devices/28-xxxxxxxxxx/w1_slave")
        # Read all of the text in the file.
        text = tfile.read()
        # Close the file now that the text has been read.
        tfile.close()
        # Split the text with new lines (\n) and select the second line.
        secondline = text.split("\n")[1]
        # Split the line into words, referring to the spaces, 
        # and select the 10th word (counting from 0).
        temperaturedata = secondline.split(" ")[9]
        # The first two characters are "t=", so get rid of those and 
        # convert the temperature from a string to a number.
        temperature = float(temperaturedata[2:])
        # Put the decimal point in the right place and display it.
        temperature = temperature / 1000
        print ("%2.2f'C" % temperature)
        time.sleep(10)

To run the script, use

python temp.py

Mine is currently showing a balmy 16-19’C in my office

19.00'C
18.94'C
19.06'C
18.81'C
18.12'C
18.12'C
18.50'C
18.56'C
18.06'C
18.06'C
18.25'C
18.06'C
17.69'C
17.81'C
17.25'C
17.12'C
16.69'C
16.31'C
16.50'C
16.81'C
17.12'C
17.38'C
17.56'C
17.56'C

Loading Drivers on Startup

The following lines are taken from raspberry.znix.com

sudo nano /etc/modprobe.d/raspi-blacklist.conf and comment out the following 2 lines

blacklist spi-bcm2708
blacklist i2c-bcm2708

sudo nano /etc/modules

w1-gpio
w1-therm

Next steps

Now that we have a temperature sensor schematic and working prototype, need to build 3 or 4 more, and then hook them up to the central server. So once they are built, I need a Restful service that each of the Pi’s can call with their temperature data. The easiest way in Python is using the Flask framework backed with a Mongo DB. Watch this space for….

Building a RESTful web service using Pythong & Flask

 

Changing the hostname on a Raspberry Pi

Changing your hostname on a Raspi is pretty straight forward,

To get a host name use

keith@fuzepi ~ $ hostname
fuzepi

We now need to change the host name for the local address 127.0.0.1

keith@fuzepi ~ $ sudo vi /etc/hosts
[sudo] password for keith:

Change the name associated with 127.0.01, then we need to change the actual name of the host

keith@fuzepi ~ $ sudo nano /etc/hostname

Change the name in this file and save
One more step is to update the system

keith@fuzepi ~ $ sudo /etc/init.d/hostname.sh

Finally, reboot the Pi

keith@fuzepi ~ $ reboot

and the new name should be available with the command ‘hostname’

keith@gertpi ~ $ hostname
gertpi

Running GGTS on OSX under Java 7

I had a look at this and it looks to be a bug somehow related to the way that RJB on OSX works. RJB is the library that Buildr uses to interact with the JVM. Essentially RJB does not seem to allow the configuration of the JVM without setting some environment variables (possibly at build time?). Seehttp://rjb.rubyforge.org/.

There are two main ways to work around this;

  • use the external compiler by adding something like “compile.using(:externaljavac)” into your buildfile.
  • use JRuby.

I will look into what is required to solve this correctly for the next release (1.4.8).

Update

Fixed in 1.4.8 – a work around for 1.4.7 is to set the JVM_LIB environment variable like

export JVM_LIB=$JAVA_HOME/jre/lib/server/libjvm.dylib

Install Python 3 Raspberry Pi

Its pretty simple to get Python 3.2 ( the latest for Pi ) installed, just a couple of apt-get commands and a little shell hacking will do it

First lets get Python 3 install

sudo apt-get install python3

Next get setup tools

sudo apt-get install python3-setuptools

And finally we want Pip installing but ensuring it works for Python 3

sudo apt-get install python3-pip

Then modify you shell script to alias ‘python’ to run ‘python3’

alias vi='vim'
alias python='python3.2'
alias pip='pip-3.2'

The best way to do this is with your favourite editor and then adding the lines you mention, followed by using source to make them available in your current session.

sudo nano ~/.bash_aliases
source ~/.bash_aliases

Jobs a good’un !

Install Python 3 on OSX

Even the the latest version of OSX ships with Python 2.7 which is getting a bit long in the tooth and Python 3.x is where its at these days. Although I wish most Python library developers would update their systems. Its not that hard

Anyway, download the latest version, in this instance Python 3.3

http://www.python.org/getit/

I went with the 64 bit version as default. Run the installer once downloaded and you have 3.3 installed but from the command line, it still shows 2.7, so we need to tell the shell to use Python 3.

Edit ~./bash_rc and add the following 2 lines

alias python=python3
PATH="/Library/Frameworks/Python.framework/Versions/3.3/bin:${PATH}" export PATH

If we want to do anything interesting with Python we will need pip, the pack installer. Get this with the following command

curl https://raw.github.com/pypa/pip/master/contrib/get-pip.py | python3

You can check its installed and working with Python 3 with the following

pip --version

Which should should something line the following

pip 1.5.2 from /Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages (python 3.3)

Which shows you are running pip on the latest version of Python you just installed

Adventures in Natural Language Processing – No 1. Setup

Natural Language Processing

So this will be a series of articles on natural language processing, this is based on Python 3, and the NL Toolkit, NLTK 3.0, which can be found at http://www.nltk.org

 

The current version of NLTK is based on Python 2,8, As we are using Python 3, and there is an alpha version available, but requires a bit of a manual process. First we need to install PyYaml 3.10 as NLTK requires a version greater than 3.9 and 3.10 is the latest version.

This can be installed at the command line using pip

> sudo pip install pyyaml

We can then check its installed with ‘pip list’

> pip list

Next head over to http://nltk.org/nltk3-alpha/, download one of the distributions, then unzip and open command line in root of unzipped files

> sudo python setup.py install

This will install NLTK into your current version of Python. You can test the install by starting the Python command line and importing the nltk module

> python
 Python 2.7.5 (default, Oct 11 2013, 15:52:19)
 [GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.2.75)] on darwin
 Type "help", "copyright", "credits" or "license" for more information.
 >>> import nltk

If this failed or throws an error, you install failed.

If there are no errors, then download the corpas ( this also tests nltk is install ), remaining in the Python command line.

>>> import nltk
>>> ntlk.download ()

A dialog will appear, select all to download all packages and then click Download

Any errors in the above and its likely nltk installation failed

We are ready to begin….. watch out for No 2. NLP Basics

Agile Smagile

I love agile, I really do, for me it provides such a close cooperation between all parties involved in a project. I love agile pragmatists, people who know what to use when, and more importantly what won’t work for their specific environment, customer or company. These are people who see agile as a toolbox that provides a suite of tools that help you to deliver great software more frequently and better aligned to what your customers want.

However I really disagree with the pursists, delivery is not guaranteed based on the number of books you have read ( and can quote ) or the number of certificates you have, its based on how you apply the core tennets of agile at the appropriate time. You don’t need to follow Scrum, or DSDM, or XP or any of the others by the book, instead you need to know that each has its place, but none are the only solution.

Certification is diluted, every man and his dog has some form of certified agile this, and agile that, its time to talk about experience not paper. Certification should be seen as getting your papers to operate within an agile environment, its says you get it, you want to get it more and you want to operate under its guiding principals.

Where I work, we love Scrum, just about every project we deliver uses Scrum, but we don’t follow it religiously, for example, we have 2 week time boxed sprints but we release as soon as a feature is ready and by release I mean straight into production from the first day of the sprint, right the way through to the last day. When its ready it ships, we don’t hang around.

Sometimes ( shock horror ), Scrum doesn’t work for us, we have a large production system and sometimes we need to get a number of small changes out the door as quickly as possible, this is when we swap between Scrum and Kanban, and allow us to push features through the entire SDLC piece by piece.

In terms of development, TDD is great, its a major innovative step in software quality, but sometimes its hard. Its hard to learn, its hard to practice, but once your get it, personal and team productivity does soar. However sometimes people like to try an idea out, explore a concept, see if it works, if it doesn’t abandon it, if it does work then refactor it with tests to make it stronger. I get no greater thrill out of sitting down at a keyboard and trying to get an idea to work in code. I know and support the power of tests and refactoring, but sometimes ( just sometimes ) they get in the way until you really understand what it is you are trying to solve. Then the discipline kicks in and I can refactor then hell out of the code and build in quality with tests.

Agile is now so close to becoming mainstream its scary, but its still fragmented, and still too many high priests ( and their followers ) telling us their way is best, or jumping on the band wagon too late and telling us their re-interpretation of agile is best. Lets get back to the 4 key principles and make them work for you.