My first cluster…

hossg   February 28, 2015   1 Comment on My first cluster…

I’ve been thinking about trying out a cluster compute home-build for a while now, but have never quite got around to it.  The recent launch of the new Raspberry Pi 2 has given me the impetus, and so this weekend I got building.

picluster

The Build

The physical assembly of the cluster is pretty straightforward:  I ordered four Pi’s, and laid them out on an A4 sheet of perspex, drilled the requisite holes for standoffs for mounting the boards, and then laid a second sheet of perspex over the top, mounted with slightly longer stand-offs.

Although I could aim for a higher density (perhaps as high as 8 Pi’s per A4-sheet – the limiting factor is the cable access), this cluster is really just a proof-of-concept for me, and a learning experience with some of the technologies.  The cluster is also stackable in as much as I can repeat the same layout on further perspex sheets if I want to extend the cluster further.

The only element of careful consideration was in providing the power to the Pi’s.  I did look carefully at powering the 5V rail via the GPIO port and some custom electrics, however doing so would circumvent the protection provided by the standard micro-USB power socket.  Instead I found a decent 6-port, 10-Amp USB power supply that can send close to 2A to each Pi, which is enough to cater even for scenarios where I have power-hungry peripherals hanging out of the cluster (not my plan, but who knows what the future holds).

I also made use of an unmanaged gigabit switch I had lying around to hook up all of the Pi’s.  In future I’ll probably replace this with a managed switch or router of some description so that the cluster is a little more isolated from my core network, and also to give me a playground to learn a little more about network configuration, etc.

Storage for the Pi’s was provided courtesy of some SANDisk 8GB Micro-SD cards, which conveniently came with a regular-SD adaptor for easy flashing in my Mac.  My only gripe is that the SD cards were strictly 8GB in size (SI Units), rather than 8GiBi in size.  Given that the common vernacular is that 1GB=1GiBi I do feel somewhat shortchanged.

Shopping List:

Item Price Price per Item Items needed per four Pi’s Cost per Pi
SanDisk Ultra 8 GB microSDHC UHS-I Class 10 Memory Card with Adapter up to 48MB/s Read  £19.60  £4.90 4  £4.90
18mm Body 50 Pcs Brass Threaded PCB Stand-off Spacer M3 Male x M3 Female  £8.47  £0.17 8  £0.34
Belkin Cat5e Snagless UTP Patch Cable (Yellow) 0.5m  £4.00  £1.00 4  £1.00
TeckNet® 50W 6-Port Family-Sized Desktop USB Wall Charger (5V 10A)  £15.97  £2.66 4  £2.66
StarTech.com PC Mounting Computer Screws M3 x 1/4in Long Standoff – 50 Pack  £1.70  £0.03 20  £0.17
2 of Clear Acrylic Sheet, A4 Size, 4mm thick   £11.47  £5.74 2  £2.87
M3 Full Nut (50 Pack) 3mm A2 Stainless Steel Hex Hexagon Nuts  £1.45  £0.03 4  £0.03
50 Pcs Brass Screw Thread PCB Stand-off Spacer M3 Male x M3 Female 6mm  £5.53  £0.11 16  £0.44
Anker® [6-Pack] Premium 1ft Micro USB Cables High Speed USB 2.0 A Male to Micro B Sync and Charge Cables  £8.99  £1.50 4  £1.50
Raspberry Pi  2 Model B  £128.00  £32.00 4  £32.00
Velcro sticky pads  £3.43  £0.14 8  £0.29
Total  £208.61  £46.19

 

The Setup

I setup each SD card on my Mac with Raspian as an OS, available on the Raspberry Pi website here: http://www.raspberrypi.org/downloads/

I may switch to something lighter-weight in future, like a custom build of ArchLinux, but that’s for another day.

To flash the SD card, do the following (assuming you are on a Mac):

diskutil list

Identify the disk (not partition) of your SD card e.g. disk4 (not disk4s1):

diskutil unmountDisk /dev/<disk# from diskutil>

e.g.

 diskutil unmountDisk /dev/disk4

then:

sudo dd bs=1m if=image.img of=/dev/DISK

e.g.

sudo dd bs=1m if=2015-02-16-wheezy-raspbian.img of=/dev/disk4

This last command can take a while – ~35mins for me – and there’s no visible output until it completes and tells you how many bytes were written, so just be patient!

Although I could have configured one Pi node, and then cloned it repeatedly, and although that is what I will probably do if I add further nodes to the cluster, to get up and going I simply repeated the above steps for each SD card.

The final steps are straightforward: whack the SD cards into the Pi’s and power up.  A quick glance at my router to see the IP’s used, and I SSH’ed to each node in turn to configure it.

There are only a handful of simple configuration steps required, but these need to be done on each node:

1. Run raspi-config to do base-level configuration.

sudo raspi-config

This offers a menu-driven approach to changing the hostname and enabling the SSH server on a permanent basis, which are the only two changes I made to start with. Also within raspi-config you can configure overclocking (something I will come back to when I can benchmark my cluster performance), as well as memory allocation for graphics, and various other miscellany.

2. Setup SSH so you can conveniently jump around and copy files between nodes as needed:

ssh-keygen

Accept the defaults, and don’t bother with a passphrase.

I named my 4 nodes pi1, pi2, pi3 and pi4 and assigned them each an IP address from 192.168.1.201-204.  You will have to update the instructions below to match your own configuration.

From each node, you will need to copy the public key you just generated to the “authorized keys” file on the other nodes.  If you first create the keys on each node it will create the folder ~/.ssh for you, otherwise you will need to do this on each node before you try and copy the public key across. For example on pi1 (192.168.1.201):

cat ~/.ssh/id_rsa.pub | ssh pi@192.168.1.202 'cat >> .ssh/authorized_keys'
cat ~/.ssh/id_rsa.pub | ssh pi@192.168.1.203 'cat >> .ssh/authorized_keys'
cat ~/.ssh/id_rsa.pub | ssh pi@192.168.1.204 'cat >> .ssh/authorized_keys'

When you do this on each node, the first time you connect to another node SSH will ask you to accept the destination-node’s fingerprint and will add it to the known-hosts file.

3. Download various other packages that will be needed later:

sudo apt-get update
sudo apt-get install python-dev
sudo apt-get install python-pip
sudo apt-get install mpich2

This last package is an implementation of MPI – Message Passing Interface – which is the first approach I will be taking to make use of the cluster’s overall compute power.

3. Next download the mpi4py package which will enable the Python usage of MPI:

wget https://bitbucket.org/mpi4py/mpi4py/downloads/mpi4py-1.3.1.tar.gz

Note: although you can install mpi4py directly using pip, this in turn installs a different implementation of MPI, called OpenMPI, which I could not get to run successfully on the Pi’s.

Unpack mpi4py, and build and install it:

gunzip mpi4py-1.3.1.tar.gz
tar -xf mpi4py-1.3.1.tar
cd mpi4py-1.3.1
sudo python setup.py build
sudo python setup.py install

4. Testing MPI and mpi4py:

All should now be setup and ready to test.  mpi4py comes with some useful tools, but for now we’ll stick to the most basic “hello world” example to prove that the cluster is up and running and the nodes can communicate with each other.

First change to the demo folder:

cd ~/mpi4py-1.3.1/demo

Next create a file that contains the IP’s of the various nodes:

nano machinefile

E.g. my machine file:

pi@pi1 ~/mpi4py-1.3.1/demo $ cat machinefile 
192.168.1.201
192.168.1.202
192.168.1.203
192.168.1.204

Note: you only need to do this on one node, the ‘master’ node.

You can then run the helloworld.py program:

pi@pi1 ~/mpi4py-1.3.1/demo $ mpiexec -f machinefile -n 8 python helloworld.py 
Hello, World! I am process 0 of 8 on pi1.
Hello, World! I am process 4 of 8 on pi1.
Hello, World! I am process 1 of 8 on pi2.
Hello, World! I am process 6 of 8 on pi3.
Hello, World! I am process 5 of 8 on pi2.
Hello, World! I am process 3 of 8 on pi4.
Hello, World! I am process 7 of 8 on pi4.
Hello, World! I am process 2 of 8 on pi3.

the -f parameter tells mpiexec where to find the nodelist, and the -n tells it how many nodes to run.  In this example the processes get split up 2 to each node.

Conclusion:

So – that’s as far as I’ve got so far – a working MPI cluster.  Pretty sweet! Next I plan to conduct some benchmarking stats, timing some sample operations, etc, and seeing how they scale across nodes, and from there learn a little of how MPI works and how to make use of it for some custom applications.

 

 

 

One thought on “My first cluster…

  1. googlefox

    Hi,
    I have some points that may enhance the reading and writing of the SD card image file and provide you the capability to monitor the writing process. I summarized them in the following:

    1. To save the SD card as uncompressed image file on your hard drive (Desktop) enter:

    sudo dd if=/dev/rdisk1 of=~/Desktop/pi.img bs=1m.

    a. The if stands for ‘input file’, in this case the SD card. diskutil will list it as /dev/disk1 
    b. Prefix ‘disk’ with ‘r’ to point to the card’s raw storage space to speed up the process: /dev/rdisk1.

    2. To store the SD card as a compressed image file, you can use the Unix ‘pipe’ symbol “|” to route the output of the dd command to the gzip utility not to a file:

    sudo dd if=/dev/rdisk1 bs=1m | gzip > ~/Desktop/pi.gz

    3. To write the compressed image file to an SD card, you need to unmount the SD card before you can write to it, so enter

    diskutil unmountDisk /dev/disk1

    then enter :

    gzip -dc ~/Desktop/pi.gz | sudo dd of=/dev/rdisk1 bs=1m

    4. Usually it takes about 3 minutes to write a 32GB image file.

    5. To know the status of the copy process while the dd command is running, you can press Control-T.

Leave a Reply