This means that installing everything that you need under Jaunty takes simply the following:

$sudo apt-get install ipython

Viola!  Now for some examples!  (Next post to come!)

First up. The sound didn’t work with the default install.
The fix for this is really easy I found it here

Basically open up ‘/etc/modprobe.d/alsa-base’ and add the line:

options snd-hda-intel model=dell

To the end of the file.  Sound worked wonderfully upon reboot.  (fixed in Jaunty)

Next up PEAP with GTC is not supported in the build of network-manager.  To further complicate things the Broadcom driver freezes up if you try to use this mode at all!  For those of us on school/corporate network’s this simply is unacceptable.  There is a fix using the ‘ndiswrapper’ style driver.  I don’t like this fix, but it works perfectly if you follow the instructions found here:

http://www.ubuntumini.com/

(Fixed in Jaunty!

$ sudo apt-get install build-essential libssl-dev python-setuptools python-dev

From there we will use “easy_install” to get the relevent python modules we need to work with.

$  sudo easy_install foolscap nose pexpect pyopenssl sphinx

Then, becuase I am a bit paranoid, I download the ipython source and run it’s confgure option to make sure that it can find all the dependant libraries.

$ wget http://ipython.scipy.org/dist/ipython-0.9.1.tar.gz
$ tar xfz ipython-0.9.1.tar.gz
$ cd ipython-0.9.1/
$ ./setup.py configure

As mentioned in a previous article, you ought to see something along the lines of:

Twisted: 8.1.0
Foolscap: 0.3.2
OpenSSL: 0.8
sphinx: 0.5.1
pygments: 1.0
nose: 0.10.4
pexpect: 2.1

Next we know the installer can find everything in the proper fashion, I install via easy_install (this makes upgrading easy later as new versions come out)

$ cd ..

$ rm ipython-0.9.1/
$ sudo easy_install ipython

From there you should be up and running!

The following is a good test as to whether all capabilities are functioning properly: everything is functioning properly:

$ ipcluster -n 4

Also notably, it seems that future versions of Ubuntu will ship with such capabilities!

This post will hopefully tell you everything you need to install similar capabilities and build a cloud of your own using ipython!

Installation on Ubuntu Gusty/Hardy/Intrepid:

First you need to get the various code bits that python modules will depend on:

$sudo apt-get install libssl-dev python-setuptools

This will install both the openssl crpytography libraries and the “easy_install” tool, that makes getting and updating python modules really effortless. There is an ongoing battle between this utility and “pip“, but that discussion is well beyond the scope of our post today. I perfer easy_install becuase it is exactly what is advertises.

Next up, we “easy_install” everything we need that ipython’s parallel capabilities will depend on:

$sudo easy_install foolscap nose pexpect pyopenssl sphinx

Depending on what you already have installed, this may take some time.

There are a lot of dependencies to ipython, so to make sure we got them all right, we grab the current ipython source:

$mkdir ipython_build
$cd ipython_build
$wget http://ipython.scipy.org/dist/ipython-0.9.1.tar.gz
$tar xfz ipython-0.9.1.tar.gz
$cd ipython-0.9.1
$./setup.py config

At this point ought to see something like:

Zope.Interface: yes
Twisted: 8.1.0
Foolscap: 0.3.2
OpenSSL: 0.8
sphinx: 0.5.1
pygments: 1.0
nose: 0.10.4
pexpect: 2.1

If you are missing something essential to ipython’s build, it will let you know :p Now that we know we have everything necessary we can go ahead with either the typical build/install whatnot from this source directory, or simply:

$cd ../..
$rm -rf ipython_build
$sudo easy_install -U ipython

And you should be all set to go!

Under *nix distros the following is a good test as to whether everything is functioning properly:

$ipcluster -n 4

Based on what I have seen the OSX / Windows install ought to be fairly similar. You might have to grab the openssl source and compile pyopenssl on you own, pointing the linker to the right directories. If anyone has any luck doing this in OSX / Windows, let me know and I’ll post an update. Some links that might be helpful ( foolscap nose pexpect pyopenssl sphinx openssl-source)

Though the particular package (called pycuda), is exceptionally well developed, it however does not allow you to escape understanding the GPU architecture and computing strategy underlying everything. Correspondingly, I will talk a little about everything. So without further ado, here is the first program I wrote with Python and CUDA working together:

from scipy import *
from scipy import misc
from pylab import *
import pycuda.autoinit
import pycuda.driver as drv
import time

mod = drv.SourceModule("""
__global__ void julia_iterate(float *a, float *b, float *max_iter)
{
  const int i = blockIdx.x * blockDim.x + threadIdx.x;
  float bounds = 16.0;

  for(int iter = 0; iter < max_iter[i]; iter++)
  {
    float new_a = (a[i] * a[i]) - (b[i] * b[i]) +.6;
    float new_b = (2 * a[i] * b[i])  - .2;
    a[i] = new_a;
    b[i] = new_b;
    if(abs((a[i]*a[i])+(b[i]*b[i])) > bounds)
    {
      max_iter[i] = iter;
      break;
    }
  }
}

__global__ void mandelbrot_iterate(float *a, float *b, float *max_iter)
{
  const int i = blockIdx.x * blockDim.x + threadIdx.x;
  float bounds = 16.0;
  float ai = 0;
  float bi = 0;
  for(int iter = 0; iter < max_iter[i]; iter++)
  {
    float new_a = (ai * ai) - (bi * bi) + a[i];
    float new_b = (2 * ai * bi)  + b[i];
    ai = new_a;
    bi = new_b;
    if(abs((ai*ai)+(bi*bi)) > bounds)
    {
      max_iter[i] = iter;
      break;
    }
  }
}
""")

mb = mod.get_function("mandelbrot_iterate")

shape = (5000,5000)pxls = shape[0] * shape[1]

x, y = mgrid[-2:1:shape[0]*1j,-1:1:shape[1]*1j]
x = x.astype(float32)
y = y.astype(float32)

x = x.flatten()
y = y.flatten()

itr  = ones_like(x) * 3000

start_time = time.time()
mb( drv.In(x), drv.In(y), drv.InOut(itr), \
    block=(500,1,1), grid = (50000,1))

print "CUDA iteration time:", time.time() - start_time

itr = itr.reshape(shape)itr = log(itr)

misc.imsave("test.png", itr.T)

The names of the CUDA functions are a bit of a giveaway to the final punch-image, but this is code for generating Mandelbrot fractals. (A Julia example is in there too, but is unused for now) What is wonderful about CUDA that thanks to its incredible paralellism, it takes more time to write the image to disk than it does for the actual computation!
On my NVIDIA 260 it took 1.8 seconds to do 3000 iterations on 25,000,000 points! The super computer is personal again.

[Click for larger image.]

It is a shame that the highest resolution blogger supports appears to be 1600×1600. The code above actually outputs a 5000×5000 “.png” (More than 9 times the area featured here!)

So lets go through this all step by step to give some idea of how this all works:

First we are going to import all the various important libraries.

from scipy import *
from scipy import misc
from pylab import *
import pycuda.autoinit
import pycuda.driver as drvimport time

The following import is worth particular note in all this:

It automatically initializes (and checks for the proper setup) of your CUDA device. There is typically code that goes into accomplishing this, so just having things setup on import is very handy. While there are provisions in pycuda to do the initialization with more control over the process, they will not be covered here. If you have more than one CUDA device, or certain SLI configurations using the autoinit import feature may not be the best choice. If you don’t know what this means, it likely will not affect you.

Next up we come to the actual CUDA/C code that we are going to run on the GPU. I am assuming a fair bit of knowledge here about the C programming language, particularly pointers, indexing, etc, but if you don’t already know about these things, they are worth learning! You can find some catchup links here.

mod = drv.SourceModule("""
__global__ void julia_iterate(float *a, float *b, float *max_iter)
{
 const int i = blockIdx.x * blockDim.x + threadIdx.x;
 float bounds = 16.0;

  for(int iter = 0; iter < max_iter[i]; iter++)
  {
    float new_a = (a[i] * a[i]) - (b[i] * b[i]) +.6;
    float new_b = (2 * a[i] * b[i])  - .2;
    a[i] = new_a;
    b[i] = new_b;
    if(abs((a[i]*a[i])+(b[i]*b[i])) > bounds)
    {
      max_iter[i] = iter;
      break;
    }
  }
}

__global__ void mandelbrot_iterate(float *a, float *b,
                                   float *max_iter)
{
  const int i = blockIdx.x * blockDim.x + threadIdx.x;
  float bounds = 16.0;
  float ai = 0;
  float bi = 0;
  for(int iter = 0; iter < max_iter[i]; iter++)
  {
    float new_a = (ai * ai) - (bi * bi) + a[i];
    float new_b = (2 * ai * bi)  + b[i];
    ai = new_a;   bi = new_b;
    if(abs((ai*ai)+(bi*bi)) > bounds)
    {
      max_iter[i] = iter;
      break;
    }
  }
}""")

The above code will compile the entire code string into a module accessible by pycuda. Pycuda ties the lifetime of objects on the GPU the the life of objects in the Python environment.

At the end of execution, all the GPU resident objects are cleaned up and deallocated appropriately. Correspondingly, if you want to remove something from the GPU you can simply delete the associated objects and cleanup will happen automagically. We will see this again in the “drv.In”/”drv.Out” objects.

Now its time to talk about some CUDA features that are worth understanding. There are two in particular that are worth note in this example:
1.) The “__global__” keyword. This is a directive that tells the CUDA compiler that this function is accessible from both the standard CPU and GPU realm of things. We use it for a function that we want to have a handle on from outside the GPU context (like through
pycuda).

2.) This line:

contains several variables that appear to be undefined. They are not defined in the normal C sense, but are special CUDA objects defined in context which they are run.
At this point we need to talk a little bit about the CUDA threading structure. Coming from a largely interpreted language background, this was a bit difficult for me. CUDA is ideally suited to the single operation, multiple data paradigm. Thusly it is _very_ good at things like fractals where you need to perform the same operation over and over on a single data set.

So, you need to conceptually break down your operation into single action that is performed on any number of data pieces.

There is two levels of threaded-ness in CUDA:
1.) Operations that can occur in any order/concurrently within a single ALU
2.) The chunks which will be slit apart to and sent to different ALU’s which also occur in any order/concurrently.

These are referred to in the NVIDIA CUDA documentation (with excellent diagrams) as threads at the lowest level. These threads are grouped into “blocks” of threads. These “blocks” are organized in a Cartesian grouping referred to as a “grid”. (Illustrated on pdf page 16 of here)

Being a modeling guy, I tend to think of this as an actual Cartesian organization of shapes. This particular example I imagine as as a vector of vectors (some number of vectors glued end to end).

So within your function, there is a way to identify your thread number, the size of your block, and what block number you are in. This is accomplished by the following three internal variables respectively:

Each of these is a structure containing variables pertaining to its dimensionality. In our example, we have both mono-dimensional blocks, and and a mono-dimensional grid.

Correspondingly, based off of simple geometric calculations:

Will store in the variable “i”, the correct index that needs to be processed by a particular thread in a particular block.
For example, the zeroth thread in the zeroth block will operate on the zeroth array index. Or the zeroth thread in the first block will be offset by the one times the block dimension.

This gets a bit more complicated for multidimensional arrays, so we will stick to 1-d for this example. More complex examples are seen in the CUDA documentation mentioned earlier. But really it boils down to a function of:

Derived from geometry of the arrays. The remainder of the code within that Python text block is industry standard C to generate either a Mandelbrot or Julia fractal. Alright, back to Python!

Next we get a handle to the actual function on the GPU:

“mb” is now an object which is callable within our Python code to execute the function we have described! Were getting there!!

Next we are going to generate a pair of vectors which are the real and imaginary component corresponding to all the points withing the interesting area of the Mandelbrot set.

We want a 5000×5000 grid so:

Next, we use the scipy function “mgrid” to generate the indices, convert them to float32 (to match the precision of NVIDIA GPU’s), then flatten them into linear vectors. We also define a linear vector of how many iterations we will maximally perform on each point. NB: All of these vectors are the same length!

Now we call the actual function that performs the calculations. While doing this, several objects are instantiated by “drv.In” and “drv.InOut”. These objects correspond to arrays of data that are resident on the GPU. Notably also, their lifetime on the GPU is tied to their lifetime in the Python interpretor environment. Therefore, instantiating them allocates their memory space on the GPU and copies the ndarray over. Also, when the object is destroyed at the end of the statement, the memory is deallocated, and cleaned up appropriately on the GPU itself.

The difference between “drv.In”, “drv.Out”, and “drv.InOut” is fairly obvious, but using them appropriately insures that time is not wasted copying back and forth unnecessarily.

In here we specify the geometry of our blocks and the grid. (Note that 500*50,000 is our pixel count) These are hard coded in this example, but it is easy to imagine juggling these around to fit the size of the problem. Notably, the dimensions of both the grid size and block size are bounded, however allow some _very_ large numbers. There is some dark voojoo in the optimization of these variables to get the best speed out of your operation. Read through the CUDA ducumentation if you want to know more!

Last up, we reshape the data back into the 2-d ndarray in scipy. I chose to log scale the coloring because it brings out the iteration difference nearest the edges for nice contrast. Then we output a .png of the image! (Using the “.T” orients the axes in those more familiar to us.)

Depending on the versions of python “misc.imsave” may or may not work for you. You can also use “imshow()” and “savefig()” from matplotlib, but this has boarders or figure formatting you would have to kill to get a pure image. The matplotlib colormap of ‘winter’ (“cm.winter”) looks very nice with this particular view, but play around! There is a lot of neat visualization fun you can have here! I love playing with the alpha channel that PIL extends you (another part of scipy).

One last closing note, is that pycuda is not dependent on scipy, only numpy (which scipy sits atop), so you should be able to manage all of this without the heady overhead of scipy if you so desire!

Comments, concerns, and critiques all welcome! I am new to this whole thing and wanted to put together a more detailed example than those available with the pycuda documentation! This in particular show you how to attack problems larger than one block size.

Next up, I am going to write a SOR solver with pycuda, so check back if that intrestes you!

I was playing with pycuda to do some intense linear algebra via my sexy new Nvidia 260 GTX, when I ran into a python hitch.

The newest version of CUDA on compilation outputs a slightly different file format that expected. Causing the Python to crash. I fixed a tid-bit of regex and every thing started rolling!

Here is the reply I got from Andreas:

Hi all,

Meawoppl reported (below) that PyCuda fails with CUDA 2.1 because nv made a whitespace change in the cubin files they write. First of all, a change like that shouldn’t be able to take PyCuda down, so I’ve made sure that even if metadata extraction fails, PyCuda doesn’t throw weird-looking exceptions.

Next, I’ve fixed the actual bug along the lines of what Meawoppl suggests. Since this bug may turn people away from PyCuda unnecessarily, I’ve released 0.91.1.

Happy hacking,
Andreas

Email:
On Mittwoch 26 November 2008, Matt G wrote:
I found a silly little bug you might wanna know about:
Some nvcc builds generate “cubin” files that your regex fails to account for:

The line:
reg = 4
(two spaces b/t ‘reg’ and ‘=’ causes the:

self.registers = int(re.search(“reg = ([0-9]+)”, cubin).group(1))

to return a None and thusly crash ungracefully. Adding a strategic + to the regex seems to fix things:

self.registers = int(re.search(“reg += ([0-9]+)”, cubin).group(1))

I attached my “cubin” file so you can take a looksee.
Thanks for the awesome tools.
I am gonna be taking on some big CFD chores only possible with your help!

Thanks.
-Meawoppl

ps:
nvcc –version
Copyright (c) 2005-2007 NVIDIA Corporation
Built on Fri_Nov__7_06:20:13_PST_2008
Cuda compilation tools, release 2.1, V0.2.1221>

NVIDIA 260 GTX

First, apt-get the dependencies:

sudo apt-get install debhelper dpatch dpkg-dev fftw-dev g77 lapack3-dev libufsparse-dev libx11-dev netcdfg-dev python-all-dev python-central refblas3-dev sharutils swig subversion build-essential

Next make a folder somewhere and cd into it.
Pull down the latest versions of scipy and numpy:

svn checkout http://svn.scipy.org/svn/numpy/trunk numpy
svn checkout http://svn.scipy.org/svn/scipy/trunk scipy

Build Numpy:
Check to see if everything looks good:

cd numpy
python setup.py
(select Configure – 0)

See if all the libraries you will be needing are found.
The output is a bit confusing, but if you are missing something crucial, it will appear in red with a “Warning:” preface.

python setup.py
(select Build – 1)

sudo python setup.py
(select Install – 2)

Next repeat this process for Scipy:
cd ../scipy

python setup.py
(select Configure – 0)

For some people, (Ubuntu Hardy Heron x86) it will fail to find umfpack.h resulting in a swig error
So first try:

UMFPACK=”/usr/include/suitesparse/”

I think this a problem cause by the fact that the modern libsuitesparse package (which contains umfpack) has a different directory than scipy is expecting for umfpack.h.

If that doesn’t work, some people can get away with:
UMFPACK = “None”

Which will default to different routines for sparse matrix solution (about an order of magnitude slower depending on your architecture) If you don’t know what this means, either option should work fine for you.

python setup.py
(select Build – 1)

sudo python setup.py
(select Install – 2)

Any you should be all set with brand spanking new scipy and numpy for epic mischief.