Buildouts and recipes

09 Aug

For the past half a year or so we have been using zc.buildout at Hexagon IT to manage most of our new development projects and also the production environments serving them. It has proved to be an invaluable tool and I can honestly recommend it for others to use.

However, the usefulness of zc.buildout is directly related to the recipes that are available to perform the steps necessary to set up your project. A cool thing about recipes is that they can be distributed as Python eggs and there are already a bunch of them available at the Python Package Index (PyPI) ready to be used.

An important thing to know about zc.buildout, if you are unfamiliar with it, is that it is a general purpose tool and not part of Zope or meant only for Zope development. It can be used to achieve many different goals. For example, we have used it to perform tasks such as managing system users, generating configuration files for system daemons and naturally building and installing a wide variety of software (including pure C code as well as Python).


Recently I released two of our most used recipes:

The download recipe is capable of downloading packages in common formats (tar.gz, tar.bz2, zip) from the net and extracting and placing them on the filesystem. The cmmi (for configure, make, make install) recipe builds on the download recipe and additionally compiles software from the extracted packages using the configure, make, make install dance.

Both recipes are inspired by other existing recipes (such as zc.recipe.cmmi and but offer some additional and useful (at least for us) features. Both recipes have good doctest coverage that also fully documents their use. Please refer to the corresponding PyPI pages for more details. The PyPI pages contain the doctests so you can get a picture of how the work even without installing them.

Buildouts for GIS-Python Laboratory projects

The GIS-Python Laboratory currently host many projects including the Python Cartographic Library (PCL) and PrimaGIS. Traditionally installing PCL has been difficult due to a large number of dependencies. Last year I made an effort to ease up installing PrimaGIS by writing an automated buildout system for it called primagis.buildout. This system was based on Chris McDonough’s BuildIt and I believe it has served us well.

However, the main goal of primagis.buildout was to simply have a system that takes the burden of building and installing the software away but it didn’t really give much extra to developers once the system was built. For a developer the next important thing after being able to install the software is to be able to run its test suite to verify that the software behaves as expected. Especially in case of PCL, I’ve always found it a bit cumbersome to set up the test fixtures, even more so when coming back to the code after a while.

So for the next generation buildouts I wanted to make them more developer friendly.

PCL buildout

The PCL buildout is by far the largest and most complicated of the GIS-Python buildouts. You can read more about in the project README.txt file, but the most interesting features are:

  • Componentized buildout. Like PCL itself, the buildout is divided into components so you can optionally build just a single component of PCL (e.g. PCL-Core, PCL-GeoRSS, etc). This can save you time if you wish to work on a smaller part of PCL when you’re fixing a bug for example.
  • Unified test runners. Regardless of whether you’ve built a single component or the full PCL, you always have a test runner script under ./bin/test that will run all the tests that are relevant to the current build. The buildout has taken care of setting up the test fixtures so you can concentrate on making sure that the tests pass.

The buildout will take care of building all the dependencies, including some larger components like GDAL, GEOS, PostGIS and MapServer. The only thing you need to have is a working C/C++ compilation environment with tools like gcc, sed, flex, bison, etc. It is also useful to have the readline library and development headers available to make the interactive prompts of some the components more usable.

If you get an error during a buildout, just see which program / library the system is missing and make sure that both the library and the corresponding development package (libraryname-dev on most systems) are installed and rerun the buildout.

Other projects

I also created buildouts for the other hosted projects:

Want to try them out?

In addition to helping developers be more productive by allowing them focus on the actual code instead of spending time on setting everything up, the buildouts are also a great way for newcomers to easily get a taste of the code. Using a buildout you can get everything up and running without too much hassle.

It’s all done in a sandbox so you don’t need to worry about messing up your existing system either. Once you’re done with it, you can simply delete the whole buildout directory and be done with it. The basic procedure to try a project is following:

  1. checkout the buildout from the repository
  2. run python
  3. run ./bin/buildout
  4. optionally run the test suite ./bin/test
  5. start playing with the interactive prompt: ./bin/python

Each buildout has a README.txt file that contains more detailed installation instructions. If you try any of the projects please leave a comment here or better yet, leave a message at our mailing list to let us know how it went.

Leave a comment

Posted by on August 9, 2007 in GIS, software engineering


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: