Category Archives: software engineering

Pro Git: Professional version control

Scott Chacon has written a new book on git  called “Pro Git: profession version control” which is freely available at http://progit.org/ and licensed under Creative Commons Attribution-Non Commercial-Share Alike 3.0 license.

Make sure to check it out if you’re in need for some additional git-fu.

Playing with Erlang

For a while now I have been following news about Erlang. I’ve always been interested in learning about new languages and paradigms, if only to get some fresh ideas into my day-to-day work using Python. I felt interested in Erlang for a couple of reasons: it seems to to be a good candidate for writing heavily concurrent software spanning processors and machines, it’s in use by some of the bigger players today, plus I’ve always enjoyed the functional paradigm starting from my early days of programming with Scheme.

Before diving in fully I wanted to write something simple just to get a feel of the language. Pattern matching is an integral part of the language but what caught my eye was that you can actually use pattern matching down to the level of individual bits when reading in binary data. This gave me the idea of porting the small SWF metadata parser package I wrote some time ago to Erlang to see how it compares with the Python implementation.

After playing with it for a while I came up with the following implementation for the parser:

parse(Filename) ->
    {ok, Swf} = file:read_file(Filename),
    {Compression, Version, _, Payload} = parse_signature(Swf),
    {Xmin, Xmax, Ymin, Ymax, Frames, Fps} = parse_payload(Payload),
    {version, Version,
     compression, Compression,
     width, Xmax-Xmin,
     height, Ymax-Ymin,
     bbox, {xmin, Xmin, xmax, Xmax, ymin, Ymin, ymax, Ymax},
     frames, Frames,
     fps, Fps}.

parse_signature(<<"FWS", Version:8, Size:32/little, Data/binary>>) ->
    {uncompressed, Version, Size, Data};
parse_signature(<<"CWS", Version:8, Size:32/little, Data/binary>>) ->
    {compressed, Version, Size, zlib:uncompress(Data)}.

parse_payload(Payload) ->
    % Extract the RECT field size
    <<Nbits:5/little, _:3, _/binary>> = Payload,
    % Calculate the byte alignment
    ByteAlign = (8 - ((5 + 4 * Nbits) rem 8)) rem 8,
    % Pattern match the RECT element
    <<_:5,
      Xmin:Nbits/big-signed,
      Xmax:Nbits/big-signed,
      Ymin:Nbits/big-signed,
      Ymax:Nbits/big-signed,
      _:ByteAlign,
      Frames:16/little-unsigned,
      Fps:16/little-unsigned,
      _/binary>> = Payload,
    % Convert the RECT values from Twips to pixels
    {Xmin / 20, Xmax / 20, Ymin / 20, Ymax / 20, Frames, Fps}.

The bit-level pattern matching really worked to its advantage in this particular case and I feel the parser implementation in Erlang is much more elegant than the code I wrote in Python. Basically the two functions: parse_signature and parse_payload do all the work in only 9 lines of Erlang code (without pretty printing and comments). I had to split the implementation into two parts because the SWF format contains variable length bit fields. A more uniform format using byte-aligned fields would have been even easier to parse. The full code is available in GitHub should anyone be interested in taking a closer look.

All in all, I enjoyed this little exercise and will keep learning more about Erlang in hopes of possibly integrating it into my list of useful tools for future projects. If you’re interested in learning more about what you can accomplish with Erlang there is a pretty cool Erlang related series of posts about creating a “Million user Comet application with MochiWeb” (part I, part II, part III).

Update 2009-02-03: Fixed a bug in the second parse_signature clause that failed to decompress SWF files correctly.

GitHub screencasts

If you’re using Git and haven’t heard about or used GitHub, which is a Git hosting service with a social networking twist, then here’s your chance! Scott Chacon is doing a screencast series called “Insider guide to GitHub” for the Pragmatic Programmers. The first episode is free of charge and a great way to get introduced to the features provided by GitHub.

collective.buildbot 0.3.0

I just pushed in a new version of collective.buildbot to the PyPI. Some highlights of the new release are:

  • Support for PyFlakes checks
  • Refactored project and poller recipes supporting multiple repositories (previously supported by the projects and pollers variants which are now gone)
  • SVN pollers work again
  • Cygwin fixes

If you were using an earlier version you will need to update your buildout configuration to accommodate the changes in the recipe configuration options.

Building Buildbots

Some time ago Tarek Ziade started a project to make it easier to configure and set up a Buildbot environment using zc.buildout. During the Paris Plone sprint I helped Jean-Francois Roche and Gael Pasgrimaud to further improve upon this work and after the sprint the collective.buildbot project was released.

I recently took some time to polish up the package with proper documentation and examples that should make it easier to deploy it for your own projects and released the changes as version 0.2.0.

Setting up a buildbot environment is pretty easy, you create a buildout for the build master that is responsible for configuring all the projects and one or more buildouts for the build slaves. The Putting it all together section in the documentation gives you an overall picture how to accomplish this.

Hopefully this will encourage people to use buildbot to improve the quality of their software. There are already some public buildbots available, check out buildbot.ingeniweb.com or buildbot.infrae.com for example. Is your buildbot next?

UPDATE: There was a bug in the “Putting it all together” example, which is fixed in 0.2.1.

SWF metadata parser

Recently I needed to be able to determine the dimensions of SWF (Flash animation) files so I could embed them properly on a web page but I couldn’t immediately find something useful with Google that would perform the task. I am aware of the Hachoir project, but it seemed a bit overkill for my simple use case and a quick try with hachoir-metadata failed to parse my particular SWF file.

Luckily the container section of the SWF file format (which contains the metadata) is rather simple and writing a parser for it turned out to be a nice distraction from my normal duties. The result is hexagonit.swfheader which is a minimal package (no dependencies outside the standard library) that provides a single function that parses SWF files and returns the metadata.

The package comes also with a console script that you can use on the command line to quickly introspect local SWF files. In a buildout you’ll need to use the zc.recipe.egg:scripts recipe to get the script installed.

Improved zc.buildout recipes with ZopeSkel

Today I worked with Tarek Ziadé on ZopeSkel. Tarek concentrated on refactoring the ZopeSkel layout to put each template in its own module and wrote doctests for all available templates. Go Tarek! The test runner actually runs tests in two layers: first testing the output of the generated items and then, if the items contain tests themselves running them also.

I concentrated on improving the template for creating new zc.buildout recipes. Many useful recipes suffer from lacking documentation and an unappealing front page on PyPI. I refactored the template to include a common set of documentation files, such as CHANGES.txt, README.txt, CONTRIBUTORS.txt etc. and added code that puts all those documents nicely together to produce a serious looking ReST document that looks good on PyPI. So now its up to the recipe author to just fill in those files accordingly.

To help recipe authors and especially people new to zc.buildout I also added comments in both the documentation files and the code to help on implementing the recipe and especially on how to document it so that other people are able to use the recipe in their own buildouts. To me, one of the most importart parts of a recipe’s documentation is the list of available options and their semantics. Looking at the PyPI pages for zc.buildout and zc.recipe.egg you can easily get information about the component. I’ve also tried to do the same with my own recipes (hexagonit.recipe.cmmi, hexagonit.recipe.download). The template provides a stub for documenting the options in the README.txt file that authors can fill in.

I also created a minimal doctest for the buildout. While being only a skeleton the test actually runs a buildout using the recipe so you can run the test case for the recipe right after ZopeSkel is finished generating it. This should help recipe authors to get started with testing the recipe while they implement it.

In addition I updated the trove classifiers to appropriate values for a buildout recipe and added support for getting the trove classifier for the license to be added automatically in the setup.py file. So now when paster asks for a license for the recipe and you answer, for example, ZPL you get ‘License :: OSI Approved :: Zope Public License’ in your setup.py automatically. This code is actually in zopeskel.base and you can easily re-use it in the other ZopeSkel templates. Just take a look at how the recipe template uses it.

If you haven’t used ZopeSkel before, give it a try!

$ easy_install ZopeSkel
$ paster create --list-templates
$ paster create -t recipe collective.recipe.foobar

If you want to try the recent changes, you need to get ZopeSkel from the collective.


http://svn.plone.org/svn/collective/ZopeSkel/trunk/

There’s been lots of interest in ZopeSkel here at the Snowsprint so expect to have cool new templates there soon!

Update: 25.01.2007

ZopeSkel 1.5 was released which contains the latest changes.

Buildouts and recipes

For the past half a year or so we have been using zc.buildout at Hexagon IT to manage most of our new development projects and also the production environments serving them. It has proved to be an invaluable tool and I can honestly recommend it for others to use.

However, the usefulness of zc.buildout is directly related to the recipes that are available to perform the steps necessary to set up your project. A cool thing about recipes is that they can be distributed as Python eggs and there are already a bunch of them available at the Python Package Index (PyPI) ready to be used.

An important thing to know about zc.buildout, if you are unfamiliar with it, is that it is a general purpose tool and not part of Zope or meant only for Zope development. It can be used to achieve many different goals. For example, we have used it to perform tasks such as managing system users, generating configuration files for system daemons and naturally building and installing a wide variety of software (including pure C code as well as Python).

Recipes

Recently I released two of our most used recipes:

The download recipe is capable of downloading packages in common formats (tar.gz, tar.bz2, zip) from the net and extracting and placing them on the filesystem. The cmmi (for configure, make, make install) recipe builds on the download recipe and additionally compiles software from the extracted packages using the configure, make, make install dance.

Both recipes are inspired by other existing recipes (such as zc.recipe.cmmi and gocept.download) but offer some additional and useful (at least for us) features. Both recipes have good doctest coverage that also fully documents their use. Please refer to the corresponding PyPI pages for more details. The PyPI pages contain the doctests so you can get a picture of how the work even without installing them.

Buildouts for GIS-Python Laboratory projects

The GIS-Python Laboratory currently host many projects including the Python Cartographic Library (PCL) and PrimaGIS. Traditionally installing PCL has been difficult due to a large number of dependencies. Last year I made an effort to ease up installing PrimaGIS by writing an automated buildout system for it called primagis.buildout. This system was based on Chris McDonough’s BuildIt and I believe it has served us well.

However, the main goal of primagis.buildout was to simply have a system that takes the burden of building and installing the software away but it didn’t really give much extra to developers once the system was built. For a developer the next important thing after being able to install the software is to be able to run its test suite to verify that the software behaves as expected. Especially in case of PCL, I’ve always found it a bit cumbersome to set up the test fixtures, even more so when coming back to the code after a while.

So for the next generation buildouts I wanted to make them more developer friendly.

PCL buildout

The PCL buildout is by far the largest and most complicated of the GIS-Python buildouts. You can read more about in the project README.txt file, but the most interesting features are:

  • Componentized buildout. Like PCL itself, the buildout is divided into components so you can optionally build just a single component of PCL (e.g. PCL-Core, PCL-GeoRSS, etc). This can save you time if you wish to work on a smaller part of PCL when you’re fixing a bug for example.
  • Unified test runners. Regardless of whether you’ve built a single component or the full PCL, you always have a test runner script under ./bin/test that will run all the tests that are relevant to the current build. The buildout has taken care of setting up the test fixtures so you can concentrate on making sure that the tests pass.

The buildout will take care of building all the dependencies, including some larger components like GDAL, GEOS, PostGIS and MapServer. The only thing you need to have is a working C/C++ compilation environment with tools like gcc, sed, flex, bison, etc. It is also useful to have the readline library and development headers available to make the interactive prompts of some the components more usable.

If you get an error during a buildout, just see which program / library the system is missing and make sure that both the library and the corresponding development package (libraryname-dev on most systems) are installed and rerun the buildout.

Other projects

I also created buildouts for the other hosted projects:

Want to try them out?

In addition to helping developers be more productive by allowing them focus on the actual code instead of spending time on setting everything up, the buildouts are also a great way for newcomers to easily get a taste of the code. Using a buildout you can get everything up and running without too much hassle.

It’s all done in a sandbox so you don’t need to worry about messing up your existing system either. Once you’re done with it, you can simply delete the whole buildout directory and be done with it. The basic procedure to try a project is following:

  1. checkout the buildout from the repository
  2. run python bootstrap.py
  3. run ./bin/buildout
  4. optionally run the test suite ./bin/test
  5. start playing with the interactive prompt: ./bin/python

Each buildout has a README.txt file that contains more detailed installation instructions. If you try any of the projects please leave a comment here or better yet, leave a message at our mailing list to let us know how it went.