PrimaGIS sprint report

The Plone Conference was held in Naples, Italy this year and, as expected, was a great experience. Like last year in Seattle, we had a PrimaGIS focused sprint where a group of people worked on the next version of PrimaGIS which is based purely on Plone 3.0.

We had a list of sprint topics that people were able to choose from listed at the official sprint page. Below is a breakdown of the sprint activities, what was accomplished and what is still left to do.

OpenLayers integration

One of the biggest changes in the upcoming PrimaGIS version is OpenLayers integration. I had initially planned that OpenLayers would be used as an alternative user interface for PrimaGIS, but after a thorough introduction and discussion with Josh Livni we decided to use OpenLayers as the sole interface to PrimaGIS and retire the old custom one. I believe that this decision will serve us well in the future and the resources of PrimaGIS developers can be put to better use without having to worry about the map UI. The OpenLayers developers seem to be doing an excellent job at that and we’re grateful to use their work.

Currently the OpenLayers view works fine but there is little user configurability. The next step is to refactor the code so that the UI can be configured using user friendly forms and to make it easier for power users to customize the relevant parts. We also want to hook in the more advanced OpenLayers features such as spatial feature editing.

We also implemented TileCache support for PrimaGIS, so that people can easily configure their TileCache servers to improve PrimaGIS performance. Each presentation layer is able to produce a TileCache configuration section that can be copy/pasted to your TileCache configuration file.

Sean Gillies participated remotely and implemented support in PCL for the Python Geo Interface which allowed us to pull in feature data using all geometry types (point, line, polygon) directly into OpenLayers. Thanks Sean!

Big thanks also to Josh Livni for giving a lightning talk on using OpenLayers maps (including examples on FeatureServer and WPServer), for helping me to understand OpenLayers a bit better and working on the actual integration during the sprint!

New layer classes

Partly inspired by the OpenLayers and partly by the changes made to PCL and ZCO we decided to refactor the PrimaGIS layer classes. Previously there were two types of layers: PrimaGISLayer and PrimaGISDataLayer. PrimaGISLayer was used to create a map layer from spatial data from outside of Zope and PrimaGISDataLayer for data from within the ZODB. In either case it was difficult to share layers between multiple maps.

It had always been a goal to get rid of the artificial difference between the layers and data layers and adding support for sharing layers between maps seemed a good idea also.

The new types of layers are: Layer and Presentation Layer. A Layer class is a combination of what PrimaGISLayer and PrimaGISDataLayer used to be. It is used to select the data source where spatial data is acquired and to define the styling of the spatial features. It doesn’t matter whether the data comes from within Zope or not. The difference is that Layers don’t need to be located within Map objects. They can be located anywhere on your site and shared between multiple maps. A map administration may choose to have a folder where all Layers are stored but this is a totally arbitrary choice.

A Presentation Layer is an object that is stored within a Map object and refers to one or many Layer objects. It is tied more tightly into OpenLayers concepts such as a layer type which can be either base, overlay or vector layer. Each presentation layer is available as a Web Map Service (WMS) layer and renderable by OpenLayers. Connecting multiple Layers into a Presentation Layer allows you create composite layers that can be switched on and off as one. Below is a simple diagram that shows how the different objects are tied together.

Buildout for Linux distributions

Alessandro Amici took on the daunting task of refactoring the primagis.buildout into an alternative buildout configuration that can take advantage of installed system libraries. By default, primagis.buildout builds all the required components from scratch which is a process that can more than an hour even on a modern machine.

The new buildout simply assumes that a given set of packages is installed system wide and compiles the rest of the dependencies against those. Currently the buildout provides a list of debian packages that users of Debian based distributions (such as Debian or Ubuntu) should install prior to running the buildout. See README-deb.txt in the buildout source tree for more specific instructions.

Eventhough the new buildout configuration was put together for Debian based distributions it is also usable on other distributions if a similar set of dependency packages are available. Users of other distributions are invited to contribute a list of packages suitable for their distributions.

The implementation of the buildout, which was initially estimated to take less then an hour :), turned out to be much more complicated task due to packaging bugs and differences in libraries between Debian and Ubuntu. We even created a new buildout recipe to help with working with C-libraries which Alessandro released as bopen.recipe.libinc. Users of 64-bit distributions should give the buildout a try since there have been reported problems when compiling some of the dependencies by hand on these systems.

Demo product

Sune Woeller and Chris Calloway worked together on a new demo product for PrimaGIS called zgeo.primagisdemo. The idea is to provide a similar effect to the createPrimaGISDemo.py script found in the previous versions. However, instead of putting the demo map in PrimaGIS itself, a separate product was created that sets up the map using a GenericSetup profile for content import. This is still work in progress so stay tuned for a release announcement soon!

Once it’s ready you can simply copy to package into your python path and run the GenericSetup import step to get a PrimaGIS demo in your Plone site.

Remaining tasks

There are still a few items I would like to see in PrimaGIS 0.7.

• Persistent data stores and symbolizers
• Better OWS properties management view
• Spatial index for ZODB data
• Feature inspection
• Feature editing

On most items the work has already been started and some are are a simple matter of hooking into the functionality provided by OpenLayers.

Thanks again for all the people who participated in the sprint! It’s always great to people meet you know from IRC in real life.

 

Buildouts and recipes

For the past half a year or so we have been using zc.buildout at Hexagon IT to manage most of our new development projects and also the production environments serving them. It has proved to be an invaluable tool and I can honestly recommend it for others to use.

However, the usefulness of zc.buildout is directly related to the recipes that are available to perform the steps necessary to set up your project. A cool thing about recipes is that they can be distributed as Python eggs and there are already a bunch of them available at the Python Package Index (PyPI) ready to be used.

An important thing to know about zc.buildout, if you are unfamiliar with it, is that it is a general purpose tool and not part of Zope or meant only for Zope development. It can be used to achieve many different goals. For example, we have used it to perform tasks such as managing system users, generating configuration files for system daemons and naturally building and installing a wide variety of software (including pure C code as well as Python).

Recipes

Recently I released two of our most used recipes:

• hexagonit.recipe.download
• hexagonit.recipe.cmmi

The download recipe is capable of downloading packages in common formats (tar.gz, tar.bz2, zip) from the net and extracting and placing them on the filesystem. The cmmi (for configure, make, make install) recipe builds on the download recipe and additionally compiles software from the extracted packages using the configure, make, make install dance.

Both recipes are inspired by other existing recipes (such as zc.recipe.cmmi and gocept.download) but offer some additional and useful (at least for us) features. Both recipes have good doctest coverage that also fully documents their use. Please refer to the corresponding PyPI pages for more details. The PyPI pages contain the doctests so you can get a picture of how the work even without installing them.

Buildouts for GIS-Python Laboratory projects

The GIS-Python Laboratory currently host many projects including the Python Cartographic Library (PCL) and PrimaGIS. Traditionally installing PCL has been difficult due to a large number of dependencies. Last year I made an effort to ease up installing PrimaGIS by writing an automated buildout system for it called primagis.buildout. This system was based on Chris McDonough’s BuildIt and I believe it has served us well.

However, the main goal of primagis.buildout was to simply have a system that takes the burden of building and installing the software away but it didn’t really give much extra to developers once the system was built. For a developer the next important thing after being able to install the software is to be able to run its test suite to verify that the software behaves as expected. Especially in case of PCL, I’ve always found it a bit cumbersome to set up the test fixtures, even more so when coming back to the code after a while.

So for the next generation buildouts I wanted to make them more developer friendly.

PCL buildout

The PCL buildout is by far the largest and most complicated of the GIS-Python buildouts. You can read more about in the project README.txt file, but the most interesting features are:

• Componentized buildout. Like PCL itself, the buildout is divided into components so you can optionally build just a single component of PCL (e.g. PCL-Core, PCL-GeoRSS, etc). This can save you time if you wish to work on a smaller part of PCL when you’re fixing a bug for example.
• Unified test runners. Regardless of whether you’ve built a single component or the full PCL, you always have a test runner script under ./bin/test that will run all the tests that are relevant to the current build. The buildout has taken care of setting up the test fixtures so you can concentrate on making sure that the tests pass.

The buildout will take care of building all the dependencies, including some larger components like GDAL, GEOS, PostGIS and MapServer. The only thing you need to have is a working C/C++ compilation environment with tools like gcc, sed, flex, bison, etc. It is also useful to have the readline library and development headers available to make the interactive prompts of some the components more usable.

If you get an error during a buildout, just see which program / library the system is missing and make sure that both the library and the corresponding development package (libraryname-dev on most systems) are installed and rerun the buildout.

Other projects

I also created buildouts for the other hosted projects:

• owslib.buildout for OWSLib (README.txt)
• geojson.buildout for GeoJSON (README.txt)
• shapely.buildout for Shapely (README.txt)
• rtree.buildout for Rtree (README.txt)

Want to try them out?

In addition to helping developers be more productive by allowing them focus on the actual code instead of spending time on setting everything up, the buildouts are also a great way for newcomers to easily get a taste of the code. Using a buildout you can get everything up and running without too much hassle.

It’s all done in a sandbox so you don’t need to worry about messing up your existing system either. Once you’re done with it, you can simply delete the whole buildout directory and be done with it. The basic procedure to try a project is following:

1. checkout the buildout from the repository
2. run python bootstrap.py
3. run ./bin/buildout
4. optionally run the test suite ./bin/test
5. start playing with the interactive prompt: ./bin/python

Each buildout has a README.txt file that contains more detailed installation instructions. If you try any of the projects please leave a comment here or better yet, leave a message at our mailing list to let us know how it went.