Now for some of my cows

We use Grok at work to (re)build our publishing platform. One of the things though that Grok does that we definitely do not want, is to have all views "open" by default. So, only when you explicitly tell Zope (through Grok) to require a permission for a particular view, Zope will trigger authentication in order to authorize the request.

There're some vague plans to have this optional. To have a switch somewhere that you can flip and have all views closed by default.

In meantime we thought of this quick and elegant hack: use a custom grokker that looks for view (-like) components and raises an error if the view component does not require any permission. Since this error will be raised "at grok time" it will effectively prevent the application from starting, if it has unprotected views:

class CheckRequireGrokker(martian.ClassGrokker):
  component_class = grok.View

  def grok(self, name, factory, module_info, config, **kw):
      if not grok.util.get_default_permission(factory):
          raise GrokError(
              'This application requires %r to use the grok.require '
              'directive!' % factory, factory)
      return True

Basically, on an application level you can check all kinds of stuff on your components!

Update: of course, to be sure all view like components are protected, you need to check on the XMLRPC, REST, JSON, and *Form components as well.

I didn't finish my report on the Neanderthal sprint. Let's see if I can wrap it up now:

On Thursday and Friday Luciano and I continued on pinning down versions for the dependencies of Grok. Thursday existed mostly of testing and trying to get as much "properly" released packaged included in the list as possible. Even if this meant we had to downgrade some of these.

One of the more restrictive criteria was to maintain compatibility with the Windows platform. Some of the eggs contain platform specific dependencies but these binaries are not always up-to-date with the most recent source releases.

At the end of Thursday we had a more or less working versions list and ideas on how to make buildout use an "official" versions.cfg file, that is published as part of a Grok release.

The solutions are described here: http://grok.zope.org/releaseinfo/readme.html

Friday consisted mostly of writing that report. But of course we had to properly wrap up the sprint as well. We gathered the groups of people that were still there (some had to leave on Wednesday and Thursday already) and reviewed the checkins that were done, had a first look at the new Grok website, and made a list of post sprint follow ups.

Last week I got postive feedback on the changes on grokproject and merged the changes to the trunk. Martijn and I got Grok-0.10.1 released and pushed grokproject-0.6 out the door as well..!

And by now, Grok turned 1 year old! And we're heading for a 1.0 release...

other people about the Neanderthal-sprint: Martijn's blog entry , Luciano's photos , and two entries by Lennart Regebro. When I find more, I'll add them.

Yesterday we had a cultural break. Aroldo arranged a trip to Neanderthal, with the Neanderthal museum in particular. Neaderthal is one of the first places where remains of prehistoric mankind have been found.

The museum is very well made and very interesting. We were guided by a lady that had to cope with this group of geeks now that happily accepted her invitation to ask questions and discuss the things she explained :-)

When we retured at the hotel in the evening we decided to order pizza's, get some beers in and basically continue the sprint just there.

I meant to jot something down after the first day of sprinting, but I was basically too tired. I'm again tired but I felt I needed to write down something now, because else it wouldn't happen anymore.

The trip to Koln by motorcycle went just fine. It is less than 300km, so you should be able to do it about 3 hours. If you don't forget to take your own wallet with you and find out only when you're about to refuel that is...

At the hotel we - I happened to bump into Lennart when I arrived - quickly met up with Martijn and Jasper and Guido and Aroldo in some restaurant. Later that night Uli and Luciano arrived - it is great to be able to meet the people in real life that you only know from the mailinglist!

Monday morning started off with identifying sprint topics and tasks and to try and do some planning.

Besides lots of other stuff, Luciano and I expressed interest to continue to work on the Grok reference documentation, because it is severly lacking. Together with Jasper, Guido and Uli we decided to continue Uli's work and to use the RestructuredText version of the original LaTex source files that Theuni started a while ago and on which I tried to work as well.

Jasper and Uli wrote some code and gathered pieces of the puzzle to render nice HTML from the ReST files, esp. in regard to the specific tags and directives that we use. We are looking at the current Python (2.5) documentation there too, because that documentation effort has apparently switched to ReST as well.

Luciano and I concentrated more on the content-level of the reference documentation: we decided on the structure of the documents, cleaned up whatever content that was already there and continued writing new sections. One of the interesting conclusions was, that, although for in Grok itself the theme of ICavemen, ICaves, IFireplaces and IClubs for writing test classes and interfaces and so on is nice and cute, for the reference documentation it might be worthwhile to come up with more realistic and pragmatic examples of how to use the code. That way we think to give better examples of how to solve actual use cases.

Today (well, actually, yesterday by now) we knew we would continue on the reference documentation, but wanted to some coding as well. First we finally concluded the seemingly simple task of making Grok skip tests and ftests modules and packages in Grok-based application when Zope is started.

If these packages are not skipped, the registrations done in the test code (for testing the application) show up in the running Zope which is usually not what you want. I had a Martian branch (where this skipping is actually done) sitting there that needed a merge and Uli did some work too on this. We reviewed Uli's work and my branch, improved it here and there, released martian-0.9 and then updated Grok to make use of the skipping-modules-during-the-scan-process- API.

Then we headed for another important task: try to see if we could come up with a list of versions of dependencies of Grok that are known to work. Currently we find ourselves in a quite messy situation, where the toolset we use to make predictable and reproducible buildouts of our code bases and its dependencies does not deliver and appears to make life more difficult instead of easier. I think most of this is not so much due to the tools, but due to the learning process the people doing releases of the software we depend on (and remember, where would we be without these people and the things they have built!!) are going through. But looking at the Zope3-dev list, I think I can see clear signs that this is another complex situation the community is sorting out.

But more on that later, now I really need to go to sleep, because tomorrow we will go to Neanderthal and maybe even "meet" the great-great-great-grandfather of Grok...

There'll be another Grok sprint from Monday on. And I'm attending! :-)

I feel quite lucky to have an employer that allows its people to attends things like sprints and conferences.

Since I had some stuff to do at home today, I decided to go by motor bike instead of taking the train with Martijn, Jasper and Guido. But its not to far away anyway, so it'll be a nice trip going there. Everything is packed now and ready to go.

Two easy_install tricks I should remember:

setup(
     ...
     install_requires=[
         ...
         'PIL'
         ]
 )

setup(
    ...
    install_requires=[
        ...
        'Imaging'
        ]
)

<a href="http://www.effbot.org/downloads/Imaging-1.1.6.tar.gz#egg=PIL-1.1.6">
  PIL-1.1.6
</a>

easy_install -f http://username:secret@domain.tld/path/to/index

Last weekend I had the opportunity to attend the second Grok sprint. Martijn invited me to join in, as we did work on some grok related issues and testcases already the last couple of weeks.

Grok makes is easier to make use of the great toolset Zope-3 has to offer. It essentially replaces the default configuration and composition system of Zope-3 (based on zcml, an XML language) with another one.

Martijn and I left for Halle, Germany on thursday morning. In Halle we would sprint at Gocept's "headquarters" for one day on friday. We decided to go by train, which would be a seven and a half hour long trip. Time passed by quite quickly actually!

We had diner in Halle and after diner we got back at Theuni's place and I got introduced to the highly addictive game called "Guitar Hero"...

On friday, Christian Theuni, Wolfgang Schnerring, Martijn and I started on making plans for the rest of the sprint weekend and identified two feasible and interesting goals to aim for:

• Make it easy to register local utilities, with being able to easily install and use the Zope catalog as pratical use case.
• Get view-based security to work.

Most of the friday was spent on this planning and figuring out how we would the security and local utility registrations to work. Since my experience with hacking on Grok was still somewhat limited, I took the role of a potential developer of applications with Grok. From that perspective I tried to give feedback on how I would perceive the API that was being designed.

This process of first designing the grok API and how the configuration directives would look like, together with the strong attention to detail in the error reporting of Grok, resembled how user interfaces can be designed to quite some extent. In that sense, Grok is actually an user interface for the application developer to Zope-3's libraries and functionality.

On saturday morning we left for Philipp von Weitershausen's place in Dresden. There we would continue the sprint. Theuni of course brought Guitar Hero there too :-) After a two hour drive to Dresden, we discussed the plans for the rest of the sprint weekend with Philipp.

We then split up in two teams, where Philipp and Theuni would work on the security topic. Martijn, Wolfgang and I would work on the local utility registration with being able to use the catalog as a clear and concrete target to reach.

Since local utilities need to registered for concrete instances of objects in Zope (unlike global utilities that get registered as soon as Zope is started), quite some infrastructure work needed to be done in Grok where core Grok code was refactored for the second, third or even fourth time.

After doing this refactoring-groundwork, Theuni and Philipp reporting succes on getting the Zope-3 security mechanisms to do what we wanted it to do, and enjoying Philipp's lasagna for diner, we played lots of Guitar Hero doing hilarious imitations of how Grok, the caveman, would play rockstar guitar. Grokstar, the dude man! :-)

On sunday we continued working on the same topics in the same teams, until Wolfgang had to leave at the end of the afternoon. We presented the teams' achievements to eachother and concluded that getting the catalog to work, using the local utility infrastructure that now was built, should be fairly easy.

And it is! With just a few lines you can now setup one or more catalog instances, including what indexes to use, for you application.

(It was a bit sad to realise Wolfgang had worked so hard on all the different test cases to cover for the local utility registrations, but had to leave just before we enjoyed actually using it and see it in action)

I think I can say that we went to diner with quite a satisfied feeling, having achieved quite something in this three days! We enjoyed the food, wine and beer and played more Guitar Hero. And then some more :-)

The trip back home yesterday, going from Dresden now, took more than nine hours... This was a bit tiring, but I'm glad I went to this sprint!

The next sprint is already being planned for somewhere in April. The goal then is to get Grok in a polished, documented and releasable state...

update: the next sprint should be somewhere around April, not March and Martijn wrote a blog entry about the sprint too!

zc.buildout is a system for managing development buildouts. Or in other words (mine): for deploying applications, for production usage or development work.

zc.buildout works by way of recipes where one or more recipes define what should be done and how. E.g. "install Zope-3, create an instance and make sure this and this egg is available on the python path". Usually a recipe needs some more information from the user to know where to install things or where to look for dependencies. This is specified in the buildout.cfg file.

I learnt using zc.buildout mainly by looking at other people's work (e.g. grok).

There you can see the specification for the buildout (in the buildout.cfg file) are part of project's hierarchy. So, you check out the project from, in this case, the Subversion repository and you have the project's sources and the specification to make a buildout for this project. You run the buildout command and voila, you have a running project set up.

This makes sense while developing on this particular project, but when you're deploying an application for production use, you do not necessarily want to checkout the sources for the project, since you probably already tagged the release version, uploaded the eggs for you project to some package index (e.g. the cheeseshop or some custom index) and created a buildout.cfg specifically for the deployment of this application.

For these situations I'm trying the following pattern "in the field":

In my project's hierarchy I add a sub directory called "deploy". This directory contains basically only a buildout.cfg. This buildout.cfg knows where to go and look for all the eggs it needs. Now you only have to get this "deployment" buildout.cfg onto the end user's machine and run it to set up the application.

todo: I need to have a look at http://svn.zope.org/zc.recipe.deployment/

There's a whole lot of things I should write about. Soon. Some of these are related to work.

Things like:

• Installing software from custom built *.deb files for easy deployment on the collection of servers we maintain.
• Exploring using Subversion as a content repository for research data and materials.

• Exploring the idea of using what I at this point call "stand alone web applications".

It's the last point that touches tonight's experiments with Paste. The experiments started off with a nice little tutorial on wsgi and how the tools Paste provide can help develop, deploy and maintain web applications for wgsi enabled application servers.

Researchers at the ING quite often do not have a internet connection available when they're "in the field" aggregating data and materials, e.g. at an archive facility.

So far it was not uncommon to use M$ Access databases installed on their laptops for entering and storing data. Apart from Access' deficiencies and its general ickyness, there's no good method of combining several of these databases once the researchers are back at the office.

I wonder if we can't build a set of more or less simple web applications running on the researcher's laptop for entering research data?

I got pretty enthousiastic about Paste Deploy that might help distributing and deploying these applications.

But well, I have no idea about the current state of the Paste developments, or if the tools gather any kind of momentum. I generally tend to like software written by Ian Bicking, but he has a certain way of doing things.

And oh, my mind is not clear at all yet on these matters. I had hoped writing this down would help...

Ian Bicking wrote an excelent little tutorial on how to build WSGI web applications and frameworks.

Tremendously useful!