Thoughts, ideas and moohoo
Awesome! Carlos de la Guardia wrote a book on Grok - it is available as of now!
Actually, I knew Carlos was writing the book for a quite a while already, as I was asked to review the draft versions of the chapters. It felt as quite an honor!
Reviewing the drafts was much fun, but more difficult and time-consuming than I anticipated at first. I sincerely hope I was of any help to Carlos.
I'm very eager to hold the end-result in my hands!
There was a time when I loathed Apple, and Macs in particular. I thought they were slow and unstable and way to expensive and frankly, Apple users would speak of other systems with dédain. At least that's how I perceived it.
This was about ten, fifteen years ago (dang - makes you feel old).
At that time I was happily using Window 95, Windows 98 and quite loved Windows 98 SE. That was then.
At some point, it must've been arround 1998, 1999, I got introduced to the wonderful world of *nix. A very good friend had already been playing around with Linux then, but though intrigued I never quite made the jump. But at my first job it was of mix of Windows, Mac and SGI and there I started fooling around with Irix.
Not long after that I got hooked by the *nix love and became very much fed up with Windows. So at home I threw it away, installed RedHat and never quite looked back. Soon after RedHat I switched to Debian, which I like(d) better.
Anyway, when I started working for Infrae I also bought a laptop. At that time Apple somehow started to attract me more and more. OS X became mainstream and its *nix underpinnings looked much much better to me than the old Mac OSs. So, I still do not know exactly how and why, I bought a Powerbook G4 Titanium. A lovely machine!
Even though it momentarily ran Gentoo, I quickly switched back a much smoother ride on OS X: suspend just worked, powersavings just worked, wifi just worked, sound just worked, attaching beamers just worked, burning a CD just worked.
It. Just. Worked.
For my desktop machines and the few servers I managed I was still using Debian, and, from its first public releases on, Ubuntu. For software development I'm actually still mainly using Ubuntu. But now on a virtual machine running on my Macbook Pro.
Why this writeup?
Today I decided to replace our wireless router. It has been the second router in two, three years that behaved erratically and basically annoyed the hell out of me. It was slow and constanly loosing connection, and whatever I tried, I couldn't get it to work reliably. I decided to replace it with an Airport Extreme.
It is not cheap of course. It is actually rather expensive. But there's a thing that makes up for it:
It. Just. Works.
So, I'm an Apple Fanboy now? Having multiple Macbooks in the house, the Mac mini, the iPhone, using iTunes, and now the Airport Extreme? Indulging the unboxing of the Apple gear, enjoying the attention to detail, like the Airport having the exact same dimensions as the Mini so they stack up nicely? Enjoy chatting with my collegues about all the cool new stuff, feeling lucky my current workplace is virtually Windows free?
I guess so.
I'd happily admit it :-)
A week ago I installed a Mac Mini as a media server. It doubles as a backup server for my web server and our laptops.
The laptops - both MacBooks - use Time Machine for backups into a shared folder on the Mini. That's fairly straight forward. For the Ubuntu-based web server I'm using rdiff-backup that has served me well for years already. Only difference now is, that the receiving end is the Mini running OS X.
Rdiff-backup itself is written in Python (jay!) and should work just fine on OS X. It uses the librsync library that's not by default available under OS X, it needs to be compiled and installed first. librsync in turns needs the popt library, that needed to be installed as well.
For both of these packages it was basically a matter of downloading and extracting the tarball and then doing a ./configure –prefix=/usr/local && make && sudo make install.
Then I tried building rdiff-backup by doing python2.5 ./setup.py build. This succeeded without warnings and installing with sudo python2.5 ./setup.py install seemed to go allright as well. However, when checking the version with /usr/local/bin/rdiff-backup –version gave a strange error message:
Traceback (most recent call last): ... ImportError: dlopen(/Library/Python/2.5/\ site-packages/rdiff_backup/_librsync.so, 2): Symbol not found: _rs_file_copy_cb Referenced from: /Library/Python/2.5/\ site-packages/rdiff_backup/_librsync.so Expected in: flat namespace in /Library/Python/2.5/site-packages/\ rdiff_backup/_librsync.so
Google didn't find me a useful hint. At a certain point I thought to try Python-2.6 instead of the Python-2.5 that comes with OS X. Building and installing rdiff-backup now worked like a charm!
The first testruns for backing up the web server revealed one more problem to solve. The backups are written to an external USB drive connected to the Mini and as it turned out, the user ids and group ids of the files are were not correctly written to this external disk - even though the backups are run by the root user.
A quick Google got me this hint: There's a setting found in the infobox for the mounted drive, where the "Ignore ownership on this volume" option can be turned off. That helps!
Backups are running fine now.
Today I decided to just cough up the $79 for an Omnifocus license. I'm trying to be bit more serious about a "kind of" GTD aproach to, well, getting done the things I want to get done.
Since I also bought the Omnifocus iPhone app and since I wanted to use Omnifocus on my work computer, some way of syncing the data between these locations was a requirement. Luckily Omnifocus is fairly flexible in this regard.
It can synchronize over MobileMe, local networks using Bonjour, shared disks or over WebDAV. WebDAV was the only viable solution for me, however my little server's HTTP daemon does not support WebDAV itself. It is able to reverse proxy to a local WebDAV daemon. And I found pywebdav to do exactly what I need: serve a directory over WebDAV.
With a short buildout.cfg file I had a server up and running in almost no time:
[buildout]
parts = config server start stop
[config]
recipe = z3c.recipe.template
directory = ${buildout:directory}/var
host = localhost
input = etc/config.ini.in
log = ${buildout:directory}/log
output = ${buildout:directory}/parts/config.ini
password = secret
port = 1234
username = foobar
[server]
recipe = zc.recipe.egg
interpreter = python
eggs = pywebdav
[start]
recipe = collective.recipe.scriptgen
cmd = ${buildout:bin-directory}/davserver
arguments = -c ${config:output}
[stop]
recipe = collective.recipe.scriptgen
cmd = ${buildout:bin-directory}/davserver
arguments = -c ${config:output} -d stop
I just completed the first alpha release of what will be Grok-1.1!
From the releasenotes:
The Grok development team is very happy to release Grok 1.1a1! This is a preview of the 1.1 release that will be based on the Zope ToolKit (ZTK). The ZTK will bring a significant number of newer versions of the libraries that Grok is built on. The ZTK will not only make available newer features and bugfixes in toolkit libraries, but also a less complex dependency structure of these libraries encouraging more reuse.
This release is not installed by default when using grokproject. It is an early alpha release, so that we can start testing our applications built with.
Most probably we will find issues and bugs in this release. Hopefully we can quickly move forward fixing issue towards a second alpha, or maybe even beta!
Personally I'd like to especially thank Michael and Souheil for working very enthusiastically on getting ZTK support in Grok!
ME GROK GROKS ZTK!
Recently Martijn Faassen blogged about Python packaging and related tools. There he mentions zc.buildout too. Of course - zc.buildout is great!
As Martijn points out, part of its greatness lies the fact the buildout uses recipes for extending its functionality.
Just look at the list of recipes already available!
http://pypi.python.org/pypi?%3Aaction=search&term=buildout+recipe&submit=search
(granted not everything in that list is a recipe persé, but there are many anyway!)
Today we reached a milestone: Grok 1.0 is released!
From the official announcement:
The Grok_ development team is very happy to release Grok 1.0. Grok 1.0 is the culmination of 3 years of work after the start of the Grok project in late 2006. It presents a stable platform for developing powerful, extensible web applications. Grok is the result of years of work by the large Grok development team. With Grok, simple projects are easy to create, but it is with increasing complexity and reuse that the Grok framework excels. Grok 1.0 is not the end of Grok development. Since Grok is extensible many powerful new features are being made available as growing collection of extension packages. The Grok development team is also working on a whole range of initiatives to improve Grok even further.
It took a while, but personally I'm very very happy with how Grok came out to be!!
Thanks for all the hard work, Grok Team!
Ever saw HTTP status code 103 appearing in your Apache access.log files?
At work we have, and we were highly confused by it as there's no "official" status code 103. We could not find a pattern for their occurrence in the logs either. The user agent didn't seem to matter, nor referrer, nor requested resource. No timing pattern, just every now and then a 103 response:
92.64.192.217 - - [23/Sep/2009:11:33:10 +0200] \ "GET /@@/jquery/jquery-1.3.2.min.js \ HTTP/1.1" 103 57254 "-" "-"
Grepping through the Apache sources (and just to be sure, through Zope's sources as well) didn't show up anything "103″ related to response or status codes.
In a moment of inspiration I looked in the error.log (duh) and found:
[Wed Sep 23 18:48:39 2009] [error] \ (103)Software caused connection abort: cache: \ error returned while trying to return disk \ cached data
Aha! Somehow the mod_cache sub system seemed involved. Indeed, for this particular site we are using Apache's caching module. Now more focussed, I had a look in the mod_cache source code again (what would you do if were you not using open source software??). There I found the error message that was in the error.log file.
I'm still not sure why exactly, but then I tried opening a connection to the server step by step and see what'd happen:
>>> import httplib >>> conn = httplib.HTTPConnection(site_address) >>> conn.request('GET', '/somepage') >>> conn.getresponse() >>> conn.close()
With this script I could reliably trigger the 103s. At the office that is, not at home… Apparently the time between the getresponse() and the close() calls is crucial for this to happen.
Our hypotheses now is that the real-life 103 errors probably come from browser windows being closed just before the page and its assets has been fully loaded.
Last day of sprinting. I had the idea most people were at least somewhat tired by now. I know I am.
We started the day with sprinter group reports (again, I hope I do not forget something here..):
Jan-Jaap kept an eye on the build slaves for Grok during our release preparations. This revealed a last minute bug in test cases of grokcore.view on Windows. Of course someone left filesystem path comparisons in the tests that could not work on Windows (hint: it was me…). So with a fast bugfix release of that we had all green flags and we were good to go for the 1.0b2.
Leonardo explored various aproaches to creating new grok projects without having to be online. Grokproject tries to do that, but it is not yet ideal. Maybe there're better eays of achieving this.
Sacha had been working on creating an Ubuntu Live-CD with one of more Grok applications installed on it to really have an "out-of-the-box" experience for people who want to give Grok a try.
Timo finished the migration of http://grok.zope.org to a more recent version of Plone. Great! It probably took him longer that he had hoped, but we're very glad he picked up this ball!
Sylvain had been working on porting five.grok and a lot of related packages to 1.0b2. It proofed not be that diffcult.
The documentation team gathered notes and remarks on the various tutorials and other documentation about using Grok. They are encouraged to send these notes to the Grok documentation mailinglist so that this useful work is not forgotten. It also makes for a nice "deliverable" for yourself to be able to share what you've been working on with others.
Vincent had been working the rdb integration libaries for Grok. There were late-night :-) checkins of an rdbexample application for Grok to demonstrate how this works. But he worked on megrok.z3cform and grokui.admin and a lot more packages too!
Souheil too worked on the grokui.admin package and made good progress for making this UI pluggable! He also updated megrok.*` packages to work with 1.0b2.
Martijn and I continued on grokcore.view and later split up: Martijn worked with on the rdb libraries and the ZTK and together with Jan-Jaap I explored how an integration of z3c.hashedresource and hurry.resource and grok.DirectoryResource could look like.
So, that was it. Now we're off to dinner somewhere in the city center of Köln and tomorrow will have early breakfast with whoever is still around.
Thank you all for a very productive sprint! See you next time, hopefully soon!
Let's also not forget to thank Alroldo and Novareto for organizing this sprint - that's always a lot of work. Thank you!!
now playing: "We will grok you" - Queen
Today we released 1.0b2. Yeah!
It took a while, but we're happy with this beta. The final 1.0 will most probably be almost identical.
But we're working on post-1.0 Grok already and there's lots of plans:
Martijn and I got grokcore.view to work with the latest martian. Next will be grokcore.viewlet and grokcore.formlib. This should be relatively straightforward.
There's also a branch of grokcore.view that improves the registry of templates that can be associated with views. Merging the registry code itself is not too diffcult. However, there's one complication: we need to make sure all templates are registered before we can actually grok the views that might associated them.
Grokkers can have a specific order, but this is the order in which the grokkers grok one module. The order in which the modules are handled is not determined.This is problem when view components in, say, module A subclass from view components in module B. The templates that get associated to the view components in B will probably not have been registered yet at the moment of grokking module A.
This needs to be solved obviously. Hopefully tomorrow.
And there're more plans still:
As always a sprint builds up lots of momentum rather quickly. We need to find a way to keep this momentum to a certain extend in between sprints. One way could maybe be to actually have more sprints.
I can imagine trying to organize smaller, very much focussed sprints on a more regular basis. Many of the Grok developers are really not that far away from eachother. Hopefully we can manage to do that.
But there's one more day of sprinting left still! Now that the release is out the door, we should get some "real" development done again :-)
We have a plan.
Today started with short reports by the sprinting groups. The things I can recall right now - I'm sorry if I forget something:
Jan-Jaap and Hanno worked on getting quite a number of buildslaves running on a buildbot dedidacted to having the ZTK, Grok and lots of related packages tested on multiple platforms on multiple versions of Python. Leonardo and Jan-Jaap then started on trying to get more "green lights" there…
Leonardo also has helped Martijn and me to get the release of Grok in good shape - more on that later. On monday Hanno and Christian Theune worked mainly on the official ZTK, making good progress there. Hopefully Grok can start to use the ZTK in the not too distant future.
Timo has been working hard on the rather thankless job of upgrading the http://grok.zope.org/ site to a newer version of Plone. Others have been working on reviewing and improving the "unofficial" documentation found there and where possible trying to promote documentation to the "official" documentation that is maintained along with Grok itself in SVN.
Uli and Christian Klinger and Sylvain and Vincent have been working on splitting the introspecter code off from grokui.admin in order to make that lighter weight. The introspector should come back as some extension to the admin view at some point - that code should not be forgotten. There're more ideas for admin view extensions like local utility configuration user interfaces and introspectors for locally granted roles and permissions.
They've also been working through the implications imposed by the infamous grok.CodeView split. Which brings me to, well, that topic again :-)
After a good nights sleep we decided to revert the split.
So there, grokcore.view will no longer have a CodeView - it was a mistake. grokcore.viewlet and grokcore.formlib and other packages need to be fixed - again - now. That may seem like a waste of effort, but most people felt this was really the right decision. Grok itself is mostly fixed already, so tomorrow Grok 1.0b2 is being released. And very short after that we will release 1.0.
Later in the afternoon we went to Bonn and vistited Ludwig von Beethoven's birthplace. We made life difficult for a nice waitress in the local noodlesbar. The food was good! But the cashregister broke down - so finding out who had to pay what turned into chaos…
Oh, right and I promised some pictures too!
Us busy sprinting at the GfU:
Beethoven's birtplace:
now playing: "I want to Grok you like an animal" - NiN
"General complexity is better than specific complexity"
Today's plan was to finish of the release we started yesterday. But we found some complications.
Martijn and I also started on finally making the Grok and the grokcore.* family work with the latest martian release. That release has fixed the long standing issues with module-level directives and inheritance.
We started with grokcore.component. The idea being that if we can this one to work, the others might not be that complicated anymore. And things fell in place quite well! grokcore.component lost some specific complexity as it used to implement its own idea of finding a context object (used by the adapter and utility grokkers), but it now relies on the general strategy in martian for it.
Another conclusion is that, if you write your own directives, in their implementation they are not allowed to use other directives for retrieving information. Since the directives work on import-time, you can quite easily get trapped in what effectively is an circular import. So that has been fixed in grokcore.component as well now. Tests pass. Yeah!
But no finished release still…
During the day we found lots of issues surrounding the recent split of grok.View and grok.CodeView. Now that the first beta was kinda out there, people actually started using grok.CodeView and wondered why the split actually happened. It didn't seem all that usefull especially as it turned out that there were some bugs to be fixed and more splitting off needed to happen (the grok.Viewlet actually needs a grok.CodeViewlet to keep things symetrical for example).
At the same time, since good progress was made on making grokcore.component compatible with latest martian, Martijn and I were in doubt whether the original technical reasons for having the split are actually still all that significant.
So, no next beta. And heavy discussions back and forth.
Tomorrow will try to find out whether it really isn't safer to not release Grok with the CodeView split off at all and having it to maybe later introduce it in a next feature release, versus now releasing a big API change with 1.0 only to have it maybe reverted later on…
We'll start off tomorrow with a report by all the sprinter groups - lots of useful things are happening. In the afternoon we're heading for a city trip in Bonn.
Like every sprint we started of with introductions and setting the agenda
Main important issue: getting one dot oh out of the door. At the very a least a first beta. Together with Leonardo and Martijn we aimed for a release at the end of the day. Unfortunately we didn't completely succeed.
Although the 1.0b1 is tagged and already on the package index, we decided to not let it be picked up by projects created by grokproject just yet.
The two things we need to resolve tomorrow morning before we can do that are:
We had a dinner at a typical German brauhaus and generally had a very nice day. Even though the goals were not completely met, it felt productive. Now it is time to go to sleep. You never know what people expect early results from us… :-)
Writing from a hotel in Köln, about to go to sleep. Tomorrow the Neanderthal II sprint will kick off.
Looking forward to it :-)
Even though I like OS X as my main day-to-day operating system, for software development I use Ubuntu. Ubuntu runs fairly smoothly in a VMWare Fusion virtual machine. Two things annoyed the h**ll out of me: I had to manually grab an ungrab the mouse pointer when switching between Ubuntu and native apps and somehow shared folders didn't work.
I found a blog entry on VMWare's company weblog that made enabling these features simple enough.
See also: http://communities.vmware.com//thread/206772#1235499
After about a year, we finally found some time at work to factor out "strict require" functionality for Grok applications from our internal code base. It's called megrok.strictrequire.
When using this package, all views that are registered in your Grok based application, are required to have an explicit grok.require directive set in the view component. This provides a first safety net for unknowingly opening up views in your application.
We tried to order a new Macbook Pro (yah!) from the Apple Store. However, both attempts failed at the moment of credit card verification.
The reported error was PaymentReentry.FullDollarFailure.
WTF. What's that supposed to mean?
Today I called the store. After explaining that somehow the credit card transaction failed, the guy immediately asked "You got the dollar failure, sir?" Sounds like a common situation.
Turns out that our order was exceeding the per-transaction limit on the card. Makes sense in a way, so I suggested the guy to get the error reporting improved. "Can't do that sir, for privacy reasons".
Huh? Why can you tell me the reason for the error over the phone, but not in the online store?
Ever since I was working on my thesis in '98 I'm using Logitech's "Marble FX" trackball. Apparently I'm not the only one hooked to it, keeping me RSI free. In the past eleven(!) years I collected some spare marbles as Logitech decided to only produce it for a year or two, without making a proper equivalent replacement.
But now even the spares are dying on me… The buttons on one of the spares aren't clicking reliably anymore. And the ball in the other one isn't moving smoothly as one of the gliders inside broke off. Luckily the third, the one I use at the office, is still working. But for how long?
I'm afraid I need to look for an alternative as even ebay only lists a couple used Marbles at the moment and looking at the pictures, they're very much used - which isn't suprising for a 11 year old trackball - or you're paying through the nose for one, or it's only shipped within the US. Bah.
Logitech lists one possible replacement: the Cordless Optical Trackman. That's the only trackball apparently where the ball is moved by the fingers, not the thumb. Unfortunately it is, duh, cordless. I'd rather have a more reliable wire, but oh well.
Maybe the Mediamarkt has one on display so I can give it try there. But if someone still has a Marble FX lying arround…
About two years ago we bought a Volvo 245. Whether you think its pretty or not, this car is a classic.
Beside being a classic, which may be nice in itself, its also very very spacious. And that's convenient when you're (still cough) renovating the house and like to go camping during holidays.
Another reason to buy this car (second hand of course, at the time already 201.000 Km on the odometer) was that it is a car that let's you do maintenance yourself. If you want to. Under the hood there's plenty of space around the engine block and all parts are big and reachable and there's not much electronics getting in your way. Great for getting your hands dirty.
Last short holiday's return trip was a bit of anxious ride. The temperature gauge wasn't in the comfort zone. It actually was reaching almost into the red zone. Does not make for a relaxed drive.
This weekend I took the plunge and replaced - myself - oil, oil filter, spark plugs. I checked and cleaned the air filter and most importantly decided to drain the cooling system, replaced the cooling liquid and also replaced the thermostat and cleaned the radiator.
Of course something needs to go wrong as well.
One of the hose connections on the radiator broke of. Just while I was filling the cooling system with new coolant. And since it was sunday afternoon by then, I could basically stop right there.
Yesterday I had to arrange for a new radiator. I ordered one at a more-or-less local car shop that does Volvo in particular. He returned my call with good news: he had a radiator in stock. So I picked it up - by bike of course.
So, yesterday evening I replaced the radiator. All connections were good to go now and I went for a test drive. The temperature gauge is happy now :-)
Just released the fourth alpha version of Grok. We're slightly overdue for a real "one dot oh" release, but there're some nasty issue that really should be solved before. This release in particular was triggered by an accidental indirect dependency on zope.container brought in by grokcore.view. Since the version for zope.container was not pinned, it could pull in newer versions. And unfortunately there was a recent release that would break Grok…
This is fixed now by getting rid of this dependency and "going back" to a dependency on zope.app.container.
Anyway, the release process is well documented by now and, even though it takes some time for the procedure to complete, it is not very complicated.
Hopefully this release solves more problems than it'll cause :)
My version of the ReST parser plugin for Zine is here: http://jw.n–tree.net/code/rest_parser
I'm used to type in ReStructured Text. So hacked up a Zine plugin parsing "ReST", including basic support for sourcecode:: python blocks that get syntax hilited.
Here's a demo:
class MyClass(object): """My cool class. Kewl, ain't? """ def __init__(self): self.foo = 'foo' self.bar = 'bar'
<root> <child_1>with text node</child_1> <child_2 with_an_attribute="value"> and more text </child_2> </root>
The plugin needs a bit of pollishing…