evolving

Sketching out an API

A long harboured niggle of mine against Template Toolkit is that it made some special rules about its fat comma that doesn't quite gel with the way that Perl uses it.

To break into example:

 statement               |  Perl                    |  Template Toolkit
 foo( bar => 'baz' )     |  foo( 'bar', 'baz' )     |  foo( { 'bar', 'baz' } )
 foo( bar => 'baz', 1 )  |  foo( 'bar', 'baz', 1 )  |  foo( 1, { 'bar', 'baz' } )

Yes, that's right, it treats => as a pair constructor, extracts the pairs, and sticks them into an anonymous array at the end of the argument list.

This used to really drive me nuts, for a number of reasons, mostly that it's a non-reversible munge.

You can try and turn things back around themselves, by writing the following preamble into your code:

 sub foo {
     push @_, %{ pop @_ } if @_ && ref $_[-1] eq 'HASH';
     ...
 }

but then you're doubly screwed when the last thing you wanted to pass actually was a hash reference. Not to mention that you've completely lost the ordering that was there.

Anyhow, I'd managed to forget most of this since I don't do a lot of work with templates these days, but a recent thread on the templates list brought it all flooding back.

While thinking about this, I hit on an idea that might be useful, if only in an Acme:: way.

Consider the chained method call: $foo-$bar->baz > Let's assume that $foo is of class Foo and $bar is of class Bar, so we can go ahead and run that:

 perl -Mstrict -e'my ($foo, $bar) = ({}, {}); 
                  bless $foo, "Foo"; bless $bar, "Bar"; 
                  $foo->$bar->baz
		 '
 Can't locate object method "Bar=HASH(0x813f2c4)" via package "Foo" at -e line 3.

So, we just tried to call a method named after the stringification of the $bar object on the $foo object. Now assuming that nobody got clever with the stringification, we're just one step away from fun evil.

Normally I'd frown on using AUTOLOAD, but in this case it's for fun, so we'll go with it. Say we were to write Foo::AUTOLOAD around Devel::Pointer, which will allow us to get ahold of the real $bar, by chasing down its pointer address.

 use strict;
 package Foo;
 use Devel::Pointer;
 use Carp;
 sub AUTOLOAD {
     our $AUTOLOAD;
     $AUTOLOAD =~ m{\((0x[a-z0-9]+)\)}
       or croak "couldn't extract an address from $AUTOLOAD";
     my $object = \deref( eval "$1" );
     return $object;
 }
 my $foo = bless {}, "Foo";
 my $bar = bless {}, "Bar";
 $foo->bar->baz;
 __END__
 perl proof_of_evil
 Can't locate object method "baz" via package "Bar" at proof_of_evil line 15.

So there we are, back to as if we'd just called $bar->baz. Now that in itself isn't hugely useful, but instead if the Foo AUTOLOAD starts returning proxying objects then you could have a proxy which wraps the push @_, %{ pop @_ } if ... idiom that's useful for some TT method calls.

Which brings us to the title, and the problem of creating an API to allow people to use this repointerizing proxy pattern. (I knew I was in trouble when I woke up this morning and declared it a pattern, but it was frightfully early).

Typically I have a couple of thoughts, two of which are well-formed, and neither of which I really like. I'll use the push @_, %{ pop @_ } idiom as an example in these.

OO-style.

 package My::Proxy;
 use base qw( Class::Proxy );
 sub _proxy_behaviour {
     push @_, %{ pop @_ } if @_ && ref $_[-1] eq 'HASH';
 }
 My::Proxy->$bar->baz;

Pros: Makes it possible to hide all the evil.

Cons: Whatever name we give _proxy_behaviour, we're going to run into a clash somewhere, as with any other methods we decide we need.

Import magic

 package My::Proxy;
 use Class::Proxy sub {
     push @_, %{ pop @_ } if @_ && ref $_[-1] eq 'HASH';
 };
 My::Proxy->$bar->baz;

Pro: Evil still fairly well hidden.

Cons: Import style can be a bit dense.

But I'm still chewing it over.

I have my proof-of-evil though, which makes me happy.

Monday 2004-06-07 18:26

And, after 3 months...

You know, I've been waiting since before the original release of Date::Holidays::UK (which CPAN has down as the 16th March) for this bug report.

I'm especially gratified to find that my pedant thinks I don't know that the regions holidays are defined in are "England and Wales", "Scotland", and "Northern Ireland", and so needed it all explaining to me.

Unfortunately, now that it's arrived, I'm still having trouble caring.

Still, I remain thankful that it's not code for correctly formatting telephone numbers.

Monday - 2004-06-07 17:16

Paging Win32 and/or Device::SerialPort users.

So pleasantly enough Device::Ericsson::AccessoryMenu turns out to have quite the life of its own, so much so that the latest release (0.8) is entirely the work of Arne Georg Gleditsch who sent me a patch last month which I shamefully only just got round to looking at.

Anyway in yesterdays mail I received a query about bluetooth devices going out of range, which is in part the disconnection/reconnection stuff mentioned in the TODO section of the pod.

The problem in play here is that when your bluetooth device goes away it's presented to the machine as 'your modem has been disconnected from your serial port' and I can't work out how to get that from the Device::SerialPort API (actually, I can't figure out very much at all from the documentation of Device::SerialPort and Win32::SerialPort, but that's a different matter).

One way around this whole mess is to avoid Device::SerialPort altogether, and handle the serial port directly in a posix stylee, the downside to which is but that means losing the plug compatibility with Win32::SerialPort and so Win32 compatibility in general.

I'm somewhat reluctant to lose an OS just to make my life easier (though if we were talking about AIX then that's quite a different matter...), but I don't really know that there are Win32 users to lose, and so on to the appeal.

If you're using Device::Ericsson::AccessoryMenu on Win32 please drop me a note to let me know. Also if you know how to get Device::SerialPort to 'fess up about the device going away, that'd be super too.

Monday - 2004-03-08 05:52

Sweet sweet release

So I just finished cleaning up my zeroconf stuff enough to get it released, you should see them arrive on your local CPAN mirror any time soon.

After a grand renaming I shoved everything into the Net::Rendezvous::Publish namespace, since after experimentation I found that Net::Rendezvous is a pure-perl browser-only implementation, but that I liked the api and implementation enough to try and usefully extend that space.

(I did try and evaluate Net::MDNS::Server and Net::MDNS::Client but their embdedded mdnsd just didn't build for me on Panther)

So about that terminology then:

Zeroconf

Zero Configuration Networking or Zeroconf to its friends, is an IETF working group who have issued specifications on various things. Together those specifications are called Zeroconf. Wasn't that nice and clear of them.

One of the Zeroconf standards is mDNS, but there are also a bunch of standards that define automatic allocation of IP addresses and the like.

ZeroConf

How I keep mistyping Zeroconf. I say it as two words too. It's all a trap laid out for me I swear.

mDNS

Multicast DNS. A fun abuse of the DNS wire standards to add multicasting of service information. This is where the publishing and browsing of services we all know and love comes from.

Rendezvous

What Apple calls their adoption of the Zeroconf protocols. It's actually just a subset which contains Dynamic Configuration of Link-Local IPv4 Addresses, Multicast DNS, and DNS-SD.

- or -

It's just what everyone calls mDNS over link-local.

Anyway I'm glad to say that's 3 modules cleared from my yak pile, I can start on finishing Hook::Queue tomorrow, and everything will be just super.

Sunday - 2004-02-22 00:46

Back with the new style

First, apologies to you people reading via rss or use.perl, the syndicating view seems to really make article-style things look like ass. It's really much better looking in the original klingon, or the html/css thing that it transforms into.

Now some more of that old fashioned 'I tweaked my blog engine' guff.

I've moved in thinking from this being a ongoing diary to more distinct entries, just like a real blog. That means now the linkable pages are entries rather than days, and I'm putting a timestamp thingy at the bottom as a =head3 block so I can still figure out the day on which I wrote it. I might even teach the RSS template about that, if I feel the urge.

There, wasn't that suitably dull.

Saturday 2004-02-21 13:49

Mac kool-aid

When I joined Fotango, I asked my new overlords if I could get a Mac laptop as my work machine. Being good masters, or as part of their open source slave programme - the distinction isn't clear, they equipped me with a shiny powerbook (and after some supplier issues a working one.)

The upshot of this is that for the last couple of months I've been migrating into it, and embracing stuff like iCal and Rendezvous.

Here's a run-down of the kool-aid I've supped to date.

Fink

I've always treated the system shipped perl with great suspicion, mostly because it's often old, the modules I want aren't packaged for it, and if I manually install/upgrade modules they get trampled by system upgrades. As a result I generally install my own into /usr/local/perl5.xx and tend that while letting the system take care of itself.

Well with the laptop I went with fink, because compiling binary modules against fink-managed libraries has bit me before, the system perl is a 5.8.1-era thingie (a late 5.8.1 RC at least) so new enough, and it's easy enough to write package descriptions so the modules I want can be made available as packages.

It's going well so far, I got a talk out of it, and more package descriptions than you'd normally shake a stick at (caveat: I rarely shake sticks, which may skew the sample).

iTunes/iPod

Actually, I drank of the iTunes kool-aid with the release of the Windows version, but I have imported all my albums into the new machine. With X-Tunes it's a dream.

New to the mix is a 15Gb iPod. Given that no new technology acquisition goes unpuninshed I ran into a typical iPod problem - my music collection is bigger than it is.

iTunes has smart playlists, so I can say 'give me 15Gb of music', but the drawback is that I can say 'of albums' or 'of songs', and 'randomly', but that either gets me the first 15Gb of albums every time, or incomplete albums.

The solution of course was to write a script that uses File::Find::Rule and Mac::Glue to populate a playlist with 15Gb of randomly selected albums. Because that's a bit slow it also uses Term::StatusBar so you can see how far through you are.

Mail.app

I read a lot of email. OK, I skim a lot of email. I used to deal with all of it using mutt on penfold, and then I switched to using Mail.app's imap mode.

That was all good, apart from Mail.app's mail filtering rules don't want to filter into imap hosted folders. That's not so much of a problem for my on penfold because everything goes through procmail and Perl in a long-tended configuration (scripts and .procmailrc in the usual places.) At the office however, the imap server is a magic multi-domain virtual-user setup, so running procmail just isn't an option.

imapfilter to the rescue! Though it's got a slightly quirky configuration file (and what doesn't), I managed to bend it to my will and now everything is all well and filtered in the office too.

Rendezvous

Or is that Zeroconf? Or should that be mDNS? Or is that just a part of the zeroconf standard? I've a dread feeling that I'll have to put together another list like I did for the vCalendar debacle.

Anyway, it's really cool to watch things show up in Rendezvous Browser, especially if you're using modules you wrote to do it. More services should advertise themselves.

While I was busy debugging Net::ZeroConf::Backend::Howl George Chlipapa released Net::Rendezvous, which I must install and play with at some point, as it could well completely obsolete my 3 dists before I even got round to releasing them.

iCal

Apart from the adventure of parsing ics, it's been really rather painless.

On my todo list is to set up a private DAV share on unixbeard.net so that we can put household events like "I'm going to FOSDEM, feed the fish" in it, as we still seem not to have bought a paper 2004 calendar for the kitchen.

Friday 20th February, 2003

Fun with Calendars

A couple of months ago, that pesky Paul Mison (if I get to be the "wily" Richard Clamp for Pod::Coverage I reserve the right to hand out one pesky to someone) planted an idea in my head, which lead me through something of a world of hurt...

First I'll tell you what the idea is, and you can tell me what the pain was.

Ready?

 It would be cool to turn a page on our intranet into ics so we can
 add it to iCal.

Okay, so now take a moment to yourselves to figure out where the pain came from.

Done?

10 points if you guessed Net::ICal.

Now I know it's disclaimed as being alpha code, and that I shouldn't be surprised if it's not too good, but the CPAN dist doesn't even pass its own tests, and the CVS version didn't even get that far.

I even sat down for a few afternoons and worked on a fork of Net::ICal CVS which got the tests passing, and it still wasn't fit for duty.

There's a whole other paragraph which goes here and questions the parentage and sanity of Class::MethodMapper, but there's only so many times you can say rude words. Let's just say it's not suitable for something you want to parse, validate, and follow the flow of control without big sheets of paper and crayons. I like crayons, but I like clear flow of control even more.

Thus cheesed off, I put the idea aside for a few weeks, expecting that when I came back to it I'd just cave and write an alternative to the whole ball of wax.

That few weeks ended this week, when Nik pointed out to me that Text::vFile should be a suitable module to base a vCalendar parser on.

an aside - iCalendars, vCalendars, ics, vFile, rfc2445, make your mind up!

So why do I keep calling things ics one moment, vCalendars the other, and iCal the next? Well it's partly due to my brain being a swiss cheesed mess, but also because someone is out to get me.

rfc2445 / iCalendar

rfc 2445

     Internet Calendaring and Scheduling Core Object Specification
                              (iCalendar)

This baby is the specification for iCalendar-formatted files.

Weighing in at 148 pages it covers pretty much anything you need to do for online interactive calendars (maybe).

ics

The common extension for iCalendar files when stored on disk.

vFile

It turns out that the object/property/parameter encoding used by rfc 2445 is the same as the one used by rfc 2426 vCard MIME Directory Profile, apart from 2445 encoded documents tend to have deeply nested objects.

We follow Text::vFiles example in calling these documents vFile formatted.

vCalendar

Okay, this one's probably made up.

It comes about because iCalendar is a kind of vFile.

I think.

Look just stop looking at me like that.

iCal

An application from Apple Computer, Ltd. Parses, generates, and generally speaks rfc 2445.

Sometimes I call it iCal.app, in order to distinguish it from the standard in the same way that use Mail.app to refer to Apple Mail.

And thus overloading occured, in my branes at least.

Sadly, Text::vFile doesn't handle nested objects - which you really need for iCalendars - and clones a little too much (ie. at all) from Class::MethodMapper in its api and thinking.

At about the time we discovered this my impatience kicked in. "Surely it can't be that hard to parse vFile to a simple data structure" I said to myself, and half an hour of coding later it turned out it wasn't. Text::vFile::asData sprang into the world.

A couple more hours tagteaming with Nik to find the corner cases and fix them and we both managed to scratch our itches. Nik's handy what's going on script to summarise upcoming events, and my wiki to ical script for the intranet, both done fairly easily with the assistance of DateTime. Version 0.01 hit the CPAN last night, with 0.02 to follow this weekend.

Friday 20th February, 2003

mmm - every home should have one

One of the most tedious things in writing a new module is just spinning up the distribution structure and the pod boilerplate to go in it. Of course, there are a whole bunch of ways to skin that cat, from the fugly h2xs, through ExtUtils::ModuleMaker, to that home grown that you've got sitting in ~/bin.

I'm using a home grown one too, but only in the sense that I'm using mmm, which my housemate Mark Fowler wrote.

It's got two big selling points for me. Firstly it creates a very sane module structure, which is an absolute requirement. Mainly though it's that it pulls all the boilerplate in using the Template Toolkit. This makes it trivial for me to just customise the sub-templates for when I'm not Mark, or I don't like one of his defaults.

If you've ever had an itch to use a module generator I'd suggest you take mmm for a spin - it offically Doesn't Suck.

Friday 20th February, 2003

Monday 17th November, 2003

Overly and repeatedly dumb.

So today I figured it'd be a good enough time to try and speedup Timesink some. For those not keenly stalking me or my pet projects Timesink is my web-based RSS aggregator which I wrote a while back on migrating away from a dying mac and NetNewsWire.

Anyway, there are a couple of parts of it which are kinda slow, one is the "what's unseen" calculation in the web frontend, and another is the scraper.

Now the unseen code looks like this:

 package Timesink::DBI::Subscriber;
 sub unseen {
     my $self = shift;
     my $feed = shift;
     my ($sub) = Timesink::DBI::Subscription->search({ feed => $feed,
                                                       subscriber => $self });
     my %seen = map { $_->item => 1 } $sub->seen;
     return grep { !$seen{ $_ } } $feed->items;
 }

That is; given the subscriber find out how many items in a given feed are unseen by that subscriber. That you get the actual objects is somewhat a side effect as the web inferface only really cares about the final count.

Okay I think, after determining that this is the slow spot with Devel::Profiler, time to rewrite that as a quick SQL query.

Small flaw in that plan, my live instance runs on mysql 4.0.16, and won't get upgraded till 4.1 until debian unstable does that for me, so I'm stuck with painful rewriting to emulate it. After about half an hour of that, I eventually admited defeat and added this comment:

 +# XXX I be the slowest routine in Christendom.  a sub-select would
 +# probably help, if mysql 4.0 wasn't lame.

So moving on to the second step. Work on the speed of the scraper.

Now the scraper itself is fairly quick, it's only really waiting on upstream servers handing out RSS documents to parse, so if I could just parallelise the downloading that it's going to take less time, even if it's not really quicker.

Now I could see two ways around that, something finicky with LWP::Parallel or the brute force forking of Proc::Queue.

Given that I didn't want to rewrite LWP::Simple's mirror routine I decided to plump for forking, just so long as I remembered to disconnect the dbh I'd be fine, or so I reasoned. So this was my first stab:

Before:

 for my $feed (@feeds) {
    my $rss = $self->get_rss( $feed ) or next;                         
    $self->scrape_feed( $feed, $rss );   
 }

After:

 for my $feed (@feeds) {
     my $pid = fork;
     die "couldn't fork $!" unless defined $pid;
     if ($pid == 0) {
         my $rss = $self->get_rss( $feed ) or next;
         $self->scrape_feed( $feed, $rss );
         exit;
     }
 }
 1 while wait != -1; # reap the kids

Spot the deliberate mistake? Well even if you did, I didn't for a time. Then Rafael asked me why I was grabbing his RSS 30 times a minute.

So I scratched my head, and eventually saw my mistake. Back in the old single process model if get_rss didn't return new rss that was your clue to check the *next* rss feed. Once I'd moved that into a multi-process model the job of the child is not to try again, but to exit gracefully. The fix was as simple as:

 -         my $rss = $self->get_rss( $feed ) or next;
 +         my $rss = $self->get_rss( $feed ) or exit;

Case solved I thought, and went off to watch teevee.

Of course it doesn't end there. But there's bonus points if you guess my next (and hopefully final mistake of the evening).

Yes that's right, I'd forgotten to install the fixed version of the module, so come the next time the script ran it picked up the old DoS-happy version of the module and looped all over again. D'oh.

Nothing for months, and then two modules come along in one day.

IO::Automatic and Parse::Debian::Packages pretty much sprung to my fingers unbidden today, the latter a side effect of adding debian support to Leon's cool new Module::Packaged module, the former is a TT-like trick extracted from some code I found myself banging on.

Enjoy.

Older stuff...

Tuesday 28th October, 2003
Friday 19th September, 2003
Thursday 18th September, 2003
Wednesday 17th September, 2003
Monday 15th September, 2003
Thursday 11th September, 2003
Monday 8th September, 2003
Saturday 6th September, 2003
Thursday 21st August, 2003
Wednesday 20th August, 2003
Intermission
Wednesday 30th July, 2003
Tuesday 29th July, 2003
Monday 28th July, 2003
Sunday 27th July, 2003
Saturday 26th July, 2003
Friday 26th July, 2003
Thursday 24th July, 2003
Wednesday 23rd July, 2003
Tuesday 22nd July, 2003
Saturday 19th July, 2003
Thursday 17th July, 2003
Wednesday 16th July, 2003
Tuesday 15th July, 2003
Monday 14th July, 2003
Saturday 12th July, 2003
Friday 11th July, 2003
Thursday 10th July, 2003
Thursday 26th June, 2003
Monday 23rd June, 2003
Sunday 22nd June, 2003
Friday 20th June, 2003
Wednesday 18th June, 2003
Monday 16th June, 2003
Sunday 15th June, 2003
Saturday 14th June, 2003
Friday 13th June, 2003
Thursday 12th June, 2003
Wednesday 11th June, 2003
Monday 9th June, 2003
Sunday 8th June, 2003
Saturday 7th June, 2003
Friday 6th June, 2003
Thursday 5th June, 2003
Tuesday 3rd June, 2003
Sunday 1st June, 2003
Saturday 31st May, 2003
Friday 30th May, 2003
Friday 23rd May, 2003
Monday 19th May, 2003
Sunday 18th May, 2003
Saturday 17th May, 2003
Monday 12th May, 2003
Sunday 11th May, 2003
Saturday 10th May, 2003
Friday 9th May, 2003
Thursday 8th May, 2003
Wednesday 7th May, 2003
Sunday 4th May, 2003
Friday 2nd May, 2003
Thursday 1st May, 2003
Wednesday 30th April, 2003
Tuesday 29th April, 2003
Monday 28th April, 2003
Sunday 27th April, 2003
Saturday 26th April, 2003
Friday 25th April, 2003
Thursday 24th April, 2003
Wednesday 23rd April, 2003
Tuesday 22nd April, 2003
Monday 21st April, 2003
About this