Two weeks off the net

I've just arrived back from a two-week family holiday in the Brecon Beacons in Wales. Before I went I confess I was a little worried that two weeks without a computer or net access would be difficult. The last two-week holiday was four years ago to see family in Thailand, and we ended up using their computer now and again. But this time it was fine - and I resisted the occasional urge to nip into the local resource centre to check email. (There were of course lots of other things to do as you can see in the picture.)

Also on holiday I read a Guardian article about BlackBerry addiction (in a real newspaper, of course) and was glad I don't have one of the blasted things...

Tim O'Reilly on Hadoop

Tim O'Reilly has a written a nice piece about Hadoop, following its starring role at OSCON. I was tickled pink to see that he linked to my article about running Hadoop on Amazon EC2 to do log analysis!

Also, from the comments: Google has some courseware on MapReduce and Hadoop. Everyone seems to love Hadoop at the moment.

Hadoop at OSCON 2007

I wasn't there, but Hadoop had two airings at OSCON this week. Doug Cutting was a part of the Open Source Radar Executive Briefing with Tim O'Reilly to talk about scaling.

He also gave a talk, with Eric Baldeschwieler, entitled "Meet Hadoop" where he gave a great exposition of the problem that Hadoop is designed to solve. In short: disk seeks are expensive, so databases built using sort-merge, which is limited by transfer speed not seek speed, scale better than traditional B-tree databases, which are limited by seek speed. More details and examples on the slides.

Eric's half of the talk gave some interesting tidbits about how Hadoop is being used at Yahoo! For example, they are running Hadoop on about 10,000 machines, and the biggest cluster is 1600 machines! With these kind of numbers I can see how Nigel Daley came to coin Nigel's Law:

In a large enough cluster, there are NO corner cases

JUnit gets flexible and philisophical

My colleague Robert Chatley pointed me to an InfoQ announcement about the release of JUnit 4.4. It looks interesting. For a start it includes Hamcrest matchers which allow you to write flexible assertions, using the assertThat construct. I've had some involvement with Hamcrest, and I really like using it since it allows me to write tests that I can read, so I'm really pleased that it's got into JUnit as this can only increase its takeup. Well done Joe for having the idea in the first place!

The new release also includes theories, adopted from the Popper JUnit extension, which are tests that apply to a (potentially infinite) set of data points. This reminds me of Andreas Leitner's talk AutoTest: Push-button testing using contracts from the Google London Test Automation Conference (in 2006) where he talked about using Eiffel contracts to generate test cases to look for contract violations. David Saff is one of the creators of JUnit theories and he also talks about the relation to contracts in his paper (authored with Marat Boshernitsan), The Practice of Theories: Adding "For-all" Statements to "There-Exists" Tests. I'm looking forward to trying this out.

Articles and blogs

Coinciding nicely with Jakob Nielson's admonition Write Articles, Not Blog Postings (via Steve Loughran) here's my latest article, Running Hadoop MapReduce on Amazon EC2 and Amazon S3, published on Amazon Web Services Developer Connection.

It's almost a year and a half since I wrote my last article and one of the reasons is because I've blogged more. Not much, but more.

Jakob's got a point about being directed to short, old, irrelevant blog postings when you're looking for something. He's not saying don't blog, but rather try to write stuff that will be long-lived. Definitely something to aim for for tech writers - but not at the cost of not writing anything (so it's OK to sprinkle blogs with lighter weight stuff).