Compiling!

When I am writing software sometimes XKCD is accurate!

Alas I can only fully participate in that activity when the boys get home from school.

The rest of the time I have to make do with other distractions.

I am currently participating in a "higher" speed broadband trial (I already have a 50Mbit service). This appears to involve the drastic step of remotely reconfiguring my Cable Modem :-)

In the last week there has been the odd request from the trial organisers to test throughput using various website based testing applications. These applications seem completely unable to cope with these 50Mbit+ connections and the results are as unreliable as expected.

To address this the trial organisers asked us to time downloading of a gigabyte file from one of their servers. I was surprised to discover that it took over 350 seconds to download the example file giving a less than stellar 3Megabyte/second rate.

So I used some "compiling" time today to look at what was going on. Firstly I went looking for an iperf like tool for http. Turns out there isn't one which came as a bit of a surprise...oh well with a little help from my friends I came up with

curl -o /dev/null http://target.domain/1GB.bin 2>&1 | tr "\r" "\n" |awk '{print $12 }' >test1.dat

Which gets a file with a "Current transfer speed" for each second of the transfer. Well ok so lets do the transfer a few times and collect the output so we have a reasonable data set.

So we have a pile of numbers...not terribly useful, lets visualise them! to the gnuplot mobile!

We need a gnuplot script something like say this:

set terminal png nocrop enhanced font arial 8 size 1024,600 xffffff

set output 'xfer.png'

set style data linespoints

set title "1Gigabyte file transfer throughput"

set ylabel "Throughput in Kilobytes per second"

set y2label "Speed in Megabits per second"

set xlabel "Seconds of transfer"

set ytics 1024

set y2tics ("10" 1220, "20" 2441, "50" 6102, "100" 12204)

set grid noxtics y2tics

set yrange [0:13000]

set datafile missing "-"

plot 'test1.dat' using 1 title 'Test1', \

'test2.dat' using 1 title 'Test2', \

'test3.dat' using 1 title 'Test3'

Once run through gnuplot I extracted a lovely graph which shows a couple of things.

Mainly that even with a nice fat downstream you are unlikely to realise the maximum throughput very often even from a server on your ISP local network.

On the other hand I now have a way to examine throughput of downloads ;-)

I like driving in my car. It is not quite a Jaguar

I work from home, this is a good thing. I benefit from a 20 metre commute, comfortable working environment and generally low carbon lifestyle.

Except on Wednesdays, on Wednesday I have to get up early and go to the office, this is not usually too much of a chore and takes around 90 minutes each way.

Today was different, I made a small five minute diversion to collect a colleague to whom I was giving a lift and then due to a little problem near the M6/M61 junction spent a fun filled three hours sitting in traffic crawling along the M56. At least I had company instead of being on my own.

Before I did the return journey I decided to check the traffic news sites. Oh dear now the M60 was stuffed, I altered the route and only had to queue on the M56 for twenty minutes or so. I dropped my colleague off at his place (avoiding the worst bits of the M62/M66 junctions by use of a rather convoluted back route) and proceeded to queue on the M62 for a while for no apparent reason.

Basically I have spent almost seven hours in the car today to do a 150 miles or a little over 20 miles per hour average. I was just going to rant about the dreadful lack of any redundancy or resilience in the UK road system which often grinds to a complete and utter halt if there is a single failure.

However a different thought has wandered across my travel weary mind. It has occurred to me that this average speed is faster than anyone could reasonably expect to do this trip for the majority of human existence.

In 1810 and indeed for all time before, your best possible speed by good horse, for 150 miles, would have been two days (and your horse would have probably been very poorly afterwards) This assumes your horse could do the 75miles (120km) each way in times consistent with modern world endurance trials... across a mountain range! Yes the Pennines are only tiny but even so!

A hundred years later, in 1910, the British railway network was nearing its zenith in most measurable terms. The influence across the north of England was profound and pushed the industrial revolution ever faster towards its climax before the first world war. Even at this point in time my best reading of the available timetables says I would have needed to change trains four times each way, purchased eight separate tickets from six different companies and taken around nine hours to make the journey allowing for hanging around on platforms.

Another fifty years on, in 1960, the trans-pennine car journey would have been on poorly maintained trunk routes through the decaying cores of the declining post-industrial northern cities. The route would probably have involved the A646, A59 or the A58 which at this time were not the well maintained (if slightly shabby) roads of today but instead were dangerous twisty and, from the looks of the archive photographs, positively heaving with traffic. On these pre-motorway strips of tarmac the 150 mile round trip would have taken in excess of seven hours (even today's mapping systems suggest over four and a half hours would be needed)

So instead of being frustrated that my commute took an extended period today I have instead decided that I shall enjoy the fact it was faster and certainly more comfortable than at any time in the past. Well that and I need to get the cars air-conditioning fixed ;-)

The turmoil of an entropy key release.

Last week we released 1.1.3 of the Entropy Key software. Poor Daniel struggled for days to get this out the door but finally he managed to build all the various debs, rpms and tars for the supported platforms and Rob got it all uploaded and announced.

The release is kinda strange in that it was the first in which the main changes were for performance. OK there is an improvement to resilience in the face of failed re-keying which some users were seeing in high load situations, but that high load was (in some cases) being caused by the daemon itself.

The process was mainly driven by one of our users, Nix, who was experiencing ekeyd using "too much" CPU on his system.

Of course on our servers during testing ekeyd it had used around a percent of CPU, certainly nothing that flagged as a problem in our own use (yes we eat our own dogfood ;-) Alas for this user on a 500MHz geode it was guzzling down 10% of his CPU which was clearly unacceptable.

This user however instead of guessing what the problem might be or simply leaving it up to us did something about it. He instrumented ekeyd, located the garbage collector tuning parameters as being incorrectly set and supplied a patch. Did he stop there? no! he then went on to profile the code further and clean up the hotspots. This resulted in ekeyd falling to less than 1% of the runtime of his system.

By reducing the CPU usage of ekeyd to this level it became more apparent where a previously reported bug was coming from, which enabled me to address it.

I know sometimes I complain about Open Source software but at times like this it makes me happy that we released the ekeyd software freely. This is how its supposed to be! Everyone working to make better software and benefiting together.

It has not just been on this occasion either, throughout the last year since our very first 1.0 release there has been helpful and useful feedback, patches from several users and even the odd thankyou mail. This project then has been a positive Open Source experience and I look forward to another constructive year maintaining this software.

You shall go to the ball!

Contrary to my last post I was able to attend the Debian UK BBQ at the weekend. My wonderful wife ditched me at Portsmouth station with permission to go play with my friends ;-)

Perhaps a bit more explanation is warranted about that last statement! We travelled back from France last Saturday. We were on the 12:15 (CET) ferry so had to be awake and on the road for the five hour France drive at "oh my gosh its early" time. The crossing to Portsmouth was slow as it was very choppy and we were leaving the Port at 15:30 at which point Melodie was good enough to let me go play with my friends while she drove home.

I did have the "fun" of doing the Portsmouth->London->Cambridge trip on UK public transport but it went pretty smoothly. Walking from Cambridge station to the BBQ location was a bit dumb, next time I am taking a cab!

The BBQ was excellent fun and big thanks for Steve for holding it again. Its always fun to meet the usual suspects. We also got to set a new occupancy record at Steves house Saturday night and discovered that certain members of Debian UK snore rather loudly (I think at one point we could measure it on the Richter scale).

Back home now of course. Work is the same as when I left so no change there and the Boys first day back at school seems to have gone smoothly too.

Sunny Brittany

Alas I did not go to Debconf 10 which looked like everyone had a blast, congratulations to the organisers. Nor will I be able to attend the traditional Steve McIntyre BBQ at the weekend, hope everyone has fun.

On the other hand I have managed to take a family holiday in sunny Brittany...

OK perhaps sunny is pushing it, we did have several nice days last week which we spent on the Le Pouldu plages but this week has been more challenging.

Fortunately the camp site where we are staying has reasonable bandwidth so I can continue to waste time online.

This has given me time to look at some Debian packaging. Specifically the mingw32-runtime packages. Their maintainer seems to be unwilling to allow an updated version to be uploaded despite there being numerous upstream releases since the last packaged release in 2007.

The packaging manual makes it clear that hijacking is not permitted and I discover my desire for having a huge, unhelpful argument about maintaining a package is non existent.

I guess when I have my updated packages available I will maybe announce them but its not the same. I guess this is one of those problems with being a Debian maintainer, we all have to rub along even with decisions we disagree with. Hmm thought I had more to say on the subject ...perhaps next time.

Anyway must go and entertain the kids for an hour or two, maybe go to the beach in the rain, hell they cannot get any soggier ;-)

Programmers are suckers for a meme

Many Open Source projects have and IRC # for developers. The NetSurf project is no different. During a discussion someone jokingly suggested that one contributor should be asked to take the FizzBuzz test. Can you guess what happened next?

Ten minutes later Michael Drake posted this solid example in C which is where it all ought to have ended.

Being programmers, of course this had a predictable result. The original question, the reason for asking it and any the serious point being made in the original article were discarded. Just so everyone (including myself) could play silly buggers over our lunch break. Coders, it would seem, simply like to produce a solution even if it is only for fun.

None of these programs took more than ten minutes (except the JAVA monstrosity), are reproduced with permission and I am to blame for none of them (ok maybe just the one ;-).

First up was Rob Kendrick with the classic solution in C (his day job is as a support team lead which seems to make those programmers who cannot do this seem even more scary bad.)

Next was Daniel Silverstone who turned this Lua solution out very quickly and berated the rest of us for not following the rules ;-)

The final C solution was my own uber silly sieve implementation

Peter Howkins decided the world required a solution in PHP

When pointed out that his solution stopped at 50 he presented this vastly superior and obviously idiomatic solution

Finally after a long time James Shaw caused mental anguish and wailing with this abomination unto Nuggin.

With Luck everyone has now got it out of their system and we will never have to put up with this again (yeah right). And now you also know why Open Source projects sometimes take ages to release ;-)

Ex Phone

I went to the Linux Plumbers Conference (LPC) last year (which was very interesting and productive). While I was there Qualcomm were in attendance giving out penguin mints and running a competition. The lucky winners of this draw were to receive an ADP1 Android phone. So I dutifully filled in the entry form, handed it in and thought no more about it.

In the break after attending a particularly interesting workshop on the last day of the conference several other people congratulated me on winning the Qualcomm draw, this was the first I knew about it! I went in search of the nice people from Qualcomm and sure enough they handed over an ADP1 after getting a couple of photos.

Once I returned home I switched to using ADP1 as my phone and started experimenting with Android kernel stuff. Then one day the Wi-Fi stopped working which while odd could be overcome by repeatedly unloading and reloading the driver until it worked once more. Then one day the USB stopped working, no more ADB, no more console, no more debugging, no more hacking.

Then one day it turned itself off and never came back. I have been forced to return to using a hand me down, very kindly given to me by Daniel Silverstone. Alas it is not a smart phone of any kind and my finances do not allow for me to spend the money to replace it.

I have repeatedly tried to contact Qualcomm open source to see if there is any kind of warranty I might be able to use to get the phone repaired alas all the contact addresses I have are now simply returning SMTP errors.

So this is pretty much a tale without a happy ending, unless anyone out there knows the right people to contact? Perhaps someone at Google maybe? I only revisit the subject at all because of the recent announcement of an Android 2.1 based edition for these devices which reminded me I like to play with these things.