In fact, we started off with two or three different shells and the shell had life of its own.

With apologies to Ken Thompson, this is a list derived from the wafflings residents of a certain UK IRC channel. The conversation happened several years ago but I was reminded of it again today from a conversation on that same channel.

No names have been kept to project the guilty.


/bin/bush - a shell that steals and lies?
/bin/cash - displays adverts before each prompt
/bin/crash - Microsoft shell
/bin/mash - requires libsausage.so
/bin/hush - terse output
/bin/irish - only shell that finds jerkcity funny
/bin/rush - über-optimised for speed, but might not actually work correctly
/bin/flash - proprietary shell that displays adverts and cartoons and hangs periodically
/bin/welsh - whinges about people not using it, but then steals features from other shells in order to actually make sense
/bin/rehash - never actually runs any program, just uses markov chains to construct output from stuff it has seen before.
/bin/wash - will remove lint in your scripts if you let it
/bin/parish - prays for the successful exit of every command
/bin/punish - symlink to /bin/csh
/bin/sheepish - apologises when $? is not zero
/bin/diminsh - decrements all result codes by one
/bin/lavish - and you thought bashisms were bad...
/bin/brainwash - once you've tried it, you'll believe it's the only shell in existence
/bin/hoggish - written in Java
/bin/reversepolish - arguments order different in go must
/bin/ganesh - no subprocess dares exit zero for fear of being removed from the system
/bin/roguish - every day is April 1st as far as it is concerned
/bin/macintosh - it's shiny, has *loads* of things you think you can poke at, yet it only actually responds to a single key on the keyboard and has no useful features for fear of confusing the user
/bin/lush - garbles output from processes like a drunkard
/bin/snobbish - you're not good enough to use it
/bin/thrush - it itches
/bin/vanquish - kills processes mercilessly
/bin/tush - pert
/bin/fetish -> /bin/zsh
/bin/skirmish - multi-user shell
/bin/whiplash - gets invoked if you break too hard
/bin/newsflash - BREAKING NEWS pid 1234 terminated
/bin/mulish - DJB does Shell
/bin/hsilop - arguments order different in go must
/bin/whitewash - government approved
/bin/trish - for tri-state hardware
/bin/jdsh - it sucks...
/bin/flesh - optimised for viewing p0rn

What man's mind can create, man's character can control.

I have a project that required me to programmatically control power to several devices. I have done this before using a Velleman vm8090 board which is relatively easy to control. However they are relatively expensive.

I turned to ebay and found a similar module at a substantially reduced cost. Upon receipt however I discovered that instead of being a simple serial USB interface it presented USB HID and the Debian system I was running it on has loaded the hiddev driver for me but it did not implement any of the standard HID Usage Pages leaving me with no way to control the device.

I did the obligatory

sudo lsusb -d 12bf:ff03 -vvv

Bus 003 Device 019: ID 12bf:ff03  
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               1.10
  bDeviceClass            0 (Defined at Interface level)
  bDeviceSubClass         0 
  bDeviceProtocol         0 
  bMaxPacketSize0         8
  idVendor           0x12bf 
  idProduct          0xff03 
  bcdDevice            1.00
  iManufacturer           1 Matrix Multimedia Ltd.
  iProduct                2 Flowcode USB HID
  iSerial                 0 
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength           41
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0x80
      (Bus Powered)
    MaxPower               50mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           2
      bInterfaceClass         3 Human Interface Device
      bInterfaceSubClass      0 No Subclass
      bInterfaceProtocol      0 None
      iInterface              0 
        HID Device Descriptor:
          bLength                 9
          bDescriptorType        33
          bcdHID               1.10
          bCountryCode            0 Not supported
          bNumDescriptors         1
          bDescriptorType        34 Report
          wDescriptorLength      54
         Report Descriptors: 
           ** UNAVAILABLE **
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0008  1x 8 bytes
        bInterval               5
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x01  EP 1 OUT
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0008  1x 8 bytes
        bInterval               5
Device Status:     0x0980
  (Bus Powered)

This simply showed me what I already knew and surprised me that lsusb did not dump HID report descriptor items. Some searching revealed that teh device had to be unbound so lussb could access the descriptor.

Thus a simple

echo -n 3-1.1.4:1.0 | sudo dd of=/sys/bus/usb/drivers/usbhid/unbind

resulted in lussb dumping the descriptor items:

Item(Global): Usage Page, data= [ 0xa0 0xff ] 65440
                (null)
Item(Local ): Usage, data= [ 0x01 ] 1
                (null)
Item(Main  ): Collection, data= [ 0x01 ] 1
                Application
Item(Local ): Usage, data= [ 0x02 ] 2
                (null)
Item(Main  ): Collection, data= [ 0x00 ] 0
                Physical
Item(Global): Usage Page, data= [ 0xa1 0xff ] 65441
                (null)
Item(Local ): Usage, data= [ 0x03 ] 3
                (null)
Item(Local ): Usage, data= [ 0x04 ] 4
                (null)
Item(Global): Logical Minimum, data= [ 0x00 ] 0
Item(Global): Logical Maximum, data= [ 0xff 0x00 ] 255
Item(Global): Physical Minimum, data= [ 0x00 ] 0
Item(Global): Physical Maximum, data= [ 0xff ] 255
Item(Global): Report Size, data= [ 0x08 ] 8
Item(Global): Report Count, data= [ 0x08 ] 8
Item(Main  ): Input, data= [ 0x02 ] 2
                Data Variable Absolute No_Wrap Linear
                Preferred_State No_Null_Position Non_Volatile Bitfield
Item(Local ): Usage, data= [ 0x05 ] 5
                (null)
Item(Local ): Usage, data= [ 0x06 ] 6
                (null)
Item(Global): Logical Minimum, data= [ 0x00 ] 0
Item(Global): Logical Maximum, data= [ 0xff 0x00 ] 255
Item(Global): Physical Minimum, data= [ 0x00 ] 0
Item(Global): Physical Maximum, data= [ 0xff ] 255
Item(Global): Report Size, data= [ 0x08 ] 8
Item(Global): Report Count, data= [ 0x08 ] 8
Item(Main  ): Output, data= [ 0x02 ] 2
                Data Variable Absolute No_Wrap Linear
                Preferred_State No_Null_Position Non_Volatile Bitfield
Item(Main  ): End Collection, data=none
Item(Main  ): End Collection, data=none

By consulting the device class definitions document I determined the device was using the "Vendor defined" Usage page (0xff00 to 0xffff) so I would definitely have to write a program to control the device.

Linux provides a really easy interface to deal with HID devices called hiddev (gosh, such adventurous naming) which I already had to unbind to get my descriptors decoded so I am fairly sure it works ;-)

The kernel documentation and header for hiddev provide the absolute basic mechanics of the interface but no example code or guidance. The obligatory web search turned up very little and even that had to be retrieved from the internet archive. So It seems I would be forced to work it through myself.

It seems the hiddev interface is orientated around HID devices generating reports which the program is expected to read. Numerous ioctl() are provided so the program can obtain the descriptor information necessary to control and process the received reports.

However in this case we need to be able to send reports to the device, all the descriptor information revealed was that there were eight (Item Report Count = 8) values with eight bits each (Item Report Size = 8) with logical and physical values representing the whole range of the octets.

Fortunately the seller provided a website with some control programs and even source. After some time rummaging through the Visual Basic program I finally found (in FrmMain.vb:2989) that the eight bytes were largely unused and the first was simply a bitmask of the eight relays coil status, set for energised clear for off. With bit 0 controlling relay labelled 1 through to bit 7 for relay 8.

To send a report to a HID device the hiddev interface uses the HIDIOCSREPORT ioctl where the report data is first set using HIDIOCSUSAGE .

The HIDIOCSUSAGE ioctl is passed a hiddev_usage_ref structure which must be initialised with information about the report descriptor identifier (constructed from the Usage Page and Usage as set by the items in the descriptor), the index of the item (named usage) we wish to set in the report (in this case the first which is 0) and the value we actually want to set.

After a great deal of debugging the final program is very short indeed but does the job, my main problem now is that if I switch too many (more than one) relays at once the whole device resets. The scope says the supply rails are behaving very badly when this happens, looks like I need to add a load of capacitance to the power well to stabilise it during the switching events.

Oh and add in the fact Relay 1 LED doest work unless you push on it and I do wonder about the wisdom of the economy in this case. Though yet again Linux makes the software side easy.

Another NetSurf Developer Workshop

The weekend saw some of the NetSurf developers gather once again to do battle against the Red Queen...no, wait. I mean plan and write code.

The successful event was once again kindly hosted by Collabora at their Cambridge office, the facilities remain as good as at the previous event and provided a comfortable environment in which to work.

Four developers managed to attend from around the UK John-Mark Bell, Vincent Sanders,  Daniel Silverstone and Rob Kendrick. The turnout was a little lower than previous events, however Michael Drake managed to assist with some bug fixes remotely.

From Friday to Sunday afternoon we each managed to put in 24 hours of work consisting of almost 90 commits changing over 100 files and removing 10,000 lines of code.

The main theme of the event was consolidation ready for the 3.0 release. While not a glamorous target this is a major release for the project which we want to achieve soon, this requires the completion of numerous ongoing tasks (like finally removing the dependency of libxml) and fixing as many reported bugs as possible.

The Major things we achieved were:

• Merging of the new javascript implementation.
• Removal of libXML from "full save" and "libSVGtiny"
• Address issues accessing some https sites (TLS 1.2 disabled for now)
• Complete integration of UI message translations from single file (fat messages)
• Significantly improve the performance of DOM event dispatch.
• Several important bugs closed.

Once the final usage of libXML is removed (from the hotlist code) NetSurf will become substantially smaller on several platforms as we will not have two separate parser implementations linked.

One big feature of the 3.0 release which was not originally on the roadmap is that of JavaScript. We initially intend to provide two builds of with and without the JavaScript interpreter compiled into NetSurf. This allows those on very small systems to continue to run NetSurf and those users who want a richer web experience to enjoy it. 

Although JavaScript is a major new feature which is frequently requested the initial implementation will be incomplete. To complete the feature several thousand JavaScript to DOM interface binding functions must still be written. In addition it should be noted that we have yet to add dynamic layout or extensions such as Canvas or WebGL

We had a Raspberry Pi which was used to run RISC OS so we could examine the operation on this system. 

The 2.9 edition as shipped works OK but is lacking the substantial progress made in the last six months. 

The CI system build was run on the system and works well so we are reassured the 3.0 edition will provide an even better browsing experience.

The event was a great success overall and we are now very close to our 3.0 release which we hope to produce before Christmas.

I thought San Diego must be Heaven on earth...It seemed to me the best spot for building a city I ever saw.

I think perhaps Alonzo Horton overstated the case but San Diego was a perfectly pleasant place to spend a week. Recently I was fortunate and attended the Linux Plumbers conference (and bits of Kernel Summit) and as on previous occasions there were some interesting sessions and some less helpful diversions.

Melodie went along with me, our first trip away together without the kids in a long time. We did the tourist thing for a few days including the USS Midway aircraft carrier tour (recommended) and the harbour boat tour (also recommended).

The conference was co-located with several other events and there seemed to be a lot of people around not connected with KS or LPC. For me the hallway track was, as usual, much more rewarding and I got to catch up with several old friends and make some new acquaintances. A lot of my colleagues were about attending the Gstreamer and Linuxcon conferences so there was the obligatory work evening event (which was pizza and beer, just done brilliantly).

A track that did stand out for me was the Clang / LLVM presentations. They gave an excellent overview of their progress towards making the Kernel compile with their tools, such innovation appears to be making both projects stronger.

One thing that occurred to me was the blandness of KS and LPC this time, usually there are at least some loud disagreements but I failed to attend even one session where there was more than small differences which were quickly resolved.

Perhaps such conferences, like the majority of people attending them, are becoming middle aged and a little complacent. An observation like that does make me wonder where the next change will come from and what it will look like. Just as open source software (including Linux) has disrupted the proprietary software industry in the last decade what will come along and disrupt us? Or is open source the end of the line and we will just continue to evolve?

Education is when you read the fine print. Experience is what you get if you don't.

Given my title I guess I just got a load of "experience" from Gandi. Previously I have written about building the NetSurf Continuous Integration (CI) server and my misjudgement of the resources it would require to be useful.

I contacted Gandi and they confirmed my suspicion that my only options to extend the VPS were to spend a great deal more money. The total spend would have been in excess of £300 per year to get a minimally satisfactory configuration.

So I contacted Mythic Beasts and they offered the NetSurf project a selection of deals they could not refuse. Once we decided on the right deal for us the system was in place, ready to use, within a morning.

Thanks to the generous open source project discount the offering from Mythic is almost double what Gandi were offering for less than half the price. I was, as previously concluded, dumb to go with Gandi in the first place when Mythic were an option.

Once the system was transferred to the shiny new server I went back to Gandi and asked how much my mistake of buying a years service for a single month worth of VPS service was going to cost. The reply was somewhat disappointing I was simply referred to section 8.1.2 of the contract (tl;dr We have your money now, tough luck, your problem). It would appear my mistake was going to cost me the full £114 I had paid.

I was somewhat disheartened at this turn of events but then Daniel Silverstone offered to take on the VPS if I could transfer it to his Gandi handle so he could administer it. this seemed like a good solution, at least the resource would be used and not wasted. Gandi response to this was initially promising but this is the final reply:

I'm really sorry to inform you that my supervisor refused the share transfer request to the other  account. He told me that we do not transfer products either in those cases. I deeply apologize since I thought it was possible. I thank you for your confidence in  Gandi, and hope that this occurrence does not deter you from continuing with us.

Now I must stress that Gandi are completely within their contractual rights to do this and are not obligated to provide any assistance to help make my blunder less costly to me, but they have firmly shown they have no willingness to be flexible in any way which means I know where I will not be spending my money in future.

Oh and then Pete from Mythic mentioned that they have a full refund and flexible policy on their services and we can always switch to the monthly billing rate and get a refund on the outstanding balance or move it or...

Basically my recommendation for an open source project looking for hosting, VPS or any network services has to be to go talk to the nice people at Mythic and steer clear of Gandi. Maybe in future all providers will realise that they are in a competitive market and certainly a little customer service flexibility might have made this posting much more about my mistake and not the attitude of the suppliers involved.

Pimm's and Punting

The Sun came out in Cambridge the other week (yes I know, a rare occurrence) and because Robert is a generous soul he decided the office should go punting on the cam.

I had not previously been out on the river but Robert proved to be the consummate punter.

We even got fruit based drinks and some serenading, although perhaps that was not a benefit.


I took many photos of the event including the rather nice meal afterwards, some were even in focus and not blurry.

Fowlers Delight?

Fowler published his paper on Continuous Integration (CI) way back in 2000. Personally at the time I was a much more naive developer, I was not really an adherent to the whole XP community and had worked for a series of companies where the idea of a release build was of a month long process and the concept that the whole codebase could be built and tested on every commit was a comical notion at best.

By the time the time the 2006 edition of the paper came out however I was convinced. Ever since then I have tried to make the projects I worked on have as many CI features as possible. It is one reason I ran the ARM Kernel autobuilder for years as an attempt to provide at least build coverage.

I especially like the way these concepts mean that releases are much less "scary" events and release anxiety is reduced to a minimum. When every commit produces something that is of release quality a project really is doing it right.

The reason I mention any of this is I have recently set up the NetSurf projects CI server. The heavy lifting is being done by a Jenkins instance running on a VPS fulfilling the "Every Commit Should Build the Mainline on an Integration Machine" part of the Fowler guidelines.

Setting up Jenkins was much less trouble than many of the previous systems I have used. I have been a little underwhelmed by the support for plain old make based projects which you have to resort to shell scripts for, but leaving that aside the rest was straightforward. The documentation could do with a purge pass for the old hudson name and the error reporting (especially under memory pressure) is not great.

And there lies my one real issue with the tool, memory. The VPS has 512MByte RAM and runs nothing beyond a web server, jenkins and the C compiler to build the project components. One would hope that was plenty of memory, doing the builds by hand it did seem gcc is happy in that space, alas Jenkins is a Java application and guzzles RAM (over 60% of RSS right now) and causes itself out of memory exceptions with distressing frequency.

The project is left with the choices:

• The project is relatively poor (and I already blew the budget) so "living with it" and keeping an eye on it manually is a possibility.
• Extending the VPS with another 512MBytes of RAM (Due to my VPS provider choice I had not anticipated the need so this option would cost almost as much as the original set-up)
• Buying another more suitable VPS from somewhere reputable like Mythic Beasts (why I did not go there in the first place...dumb, sometimes I am just dumb)

It is such a pity too, aside from the RAM issues it is all going well and it has already encouraged us to get the toolchains for the less commonly built targets (ppc amiga and windows for a start) working.