Gource

I have used the Gource tool for a few years now to produce visualisations of project history. The results are pretty but not especially informative and mainly serve to show how well maintained a projects revision control history is.

The results do however provide something pretty to put on projectors and screens at shows when there is nothing better to be displayed.

Recently I noticed the Gource tool got updated and I decided to compile it and give it a try. After the usual building of the dependencies (including all of libboost!) the new version (0.38) gives much better results than the previous edition I had been using (0.27).

I tested it on the NetSurf git repository generating an overview for the whole ten years the project has been running which produced a six minute video which I shall be using on the NetSurf stand at our next show.

Overall if you need a historical visualisation of your projects revision history Gource is a pretty good tool. I have also used the alternative "code swarm" tool in the past but that seems to have bitrotted to death so I cannot recommend it.

I don't drink coffee I take tea my dear, I like my toast done on one side...

Choices, options, selections if you rather. These are what set us all apart from our fellow man, perhaps it is only the appearance of free will and individuality but our world is full of choice.

In software it might be argued programming is nothing more than making thousands perhaps millions of choices which result in a list of instructions for a machine incapable of making decisions of its own.

On a more mundane level sometimes a programmer cannot make a choice suitable for all expected use cases and a user option is created. These options have become something of a cause for arguments within certain sections of the open source software community especially amongst the groups that influence the graphical user experience.

The discussion should be nuanced and varied (perhaps that is my age, or maybe I am more diplomatic than I thought?) but there seems to be little compromise on this discussion which (from an outsiders point of view) splits into two viewpoints:

• On one side of the argument, which I shall label reductionist, is that all options should be removed with the software just doing the correct thing 
• On the other, whom I shall refer to as maximalist, is that users should be presented with options to customise everything.

The reductionist group is currently winning the argument in the popular graphical environments and seem to be removing functionality which requires user choice wherever they can find it. This results in the absurd "joke" that the UI will become a single button, which they are trying to remove.

Personally my view is that an option should be present only when an option cannot be satisfactorily selected by the computer and even then a default suitable for as many users as possible should be picked.

You may ask why I have raised this topic at all? well over the last few days I have been trying to fix the preferences selection for the GTK port of NetSurf. The NetSurf project follows my view on this subject pretty closely but being a browser its very difficult to do the right thing for everyone

NetSurf has numerous frontends for the core browser, I use the term frontend because in some places the toolkit and OS are conflated (windows, cocoa) and not in others (gtk). For each frontend NetSurf is a native application, this is an important distinction, the windows and widgets a user interfaces with are produced by that platforms toolkit.

Old NetSurf Preferences Dialog

This is a deliberate choice unlike other browsers which render their UI themselves as web content, this is a beguiling solution when the authors wish there to be a single browsing "experience" with a common look and feel. However NetSurf looks and feels like a native browser for each frontend, which is what the project decided it wanted to achieve.

Given this the gtk frontend (is it a Gnome application? not sure of the distinction TBH) has had no dedicated maintainer for some time it has suffered from a bad case of bitrot from both GTK library changes and general neglect.

I have slowly been improving this situation, the browser can now be compiled with GTK version 2 (2.12 onwards) or 3, the menus and other UI elements have been translated for the supported languages and now the turn has come for the options dialog.

Oh, right, its "Preferences" not options, fair enough a common semantic throughout all applications does give a degree of homogeneousness but the word choice does seem to indicate the users control has been reduced. At this point I gained an education in just how unfriendly GTK and its tools have become towards the casual programmer.

The dialog I wanted to construct was a pretty standard tabbed layout which reflected the current options and allowed the user to change them in an obvious way. Given that I have constructed and equivalent interface in idiomatic manner for cocoa and windows I thought  this would be straightforward, I was very wrong.

The interface construction tool is called glade which used to be the name of the UI "resource" file and interfaces. However the tool is still called glade but the interface is now GtkBuilder which has a different (but similar) XML based file format. Then we discover that despite it being an extensible file format the UI files are specifically versioned against the GTK library. Also why on earth can these resources not be compiled into my program? OK make them over-ridable perhaps. Generally it is yet another file to distribute and update in step with the executable.

New NetSurf Preferences

So because I want to support back to version 2.12 of GTK I do not get to use any of the features from 2.16 in my UI unless I load a different UI builder file...oh and GTK3? requires a completely different version of glade and the UI files which are incompatible with gtk 2. Once this was worked round with having multiple UI files I then moved on to the next issue.

The GTK model uses function callbacks on events, these are known as signals. Perfectly reasonable but because the ui files are loaded at runtime and not compiled in there must be a way for the GTKBuilder to map the textural signal names to function pointers.

The way the GTK developers have chosen to do this is to search the global function table which means any signal handler function symbol has also to be global which means unnecessary overhead for the ELF loader increasing load times.

This lookup should have been confined to a single object or even placed in an alternative section to avoid these issues, this would not have seemed especially challenging to implement as all callback handlers have to use a preprocessor to define already (G_MODULE_EXPORT).

Another thing that makes developing GTK applications worse is the documentation, this seems to be a continuous complaint about may open source projects, which is taciturn or simply missing in many cases. GTK seems to suffer dreadfully from having multiple API versions all subtly different resulting in a lot of work on the developers part to simply find what they want.

A specific example of this is the signals and under what circumstance they occur  I wanted to update all the widgets with the current configuration state whenever the options window is shown (using gtk_widget_show()) so I figured that would be the "show" signal...right? nope, never did find the right one to use, ended up with "realize" which occurs once when the dialog is created, not what I wanted, but is at least consistent and works.

NetSurf with Preferences Dialog

Overall my impression of developing just one small dialog (60 widgets total exclusing labels and containers) for a GTK program is that the toolkit and its tooling is missing the facilities to make developers life easier when doing the drudge work that a good proportion of graphical program development.

It is not the case that one cannot do things, just that everything has to be done manually instead of having the tools do the work, and because of the documentation that tedium is magnified.

I did eventually reach the stage that the thought of writing the boilerplate to add another check button to enable "do not track" had me thinking "do they really need this or can I avoid the work?" perhaps that is why they are all reductionist?

In fact, we started off with two or three different shells and the shell had life of its own.

With apologies to Ken Thompson, this is a list derived from the wafflings residents of a certain UK IRC channel. The conversation happened several years ago but I was reminded of it again today from a conversation on that same channel.

No names have been kept to project the guilty.


/bin/bush - a shell that steals and lies?
/bin/cash - displays adverts before each prompt
/bin/crash - Microsoft shell
/bin/mash - requires libsausage.so
/bin/hush - terse output
/bin/irish - only shell that finds jerkcity funny
/bin/rush - über-optimised for speed, but might not actually work correctly
/bin/flash - proprietary shell that displays adverts and cartoons and hangs periodically
/bin/welsh - whinges about people not using it, but then steals features from other shells in order to actually make sense
/bin/rehash - never actually runs any program, just uses markov chains to construct output from stuff it has seen before.
/bin/wash - will remove lint in your scripts if you let it
/bin/parish - prays for the successful exit of every command
/bin/punish - symlink to /bin/csh
/bin/sheepish - apologises when $? is not zero
/bin/diminsh - decrements all result codes by one
/bin/lavish - and you thought bashisms were bad...
/bin/brainwash - once you've tried it, you'll believe it's the only shell in existence
/bin/hoggish - written in Java
/bin/reversepolish - arguments order different in go must
/bin/ganesh - no subprocess dares exit zero for fear of being removed from the system
/bin/roguish - every day is April 1st as far as it is concerned
/bin/macintosh - it's shiny, has *loads* of things you think you can poke at, yet it only actually responds to a single key on the keyboard and has no useful features for fear of confusing the user
/bin/lush - garbles output from processes like a drunkard
/bin/snobbish - you're not good enough to use it
/bin/thrush - it itches
/bin/vanquish - kills processes mercilessly
/bin/tush - pert
/bin/fetish -> /bin/zsh
/bin/skirmish - multi-user shell
/bin/whiplash - gets invoked if you break too hard
/bin/newsflash - BREAKING NEWS pid 1234 terminated
/bin/mulish - DJB does Shell
/bin/hsilop - arguments order different in go must
/bin/whitewash - government approved
/bin/trish - for tri-state hardware
/bin/jdsh - it sucks...
/bin/flesh - optimised for viewing p0rn

What man's mind can create, man's character can control.

I have a project that required me to programmatically control power to several devices. I have done this before using a Velleman vm8090 board which is relatively easy to control. However they are relatively expensive.

I turned to ebay and found a similar module at a substantially reduced cost. Upon receipt however I discovered that instead of being a simple serial USB interface it presented USB HID and the Debian system I was running it on has loaded the hiddev driver for me but it did not implement any of the standard HID Usage Pages leaving me with no way to control the device.

I did the obligatory

sudo lsusb -d 12bf:ff03 -vvv

Bus 003 Device 019: ID 12bf:ff03  
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               1.10
  bDeviceClass            0 (Defined at Interface level)
  bDeviceSubClass         0 
  bDeviceProtocol         0 
  bMaxPacketSize0         8
  idVendor           0x12bf 
  idProduct          0xff03 
  bcdDevice            1.00
  iManufacturer           1 Matrix Multimedia Ltd.
  iProduct                2 Flowcode USB HID
  iSerial                 0 
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength           41
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0x80
      (Bus Powered)
    MaxPower               50mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           2
      bInterfaceClass         3 Human Interface Device
      bInterfaceSubClass      0 No Subclass
      bInterfaceProtocol      0 None
      iInterface              0 
        HID Device Descriptor:
          bLength                 9
          bDescriptorType        33
          bcdHID               1.10
          bCountryCode            0 Not supported
          bNumDescriptors         1
          bDescriptorType        34 Report
          wDescriptorLength      54
         Report Descriptors: 
           ** UNAVAILABLE **
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0008  1x 8 bytes
        bInterval               5
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x01  EP 1 OUT
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0008  1x 8 bytes
        bInterval               5
Device Status:     0x0980
  (Bus Powered)

This simply showed me what I already knew and surprised me that lsusb did not dump HID report descriptor items. Some searching revealed that teh device had to be unbound so lussb could access the descriptor.

Thus a simple

echo -n 3-1.1.4:1.0 | sudo dd of=/sys/bus/usb/drivers/usbhid/unbind

resulted in lussb dumping the descriptor items:

Item(Global): Usage Page, data= [ 0xa0 0xff ] 65440
                (null)
Item(Local ): Usage, data= [ 0x01 ] 1
                (null)
Item(Main  ): Collection, data= [ 0x01 ] 1
                Application
Item(Local ): Usage, data= [ 0x02 ] 2
                (null)
Item(Main  ): Collection, data= [ 0x00 ] 0
                Physical
Item(Global): Usage Page, data= [ 0xa1 0xff ] 65441
                (null)
Item(Local ): Usage, data= [ 0x03 ] 3
                (null)
Item(Local ): Usage, data= [ 0x04 ] 4
                (null)
Item(Global): Logical Minimum, data= [ 0x00 ] 0
Item(Global): Logical Maximum, data= [ 0xff 0x00 ] 255
Item(Global): Physical Minimum, data= [ 0x00 ] 0
Item(Global): Physical Maximum, data= [ 0xff ] 255
Item(Global): Report Size, data= [ 0x08 ] 8
Item(Global): Report Count, data= [ 0x08 ] 8
Item(Main  ): Input, data= [ 0x02 ] 2
                Data Variable Absolute No_Wrap Linear
                Preferred_State No_Null_Position Non_Volatile Bitfield
Item(Local ): Usage, data= [ 0x05 ] 5
                (null)
Item(Local ): Usage, data= [ 0x06 ] 6
                (null)
Item(Global): Logical Minimum, data= [ 0x00 ] 0
Item(Global): Logical Maximum, data= [ 0xff 0x00 ] 255
Item(Global): Physical Minimum, data= [ 0x00 ] 0
Item(Global): Physical Maximum, data= [ 0xff ] 255
Item(Global): Report Size, data= [ 0x08 ] 8
Item(Global): Report Count, data= [ 0x08 ] 8
Item(Main  ): Output, data= [ 0x02 ] 2
                Data Variable Absolute No_Wrap Linear
                Preferred_State No_Null_Position Non_Volatile Bitfield
Item(Main  ): End Collection, data=none
Item(Main  ): End Collection, data=none

By consulting the device class definitions document I determined the device was using the "Vendor defined" Usage page (0xff00 to 0xffff) so I would definitely have to write a program to control the device.

Linux provides a really easy interface to deal with HID devices called hiddev (gosh, such adventurous naming) which I already had to unbind to get my descriptors decoded so I am fairly sure it works ;-)

The kernel documentation and header for hiddev provide the absolute basic mechanics of the interface but no example code or guidance. The obligatory web search turned up very little and even that had to be retrieved from the internet archive. So It seems I would be forced to work it through myself.

It seems the hiddev interface is orientated around HID devices generating reports which the program is expected to read. Numerous ioctl() are provided so the program can obtain the descriptor information necessary to control and process the received reports.

However in this case we need to be able to send reports to the device, all the descriptor information revealed was that there were eight (Item Report Count = 8) values with eight bits each (Item Report Size = 8) with logical and physical values representing the whole range of the octets.

Fortunately the seller provided a website with some control programs and even source. After some time rummaging through the Visual Basic program I finally found (in FrmMain.vb:2989) that the eight bytes were largely unused and the first was simply a bitmask of the eight relays coil status, set for energised clear for off. With bit 0 controlling relay labelled 1 through to bit 7 for relay 8.

To send a report to a HID device the hiddev interface uses the HIDIOCSREPORT ioctl where the report data is first set using HIDIOCSUSAGE .

The HIDIOCSUSAGE ioctl is passed a hiddev_usage_ref structure which must be initialised with information about the report descriptor identifier (constructed from the Usage Page and Usage as set by the items in the descriptor), the index of the item (named usage) we wish to set in the report (in this case the first which is 0) and the value we actually want to set.

After a great deal of debugging the final program is very short indeed but does the job, my main problem now is that if I switch too many (more than one) relays at once the whole device resets. The scope says the supply rails are behaving very badly when this happens, looks like I need to add a load of capacitance to the power well to stabilise it during the switching events.

Oh and add in the fact Relay 1 LED doest work unless you push on it and I do wonder about the wisdom of the economy in this case. Though yet again Linux makes the software side easy.

Another NetSurf Developer Workshop

The weekend saw some of the NetSurf developers gather once again to do battle against the Red Queen...no, wait. I mean plan and write code.

The successful event was once again kindly hosted by Collabora at their Cambridge office, the facilities remain as good as at the previous event and provided a comfortable environment in which to work.

Four developers managed to attend from around the UK John-Mark Bell, Vincent Sanders,  Daniel Silverstone and Rob Kendrick. The turnout was a little lower than previous events, however Michael Drake managed to assist with some bug fixes remotely.

From Friday to Sunday afternoon we each managed to put in 24 hours of work consisting of almost 90 commits changing over 100 files and removing 10,000 lines of code.

The main theme of the event was consolidation ready for the 3.0 release. While not a glamorous target this is a major release for the project which we want to achieve soon, this requires the completion of numerous ongoing tasks (like finally removing the dependency of libxml) and fixing as many reported bugs as possible.

The Major things we achieved were:

• Merging of the new javascript implementation.
• Removal of libXML from "full save" and "libSVGtiny"
• Address issues accessing some https sites (TLS 1.2 disabled for now)
• Complete integration of UI message translations from single file (fat messages)
• Significantly improve the performance of DOM event dispatch.
• Several important bugs closed.

Once the final usage of libXML is removed (from the hotlist code) NetSurf will become substantially smaller on several platforms as we will not have two separate parser implementations linked.

One big feature of the 3.0 release which was not originally on the roadmap is that of JavaScript. We initially intend to provide two builds of with and without the JavaScript interpreter compiled into NetSurf. This allows those on very small systems to continue to run NetSurf and those users who want a richer web experience to enjoy it. 

Although JavaScript is a major new feature which is frequently requested the initial implementation will be incomplete. To complete the feature several thousand JavaScript to DOM interface binding functions must still be written. In addition it should be noted that we have yet to add dynamic layout or extensions such as Canvas or WebGL

We had a Raspberry Pi which was used to run RISC OS so we could examine the operation on this system. 

The 2.9 edition as shipped works OK but is lacking the substantial progress made in the last six months. 

The CI system build was run on the system and works well so we are reassured the 3.0 edition will provide an even better browsing experience.

The event was a great success overall and we are now very close to our 3.0 release which we hope to produce before Christmas.

I thought San Diego must be Heaven on earth...It seemed to me the best spot for building a city I ever saw.

I think perhaps Alonzo Horton overstated the case but San Diego was a perfectly pleasant place to spend a week. Recently I was fortunate and attended the Linux Plumbers conference (and bits of Kernel Summit) and as on previous occasions there were some interesting sessions and some less helpful diversions.

Melodie went along with me, our first trip away together without the kids in a long time. We did the tourist thing for a few days including the USS Midway aircraft carrier tour (recommended) and the harbour boat tour (also recommended).

The conference was co-located with several other events and there seemed to be a lot of people around not connected with KS or LPC. For me the hallway track was, as usual, much more rewarding and I got to catch up with several old friends and make some new acquaintances. A lot of my colleagues were about attending the Gstreamer and Linuxcon conferences so there was the obligatory work evening event (which was pizza and beer, just done brilliantly).

A track that did stand out for me was the Clang / LLVM presentations. They gave an excellent overview of their progress towards making the Kernel compile with their tools, such innovation appears to be making both projects stronger.

One thing that occurred to me was the blandness of KS and LPC this time, usually there are at least some loud disagreements but I failed to attend even one session where there was more than small differences which were quickly resolved.

Perhaps such conferences, like the majority of people attending them, are becoming middle aged and a little complacent. An observation like that does make me wonder where the next change will come from and what it will look like. Just as open source software (including Linux) has disrupted the proprietary software industry in the last decade what will come along and disrupt us? Or is open source the end of the line and we will just continue to evolve?

Education is when you read the fine print. Experience is what you get if you don't.

Given my title I guess I just got a load of "experience" from Gandi. Previously I have written about building the NetSurf Continuous Integration (CI) server and my misjudgement of the resources it would require to be useful.

I contacted Gandi and they confirmed my suspicion that my only options to extend the VPS were to spend a great deal more money. The total spend would have been in excess of £300 per year to get a minimally satisfactory configuration.

So I contacted Mythic Beasts and they offered the NetSurf project a selection of deals they could not refuse. Once we decided on the right deal for us the system was in place, ready to use, within a morning.

Thanks to the generous open source project discount the offering from Mythic is almost double what Gandi were offering for less than half the price. I was, as previously concluded, dumb to go with Gandi in the first place when Mythic were an option.

Once the system was transferred to the shiny new server I went back to Gandi and asked how much my mistake of buying a years service for a single month worth of VPS service was going to cost. The reply was somewhat disappointing I was simply referred to section 8.1.2 of the contract (tl;dr We have your money now, tough luck, your problem). It would appear my mistake was going to cost me the full £114 I had paid.

I was somewhat disheartened at this turn of events but then Daniel Silverstone offered to take on the VPS if I could transfer it to his Gandi handle so he could administer it. this seemed like a good solution, at least the resource would be used and not wasted. Gandi response to this was initially promising but this is the final reply:

I'm really sorry to inform you that my supervisor refused the share transfer request to the other  account. He told me that we do not transfer products either in those cases. I deeply apologize since I thought it was possible. I thank you for your confidence in  Gandi, and hope that this occurrence does not deter you from continuing with us.

Now I must stress that Gandi are completely within their contractual rights to do this and are not obligated to provide any assistance to help make my blunder less costly to me, but they have firmly shown they have no willingness to be flexible in any way which means I know where I will not be spending my money in future.

Oh and then Pete from Mythic mentioned that they have a full refund and flexible policy on their services and we can always switch to the monthly billing rate and get a refund on the outstanding balance or move it or...

Basically my recommendation for an open source project looking for hosting, VPS or any network services has to be to go talk to the nice people at Mythic and steer clear of Gandi. Maybe in future all providers will realise that they are in a competitive market and certainly a little customer service flexibility might have made this posting much more about my mistake and not the attitude of the suppliers involved.