What is it with embedded computing?

Last week I was given a development board for a system on chip (SoC) we were looking at using. All i had to do was ensure I could get a reasonable, supportable, bootloader and Linux kernel running on it.

Now I am a fairly competent engineer, I have worked with the ARM processors and SoC for many years. It should not therefor take me four days of almost continuous frustration to get a bootloader and kernel onto a system, but it has.

Firstly the system arrives and we look it over, the hardware design is not exactly elegant but seems like it will get the job done. That is at least a step up from several previous offerings which have been just plain broken. So I power it up and it boots into Fedora core 8, possibly not my first OS choice for a diskless system, but it does work.

Perhaps its worth mentioning that my experience with doing any Open Source development has caused me to have a severe aversion to software which is not upstream, especially kernels. The effort required to run your own patch series and keep it up to date is nothing short of Herculean.

Let us be honest, how many developers do you know who are talented enough to generate good code and patient enough to separately maintain it over a long period of time? I shall not belabor this point as others have repeatedly made it for me in much more eloquent ways. Lets us just say from my point of view. Upstream - good. Long term forks - bad.

With this in mind, the solutions Simtec develop are always with a view to our customers being able to simply download and use the standard upstream software wherever possible. Here is where the story stops being smooth and becomes a rant.

The system was running 2.6.22, one that was patched all which ways. "maybe they just shipped an old kernel" I thought. I opened the CD they provided and...oh, OK its just zips and tarballs of the binaries and source they used (actually that in itself is a minor miracle with a lot of vendors right now - see the almost continuous stream of copyright infringement litigation from the FSF) but sill stuck back in July 2007.

So i did teh normal thing, entered the URL printed on the outside of the box and went looking for the updates. No dice, just the same old stuff. So I contacted the company and recived the news that no there were no updates, that was what was published for the device, thats what I could have.

My initial reaction was of tired resignation...this would not be my first SoC port (probably around the fifth or sixth!) So I responded with cd linux-2.6;git checkout master;git pull and went to see what I could see. and lo and behold, there in the kernel, was full support for not only the SoC but for the other version of the very board I had.

Fortunately after mentioning this on IRC my giraffe loving friend. pointed me at the correct community git tree to find the tiny patch required to support my board.

So the vendor was not only shipping old code but actively dissuading customers from even looking for new stuff, which it turns out was mainline anyway! To add to this somewhere along the lines the company name on the box and contact information has changed and...well confusion reigned.

Right Plain sailing from here on? not quite...that would have been too easy. So you can build a mainline kernel...but you cannot boot it. you guessed it the bootloader they ship on the board is ancient, specifically a u-boot fork from December 2005. Having finally figured out how this was going to go, I went to the uboot repo, pulled the latest and built it for my target. And then I needed to upgrade the u-boot and joy of joys bricked the system.

Fortunately the system has a integrated FDTI serial/JTAG port so re-writing the flash should be as simple as running openOCD configured correctly. Of course this did not work...in the end I resorted to running their binary built versions from the CD (tarfile inside a zip inside a tarfile...whatever) and wrote their original image back to the board...which worked.

Long story short, next day I discover that i need a magic different uboot target (make u-boot.kwb for those that care) write this with the binary openocd build and...success
From then on its all been a bit anticlimactic and straightforward and I now have a system capable of starting a kernel over the network and a root fs on network, disc or flash.

My main complaint from all of this is what the hell are vendors doing shipping crap like this? Why do they all insist on taking a dreadful code drop from the SoC manufacturer when they first brought the part up and shipping that prototype junk until the part reaches the end of its life?!?

All the usual excuses about support etc. will come out, but seriously, you recive *no* support from vendors unless you pay for it and even then they will simply tell you whats on the CD is "it". Btw the diff for u-boot to support this board is larger than the original sources and the 2.6.22 is attacked with a 65MB diff which cannot be good.

The Open Source option is continuously innovating and improving, producing ever better software...but the embedded world seems to be forever stuck shipping crap. Is this just me? Does anyone have any solutions? Or am I stuck with this until I change profession?