Making BeagleBone Black serial number available to user applications

The BeagleBone Black is a handy little board. It has a 1GHz ARM processor and 4GB of on-board Flash storage, and runs Linux very easily. In particular, the Debian distribution is available ready-made to just copy on to the board from an SD card. I’ve been using them in various embedded applications recently.

IMG_20160812_162532

One of the applications I’m working on at the moment will be deploying BeagleBone Black boards in hundreds of locations, and we have to manage them sensibly. When managing all these boards, it will make our lives much easier to know their identities. Happily, the BeagleBone Black is fitted with an EEPROM which contains various useful information about it, including its serial number and a copy of the barcode stuck to the board.

The EEPROM is accessible via the I2C bus. The bus itself is accessible via /dev/i2c-0 at address 0x50. As standard, Debian Linux is set up so that the group i2c has access to this device. However, the kernel device tree is one step ahead of us. A special at24 driver takes over access to the EEPROM, so any attempt to access it directly via /dev/i2c-0 just gets denied with ‘Device or resource busy’. The contents of the EEPROM are available, however, in sysfs under /sys/bus/i2c/0-0050/eeprom.

There is also a bone-capemgr driver which presents some of the EEPROM contents in a more convenient way, as a series of files elsewhere in sysfs, under /sys/devices/bone_capemgr.9/baseboard/

This is all very handy, but all the files in sysfs are only readable by root. That’s not helpful for my applications. I don’t want to be running things as root if I can possibly help it. I wanted to find a reasonably legitimate way to make the relevant sysfs files available to a group of users. We can’t just change the permissions on those files because sysfs is created dynamically, so any changes would be lost after a reboot.

The obvious way to do this would be via a udev rule so that the permissions are set up when the relevant devices are found. However, udev rules don’t seem to have a direct way to change the permissions and ownership of files in sysfs. After some experimentation I came up with a couple of rules which did what I wanted. They match on the narrowest set of keys I could work out, and run chmod and chown to set the permissions the way I’d like them:

DRIVER=="bone-capemgr", RUN+="/bin/chown root:i2c /sys$env{DEVPATH}/baseboard/serial-number"
SUBSYSTEM=="i2c", DEVPATH=="*0-0050", RUN+="/bin/chown root:i2c /sys$env{DEVPATH}/eeprom", RUN+="/bin/chmod 0640 /sys$env{DEVPATH}/eeprom"

Put those lines in a suitable rules file. I used /etc/udev/rules.d/30-bone-capemgr.rules. Then, running

udevadm trigger

Should trigger the rules and set the permissions. Because sysfs is recreated at boot time when the various devices are found and added, the rules will trigger each time and leave the permissions the way we want them.

The board serial number is to be found in bytes 16-28 of the EEPROM, and the barcode is in bytes 80-103. The hexdump utility is handy for extracting the data. For example,

hexdump -e '8/1 "%c"' /sys/bus/i2c/devices/0-0050/eeprom -s 80 -n 23

should show the barcode. Wrap that up in a script and we have a programmatic way of finding the board’s identity.

Super Breakout Saga

In the beginning, or at least in 1972, there was Pong. It was one of the first video games to become widely popular, and made Atari’s name in the games industry. It wouldn’t be overstating the case to call it seminal, even though the game itself involved only bouncing a ball between two bats. Four years later, Pong begat Breakout, which extended the bat-and-ball mechanics of Pong to knocking bricks out of a wall. Then came Super Breakout, a similar game, but more complex, with three different modes of play and even multiple bats and balls. Adding the new features needed a new, powerful ingredient: Super Breakout was one of the earliest arcade games to feature a microprocessor. This was in 1977, before Space Invaders, before Pac-Man, before Donkey Kong.

super-breakout-cabinet

I picked up this PCB several years ago on eBay. The price was keen because, of course, it was ‘untested’. We all know what that means. Broken. That’s the way I like them. It’s hard to get to know a game board that’s in perfect working order.

IMG_20160807_180709

IMG_20160807_180717

Getting this one running was more than the usual challenge. From about 1980 onwards, a few conventions about how arcade games were built and wired got established. This game pre-dates those, so the usual rules don’t apply. Power? Instead of 5V and 12V DC supplies, it wants 10V and 22V AC, centre-tapped. Video? Instead of the usual colour RGB with separate sync, it delivers black-and-white composite video, and expects a screen with strips of coloured film stuck to it. Controls? No joystick here. Not even a trackball. Just an analogue potentiometer to control the paddle, and various flashing lights and switches.

My aim with every arcade game I acquire is to adapt it to the JAMMA standard, so it’s easy to plug it in and play it. This one was going need more than just a bit of wiring.

The first job was to get the board running. I rigged up temporary power supplies and a black-and-white monitor which could handle its video output.

The most serious fault was faulty RAM chips. Super Breakout has eight RAM chips of a whole one kilobit each, totalling a spectacular 1 Kbyte of RAM. Six of them were faulty. Fortunately a kind member of the most excellent UKVAC forum had some available. Identifying the faults was made easier by the board’s test mode, which makes beep-boop-boop noises to indicate which chips it thinks are faulty.

IMG_20160807_180733

The sockets holding the program ROMs and the processor were all rubbish and had to be replaced with modern ones. Cleaning them never worked reliably. This particular version of the game uses 12 ROM chips, each holding 4 kilobits. That’s a generous 6 Kbyte of program code, but it’s also 216 pins in sockets to go wrong.

IMG_20160807_180742

There was one faulty logic gate ( a 7420) associated with the ‘game select’ and ‘serve’ switches, and the DIP switches for choosing the game options needed a squirt of DeoxIT switch cleaner. Success!

IMG_20160807_220827

With those problems attended to the board passed all its self tests and seemed to run quite happily. Now to design that JAMMA adapter.

Tektronix 549 Storage Oscilloscope, Restored

A couple of years ago, I restored this exquisite brute of an oscilloscope to working order. You can read the story starting with part 1. I never posted any pictures of the finished article, so here they are.

You can read more about the machinery at the TekWiki.

549_34a

549_front_off

549_34c

549_34b

549_rear

1a4

manuals

Switched on and working, with all four traces showing.

549_front_light

549_front_dark

I made a short movie of the storage functions working.

Cracking a password-protected PDF file

Suppose you asked an insurance company for a letter. The insurance company kindly sent it as a PDF attached to an email. Sensibly, they protected that PDF with a password which they told you over the phone. You wrote it in a notebook and then left the notebook at work over the weekend.

pdf-password

How could you read the letter in the password-protected file at home, then? Remembering that the password was definitely an English word, and all in lower case, a dictionary attack has got to be worth a try.

Linux provides some handy tools for this. There’s a list of English words in /usr/share/dict/words, and a suite of PDF tools which can attempt to open the file using a password, indicating success or failure. A few minutes with Python and:

#!/usr/bin/python
import os,sys

wf=open('/usr/share/dict/words','r')
while True:
  word = wf.readline().strip().lower()
  if word == '':
    print "No solution found"
    break
  print word
  cmdline = 'pdftotext -upw "'+word+'" '+sys.argv[1]
  result = os.system(cmdline)
  if result == 0:
    break

The same thing must be possible in a more hipsterly fashion using awk, but I couldn’t be bothered to figure out a sufficiently baroque command line.

By the way, the password was ‘orange’. Don’t tell anybody.

Reliable I2C with a Raspberry Pi and Arduino

There are many ways of connecting sensors and devices to a Raspberry Pi. One of the most popular is the I2C bus. It’s great for devices which don’t need to transfer too much data, like simple sensors and motor controllers, and it’s handy because lots of devices (up to 127, or even more) can be connected to the same pair of wires, which makes life really simple for the experimenter. I’ve mentioned using the I2C bus in another blog post, because sometimes a bit of software fiddling is needed to get it to work.

Recently I’ve been working on a project involving various devices connected to a Raspberry Pi. Some of them use I2C. The project is based around a breakout board I designed for the Multidisciplinary Design Project at Cambridge University Department of Engineering, in which students collaborate in teams to put together a robot. The breakout board is shown next to the Raspberry Pi in the photo below.

IMG_1381

It fits on top of the Pi, and has lots of useful features including a student-proof power supply, real time clock, accelerometer, space for a Zigbee module, analogue inputs, diagnostic LEDs and four motor driving outputs, all wired to convenient connectors.

The analogue inputs and motor outputs are implemented by a PIC microcontroller connected to the I2C bus. The software for the PIC was written by an undergraduate several years ago. It works well, but seems to have some odd habits. I found that it would apparently work, but sometimes an attempt to read data from the PIC would just fail, or return wrong data, and sometimes data would get written to the wrong register. At first I suspected a wiring problem, but examining the SDA and SCL signals with a scope showed nothing wrong. I tested another device on the same bus – a Philips PCF8575 I/O expander – and it worked perfectly every time. That narrowed the problem down to the PIC. Since there was nothing I could do about the PIC’s software, I had to find a workaround.

I spent some time experimenting with where the communications seemed to go wrong. Reading from an I2C device usually involves two separate operations on the bus. The first one tells the I2C device which register address we want to read, and the second does the actual read. The diagram below shows the sequence. The ‘control byte’ in each case sends the address of the I2C device (0x30 in this case) plus a bit indicating read or write.

smbus-transaction

I found a pattern in the failures. From time to time, the write operation which sets the register address would fail, reporting ‘I/O error’. After that, reading the data would return the wrong value. I modified my code so that if the write operation failed, it would retry a couple of times before giving up. It turned out that retrying was always successful, if not on the first attempt then on the second. However, the data read would still return the wrong value. The value returned was always the address of the register I wanted! It seemed as if something was getting stuck somewhere in the I2C system. Whether it was in the Linux drivers, or the PIC software, I don’t know, and I didn’t spend long enough to find out. My assumption is that the PIC software is sometimes just too busy to respond to the I2C operations correctly.

I tried the retry strategy again, and it turned out that the second attempt to read the data byte always got the right value. The algorithm to read reliably looks like this, in pseudo-code:

  if (write_register_address() fails)
    retry up to 3 times;

  read_data();
  if (we had to retry writing register address)
    read_data();

In practice I was using the Linux I2C dev interface to implement this. Yes, it’s a bit of a nasty hacky workaround, but it did get the communications working reliably.

There was another device I wanted to talk to: an Arduino Mini running a very simple sketch to return some sensor data. This also used the I2C bus. There are handy tutorials about how to get an Arduino to behave as an I2C slave device, like this one. The I2C interface is implemented nicely by the Wire library. Implementing a slave involves responding to two events: onReceive and onRequest.

The onReceive event is called when data, like the register address, is written to the slave, and the onRequest event is called when the master wants to read data. My initial code looked like this:

Wire.begin(I2C_ADDRESS)
Wire.onReceive(receiveEvent)
Wire.onRequest(requestEvent)

void receiveEvent(int bytes) {
  registerNumber = Wire.read();
}
void requestEvent() {
  Wire.write(registers[registerNumber];
}

This worked most of the time, but after a few thousand transactions, it would appear to ‘lock up’ and ignore any attempt to change registers – it would always return the same register, and in fact no more onReceive events were ever generated. Of course, it turned out to be my fault. When reading data in the onReceive event code, it turns out to be important to make sure that data is actually available, like this:

void receiveEvent(int bytes) {
  while(Wire.available())
    registerNumber = Wire.read();
}

That solved the problem. It’s annoying that reading non-existent data can lock up the whole I2C interface, so watch out for this one if you’re using an Arduino as an I2C slave.

Systemd for Embedded Linux

Over the last few years, there has been a lot of controversy in the Linux world about systemd. As I understand it, systemd is intended to be a better-engineered, more powerful version of the motley collection of little programs and scripts which keeps the essential services on a Linux system running.

systemctl

The controversy arises because the original 1970s Unix way of doing things was to rely on a motley collection of little programs and scripts for everything, each of which was simple but well understood, and to knit them together to form a complete operating system. Systemd takes a different approach, using larger and more sophisticated components which are more dedicated to particular tasks, such as managing services or network connections. This is supposed to make it more efficient and easier to manage in the twenty-first century.

I’ve been doing some work recently on an embedded Linux system which runs on the latest version of Debian Linux, version 8 (‘Jessie’). Debian Jessie fully supports systemd to the extent that it seems to be the default way of doing things. I thought I’d experiment with it a bit.

When working on an embedded Linux system, I very frequently want to have a piece of my software run reliably at startup, get restarted if it fails, and be able to output logging information to an easily-managed place. In this case, my software provides a D-Bus interface to a piece of industrial electronics.

In the past I’ve relied on copying and pasting scripts from other pieces of software, and managing log files has always been a bit of a mess. It’s hard to do these things right, so re-inventing the wheel is too risky, which means that the best strategy is to copy somebody else’s scripts. I have never counted the hours of my time which have been wasted by dealing with awkward corner cases and peculiar bugs due to recycled scripts behaving in ways I hadn’t anticipated.

What does it look like with systemd? There are some helpful tutorials out there, including this one from Alexander Patrakov, so it didn’t take me too long to put together a service file which looks like this:

[Unit]
Description=My D-Bus Gateway
[Service]
Type=dbus
BusName=com.martin-jones.gateway
ExecStart=/usr/bin/my_dbus_gateway
Restart=always
[Install]
WantedBy=multi-user.target

I’ve changed the names to protect the innocent, but the contents of the file are pretty self-explanatory. The [Unit] section just includes a description which is readable to a human being. The [Service] section describes the service itself. In this case it’s of type  dbus, which means that systemd will check that the service name (com.martin-jones.gateway in this case) gets correctly published on to D-Bus. The Restart=always setting means that my software gets restarted if it exits. The [Install] section just indicates that this service should run when the system comes up in multi-user mode (like the old runlevel 5).

Having created this file, I simply copied it into /etc/systemd/system/my_dbus_gateway.service and, lo and behold, my new service worked. It was immediately possible to manage the service using commands like

systemctl start my_dbus_gateway.service
systemctl stop my_dbus_gateway.service
systemctl status my_dbus_gateway.service

Great! That’s exactly what I wanted.

Now for logging. I’d heard that systemd would log the stdout and stderr outputs of services into its own journal, and forward that to syslog as required. It does, but there’s a subtlety. Output from stderr appears in /var/log/syslog immediately line-by-line, but output from stdout gets aggressively buffered. This means that it gives the appearance of not working at all unless you explicitly flush the stdout buffer in your code using something like

fflush(stdout)

That’s the only wrinkle I came across, though.

In summary, using systemd’s facilities has made my life as an embedded Linux developer much, much easier and hopefully more reliable. That’s a good thing. My top tips for getting your software working under systemd are these:

  • Create your .service file using the recipe above and the documentation
  • Don’t forget to flush stdout if you want to see it in syslog.

Lenovo Thinkpad T61 GPU fix. Or not.

My mobile workhorse is a trusty Lenovo T61, a close cousin of the ones they use on the International Space Station. How cool is that? It’s built like a tank, and weighs about as much, but the feature I appreciate most is its screen: a lovely 1680 x 1050 resolution, which is actually enough pixels to get some work done. Most laptops have rubbish screens which were clearly only designed for watching DVDs and reading Facebook. It’s a pet hate of mine.

IMG_1276

Sadly the Thinkpad has blotted its copybook. The Nvidia graphics chip has a reputation for expiring earlier than it should do, and this one has. It started with the screen going blank a couple of months ago. I replaced the screen, and then it worked but only with every other column of pixels. Then it spontaneously started working almost-properly, but with a peculiar green shimmering effect on bright colours which was just about possible to avoid by fiddling with the display settings. Then it died altogether, and just gave the ominous beep-bip-bip code at startup which means ‘my graphics hardware isn’t working’.

In a last ditch attempt to revive it, I thought I’d try reflowing the solder on the graphics chip. It helps on some laptops. In theory you need very specialised equipment for this, but I’ve had success in my professional life doing it with a more, shall we say, agricultural approach. With nothing to lose, I had a go. Here’s what I did.

Reflowing the solder on the graphics chip involves removing the motherboard, which requires completely disassembling the laptop.

Remove all the bits the come out: the battery, DVD drive, hard drive, and any PC card and SD card.

IMG_1237

Remove the marked screws  on the bottom to remove the keyboard, touch pad and palm rest.

IMG_1235

Unplug the three wires to the Wifi module. Plug 1 is grey, 3 is white, 2 is black.

IMG_1238

Remove the keyboard surround/speaker grills. Two short screws on the top, one long one from the bottom rear right corner, one long one from the outer rear bottom left corner, four little flat ones from the metalwork near the CPU.

Unplug the screen connector, the grey cloth one near the fan.

IMG_1242

Remove the two tiny screws holding the left speaker and move it to one side.

IMG_1243

Remove 2 medium-length screws from the bottom rear edge and two short screws from the screen brackets on the top side. Take the screen off.

IMG_1245

Remove one tiny silver screw from the front right hand side, 9 short screws from the bottom, three more long ones from the bottom, and four short ones with big flat heads near the docking connector.

From the top, remove the four medium-length screws holding the heatsink down.

IMG_1249

Remove the two short silver screws holding the two silver brackets on the right side of the heat sink.

IMG_1251

Unplug the fan connector and ease the heatsink/fan assembly free.

 

Img_1252

Unplug the multiway connector in the rear centre which feeds the USB sub-board. Tease the wire free from under its sellotape.IMG_1253

Remove the two medium screws holding the wifi module in and remove the wifi module.

IMG_1255.JPG

The motherboard and frame should now be free of the bottom case.

IMG_1256.JPG

Turn the motherboard over. Remove the 8 remaining screws with big flat heads. Remove the medium screw by the DVD connector. Remove the small silver screw by the SD socket. Remove the short screw holding the heatsink brace. Remove the heatsink brace.

Returning to the top of the motherboard, unplug the charger connector, the small connector by the phone socket, the speaker connector by the RAM and the little black connector at the back where the wifi module was. Leave the backup battery connected.

The magnesium frame should now be free of the motherboard. Take it off. Clean the heatsink compound from the tops of the large chips. U47, with the Nvidia logo on it, is the GPU.

To try and reflow it, I made a makeshift heatshield out of a doubled-up piece of kitchen foil. I’ve done some emergency BGA repairs in the past so I had some idea of what I was aiming at. My strategy is to heat the whole chip area while carefully poking some nearby easy-to-repair part to see when the solder has melted. I expected it to take a couple of minutes.

Img_1273.jpg

I did exactly that, but ran into a nasty problem: there is self-adhesive tape on the top of the GPU and nearby on the PCB, which shrinks in the heat. Unfortunately it takes components with it, so they end up on the tape instead of the PCB. It turns out that there are a couple of dozen parts – decoupling capacitors, I think – on the top of the GPU. Most of them had come off and stuck to the shrunken tape.

IMG_1274

In addition, the backup battery got itself in the way and overheated and burst during the reflow. There were also solder balls visible round the end of the GPU. I thought I’d pretty much wrecked it. But, having come this far, I decided to try and repair the damage and put it back together.

Reassembly is the reverse of disassembly, as they say in the Haynes manuals. Don’t forget to apply new heatsink compound to the tops of the chips in contact with the heatsink. I didn’t hold out much hope.

I got it all back together with no screws left over. I put the battery in, pressed the power switch, and apart from a brief flicker of the power light, nothing. Not a sausage, or even a beep. Game over. Time to go shopping for a new motherboard.

As luck would have it, the Thinkpad T61 was built in various versions. Only the really upmarket ones have the Nvidia graphics chip with the bad reputation. The cheaper versions use the graphics provided by the Intel motherboard chipset. I care more about reliability than I do about ultimate 3D graphics performance, so I decided to do a motherboard swap once I’d discovered that replacement motherboards were available from the US at about $30. It’s useful to know the part numbers. My original motherboard, an early one with Nvidia graphics, was a 42W7652. The ones with the Intel graphics are 42W7651 (early version, supports Intel Merom processors) and 42W7875 (later version, supports both Merom and Penryn processors). On the left the old motherboard, on the right the replacement one.

Img_1377

Swap the CPU and RAM to the new motherboard, and make sure you put the little foam block on top of the transistors by the VGA socket – it conducts heat to the heatsink. Also swap the PC Card cage, which is held by two screws underneath the motherboard. It pulls out vertically from the motherboard.

Getting the machine to boot after I’d swapped the motherboard took a bit of fiddling. It turns out that, because the backup battery had been removed, all the BIOS settings were lost and the SATA interface mode had changed from ‘Emulation’ to ‘AHCI’. The symptom was that the machine would start booting but Windows 7 would just bluescreen immediately. Changing that BIOS setting fixed the problem.

I didn’t have to re-validate Windows, but a couple of pieces of software got unhappy. The Visagesoft Expert PDF tools required me to re-enter the registration information I’d already paid for, which was no trouble, and the Xilinx and Lattice FPGA tools needed new host-locked licence keys generating, which was free.

The Thinkpad lives to fight another day, and to let me write this blog post.