Tag Archives: beaglebone

Systemd for Embedded Linux

Over the last few years, there has been a lot of controversy in the Linux world about systemd. As I understand it, systemd is intended to be a better-engineered, more powerful version of the motley collection of little programs and scripts which keeps the essential services on a Linux system running.

systemctl

The controversy arises because the original 1970s Unix way of doing things was to rely on a motley collection of little programs and scripts for everything, each of which was simple but well understood, and to knit them together to form a complete operating system. Systemd takes a different approach, using larger and more sophisticated components which are more dedicated to particular tasks, such as managing services or network connections. This is supposed to make it more efficient and easier to manage in the twenty-first century.

I’ve been doing some work recently on an embedded Linux system which runs on the latest version of Debian Linux, version 8 (‘Jessie’). Debian Jessie fully supports systemd to the extent that it seems to be the default way of doing things. I thought I’d experiment with it a bit.

When working on an embedded Linux system, I very frequently want to have a piece of my software run reliably at startup, get restarted if it fails, and be able to output logging information to an easily-managed place. In this case, my software provides a D-Bus interface to a piece of industrial electronics.

In the past I’ve relied on copying and pasting scripts from other pieces of software, and managing log files has always been a bit of a mess. It’s hard to do these things right, so re-inventing the wheel is too risky, which means that the best strategy is to copy somebody else’s scripts. I have never counted the hours of my time which have been wasted by dealing with awkward corner cases and peculiar bugs due to recycled scripts behaving in ways I hadn’t anticipated.

What does it look like with systemd? There are some helpful tutorials out there, including this one from Alexander Patrakov, so it didn’t take me too long to put together a service file which looks like this:

[Unit]
Description=My D-Bus Gateway
[Service]
Type=dbus
BusName=com.martin-jones.gateway
ExecStart=/usr/bin/my_dbus_gateway
Restart=always
[Install]
WantedBy=multi-user.target

I’ve changed the names to protect the innocent, but the contents of the file are pretty self-explanatory. The [Unit] section just includes a description which is readable to a human being. The [Service] section describes the service itself. In this case it’s of type  dbus, which means that systemd will check that the service name (com.martin-jones.gateway in this case) gets correctly published on to D-Bus. The Restart=always setting means that my software gets restarted if it exits. The [Install] section just indicates that this service should run when the system comes up in multi-user mode (like the old runlevel 5).

Having created this file, I simply copied it into /etc/systemd/system/my_dbus_gateway.service and, lo and behold, my new service worked. It was immediately possible to manage the service using commands like

systemctl start my_dbus_gateway.service
systemctl stop my_dbus_gateway.service
systemctl status my_dbus_gateway.service

Great! That’s exactly what I wanted.

Now for logging. I’d heard that systemd would log the stdout and stderr outputs of services into its own journal, and forward that to syslog as required. It does, but there’s a subtlety. Output from stderr appears in /var/log/syslog immediately line-by-line, but output from stdout gets aggressively buffered. This means that it gives the appearance of not working at all unless you explicitly flush the stdout buffer in your code using something like

fflush(stdout)

That’s the only wrinkle I came across, though.

In summary, using systemd’s facilities has made my life as an embedded Linux developer much, much easier and hopefully more reliable. That’s a good thing. My top tips for getting your software working under systemd are these:

  • Create your .service file using the recipe above and the documentation
  • Don’t forget to flush stdout if you want to see it in syslog.
Advertisements

Embedded serial port debugging on the cheap

I do a lot of electronics design and debugging. It’s how I make my living. Most of the things I work on involve embedded computers of one sort or another talking to various peripherals. It could be an 8-bit microcontroller sending data to a simple data logger, or a sophisticated 32-bit Linux system communicating with a wireless modem. Many of these connections involve asynchronous serial ports. If something goes wrong, as it usually does, It’s really useful to be able to see what’s going on with the communications. Here’s a picture of the sort of thing I’m dealing with. It’s a board I designed which contains a load of power electronics as well as a microcontroller and a USB data logger which talk to each other using a serial connection. Note the 175A fuse next to the USB connector!

DSC_0511

Monitoring a serial port on a PC is relatively easy, but the ones I deal with are separate from the PC, stuck on some circuit board on the workbench or buried inside a piece of equipment. To complicate matters, I need to monitor both directions of traffic at the same time, and have some idea of what order things happened in. There are many protocol analyser tools out there which will do all sorts of clever things, but it’s hard to justify them for small projects.

Faced with a such problem to solve recently, I put together a cheap and simple system which allows a developer to monitor both directions of an embedded serial port simultaneously, and have a record of which end transmitted what, when. It consists of a software tool and a bit of hardware.

Software

The software is a simple tool I wrote in C. It compiles and runs under Linux. Given the names of two devices, it opens both of them. It waits for a character on either of them and displays it on the screen in hexadecimal and ASCII. Each device has a separate column. Every time data arrives from the other device, it starts a new line. Each line is timestamped. In this way, it produces a dump of the conversation between the two devices, and it’s clear what order things happened in, and who said what.

It has one other handy feature: it has a ‘-l’ for (‘live’) command line option. In this mode, it will update the display every time a character arrives so you can see occasional data as it happens. This is great for monitoring things like keystrokes. This mode involves a whole load of backspacing and overprinting on the display, so it’s not really suitable if you want to record the output of the tool for later examination. The non-live mode is better for that – it outputs data only when it has a complete line.

Here’s an example session using the software.

hdump2Here you can see that I used stty to set the baud rate of two serial devices attached to the PC. Then I ran the tool (provisionally called hdump2). You can see the data after that. The timestamps are on the left, followed by hexadecimal representations of the data, then the ASCII version. In this case we’re watching the initialisation of a Vinculum USB host controller. The data isn’t guaranteed to be in exactly the right order character-for-character, since it’s had to travel from two serial ports through the operating system, but it’s good enough for debugging.

The source is available to download here:

hdump2-1.0-source.tar.gz

To build it:

tar xzf hdump2-1.0-source.tar.gz
make

It’s free software, licensed under the GNU General Public License.

Hardware

The hardware can be any pair of serial ports suitable to connect to the device under test, but I made a little module which makes it easier. It’s based on an FTDI FT2232H Mini Module. In fact, that’s all it is! There are just a couple of wire links to configure its power supplies and bring the receive pin of each of its two serial ports out to a connector. There’s a load of spare space on my board which I intend to use eventually to fit various useful connectors and buffers to hook up to the different serial port standards I come across.

DSC_0516

The wiring looks like this:

  • Join pin CN3-1 to CN3-3
  • Join pin CN2-1 to CN2-11 and CN3-12
  • serial input 1 is on CN2-10
  • serial input 2 is on CN3-25
  • input ground is on CN3-2 and CN3-4

When plugged in to a PC, two devices (typically /dev/ttyUSB0 and /dev/ttyUSB1, assuming you don’t have any other USB serial ports attached) appear. They can be used as the device arguments to the tool.

Do let me know if any of this is useful to you, or if you have comments or improvements to suggest or contribute.

BeagleBone Black Debian build from balloonboard.org

The Balloon Board project set out more than a decade ago to produce a high-performance, low-power, open hardware embedded Linux computing platform. It successfully did that, and now thousands of them are in use in all sorts of products all over the world.

One of the major achievements of the project was a build system which could configure, download, patch and build a boot loader, kernel and root filesystem using a relatively simple menu-driven interface. The menu allows the user to select which type of board to build for, and which customisations to apply. These can include custom kernel drivers for a particular application, or a different Linux distribution (Debian and Emdebian are supported). It doesn’t need an especially powerful machine to run on, but can make use of multiple CPU cores to speed up building.

My colleague Nick Bane has recently, in collaboration with Cambridge University Engineering Department, updated the build system so that it can build software for the Raspberry Pi and BeagleBone Black, both of which are somewhat newer hardware than the Balloon Board. This should make it easier to port software and device drivers between the boards without having to start with a completely new development system.

It might also be of interest to developers working with the Raspberry Pi and BeagleBone boards who want a simple, repeatable way to create the necessary software. This is especially important for use in manufactured products, where the process for creating a whole software distribution needs to be well understood.

This page describes my first attempts at building Debian Linux and the kernel for the BeagleBone Black. This is extremely preliminary, and it doesn’t work yet,

Software build options

Get the software:

svn checkout svn://balloonboard.org/balloon/branches/menuconfig2
make menuconfig
  • Mode Expert mode
  • Balloon Board BeagleBone Black
  • Choose which buildroot version to build -> Feb 2013
  • Select kernel version 3.8
  • make kernel-menuconfig
  • Select Kernel Features -> unselect Compile the kernel in Thumb-2 mode
  • make buildroot-menuconfig
  • Package Selection for the target -> Hardware handling -> Misc devices firmwares -> unselect rpi-firmware
  • Hardware handling -> unselect rpi-userland
  • Debian rootfs
  • Select kernel and module installation method -> Build Debian rootfs with kernel modules
make

Create Micro SD card

I used a 2GB card in a USB card reader connected to my Debian Squeeze PC. I created two partitions using fdisk. Partition 1 is a 50MB FAT16 (type 6) partition, and partition 2 is the rest of the card formatted as ext3 (type 83). On my system, the card appeared as /dev/sdc.

sudo mkfs.msdos /dev/sdc1
sudo mkfs.ext3 /dev/sdc2

Copy the files MLO, u-boot.img and uEnv.txt from the BeagleBone USB drive to the FAT partition.

Make uboot image

sudo apt-get install uboot-mkimage

In build/kernel, do

mkimage -A arm -O linux -T kernel -C none -a 0x80008000 -e 0x80008000 -n "Linux" -d ImageBoot uImage

copy uImage into /boot on ext3 partition

copy am335x-boneblack.dtb from /boot on the original BeagleBone Black board into /boot on ext3 partition

The board tries to boot, but kernel dies with a stack dump:

## Booting kernel from Legacy Image at 80007fc0 ...
 Image Name: Linux
 Image Type: ARM Linux Kernel Image (uncompressed)
 Data Size: 8750560 Bytes = 8.3 MiB
 Load Address: 80008000
 Entry Point: 80008000
 Verifying Checksum ... OK
## Flattened Device Tree blob at 80f80000
 Booting using the fdt blob at 0x80f80000
 XIP Kernel Image ... OK
OK
 Using Device Tree in place at 80f80000, end 80f88bac
Starting kernel ...
[ 0.113797] platform 53100000.sham: Cannot lookup hwmod 'sham'
[ 0.114013] platform 53500000.aes: Cannot lookup hwmod 'aes'
[ 0.114792] omap_init_sham: platform not supported
[ 0.114807] omap_init_aes: platform not supported
[ 0.156004] omap2_mbox_probe: platform not supported
[ 0.225933] Unhandled fault: external abort on non-linefetch (0x1028) at 0xf4
[ 0.233938] Internal error: : 1028 [#1] SMP ARM
[ 0.238662] Modules linked in:
[ 0.241861] CPU: 0 Not tainted (3.8.0 #1)
[ 0.246424] PC is at rtc_wait_not_busy+0x14/0x50
[ 0.251241] LR is at omap_rtc_read_time+0x10/0xa4
[ 0.256147] pc : [<c03eee84>] lr : [<c03ef490>] psr: 40000193
[ 0.256147] sp : df053d60 ip : 00000000 fp : c08b0f28
[ 0.268112] r10: df053e18 r9 : c07d31e4 r8 : df2b7000
[ 0.273560] r7 : df2b71c8 r6 : c08b0f28 r5 : c083a9cc r4 : 00000000
[ 0.280365] r3 : f9e3e000 r2 : 00000000 r1 : df053dc4 r0 : df101010
[ 0.287172] Flags: nZcv IRQs off FIQs on Mode SVC_32 ISA ARM Segment kel
[ 0.294880] Control: 10c5387d Table: 80004019 DAC: 00000015
[ 0.300873] Process swapper/0 (pid: 1, stack limit = 0xdf052240)
[ 0.307134] Stack: (0xdf053d60 to 0xdf054000)
[ 0.311684] 3d60: df053dc4 df053dc4 df053dc4 c03ef490 df2b7000 c03ebbd8 c08b0
[ 0.320217] 3d80: df2b7000 c03ec884 00000000 00000000 00000001 df053dc4 00004
[ 0.328746] 3da0: df053df4 c084fcd4 60000113 df2b54b0 00000000 00000000 80000
[ 0.337275] 3dc0: c084fcc4 00000000 00000000 00000000 00000000 00000000 00000
[ 0.345805] 3de0: 00000000 00000000 00000000 c084fe34 c08b0f28 00000000 00000
[ 0.354335] 3e00: 00000000 df101010 c07d31e4 df103640 c08b0f28 c03eba44 dfef0
[ 0.362866] 3e20: 60000113 df101080 df101010 60000113 00000004 60000113 0000c
[ 0.371394] 3e40: df101080 df101000 df101010 c08b0f30 df104cc0 c08b0f2c c07f8
[ 0.379923] 3e60: 00000000 df101044 df2ccb80 c084fe78 c084fdf4 df101010 df104
[ 0.388453] 3e80: c084fdf4 c08af648 c07d31e4 c07ddb14 00000000 c031e1cc c084c
[ 0.396982] 3ea0: df101010 df101044 c084fdf4 c031cf88 df052000 c031d014 00008
[ 0.405510] 3ec0: c084fdf4 c031b66c df045478 df104c80 df052000 df001840 df2c4
[ 0.414040] 3ee0: c0841ea0 c031bf14 c0725ca0 c084fde0 c084fdf4 c084fde0 c0847
[ 0.422570] 3f00: c0860608 c031d5fc c0860608 c084fde0 c07ddb4c 00000007 c086c
[ 0.431102] 3f20: c07ddb4c c07eff94 c07ddb4c c0008858 00000006 00000006 c0780
[ 0.439633] 3f40: 00000000 c07efcac c07eff94 c07ddb4c 000000dd c086060c c07a4
[ 0.448163] 3f60: c0860600 c07ac314 00000006 00000006 c07ac3f0 c0cc8d80 c0570
[ 0.456693] 3f80: c0577e98 00000000 00000000 00000000 00000000 00000000 00004
[ 0.465222] 3fa0: 00000000 00000000 c0577e98 c000e6d8 00000000 00000000 00000
[ 0.473751] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000
[ 0.482280] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000 b637d
[ 0.490833] [<c03eee84>] (rtc_wait_not_busy+0x14/0x50) from [<c03ef490>] (om)
[ 0.500636] [<c03ef490>] (omap_rtc_read_time+0x10/0xa4) from [<c03ebbd8>] (_)
[ 0.510257] [<c03ebbd8>] (__rtc_read_time+0x58/0x5c) from [<c03ec884>] (rtc_)
[ 0.519423] [<c03ec884>] (rtc_read_time+0x30/0x48) from [<c03ecad4>] (__rtc_)
[ 0.528773] [<c03ecad4>] (__rtc_read_alarm+0x1c/0x290) from [<c03eba44>] (rt)
[ 0.538858] [<c03eba44>] (rtc_device_register+0x140/0x268) from [<c07d3358>])
[ 0.548850] [<c07d3358>] (omap_rtc_probe+0x160/0x3b8) from [<c031e1cc>] (pla)
[ 0.558576] [<c031e1cc>] (platform_drv_probe+0x1c/0x24) from [<c031cdec>] (d)
[ 0.568657] [<c031cdec>] (driver_probe_device+0x80/0x21c) from [<c031d014>] )
[ 0.578459] [<c031d014>] (__driver_attach+0x8c/0x90) from [<c031b66c>] (bus_)
[ 0.587902] [<c031b66c>] (bus_for_each_dev+0x60/0x8c) from [<c031bf14>] (bus)
[ 0.597431] [<c031bf14>] (bus_add_driver+0x178/0x23c) from [<c031d5fc>] (dri)
[ 0.606963] [<c031d5fc>] (driver_register+0x5c/0x150) from [<c031e58c>] (pla)
[ 0.616948] [<c031e58c>] (platform_driver_probe+0x1c/0xa4) from [<c0008858>])
[ 0.626936] [<c0008858>] (do_one_initcall+0x2c/0x164) from [<c07ac314>] (ker)
[ 0.637019] [<c07ac314>] (kernel_init_freeable+0x108/0x1e4) from [<c0577ea4>)
[ 0.646648] [<c0577ea4>] (kernel_init+0xc/0x164) from [<c000e6d8>] (ret_from)
[ 0.655452] Code: e59f6038 e59f5038 e3a04000 e5963000 (e5d32044)
[ 0.661858] ---[ end trace 33e6cc13d62fdb11 ]---
[ 0.666954] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0b
[ 0.666954]