Category Archives: Projects

Quick-and-dirty partition resize with Acronis True Image

A little embedded Linux magic comes in unexpectedly handy. Years ago, I started using Acronis True Image to back up my office PC. Other backup packages are available, as they say, but it’s worked well enough for me. This week I had reason to use it in earnest.

img_20180619_112420

The PC is set up with two main partitions on the disk. There’s one for the operating system (Windows 10) and applications, and another with data on it, analogous to the /home partition on a Linux system. Inevitably, it turned out that the boundary between these partitions was in the wrong place. The operating system partition was full, and things were starting to fail. Time to make sure the backup is up-to-date and then to restore to differently-sized partitions.

The operation fell at the first hurdle. Acronis True Image will create a ‘recovery’ USB stick from which the PC boots. This contains a small Linux distribution (Acronis Linux) and a GUI for managing the recovery operation. My backups are on a network drive, which happens to be an Apple Time Capsule. Would the Acronis recovery system connect to the Time Capsule? Would it ever. I couldn’t even persuade it to ask for the username and password, though it could see the Time Capsule on the network.

In Acronis Linux, pressing ctrl-alt-F2 in time-honoured fashion brings up a command line. Yay! Power! A bit of digging reveals a command called asamba which can connect to SMB shares. Using that, I could connect to the Time Capsule absolutely fine and mount it to the local filesystem. So why couldn’t the GUI do it? Annoying.

My next trick was to copy the backup from the Time Capsule to a USB hard drive using another Linux PC. It took a while for the several hundred gigabytes, but it got there. Acronis recovery will read USB drives, so the next step should be easy, right? No. My USB drive happened to be formatted ext4. Though Acronis Linux is perfectly capable of mounting ext4 filesystems, the recovery GUI only acknowledges the presence of FAT and NTFS discs. Aargh!

The solution was to mount the USB drive on my Linux PC, then share it using Samba. Acronis recovery found and let me log on to the Samba share, and could thus see the drive and backups.

Changing the partition sizes was the next problem. The GUI only allows you to shrink partitions or to grow them in to free space. My disk didn’t have any free space. It was all partitioned and in use. I thought that shrinking the data partition would create free space that the operating system partition could grow in to, but I was caught out by the presence of a 500MB ‘recovery’ partition which Windows had sneaked in between the two. I don’t know what it’s for but I didn’t feel like taking the risk of destroying it. There are no tools in the Acronis GUI for deleting partitions and starting again, so I was stuck.

img_20180619_113338

Command line to the rescue! Ctrl-alt-F2 again, and there’s a perfectly good copy of fdisk available. It was a moment’s work to delete the existing recovery and data partitions, then ctrl-alt-F1 to the GUI. Restart the restore process and there’s free space available for the operating system partition to grow in to, and plenty of room to create new recovery and data partitions.

img_20180619_112939

The restore took a few hours, as expected, but was successful. The only wrinkle was that Acronis recovery had marked the recovery partition as active rather than the boot partition, so the PC complained that there was no operating system. Another trip in to fdisk to mark the boot partition active sorted that out, and now I’m typing this very text on the PC restored to working order.

Footnote: there is another Acronis restore tool intended for restoring backups to dissimilar hardware. This may have handled the partition resizing more elegantly, but I didn’t try it.

Advertisements

Good accuracy from a low cost Real Time Clock

One product I work on has a built-in data logger. This helps us a lot if a problem occurs: we can see the history of any faults. Every log entry is time stamped, which is important. We need to know when it’s been used and how often. However, good timekeeping is a challenge. The product has no Internet connection, it gets stored and moved around a lot, and it’s nobody’s job to check or adjust its clock, so there’s a real problem with clock accuracy.

The real time clock is based on the Microchip MCP7940N chip. The chip uses a standard 32.768kHz crystal for its timekeeping. These crystals are fickle beasts, partly because of the very low-power oscillator in the chip. The oscillator frequency, which is critical for accurate timekeeping, is very dependent on the load capacitance, which itself can vary with different builds of the PCB. The heat of soldering during manufacture also affects the crystal. I’ve seen plenty which have failed altogether, and others whose frequency has shifted significantly. Note the soldering on the crystal’s load capacitors C9 and C10 in the photo below, part of an attempt to find the optimum load capacitance on a prototype board.

img_20180530_100756

All of these issues mean that just assembling the PCB and hoping for the best doesn’t work well. The frequency of an apparently working crystal can be anything up to 100ppm (parts per million) wrong. That doesn’t sound like much, but it works out to nearly an hour’s error per year, which is pretty bad.

Fortunately the MCP7940N has a neat feature which helps a lot. One of its registers, called CAL, holds a value which speeds up or slows down the clock by a small amount, like the regulator on a mechanical clock. But how do we know what the error is, and whether it’s been successfully corrected?

The MCP7940N also has a pin which can output a 1Hz square wave derived from the oscillator. With a sufficiently accurate timer, it’s possible to measure the error in the crystal’s frequency this way.

Checking the correction is more difficult. Because the chip only works on whole oscillator clock cycles, it does the adjustment to the clock’s speed by adding or removing a few clock cycles each minute. It’s therefore necessary to measure the period of exactly 60 of the chip’s seconds to find out how long its minute is, and therefore how accurate the whole clock is.

Getting a sufficiently accurate timer is the first problem. My aim was to get the MCP7940N to be accurate to within 1ppm, or about 30 seconds per year. A useful rule of thumb in metrology is that the measuring instrument needs to be ten times more precise that the quantity being measured, so we need a timer accurate to 0.1ppm, or one part in ten million. To the rescue comes my trusty Hewlett Packard 5335A universal counter. It’s an oldie but a goodie. Mine is fitted with the optional oven-controlled crystal oscillator, an HP 10544A. I checked it and set it up against a Rubidium frequency standard about 8 years ago and it hasn’t been touched since. I checked it this month against the same frequency standard, and it still agrees to within 0.1ppm. Not bad, and certainly good enough for this job.

Measuring the initial clock error is easy enough: connect the counter to the MCP7940N’s 1Hz output and look at the error. The 5335A counter has handy built-in maths functions to make this easier, so it will directly display the difference between its idea of a second and the chip’s attempt.

To measure the corrected clock output over a minute needs a bit more trickery. The 5335A counter has an external ‘arm’ input, and can average a period reading over the length of the ‘arm’ signal. All that’s needed is to arm the counter for 60 seconds and the counter will do the rest. I couldn’t find a way to make the counter do this for itself, so I cheated and used a spare Arduino mini that happened to be lying around. All it had to do was wait for a clock pulse on a GPIO pin, take another GPIO pin high to arm the counter, count 60 clock pulses, then take the arm signal low. Simple.

The test setup looked like this. The scope is there for ease of probing (note the cable from its ‘sig out’ connector to the counter) and it also includes a Tektronix 7D15 timer/counter module connected to the chip’s output. The 7D15 is a lot less accurate (about 1ppm) than the 5335A but it’s good enough to give an idea of what correction is required. The Arduino mini is just about visible at the bottom of the photo.

img_20180427_171833

Here’s a closeup of the scope screen, showing the measured period of the clock’s 1Hz output, 999.9818ms. That’s just over 18ppm too fast.

img_20180427_171723

This the setup screen of the product, showing the 18ppm correction applied to the MCP7940N’s CAL register.

img_20180427_171715

Finally the error in the measured minute, calculated by the 5335A counter gated by the Arduino.

img_20180427_171823

It’s showing that the corrected minute is 301.3 parts per billion, so 0.3ppm, too fast. That’s as good as it’s going to get – about 10 seconds per year. With the clock set up using this process, it has a fighting chance of staying accurate in the real world.

The Euroquadruped, or the solution to wall wart Tetris

Having relocated from the UK to Poland, the change in mains plugs has never quite been resolved. All the equipment that came from the UK has 13A plugs on it, and all the local sockets are the CEE 7/5 type. Then there are locally-acquired bits and pieces  which generally have two-pin Europlugs. No matter how many adapter cables and extension leads I make, or change the plug or socket on, there’s always something that doesn’t fit, or won’t reach the nearest compatible socket.

Of particular annoyance are various chargers and wall warts. They’re all bigger than a standard Europlug, and there’s no standard for which way the ‘lump’ of the charger will be attached, so connecting several of them in to a multi-socket extension lead doesn’t work well. Often one charger will obscure one or more sockets in such a way that all the things you need just won’t fit, or the whole assembly becomes so big and unwieldy that things start falling out under their own weight.

It is therefore with some relief that I announce the debut of the Euroquadraped: one 13A plug with short cables to four Europlug sockets. It reminded me of an octopus but only has four ‘legs’, hence the name.

IMG_20180211_221335.jpg

Getting all four cables in to the 13A plug was a bit fiddly, but choosing a roomy plug helped. The little Europlug sockets are neat. I’ve only seen them here in Poland. The finger ring to discourage pulling on the cable is a nice touch, and doubles as a cord grip internally. They even (finally) have safety shutters on the socket holes, roughly 60 years after shutters became standard on the British 13A socket…

The great advantage of this thing is that each socket is independent of the others and can find its own equilibrium with whatever electrical carbuncle is plugged in to it.

IMG_20180211_221527.jpg

It’s trivially simple, but it’s going to save a lot of frustration in the workshop.

Garage door opener life extension

This little thing is really annoying. It’s the remote control for our electric garage door, and I use it several times a day to get my bike in and out. Luxury, pure luxury. Or it would be, if it worked properly. The trouble is that about half the time after pressing the button, it locks up, stuck either on, with its little red LED lit, or off, so it won’t open or close the door, and drains the battery. The only way to fix it is to pull it apart, take out the battery, and put it back in without losing any of the bits of plastic. You can imagine how much fun that is with gloves on, in the rain, when you’re in a hurry. Yeah, yeah, first world problems, I know. But there doesn’t seem to be any way of opening our garage door manually, which is a nuisance.

IMG_20171215_155448.jpg

Inside is a little PCB containing a PIC12F635 microcontroller and a simple 433MHz transmitter circuit using a couple of transistors and a resonator. I did all the obvious things: replace the battery, clean the board with solvent, all of that. Nothing helped. It seems there’s a bug in the PIC’s software which makes it prone to locking up. The switch under the button is rather worn and thus suffers from contact bounce, and I think that’s what’s upsetting it.

I tried filtering the signal from the button to the PIC with an RC network. That didn’t help: the PIC didn’t even respond to the button any more. Presumably it needs a fairly quick risetime to trigger an interrupt, or something similar.

There is a second button which is less worn. It transmits a different code which doesn’t seem to do anything on our garage door, but it never crashes the PIC. I considered swapping the switches over, but they’re surface-mount and glued down, and the chances of removing both of them and successfully soldering them down again undamaged were slim.

I wondered about rewiring the PCB so that the input to the PIC is always active, and the button just switches the power to the PIC. A quick test by adding a blob of solder and removing and reconnecting the battery showed that wasn’t going to work: the PIC just didn’t transmit. It needed to have the power applied and then have the button activated.

There was another issue to address. The button, when it worked, was prone to getting pressed by accident as the bunch of keys rattled around in my pocket. That led to the garage door opening itself at odd times, which wasn’t great for the security of our bikes. Somehow modifying it to be less prone to accidental operation would be nice. 

Design changes were called for. Since the second button is redundant, I decided to convert it in to a power switch, so it has to be held down while the real button is pressed, and releasing it will guarantee that the power is cut from the PIC so it can’t remain locked up. I cut the power track to the PIC and took advantage of the PCB layout to add a P-channel MOSFET in series with it, then cut the track from the second button to the PIC’s input and wired it to the MOSFET’s gate. The buttons both have one terminal to battery negative. A pullup resistor between gate and source completes the circuit.

IMG_20171215_155001.jpg

Here’s the modified board. You can see the remnants of some of my previous repair attempts, and the extra MOSFET and pullup resistor at top right. Now, to open or shut the door, I have to hold down the button on the right and press the left button, rather like using a shift key on a keyboard.

It seems to work reliably now, so leaving the house and returning are considerably quicker and less stressful. 

Incidentally, I did the repairs and modifications in the office, which is several miles away from the offending garage door. To check that the remote was actually transmitting properly, I used my Tektronix 7912AD programmable digitizer with 7A29 vertical plugin. Originally intended for high-speed laser and nuclear physics research, it’s kind of overkill for testing a garage door opener, but its 700MHz bandwidth is plenty fast enough to get a clear picture of the transmission using just a loop of wire connected to the input. Analogue technology at its very finest. Here’s its TV output connected to a little LCD monitor showing the 433MHz signal from the remote.

IMG_20170814_130325.jpg

Cracking a password-protected PDF file

Suppose you asked an insurance company for a letter. The insurance company kindly sent it as a PDF attached to an email. Sensibly, they protected that PDF with a password which they told you over the phone. You wrote it in a notebook and then left the notebook at work over the weekend.

pdf-password

How could you read the letter in the password-protected file at home, then? Remembering that the password was definitely an English word, and all in lower case, a dictionary attack has got to be worth a try.

Linux provides some handy tools for this. There’s a list of English words in /usr/share/dict/words, and a suite of PDF tools which can attempt to open the file using a password, indicating success or failure. A few minutes with Python and:

#!/usr/bin/python
import os,sys

wf=open('/usr/share/dict/words','r')
while True:
  word = wf.readline().strip().lower()
  if word == '':
    print "No solution found"
    break
  print word
  cmdline = 'pdftotext -upw "'+word+'" '+sys.argv[1]
  result = os.system(cmdline)
  if result == 0:
    break

The same thing must be possible in a more hipsterly fashion using awk, but I couldn’t be bothered to figure out a sufficiently baroque command line.

By the way, the password was ‘orange’. Don’t tell anybody.

Systemd for Embedded Linux

Over the last few years, there has been a lot of controversy in the Linux world about systemd. As I understand it, systemd is intended to be a better-engineered, more powerful version of the motley collection of little programs and scripts which keeps the essential services on a Linux system running.

systemctl

The controversy arises because the original 1970s Unix way of doing things was to rely on a motley collection of little programs and scripts for everything, each of which was simple but well understood, and to knit them together to form a complete operating system. Systemd takes a different approach, using larger and more sophisticated components which are more dedicated to particular tasks, such as managing services or network connections. This is supposed to make it more efficient and easier to manage in the twenty-first century.

I’ve been doing some work recently on an embedded Linux system which runs on the latest version of Debian Linux, version 8 (‘Jessie’). Debian Jessie fully supports systemd to the extent that it seems to be the default way of doing things. I thought I’d experiment with it a bit.

When working on an embedded Linux system, I very frequently want to have a piece of my software run reliably at startup, get restarted if it fails, and be able to output logging information to an easily-managed place. In this case, my software provides a D-Bus interface to a piece of industrial electronics.

In the past I’ve relied on copying and pasting scripts from other pieces of software, and managing log files has always been a bit of a mess. It’s hard to do these things right, so re-inventing the wheel is too risky, which means that the best strategy is to copy somebody else’s scripts. I have never counted the hours of my time which have been wasted by dealing with awkward corner cases and peculiar bugs due to recycled scripts behaving in ways I hadn’t anticipated.

What does it look like with systemd? There are some helpful tutorials out there, including this one from Alexander Patrakov, so it didn’t take me too long to put together a service file which looks like this:

[Unit]
Description=My D-Bus Gateway
[Service]
Type=dbus
BusName=com.martin-jones.gateway
ExecStart=/usr/bin/my_dbus_gateway
Restart=always
[Install]
WantedBy=multi-user.target

I’ve changed the names to protect the innocent, but the contents of the file are pretty self-explanatory. The [Unit] section just includes a description which is readable to a human being. The [Service] section describes the service itself. In this case it’s of type  dbus, which means that systemd will check that the service name (com.martin-jones.gateway in this case) gets correctly published on to D-Bus. The Restart=always setting means that my software gets restarted if it exits. The [Install] section just indicates that this service should run when the system comes up in multi-user mode (like the old runlevel 5).

Having created this file, I simply copied it into /etc/systemd/system/my_dbus_gateway.service and, lo and behold, my new service worked. It was immediately possible to manage the service using commands like

systemctl start my_dbus_gateway.service
systemctl stop my_dbus_gateway.service
systemctl status my_dbus_gateway.service

Great! That’s exactly what I wanted.

Now for logging. I’d heard that systemd would log the stdout and stderr outputs of services into its own journal, and forward that to syslog as required. It does, but there’s a subtlety. Output from stderr appears in /var/log/syslog immediately line-by-line, but output from stdout gets aggressively buffered. This means that it gives the appearance of not working at all unless you explicitly flush the stdout buffer in your code using something like

fflush(stdout)

That’s the only wrinkle I came across, though.

In summary, using systemd’s facilities has made my life as an embedded Linux developer much, much easier and hopefully more reliable. That’s a good thing. My top tips for getting your software working under systemd are these:

  • Create your .service file using the recipe above and the documentation
  • Don’t forget to flush stdout if you want to see it in syslog.

Lattice FPGA programming adapter from the junk box

Working with Lattice FPGAs recently, I had a need to program one but couldn’t find my ‘proper’ (Chinese clone, bought from eBay) programming adapter. When I started the Diamond Programmer software, though, it claimed it could see a USB programming adapter. It turned out that I’d left an FTDI ‘FT2232H Mini Module‘ attached to the PC. I use the module for all sorts of little debugging exercises: most often as a dual serial port for serial port debugging, but it also works for programming Parallax Propeller microcontrollers.

Img_0603

As luck would have it, the Diamond software recognises the unadulterated FT2232H as a legitimate USB programmer, and pressing the ‘Detect Cable’ button finds it. Note that if you plug in a new USB device, the Diamond Programmer software needs restarting before it can see it.

The FT2232H has two ports, A and B, and these appear as ports FTUSB-0 and FTUSB-1 in the Diamond software. All that remained was to figure out the wiring. Fortunately, there are a lot of clues in the schematics of various Lattice evaluation boards, particularly the MachXO2 Pico Board and the iCE40 Ultra Breakout Board.

diamond-programmer

Here’s the wiring, both for SPI and JTAG, referred to the pins on the Mini Module. I chose to use port B since it was more convenient for my prototype board. Translating the wiring to port A is left as an exercise for the reader.

SPI    JTAG  FT2232H  Mini Module
SO     TDI   DBUS1    CN3-25
SI     TDO   DBUS2    CN3-24
SCK    TCK   DBUS0    CN3-26
SS_B   ISPEN DBUS4    CN3-21
CRESET TRST  DBUS7    CN3-18
GND    GND   GND      CN3-2,4

It works well, and does exactly what it should.