Category Archives: Projects

Cracking a password-protected PDF file

Suppose you asked an insurance company for a letter. The insurance company kindly sent it as a PDF attached to an email. Sensibly, they protected that PDF with a password which they told you over the phone. You wrote it in a notebook and then left the notebook at work over the weekend.

pdf-password

How could you read the letter in the password-protected file at home, then? Remembering that the password was definitely an English word, and all in lower case, a dictionary attack has got to be worth a try.

Linux provides some handy tools for this. There’s a list of English words in /usr/share/dict/words, and a suite of PDF tools which can attempt to open the file using a password, indicating success or failure. A few minutes with Python and:

#!/usr/bin/python
import os,sys

wf=open('/usr/share/dict/words','r')
while True:
  word = wf.readline().strip().lower()
  if word == '':
    print "No solution found"
    break
  print word
  cmdline = 'pdftotext -upw "'+word+'" '+sys.argv[1]
  result = os.system(cmdline)
  if result == 0:
    break

The same thing must be possible in a more hipsterly fashion using awk, but I couldn’t be bothered to figure out a sufficiently baroque command line.

By the way, the password was ‘orange’. Don’t tell anybody.

Systemd for Embedded Linux

Over the last few years, there has been a lot of controversy in the Linux world about systemd. As I understand it, systemd is intended to be a better-engineered, more powerful version of the motley collection of little programs and scripts which keeps the essential services on a Linux system running.

systemctl

The controversy arises because the original 1970s Unix way of doing things was to rely on a motley collection of little programs and scripts for everything, each of which was simple but well understood, and to knit them together to form a complete operating system. Systemd takes a different approach, using larger and more sophisticated components which are more dedicated to particular tasks, such as managing services or network connections. This is supposed to make it more efficient and easier to manage in the twenty-first century.

I’ve been doing some work recently on an embedded Linux system which runs on the latest version of Debian Linux, version 8 (‘Jessie’). Debian Jessie fully supports systemd to the extent that it seems to be the default way of doing things. I thought I’d experiment with it a bit.

When working on an embedded Linux system, I very frequently want to have a piece of my software run reliably at startup, get restarted if it fails, and be able to output logging information to an easily-managed place. In this case, my software provides a D-Bus interface to a piece of industrial electronics.

In the past I’ve relied on copying and pasting scripts from other pieces of software, and managing log files has always been a bit of a mess. It’s hard to do these things right, so re-inventing the wheel is too risky, which means that the best strategy is to copy somebody else’s scripts. I have never counted the hours of my time which have been wasted by dealing with awkward corner cases and peculiar bugs due to recycled scripts behaving in ways I hadn’t anticipated.

What does it look like with systemd? There are some helpful tutorials out there, including this one from Alexander Patrakov, so it didn’t take me too long to put together a service file which looks like this:

[Unit]
Description=My D-Bus Gateway
[Service]
Type=dbus
BusName=com.martin-jones.gateway
ExecStart=/usr/bin/my_dbus_gateway
Restart=always
[Install]
WantedBy=multi-user.target

I’ve changed the names to protect the innocent, but the contents of the file are pretty self-explanatory. The [Unit] section just includes a description which is readable to a human being. The [Service] section describes the service itself. In this case it’s of type  dbus, which means that systemd will check that the service name (com.martin-jones.gateway in this case) gets correctly published on to D-Bus. The Restart=always setting means that my software gets restarted if it exits. The [Install] section just indicates that this service should run when the system comes up in multi-user mode (like the old runlevel 5).

Having created this file, I simply copied it into /etc/systemd/system/my_dbus_gateway.service and, lo and behold, my new service worked. It was immediately possible to manage the service using commands like

systemctl start my_dbus_gateway.service
systemctl stop my_dbus_gateway.service
systemctl status my_dbus_gateway.service

Great! That’s exactly what I wanted.

Now for logging. I’d heard that systemd would log the stdout and stderr outputs of services into its own journal, and forward that to syslog as required. It does, but there’s a subtlety. Output from stderr appears in /var/log/syslog immediately line-by-line, but output from stdout gets aggressively buffered. This means that it gives the appearance of not working at all unless you explicitly flush the stdout buffer in your code using something like

fflush(stdout)

That’s the only wrinkle I came across, though.

In summary, using systemd’s facilities has made my life as an embedded Linux developer much, much easier and hopefully more reliable. That’s a good thing. My top tips for getting your software working under systemd are these:

  • Create your .service file using the recipe above and the documentation
  • Don’t forget to flush stdout if you want to see it in syslog.

Lattice FPGA programming adapter from the junk box

Working with Lattice FPGAs recently, I had a need to program one but couldn’t find my ‘proper’ (Chinese clone, bought from eBay) programming adapter. When I started the Diamond Programmer software, though, it claimed it could see a USB programming adapter. It turned out that I’d left an FTDI ‘FT2232H Mini Module‘ attached to the PC. I use the module for all sorts of little debugging exercises: most often as a dual serial port for serial port debugging, but it also works for programming Parallax Propeller microcontrollers.

Img_0603

As luck would have it, the Diamond software recognises the unadulterated FT2232H as a legitimate USB programmer, and pressing the ‘Detect Cable’ button finds it. Note that if you plug in a new USB device, the Diamond Programmer software needs restarting before it can see it.

The FT2232H has two ports, A and B, and these appear as ports FTUSB-0 and FTUSB-1 in the Diamond software. All that remained was to figure out the wiring. Fortunately, there are a lot of clues in the schematics of various Lattice evaluation boards, particularly the MachXO2 Pico Board and the iCE40 Ultra Breakout Board.

diamond-programmer

Here’s the wiring, both for SPI and JTAG, referred to the pins on the Mini Module. I chose to use port B since it was more convenient for my prototype board. Translating the wiring to port A is left as an exercise for the reader.

SPI    JTAG  FT2232H  Mini Module
SO     TDI   DBUS1    CN3-25
SI     TDO   DBUS2    CN3-24
SCK    TCK   DBUS0    CN3-26
SS_B   ISPEN DBUS4    CN3-21
CRESET TRST  DBUS7    CN3-18
GND    GND   GND      CN3-2,4

It works well, and does exactly what it should.

First steps with a Lattice iCE40 FPGA

I’ve just been doing some work with the iCE40 series of FPGAs from Lattice Semiconductor. They’re small FPGAs, with up to 7680 gates, and they’re very low-power, which is nice for mobile applications. From what I can gather, Lattice acquired the designs when they bought a company called SiliconBlue in 2011. I’ve been used to using the Lattice Diamond software with their other chips, but the iCE40 chips aren’t supported by Diamond. Instead, they get their own software called iCEcube2. It’s a bit of a pain to use and not very well documented. I’ve just been through the process of starting a project and getting a very basic design working, and I’m writing about it here in case someone else finds it useful.

Img_0602

The iCEcube2 software looks convincingly like an IDE, but it isn’t, really. It doesn’t even seem to have a way of creating new source code files, and the order in which some things have to be done is not at all obvious. I think iCEcube2 is really designed for taking existing designs and implementing them on the Lattice iCE40 chips. While the software is a complete dog’s breakfast, it does have the key advantage of being free. You do need to create a node-locked licence for it using their licencing page.

iCEcube-blank

To start an empty project, double click Project -> New Project. Select the chip you’re going to use. This creates a folder with the title of the project, containing:

  • <project>_sbt.project
  • <project>_syn.prj
  • folder <project>_Implmnt, containing folder sbt, containing folders constraint, log and outputs. All are empty apart from iceCube0.log in log folder.

Now you can add your source files. If you click on ‘Synthesis Tool’, then an ‘Add Synthesis Files’ menu item appears, but clicking on this doesn’t do anything useful. You have to right-click on ‘Add Synthesis Files’ and select ‘Add Files…’ from the pop-up menu. Go figure. I used a very simple VHDL source file:

LIBRARY ieee;
USE ieee.std_logic_1164.ALL;

ENTITY test IS
 PORT
 (
 d: in std_logic;
 q: out std_logic;
 qn: out std_logic
 );
END test;

ARCHITECTURE rtl OF test IS
BEGIN

 q <= d;
 qn <= not d;
 
END rtl;

At this point I’d expect to be able to allocate signal names (d, q and qn, in this case) to pins on the device package. But you can’t do that yet in the wonderful world of iCEcube2. All the buttons on the toolbar are greyed out. The way to proceed is to double click ‘Run Synplify Pro Synthesis’. Hopefully your code will compile without errors, and lots of files get created.

The project folder now contains:

  • stdout.log and stdout.log.bak
  • synlog.tcl
  • loads of stuff under <project>_Implmnt

Two new files appear in the project under ‘P&R Flow’: <project>.edf and <project>.scf.

Now double-click ‘Run P&R’. The design will get placed and routed, and a bitmap gets generated for programming the chip.

At this point the toolbar buttons for timing constraints, pin constraints, floor planner, package view, power estimator and timing analysis become active. Hurrah! Now you can change your pin constraints.

iCEcube-toolbar-enabled

Click on ‘Pin Constraints Editor’, the fourth icon from the left. Put in the pin locations for the signals you want. Make sure you click the ‘locked’ checkboxes on the left hand side, otherwise the place and route process is likely to move them. Press ctrl-S to save. The constraints get saved in <project>_Implmnt\sbt\constraint\<top design file>_pcf_sbt.pcf. You will then get asked to add the file to the project. Say yes.

If you’re using source control, it’s a good idea to add this file to it. I’m not so sure about all the other junk that iCEcube generates.

Now double-click ‘Run P&R’ again and the new bitmap file will be generated, using your pin constraints.

Programming an actual chip (or at least its SPI Flash ROM) needs the Diamond Programming tool, which comes as part of the Lattice Diamond software and *not* as part of iCEcube2. That’s just another couple of gigabytes to download, and another licence (free) to acquire, so it’s a pain, but it does work.

Orange Internet and port forwarding, continued

A long time ago, I wrote an article about how I worked around the lack of NAT loopback support on the Orange LiveBox broadband router. At the time, it was a pain to get everything working right. Having just moved house, we made the sensible decision to stay with the same broadband provider, in order to avoid having to re-invent or at least re-configure all this stuff.

Well, it turned out not to be as easy as that. Nothing ever is, it seems. The new Orange broadband service comes with a ‘FunBox‘ instead of a ‘LiveBox’. Fun? Who said? Not in my experience. The web interface to the box looks comfortingly similar to the old LiveBox, so I thought it would work the same way. No chance.

funbox2

What’s the problem I’m trying to solve here? Well, I have a little SheevaPlug which hosts some services I need to use from various locations when I’m working. Yes, I know I should put those services ‘in the cloud’, but that would involve both paying money and solving a whole load of other problems. For years and years, I’ve simply had a port forwarding rule set up on my home broadband router so that the SheevaPlug is accessible from the internet. A little touch of dynamic DNS courtesy of dyn.com and it’s all worked fine.

Fast forward to 2015. I tried to recreate the setup I’d always used, but with the FunBox (hah!). The old setup went like this:

home_net_old

  • the broadband router just does the usual NAT routing and behaves as a dumb wi-fi access point. It has a port forwarding rule set up to forward port 22 (ssh) to the SheevaPlug
  • the SheevaPlug hosts my ssh server, and provides the DHCP and DNS services to everything on the network, so I get proper local hostname lookups, easy-to-manage IP addresses, and can solve the NAT loopback problem.

That’s it. I tried to set up the FunBox the same way. What could possibly go wrong?

I boldly switched off the DHCP and DNS servers in the FunBox and switched over to using the ones on my SheevaPlug. Everything seemed to work fine, except…the TV. Yes, the TV decoder is connected to the FunBox via Ethernet, so the whole shebang comes down the wire. No aerial required. Trouble is, the FunBox seems to need to set up the TV decoder by DHCP otherwise it doesn’t know it’s there, so you get no telly. Oh well, I’ll use the DHCP server in the FunBox and put up with the inconvenience.

Even that doesn’t work out. In order to solve the NAT loopback problem, which the miserable FunBox suffers from just like the LiveBox, I need to run my local DNS server. Except that the brain-dead FunBox won’t let you change the DNS settings on its built-in DHCP server. How annoying is that?

OK, accept that NAT loopback will remain a problem. Maybe I’ll find another way round that. Now for the showstopper: the blessed FunBox refuses to forward port 22. It will forward every other port under the sun, but not 22. The web configuration interface just won’t accept the setting: it ignores it. Doesn’t even give an error message. You’re just not having it. Oh well, maybe I have to expose my ssh server on a different port and put up with changing all the gazillion clients which know about it. Except I’ve still got the loopback problem. This is getting unpleasant.

Before tearing out what little remained of my hair by this point, I slept on the problem and had a brainwave. The FunBox supports a ‘DMZ’ feature, in which it’ll forward all incoming internet traffic to a particular IP address on the LAN. I tried it, experimentally sending all internet traffic to the poor, naked SheevaPlug, and it worked! At last, sweet relief: something which does what it says on the tin.

Clearly exposing the server in all its complacent insecurity to the internet isn’t a good idea. I needed to put a firewall in the way. A rummage in the cupboard produced a spare Raspberry Pi and a USB to Ethernet adapter. I programmed OpenWRT on to an SD card and booted up. It turned out to be easier to configure it through the command line than with the web interface, LuCI, which isn’t exactly finished yet. Some fiddling later and I’d managed to disable its DHCP and DNS features and enable incoming connections only on port 22.

Adding the firewall had another bonus: it meant I could re-enable my DNS and DHCP servers and let them look after the LAN without the wretched FunBox knowing anything about it. It just has to live a simple life, looking after the telly and the firewall. I disabled its built-in Wi-fi access point and added an ageing Linksys WAP54G running DD-WRT software after the firewall. Lovely.

home_net_new

There is one fly in the ointment: the Raspberry Pi turns out to make a rubbish router. Our 30 megabit internet connection is reduced to 3 megabits on its way through the Pi. I don’t know if it’s something to do with my configuration, or the release of OpenWRT (15.05-rc3) I’m using. When I’ve unpacked enough boxes to find another router, I’ll try that.

Servicing a Fluke 12 Multimeter

One of my most-used tools on the workbench is my Fluke ’12’ multimeter. I’ve had it almost 20 years, and it’s my favourite meter because it was clearly designed by someone who had to fix things. It’s rugged, especially in its bright yellow holster, and has so many thoughtful features: it has big buttons and a switch, instead of a rotary knob, so it’s easy to use with one hand. It autoranges quickly and reliably. In continuity and resistance modes, it automatically switches to measuring voltage if it detects one, so you don’t need to worry about changing ranges when debugging things. It doesn’t have fiddly extra features. It doesn’t even have a current range, because it would make no sense: measuring current usually involves breaking a circuit, which you can’t easily do when working on a circuit board. It switches itself off when unused for half an hour or so, saving the battery.

Img_0007

It’s been completely reliable apart from needing a new set of test leads last year (for the first time). However, recently the big, chunky buttons had become reluctant to respond, and needed firmer and firmer presses until they didn’t work at all. That meant I was stuck measuring either DC voltage or continuity. Time to pull it apart and see what’s wrong.

Removing the test leads and holster, then taking out the four screws, reveals the view you get when changing the battery. I’m sure Dave at EEVBlog, master of the multimeter teardown, would approve. Lots of chunky components and very solid construction.

Img_0008

The PCB is held in by the plastic clips at the top and sides. Easing them back lets it and the plastic frame underneath it come out:

Img_0010

The black plastic internal frame (under the PCB in this photo) is marked as being made of polycarbonate, so it’s very strong. None of that high-impact polystyrene rubbish. It’s nice that the membrane for the buttons bears against the frame, not straight against the PCB. This is nice industrial design.

The button membrane is connected by a zebra strip to the PCB. The PCB itself looks nice and clean, and the zebra strip is OK, but the contacts on the membrane look tarnished.

Img_0012

I wanted to test the membrane using the continuity check function of my trusty Fluke 12…oh, hang on, it’s in pieces. Break out the equally trusty Avo 8.

Img_0013

The membrane itself works fine. I cleaned up the contacts using DeoxIT D5 on a piece of paper. I also cleaned all the plastic parts, including the holster, in the office sink with washing-up liquid. Here’s the result, showing resistance mode to prove that the buttons work.

Img_0014

Looking good, working as well as it did when new, and ready for the next 20 years.

Calculating days of the week on a Propeller

I’m doing a project using a Parallax Propeller microcontroller at the moment. It’s like a normal microcontroller, but psychedelic. Eight cores, peripherals defined almost entirely in software, and primarily programmed in a cunning language called Spin. Spin is a very efficient interpreted language and makes it possible to squeeze a lot into the Propeller’s meagre 32K of RAM.

The project has a real-time clock maintained by an off-the-shelf clock chip, but this application needs to know the day of the week as well. Of course, I can get the user to set the day of the week when they set the clock, but it’s unintuitive to have to set it manually, and this thing is supposed to be easy to use.

IMG_0006

I looked in to algorithms for calculating the day of the week given a year, month and date. There are loads of them. The Wikipedia page has lots, and this Stackoverflow page has a load of useful suggestions. There was one particularly simple one which appeared in both places, credited to Tomohiko Sakamoto on the comp.lang.c Usenet newsgroup in 1993. I don’t care about the theory behind the algorithm, I just want it to work and to be simple to program in Spin. The original is in C and looks like this:

dayofweek(y, m, d)	/* 1 <= m <= 12,  y > 1752 (in the U.K.) */
    {
        static int t[] = {0, 3, 2, 5, 0, 3, 5, 1, 4, 6, 2, 4};
        y -= m < 3;
        return (y + y/4 - y/100 + y/400 + t[m-1] + d) % 7;
    }

My Spin version looks like this:

DAT
 dayTable BYTE 0, 3, 2, 5, 0, 3, 5, 1, 4, 6, 2, 4

PRI decFromBCD(n)
 return (n&$f)+10*((n&$f0)>>4)

PRI calcDayFromDate | y
 y := decFromBCD(year) + 2000
 if month < 3
   y--

 day := (y + (y/4) - (y/100) + (y/400)
       + dayTable[decFromBCD(month)-1]
       + decFromBCD(date)) // 7

Mine looks more complex, and the one reason for that is that the clock chip maintains its counts in BCD (binary-coded decimal) rather than straight decimal. There might be a cunning way to modify the algorithm to take BCD values directly, but I took the easy route of implementing a function which converts an 8-bit BCD value into decimal.

The code assumes the presence of three variables: year (0 to 99, starting at the year 2000), month (1 to 12) and date (1 to 31) and sets a variable day in the range 0 to 6, where 0 is Sunday.

When implementing the clock in your project, don’t forget to compensate for the leap second tonight.