chroot shenanigans: Running Arch Linux on OpenWRT (LEDE) routers

Here's some notes on how to get Arch Linux running on OpenWRT devices. I'm using an Inteno IOPSYS (OpenWRT-based) DG400 for this, which has a Broadcom BCM963138 SoC - reportedly ARMv7 but not really (I'll get to that later).

I figured it would be fun trying to run Arch on such an unconventional device. I ran into 3 issues which I will be discussing, and the workarounds for them.

I've already "hacked" my router and have direct root access to the system, so I won't be discussing that in this post. If you're interested, check out any of my older posts with a CVE label for more information, or if you're brave and want to compile and flash custom firmware on your Inteno router, check out this post.

I used the lovely Arch Linux ARM community project as the basis for this. The plan of action: Grab a tarball of a compiled system for my architecture (ARMv7), extract it on the router and use chroot to effectively "run" it as if it was the root filesystem. Seems simple enough.

Issue 1: Space

These sort of devices are usually built with very limited storage to keep production costs down. The firmware just about fits on the onboard flash with some extra space for temporary files. It's not meant to be used as your conventional system.

df -h reported my root filesystem to only have 304 Kb of available space, and my tmp filesystem to have 100 Mb. Considering that the Arch tarball itself is already over 500 Mb, the device doesn't have nearly enough space to fit another OS on it.

The solution for this is quite simple: Use a USB drive. Indeed, my DG400 router has a USB2.0 and 3.0 port presumably for sticking pen drives into them. Evidently, seeing as any drives inserted are automatically mounted in /mnt (I'm unsure whether this is done by OpenWRT by default or if it's an IOPSYS feature).

It's settled then. I used my PC to format a pen drive as ext4 (FAT won't work for this very well), downloaded the ARMv7 tarball and extracted it onto the pen drive:

# umount /dev/sdc1 # (replace with your USB drive)
# mkfs.ext4 /dev/sdc1
# mount /dev/sdc1 /mnt
# mkdir /mnt/archfs
# wget http://os.archlinuxarm.org/os/ArchLinuxARM-armv7-latest.tar.gz
# bsdtar -xpf ArchLinuxARM-armv7-latest.tar.gz -C /mnt/archfs

Done. After plugging the USB drive into the router, it got automatically mounted at /mnt/usb0 (might differ). However, it got mounted with the noexec flag, which will prevent executables being run. It's easy enough to remount it. On the router:

# mount /mnt/usb0 -o exec,remount

Great! It's time to test if we can now actually chroot into it:

# chroot /mnt/usb0/archfs /bin/bash
Illegal instruction (core dumped)

Uh oh. Looks like something is still wrong. Which brings us to…

Issue 2: Not all ARM is created equal

Looks like we're running into some instructions while running bash that our processor doesn't support. Let's see if we're still ARMv7 and I hadn't messed up:

# cat /proc/cpuinfo 
processor       : 0
model name      : ARMv7 Processor rev 1 (v7l)
BogoMIPS        : 1325.05
Features        : half thumb fastmult edsp tls 
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x4
CPU part        : 0xc09
CPU revision    : 1

Strange. We're using the ARMv7 tarball, it should all be groovy. My custom firmware is compiled with GDB, which I could use to see exactly which instruction it's failing on. Since there's no way of running GDB + any of my Arch binaries natively without library mismatches, I opted to simply grab the core dump and use that instead. I looked into /proc/sys/kernel/core_pattern to identify the script responsible for handling coredumps and modified it to dump it to the root of my USB stick instead. I could then use GDB to look through the backtrace:

# gdb /mnt/usb0/archfs/bin/grep /mnt/usb0/coredump -q
Reading symbols from archfs/bin/grep...(no debugging symbols found)...done.
[New LWP 14713]

warning: Could not load shared library symbols for /lib/ld-linux-armhf.so.3.
Do you need "set solib-search-path" or "set sysroot"?
Core was generated by `/bin/grep'.
Program terminated with signal SIGILL, Illegal instruction.
#0  0xb6fe5ba4 in ?? ()

I needed to set the proper sysroot as well, to fetch proper library symbols:

(gdb) set sysroot /mnt/usb0/archfs/
Reading symbols from /mnt/usb0/archfs/lib/ld-linux-armhf.so.3...(no debugging symbols found)...done.
(gdb) disas 0xb6fe5ba4
Dump of assembler code for function __sigsetjmp:
   0xb6fe5b70 <+0>:	movw	r12, #28028	; 0x6d7c
   0xb6fe5b74 <+4>:	movt	r12, #1
   0xb6fe5b78 <+8>:	ldr	r2, [pc, r12]
   0xb6fe5b7c <+12>:	mov	r12, r0
   0xb6fe5b80 <+16>:	mov	r3, sp
   0xb6fe5b84 <+20>:	eor	r3, r3, r2
   0xb6fe5b88 <+24>:	str	r3, [r12], #4
   0xb6fe5b8c <+28>:	eor	r3, lr, r2
   0xb6fe5b90 <+32>:	str	r3, [r12], #4
   0xb6fe5b94 <+36>:	stmia	r12!, {r4, r5, r6, r7, r8, r9, r10, r11}
   0xb6fe5b98 <+40>:	movw	r3, #28064	; 0x6da0
   0xb6fe5b9c <+44>:	movt	r3, #1
   0xb6fe5ba0 <+48>:	ldr	r2, [pc, r3]
=> 0xb6fe5ba4 <+52>:	vstmia	r12!, {d8-d15}
   0xb6fe5ba8 <+56>:	tst	r2, #512	; 0x200
   0xb6fe5bac <+60>:	beq	0xb6fe5bc8 <__sigsetjmp+88>
   0xb6fe5bb0 <+64>:	stfp	f2, [r12], #8
   0xb6fe5bb4 <+68>:	stfp	f3, [r12], #8
   0xb6fe5bb8 <+72>:	stfp	f4, [r12], #8
   0xb6fe5bbc <+76>:	stfp	f5, [r12], #8
   0xb6fe5bc0 <+80>:	stfp	f6, [r12], #8
   0xb6fe5bc4 <+84>:	stfp	f7, [r12], #8
   0xb6fe5bc8 <+88>:	b	0xb6fe39d8 <__sigjmp_save>
End of assembler dump.

Looks like our processor didn't like the vstmia instruction. Can't imagine why - it seems to be a valid ARMv7 instruction.

After reading through some reference manuals and consulting others online, it turned out that my SoC processor is crippled: A set of instructions simply wasn't supported by my processor. Luckily, thanks to those instructions not existing in ARMv5 and ARM being backwards-compatible, I could simply use the ARMv5-compiled system instead.

Repeating the steps to create the root filesystem, this time using the ArchLinuxARM-armv5-latest.tar.gz tarball instead, showed promising results. I could finally:

# chroot /mnt/usb0/archfs /bin/bash
[root@iopsys /]# cat /etc/os-release
NAME="Arch Linux ARM"
PRETTY_NAME="Arch Linux ARM"
ID=archarm

I exited the chroot after seeing it works. We still needed to mount some partitions so the chroot could see and interact with them and copy some files over. I wrote a helper script for all of that which you can find here.

Great, we can now initialise pacman and try upgrading the system.

# pacman-key --init
# pacman-key --populate archlinuxarm
# pacman -Syu

error: out of memory

Issue 3: Memory problems

Honestly, should've seen this one coming. free -m showed that I was working with around 100 Mb of usable memory, which is not much - no wonder pacman crapped out. Luckily, my device kernel was compiled with swap support. This essentially allows the system to "swap" memory contents out to the filesystem and load them later when necessary. It's very slow compared to real memory, but it gets the job done in a pinch. I created a 1G swapfile on my USB drive and activated it, whilst inside the chroot:

# truncate -s 0   /swapfile
# chattr +C       /swapfile
# fallocate -l 1G /swapfile
# chmod 600       /swapfile
# mkswap          /swapfile
# swapon          /swapfile

Running pacman again allowed me to continue upgrading the system, which it finished successfully.

At this point, I had a fully functional Arch Linux system which I could chroot into and utilise pretty much to the maximum. I've successfully set up Python bots, compiled software with gcc/g++, etc. what you'd expect to see from a normal system. I don't know why you would want to do this, but it's definitely possible.

I realise that it may not go this smoothly on other systems. For example, a large portion of routers utilise the MIPS architecture instead of ARM. If this is the case for you, it unfortunately means that Arch Linux is off the table, as it doesn't have any functioning MIPS builds. However, the Debian community maintains an active MIPS port of Debian which you might want to look into instead. Everything in this post should still pretty much apply to Debian/MIPS as well, with some minor differences.

This has also been done on other unconventional devices. Reddit user parkerlreed used a similar procedure to run Arch Linux on a Steamlink, which you can read here - it even has instructions on how to compile applications natively on it.