The Sequoia seq_file vulnerability

A local root hole in the Linux kernel, called Sequoia, was disclosed by Qualys on July 20. A full system compromise is possible until the kernel is patched (or mitigations that may not be fully effective are applied). At its core, the vulnerability relies on a path through the kernel where 64-bit size_t values are "converted" to signed integers, which effectively results in an overflow. The flaw was reported to Red Hat on June 9, along with a local systemd denial-of-service vulnerability, leading to a kernel crash, found at the same time. Systems with untrusted local users need updates for both problems applied as soon as they are available—out of an abundance of caution, other systems likely should be updated as well.

Down in the guts of the kernel's seq_file interface, which is used for handling virtual files in /proc and the like, buffers are needed to store each line of the file's "contents". To start, a page of memory is allocated for the buffer, but if that is not sufficient, a new buffer that is twice the size of the old one is allocated. This is all done using a size_t, which is an unsigned 64-bit quantity (on x86_64) that is large enough to hold the results, so "the system would run out of memory long before this multiplication overflows".

But that value (m->size in the advisory) is passed to other functions that expect a signed 32-bit integer. In particular, the exploit uses the output of /proc/self/mountinfo to get to a place in the kernel where that is the case. The attacker can create a directory hierarchy with a path length larger than 1GB (roughly one-million nested directories), bind-mount it into a user namespace, and then delete the directory. In the namespace, the attacker opens and reads mountinfo, which causes the string "//deleted" to be written outside of the seq_file buffer. The seq_file interface creates a buffer that is 2GB in size, but down in the code that writes the string, that value gets interpreted as -2GB, which is used to calculate where to do the write. That is far outside the buffer, of course, and so it overwrites memory at a known offset elsewhere in the vmalloc() region.

As might be guessed, writing a ten-byte fixed string outside of the buffer is only the start of a complicated, but easily replicated, series of gyrations leading to a root shell. The Qualys advisory goes into great detail about the journey, which involves several different kernel features that have come about over the last decade or so: user namespaces, BPF, and user-space page fault handling (or FUSE, which is much older). But in the end, it is the mishandling of the integer buffer size that gets the foot in the door; the fix is something of a band-aid to simply reject seq_buf allocations that get "too large".

Qualys said that it has exploit code that can gain root on "default installations of Ubuntu 20.04, Ubuntu 20.10, Ubuntu 21.04, Debian 11, and Fedora 34 Workstation", furthermore "other Linux distributions are certainly vulnerable, and probably exploitable". One of the suggested mitigations is turning off the ability for unprivileged users to create user namespaces (via /proc/sys/kernel/unprivileged_userns_clone), because that will stop their ability to mount the deep directory hierarchy that results in the 1GB+ path name. But even without user namespaces, there may be an alternative:

However, the attacker may mount a long directory via FUSE instead; we have not fully explored this possibility, because we accidentally stumbled upon CVE-2021-33910 in systemd: if an attacker FUSE-mounts a long directory (longer than 8MB), then systemd exhausts its stack, crashes, and therefore crashes the entire operating system (a kernel panic).

The systemd bug, which Qualys also describes well, stems from switching from strdup() to strdupa() in the mount-path handling code. That switched from heap-based allocations to stack-based ones, which provides the opportunity to exhaust the 8MB (by default) stack. So an overlong path that is found when systemd is parsing /proc/self/mountinfo will result in a segmentation fault; Linux is decidedly unhappy about running without an init process, so it panics as well.

A user can trigger the crash by mounting a FUSE filesystem, creating a directory path longer than 8MB, moving the FUSE filesystem to that directory, and causing systemd to parse the mountinfo file again. The fix in this case is to switch back to using strdup() as it was before an April 2015 commit.

The systemd crash was found while developing Sequoia exploit, which uses BPF code to find and then overwrite the modprobe_path variable in the kernel. That path is used to run an executable as root (normally /sbin/modprobe) when a kernel module gets loaded. Pointing that at a different path, where an attacker-controlled executable lives, gives root privileges.

One would expect the BPF verifier to thwart any attempts to execute a harmful program, since that is the primary mechanism to restrict users from loading unsafe BPF programs. But that is where user-space page-fault handling comes into play: the userfaultfd() system call allows the exploit to effectively pause the kernel after the BPF verifier has been run. Then the overwrite of "//deleted" can be arranged to land at the "right" place in the BPF code, which is loaded into the vmalloc() region. userfaultfd() allows the exploit to change the BPF code after it has been verified but before it gets JIT-compiled, thus evading the verifier.

The out-of-bounds write of "//deleted" is turned into an information disclosure that allows the exploit code to have limited control over what gets overwritten. After that, the techniques from a 2020 BPF vulnerability found by Manfred Paul were used to "transform this limited out-of-bounds write into an arbitrary read and write of kernel memory" The six-step "Exploitation overview" section of the advisory gives a nice overview of the exploit, while the following section gives all the gory details. It makes for interesting reading for those who are curious about kernel exploits and the intricate steps that are needed to make them work.

The reports of both flaws were in the hands of Red Hat for nearly a month until they were reported to the closed mailing lists for kernel and distribution security reports on July 6. Two weeks after that, the coordinated release of the advisories was made and distribution updates started rolling out. It is not at all clear why there was a month-long delay, since neither of the fixes seems particularly challenging. Maybe that time was spent looking for other similar problems in the kernel and systemd.

Clearly the integer conversion at the heart of the exploit needed fixing; one wonders how many other size_t-to-int problems of that sort still linger in the kernel. One might also wonder what kind of a can of worms was opened when userfaultfd() was added to the kernel; it provides a way for user space to semi-arbitrarily pause the kernel in spots of its choosing. That may come in handy again for kernel exploits down the road.

Meanwhile, systemd has hopefully scrutinized any other stack-based allocations it is doing, especially for user-controllable values like paths. While exhausting the stack is often not a security problem for user-space programs (except, of course, for exploits like Stack Clash), systemd sits in a sensitive place in many Linux systems. Any user-controlled means to bring it down is a clear route to a denial of service on the system.

Integer-conversion problems of the sort we see here are something that would likely be impossible in some other languages (e.g. Rust). Stack (or memory) exhaustion, on the other hand, is not really something that can be handled at the language level; hitting a resource limit must be handled somehow and crashing a user-space program is often the least harmful thing the operating system can do. But at least we now have N-2 bugs in our systems; unfortunately, the unknown N is likely distressingly large.