Intro
Goal
When stumbling upon the great American Fuzzy Lop and trying its awesome deterministic fuzzing capabilities and instrumentation soon we find out that this fuzzer was built to fuzz programs that take input from the command line (or standard input) instead of a network socket. Because of that, I thought I would give it a try and make AFL fuzz against Apache's httpd server. First the AFL way by adding a new option to Apache's command line and the second way, by using the persistence fuzzing (afl-clang-fast) by shamelessly copying the way Robert Swiecky fuzzes Apache with honggfuzz.
Takeaways for the reader
- Learn to fuzz network based programs with AFL
- Code to start fuzzing Apache with AFL in no time
- Testcases scrapped from wikipedia
- A push in the interwebz fuzzing race
Let's do it!
Setup part 1
I will be using a Debian GNU/Linux 8 64bit with the kernel 4.9.0-0.bpo.2-rt-amd64. You don't really need that setup, this can be done on Ubuntu as well. All that you need is an operating system (under a virtual machine or not) that can compile and run AFL with the afl-clang feature.
But, before getting into any compilations/installations/fuzzing, I encourage you to set an organised folder structure that suits you best but, in case you haven't got one already, I am sharing mine.
But, before getting into any compilations/installations/fuzzing, I encourage you to set an organised folder structure that suits you best but, in case you haven't got one already, I am sharing mine.
Under the Fuzzing folder I have:
- Victims - For the target programs that we are about to fuzz
- Fuzzers - AFL, honggfuzz, radamsa, etc. go here
- Testcases - The samples we are going to feed the fuzzer to throw against our Victim
- Sessions - For storing the fuzzing sessions
- Compilers - To store compilers such as clang-4.0 and binaries needed to compile
Getting clang-4.0 and llvm-tools
Getting pre-built binaries for clang-4.0 and the llvm-tools is fairly easy if you have Debian or Ubuntu. You can get these from here http://releases.llvm.org/download.html. In my case the clang+llvm-4.0.0-x86_64-linux-gnu-debian8.tar.xz tarball.
If you are following the structure mentioned above, you can cd into your Compilers folder and drop the tarball there, extract it and then add the binaries folder to your path by adding the following line to the end of your ~/.bashrc file (~/.profile nor /etc/environment worked for me - It seems that you need to logout and login for these changes to take place).
PATH="$HOME/Fuzzing/Compilers/clang+llvm-4.0.0-x86_64-linux-gnu-debian8/bin:$PATH"
Now issuing the which command on a new shell we should have the following output:
which clang /home/javier/Fuzzing/Compilers/llvm-clang-binaries/clang+llvm-4.0.0-x86_64-linux-gnu-debian8/bin/clang
Compiling and Installing AFL
Compiling AFL should be pretty straight forward but, for the lazy, you can just copy paste these commands and you should be ready to go:sudo apt install build-essential wget http://lcamtuf.coredump.cx/afl/releases/afl-latest.tgz tar -xzf afl-latest.tgz cd afl* make && sudo make install && echo -e "\n[+] AFL ready to fuzz at $(which afl-fuzz)"
That's it, the binary afl-fuzz should be in your path now ready to be unleashed.
Before compiling Apache we are going to need the Apache Portable Runtime (APR), APR Utils and support for HTTP/2 through nghttp2.
No lazyness this time, go download:
Compiling and installing Apache
First move to a folder where we are about to download all the dependencies needed by Apache and apache itself. In my case the folder is at ~/Fuzzing/Victims/apache_afl/.Before compiling Apache we are going to need the Apache Portable Runtime (APR), APR Utils and support for HTTP/2 through nghttp2.
No lazyness this time, go download:
- APR and APR Util
- nghttp
- The following dependencies:
- sudo apt install pkg-config
- sudo apt install libssl-dev
- The latest pcre-8.x (DON'T DOWNLOAD PCRE2!!)
sudo apt install subversion svn checkout http://svn.apache.org/repos/asf/httpd/httpd/branches/2.4.x httpd-2.4.xNow if you downloaded and unpacked everything, you should have a similiar output from ls -l command:
drwxr-xr-x 28 javier javier 4096 Apr 9 23:12 apr-1.5.2 drwxr-xr-x 20 javier javier 4096 Apr 9 23:13 apr-util-1.5.4 -rwxr-xr-x 1 javier javier 1445 May 6 20:56 compile_httpd_with_flags.sh drwxr-xr-x 11 javier javier 4096 Apr 29 01:22 httpd-2.4.x drwxr-xr-x 14 javier javier 4096 Apr 9 23:14 nghttp2-1.21.0 drwxr-xr-x 9 javier javier 12288 Apr 9 22:53 pcre-8.40
I made the following file to compile and link it all since I found myself often changing flags for the compiler and it was too time consuming compiling each dependency one by one with its own flags. Get it with:
wget https://gist.githubusercontent.com/n30m1nd/14418fd425a3b2d14b64650710fae301/raw/e1cff738eb1ffaa55cb8a1a66bb1a2b06ed7f97e/compile_httpd_with_flags.sh
Before editing any files yet lets run the bash script and see that we can compile everything cleanly without any missing dependencies whatsoever:
wget https://gist.githubusercontent.com/n30m1nd/14418fd425a3b2d14b64650710fae301/raw/e1cff738eb1ffaa55cb8a1a66bb1a2b06ed7f97e/compile_httpd_with_flags.sh
Before editing any files yet lets run the bash script and see that we can compile everything cleanly without any missing dependencies whatsoever:
CC="clang" CXX="clang++" PREFIX="/usr/local/apache_clean_test/" ./compile_httpd_with_flags.shPlease see the next asciinema for reference of a nice compilation run.
Fuzzing Apache with AFL through an input file
As you might know by now, AFL in its basic usage feeds a file into the target program through its "argv" array in the following form:afl-fuzz -i testcases/ -o session_1/ -m none -t 2000 -- ./victim -d -v -f @@The problem with Apache is that it doesn't have such functionality so we will have to patch it our own way.
Patching Apache ... Apatching
Taking into account the aforementioned problem, we need to write some lines into Apache's main.c file to make it able to read files from input.
You can patch Apache with the following patch file here. Now apply it by cd'ing into the base path of Apache httpd's source code and issuing the following command:
The first and only time that I have seen the following technique was by Robert Swiecky, an information security researcher at Google when fuzzing Apache with honggfuzz. It is pretty clever and pretty obvious once you see the way it is done. It basically consists of launching a new thread inside Apache that will create a connection to the web server itself and send our fuzzed input; all happening within the same unique process so we can get all the instrumentation data into AFL. Clever! Right?
To achieve this it uses the unshare function that disassociates parts of this thread's context from the others without the need of creating a new process. Specifically, the network and mount namespaces are separated. This is done so we can have several processes with the same settings (listening on the same loopback interface and port with the help of netIfaceUp on line 44 of the patch file and writing logs to /tmp on line 75) running at the same time on each process we launch.
We can see that the unshare function is indirectly called on line 188 previous to firing the new thread that will receive the fuzzed input at line 189.
The process of reading a file through the "-F" switch starts on line 156 and when the file is read into a buffer, this buffer is passed onto the function responsible to launch the new thread (189) that will, in turn, send the fuzzed file inside the SENDFILE function on line 119.
You can patch Apache with the following patch file here. Now apply it by cd'ing into the base path of Apache httpd's source code and issuing the following command:
patch -p0 -i apatching_apache_for_AFL_fuzzing.diffI am not going to cover all the patch in detail but some parts are worth mentioning.
The first and only time that I have seen the following technique was by Robert Swiecky, an information security researcher at Google when fuzzing Apache with honggfuzz. It is pretty clever and pretty obvious once you see the way it is done. It basically consists of launching a new thread inside Apache that will create a connection to the web server itself and send our fuzzed input; all happening within the same unique process so we can get all the instrumentation data into AFL. Clever! Right?
To achieve this it uses the unshare function that disassociates parts of this thread's context from the others without the need of creating a new process. Specifically, the network and mount namespaces are separated. This is done so we can have several processes with the same settings (listening on the same loopback interface and port with the help of netIfaceUp on line 44 of the patch file and writing logs to /tmp on line 75) running at the same time on each process we launch.
We can see that the unshare function is indirectly called on line 188 previous to firing the new thread that will receive the fuzzed input at line 189.
The process of reading a file through the "-F" switch starts on line 156 and when the file is read into a buffer, this buffer is passed onto the function responsible to launch the new thread (189) that will, in turn, send the fuzzed file inside the SENDFILE function on line 119.
Fuzzing Apache ... Fapache
Yes! We are ready now! Let's compile Apache:
CC="afl-clang" CXX="afl-clang++" PREFIX="/usr/local/apache_afl_blogpost/" ./compile_httpd_with_flags.sh
If you are familiar with AFL and how it works you probably have your own testcases to feed it with, in case you don't the following video shows how to launch AFL and create two very simple test cases - remember that we need to be root in order to use the unshare function AND MORE IMPORTANT TO LAUNCH APACHE WITH THE "-X" FLAG AND "-m none -t 5000" FOR AFL SO IT CAN BOOT APACHE:
Well, that was not too fast, was it? 5 execs per second on my laptop... how can we speed things up a bit?
Setup part 2
Compiling afl-clang-fast
Remember we downloaded clang-4.0 and the llvm-tools before and set it in our path? This is where it comes most handy. Inside your AFL folder, navigate to the llvm_mode and run make and sudo make install in the root folder of AFL. What we have just done is compiling an experimental feature of AFL that will run a certain number of fuzzed inputs against a program without having to run the whole program per fuzzing input.
Patching Apache with Persistence ... Apatchistence
Following the same dance as before, download this patch, patch it and ready to fuzz!Fuzzing Apache with AFL on Persistent mode
Let the video speak for itself but again remember the previously mentioned "-X" flag for Apache server and the "-m none -t 5000" flags for AFL
Update:
As pointed by Robert himself, you don't need to run everything as root as I did in the examples which, obviously imposes security risks. You can make use of the following command line (this one to be run as root :P) to let non-root users make use of the unshare function:sysctl -w kernel.unprivileged_userns_clone=1
Conclusions
We have learned how to effectively fuzz server programs such as web servers by using Robert's technique of launching a different thread and different context through the unshare function.
While relatively fast, it is not as fast as honggfuzz, which can go up to 20k iterations per second with 8 processes running. Also after a few days of fuzzing AFL's stability goes way down 50% because of the multithreading that Apache is implemented on and so, any reported crashes, can be false positives or would either be needing the last iterations launched which AFL lacks of.
It is left as an exercise to the reader to implement into Apache a way to save the last 1k sent inputs into a file and to think which other ways would improve stability and/or speed. Hint: Don't instrument everything.
Thanks for your time! In the meantime you can check the post I did on SensePost's blog about ptmalloc2 implementation basics.
