domingo, 7 de mayo de 2017

Fuzzing Apache httpd server with American Fuzzy Lop + persistent mode



When stumbling upon the great American Fuzzy Lop and trying its awesome deterministic fuzzing capabilities and instrumentation soon we find out that this fuzzer was built to fuzz programs that take input from the command line (or standard input)  instead of a network socket. Because of that, I thought I would give it a try and make AFL fuzz against Apache's httpd server. First the AFL way by adding a new option to Apache's command line and the second way, by using the persistence fuzzing (afl-clang-fast) by shamelessly copying the way Robert Swiecky fuzzes Apache with honggfuzz.

Takeaways for the reader

  • Learn to fuzz network based programs with AFL
  • Code to start fuzzing Apache with AFL in no time
  • Testcases scrapped from wikipedia
  • A push in the interwebz fuzzing race
Let's do it!

Setup part 1

I will be using a Debian GNU/Linux 8 64bit with the kernel 4.9.0-0.bpo.2-rt-amd64. You don't really need that setup, this can be done on Ubuntu as well. All that you need is an operating system (under a virtual machine or not) that can compile and run AFL with the afl-clang feature.

But, before getting into any compilations/installations/fuzzing, I encourage you to set an organised folder structure that suits you best but, in case you haven't got one already, I am sharing mine.
Under the Fuzzing folder I have:
  • Victims - For the target programs that we are about to fuzz
  • Fuzzers - AFL, honggfuzz, radamsa, etc. go here
  • Testcases  - The samples we are going to feed the fuzzer to throw against our Victim
  • Sessions - For storing the fuzzing sessions
  • Compilers - To store compilers such as clang-4.0 and binaries needed to compile

Getting clang-4.0 and llvm-tools

Getting pre-built binaries for clang-4.0 and the llvm-tools is fairly easy if you have Debian or Ubuntu. You can get these from here In my case the clang+llvm-4.0.0-x86_64-linux-gnu-debian8.tar.xz tarball.
If you are following the structure mentioned above, you can cd into your Compilers folder and drop the tarball there, extract it and then add the binaries folder to your path by adding the following line to the end of your ~/.bashrc file (~/.profile nor /etc/environment worked for me - It seems that you need to logout and login for these changes to take place).
Now issuing the which command on a new shell we should have the following output:
which clang

Compiling and Installing AFL

Compiling AFL should be pretty straight forward but, for the lazy, you can just copy paste these commands and you should be ready to go:
sudo apt install build-essential
tar -xzf afl-latest.tgz
cd afl*
make && sudo make install && 
echo -e "\n[+] AFL ready to fuzz at $(which afl-fuzz)"

That's it, the binary afl-fuzz should be in your path now ready to be unleashed.

Compiling and installing Apache

First move to a folder where we are about to download all the dependencies needed by Apache and apache itself. In my case the folder is at ~/Fuzzing/Victims/apache_afl/.

Before compiling Apache we are going to need the Apache Portable Runtime (APR), APR Utils and support for HTTP/2 through nghttp2.
No lazyness this time, go download:
Now we need to get the latest Apache build, which I recommend you do from their subversion repository by doing so:
sudo apt install subversion
svn checkout httpd-2.4.x
Now if you downloaded and unpacked everything, you should have a similiar output from ls -l command:
drwxr-xr-x 28 javier javier  4096 Apr  9 23:12 apr-1.5.2
drwxr-xr-x 20 javier javier  4096 Apr  9 23:13 apr-util-1.5.4
-rwxr-xr-x  1 javier javier  1445 May  6 20:56
drwxr-xr-x 11 javier javier  4096 Apr 29 01:22 httpd-2.4.x
drwxr-xr-x 14 javier javier  4096 Apr  9 23:14 nghttp2-1.21.0
drwxr-xr-x  9 javier javier 12288 Apr  9 22:53 pcre-8.40

I made the following file to compile and link it all since I found myself often changing flags for the compiler and it was too time consuming compiling each dependency one by one with its own flags. Get it with:

Before editing any files yet lets run the bash script and see that we can compile everything cleanly without any missing dependencies whatsoever:

CC="clang" CXX="clang++" PREFIX="/usr/local/apache_clean_test/" ./
Please see the next asciinema for reference of a nice compilation run.

Fuzzing Apache with AFL through an input file

As you might know by now, AFL in its basic usage feeds a file into the target program through its "argv" array in the following form:
afl-fuzz -i testcases/ -o session_1/ -m none -t 2000 -- ./victim -d -v -f @@
The problem with Apache is that it doesn't have such functionality so we will have to patch it our own way.

Patching Apache ... Apatching

Taking into account the aforementioned problem, we need to write some lines into Apache's main.c file to make it able to read files from input.
You can patch Apache with the following patch file here. Now apply it by cd'ing into the base path of Apache httpd's source code and issuing the following command:
patch -p0 -i apatching_apache_for_AFL_fuzzing.diff
I am not going to cover all the patch in detail but some parts are worth mentioning.

The first and only time that I have seen the following technique was by Robert Swiecky, an information security researcher at Google when fuzzing Apache with honggfuzz. It is pretty clever and pretty obvious once you see the way it is done. It basically consists of launching a new thread inside Apache that will create a connection to the web server itself and send our fuzzed input;  all happening within the same unique process so we can get all the instrumentation data into AFL. Clever! Right?
To achieve this it uses the unshare function that disassociates parts of this thread's context from the others without the need of creating a new process. Specifically, the network and  mount namespaces are separated. This is done so we can have several processes with the same settings (listening on the same loopback interface and port with the help of netIfaceUp on line 44 of the patch file and writing logs to /tmp on line 75) running at the same time on each process we launch.

We can see that the unshare function is indirectly called on line 188 previous to firing the new thread that will receive the fuzzed input at line 189.

The process of reading a file through the "-F" switch starts on line 156 and when the file is read into a buffer, this buffer is passed onto the function responsible to launch the new thread (189) that will, in turn, send the fuzzed file inside the SENDFILE function on line 119.

Fuzzing Apache ... Fapache

Yes! We are ready now! Let's compile Apache:
CC="afl-clang" CXX="afl-clang++" PREFIX="/usr/local/apache_afl_blogpost/" ./
If you are familiar with AFL and how it works you probably have your own testcases to feed it with, in case you don't the following video shows how to launch AFL and create two very simple test cases - remember that we need to be root in order to use the unshare function AND MORE IMPORTANT TO LAUNCH APACHE WITH THE "-X" FLAG AND "-m none -t 5000" FOR AFL SO IT CAN BOOT APACHE:
Well, that was not too fast, was it? 5 execs per second on my laptop... how can we speed things up a bit?

Setup part 2

Compiling afl-clang-fast

Remember we downloaded clang-4.0 and the llvm-tools before and set it in our path? This is where it comes most handy. Inside your AFL folder, navigate to the llvm_mode and run make and sudo make install in the root folder of AFL. What we have just done is compiling an experimental feature of AFL that will run a certain number of fuzzed inputs against a program without having to run the whole program per fuzzing input.

Patching Apache with Persistence ... Apatchistence

Following the same dance as before, download this patch, patch it and ready to fuzz!

Fuzzing Apache with AFL on Persistent mode

Let the video speak for itself but again remember the previously mentioned "-X" flag for Apache server and the "-m none -t 5000" flags for AFL


As pointed by Robert himself, you don't need to run everything as root as I did in the examples which, obviously imposes security risks. You can make use of the following command line (this one to be run as root :P) to let non-root users make use of the unshare function:
sysctl -w kernel.unprivileged_userns_clone=1


We have learned how to effectively fuzz server programs such as web servers by using Robert's technique of launching a different thread and different context through the unshare function.
While relatively fast, it is not as fast as honggfuzz, which can go up to 20k iterations per second with 8 processes running. Also after a few days of fuzzing AFL's stability goes way down 50% because of the multithreading that Apache is implemented on and so, any reported crashes, can be false positives or would either be needing the last iterations launched which AFL lacks of.

It is left as an exercise to the reader to implement into Apache a way to save the last 1k sent inputs into a file and to think which other ways would improve stability and/or speed. Hint: Don't instrument everything.

Thanks for your time! In the meantime you can check the post I did on SensePost's blog about ptmalloc2 implementation basics.

13 comentarios:

  1. Pretty sure changes to /etc/environment require a logoff

  2. I have seen a lot of blogs and Info. on other Blogs and Web sites But in this Hadoop Blog Information is useful very thanks for sharing it........

    1. My first spam comment, YAY!

      Further comments like this one will be removed :)

  3. Este comentario ha sido eliminado por el autor.

  4. Hi,but I have trouble in running httpd,some errors are as follow:
    [+] Stdin file (null)
    [-] Couldn't fopen

    So,could you help me or give me some idea? Thanks a lot!

    1. Try to use the "strace" command and see what is happening at the time of reading the file.

  5. Hi,

    Great post! But when I try to start the second httpd instance (the first one is already running), I receive :
    httpd (pid 15951) already running

    Did I forget something ?

    Thanks in advance.

    1. Indeed! I do set a different file name in each config file, and use a different config file for each fuzzer.
      Also, this is actually a key point to fuzz Apache as, depending on the configuration file you set, AFL will also cover different parts of code (e.g. enable diferent Apache mods)

  6. So when I launch: afl-fuzz -i testcases/ -o sess_1/ -m none -t 2000 -- /usr/local/apache_afl/bin/httpd -X -F @@
    I get a message saying the target binary is not instrumented. If I follow your instructions (which are quite confusing and they get even worse on the sunspots blog) apache is never built with afl-clang since your CC=afl-clang before running your script is ignored.
    Do what is the trick?

    1. I found on some Linux Distros (such as Ubuntu) if you run:
      CC=afl-clang make

      It won't get the ENV variable (this might be due to several things) and found that, if you swap the ENV variables like so:
      make CC=afl-clang

      It will work. Haven't figured why yet but, please do tell if you came up with a solution!

  7. I'm trying the two examples: Fapache and Apachistence, and both experienced such warnings:
    > Some test cases look useless. Consider using a smaller set.

    Also, I could execute ~3000 tests per second, but the coverage is poor:
    > last new path : none yet (odd, check syntax!)

    I've noticed that each cycle takes only a few seconds, and am wondering which part I should fix.
    My repo is: , whose subdirectories contain the corresponding `Dockerfile`s.

    1. I am afraid that you will have to be a little bit more specific. It seems that either you aren't instrumenting Apache properly or the fuzzer is straight not processing the inputs and sending these to the Apache process. This could be by a variety of reasons, from buggy code (bear in mind Apache and AFL have changed in the last year) to memory issues such as the "-m" option.
      Furthermore, if you are using a Docker image, you might need the --privileged switch for Docker.