Domain fronting through Cloudflare
Thur 21st Feb 19
Before jumping into this post I feel I've got to discuss the term "Domain Fronting" and some of the various possible definitions. If you just want to find out about hiding your comms, feel free to skip this section.
Defining "Domain Fronting"
I'm coming into this from a red team perspective, and, for me, domain fronting means hiding a request to a malicious website inside a request that looks like it is going to an innocent site. From a technical perspective, I can describe this as decoupling the network layer setup from the application layer request for a particular virtual host. If all the observer sees is the network setup going to a "good" domain (or one that isn't flagged as bad), then I win. This is backed up by the Wikipedia page on Domain fronting which describes it as obfuscating the actual domain requested to avoid it being blocked.
In this Twitter thread, Nick argues that using ESNI (I'll come on to this later) to obscure the traffic is not quite the same as domain fronting. Nick has been at this for a lot longer than I have and so could well be right and I'll be interested in his comments once this post comes out, maybe over something more than 280 character chunks.
Other people I've asked have been on the fence, saying the techniques here have the same final result as the original domain fronting concept but may not match it in a truly technical sense.
Despite all this, I'm going to stick with the term here and accept whatever flack is thrown. If you are really upset by this, you can drop this into a JavaScript console to use hide the offensive term:
document.body.innerHTML = document.body.innerHTML.replace (/domain fronting/ig, "kitten hugging")And now, our feature presentation...
The Actual Content
When I put out my 101 on domain fronting article, @sehnaoui tweeted a link to it and got this reply from Suzanne who is a "Solutions Engineer Team Lead @Cloudflare":
First, I didn't claim that it worked on Cloudflare, my only mention of Cloudflare was that it was a CDN along with a rough description of how a CDN works.
Second, here is a demonstration of domain fronting working on Cloudflare using the technique I discussed in the article.
$ curl -s -H $'Host: frontmecf.vuln-demo.com' http://digininja.org.uk
<p>Vuln Demo site fronted by Cloudflare</p>The setup for this was fairly simple, I signed up for an account with Cloudflare, added digininja.org.uk as a domain with them and completed the setup process, which involves moving the nameservers for the domain over to Cloudflare. I also added an account for vuln-demo.com but never completed the setup so the domain remains in the "Pending Nameserver Update" stage.
And in case anyone feels I'm cheating having both the domains setup in the same account, lets try another domain hosted by Cloudflare, one that should have a good reputation and if there were any extra protections available, would have them all enabled.
$ curl -s -H $'Host: frontmecf.vuln-demo.com' http://cloudflare.com
<p>Vuln Demo site fronted by Cloudflare</p>Yes, I both technically and literally just domain fronted through Cloudflare!
A quick step back to definitions, can you domain front over HTTP? I don't see why not, it is separating the request for the network setup from the request for the HTTP data and anyone who is only watching the DNS traffic will not see the real request. Is there any point to doing this over HTTP? No, none at all, as the content goes in clear text and so nothing is really hidden.
At this point, I wish I'd been satisfied and gone off to do something different, but instead, I thought I'd try it over HTTPS. That lead me down a rabbit hole of cipher suites, SNI, ESNI and packet traces and resulted in the conclusion that, depending on semantics, Suzanne was partially right in what she said, domain fronting through Cloudflare over HTTPS doesn't work using the method I'd showed. Lets have a look:
$ curl -s -H $'Host: frontmecf.vuln-demo.com' https://cloudflare.com
<html>
<head><title>403 Forbidden</title></head>
<body bgcolor="white">
<center><h1>403 Forbidden</h1></center>
<hr><center>cloudflare</center>
</body>
</html>The result was the same with digininja.org.uk which was fully setup on the Cloudflare system:
$ curl -s -H $'Host: digininja.org.uk' https://cloudflare.com
<html>
<head><title>403 Forbidden</title></head>
<body bgcolor="white">
<center><h1>403 Forbidden</h1></center>
<hr><center>cloudflare</center>
</body>
</html>It took a bit of digging, but I found that Cloudflare check the SNI field as well, as the Host header, when setting up the connection, if the two do not match, then the connection is refused. If you have not heard of SNI, here are a couple of articles describing what it is and how it works:
The TLDR; is that SNI acts in a similar way to the Host header, except rather than going in the HTTP header and telling the web server which virtual host to serve from, it goes in the Client Hello when setting up the TLS connection and is designed to allow the server to pick the right certificate to offer to the client to avoid certificate mismatch errors when there are multiple virtual hosts and certificates on the one server.
Curl takes the domain name used to setup the network connection and puts that in the SNI field. In a normal request, this will match the value in the Host header and all will be good, but in our example above, this gives a mismatch between SNI and the manually modified Host header which causes the rejection. The SNI field can be seen by examining the connection setup in Wireshark:
As far as I can tell, there is no way to modify the SNI field in curl so I decided to go lower level and try OpenSSL. From reading the docs, by default, OpenSSL does not send the SNI extension and I thought that this might be a quick way round the restriction. If Cloudflare failed open when there was no SNI extension present, then that would give an easy way to bypass the check. Let's try it:
$ cat get_digininja.org
GET / HTTP/1.1
Host: digininja.org.uk
User-Agent: front/1
Accept: */*
$ (cat get_digininja.org;sleep 5) | openssl s_client -connect www.cloudflare.com:443
<Lots of verbose setup stuff...>
---
HTTP/1.1 403 Forbidden
Server: cloudflare
Date: Tue, 19 Feb 2019 15:43:01 GMT
Content-Type: text/html
Content-Length: 167
Connection: close
CF-RAY: 4ab9d8c079cc3614-LHR
<html>
<head><title>403 Forbidden</title></head>
<body bgcolor="white">
<center><h1>403 Forbidden</h1></center>
<hr><center>cloudflare</center>
</body>
</html>No luck, it looks like Cloudflare insist on the SNI field being present.
As I said, OpenSSL does not send the SNI extension by default, but it can be told to send one by setting the "servername" parameter so my next attempt was to force the SNI name to match the one I send in the Host header:
$ (cat get_digininja.org;sleep 5) | openssl s_client -connect www.cloudflare.com:443 \
-servername digininja.org.uk
<Lots of verbose setup stuff...>
---
HTTP/1.1 200 OK
Date: Tue, 19 Feb 2019 20:35:45 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: __cfduid=dc9c79f93a9dd6cb5da6508ecdee4fa261550608545; expires=Wed, 19-Feb-20 20:35:45 GMT; path=/; domain=.digininja.org.uk; HttpOnly
X-Content-Type-Options: nosniff
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Server: cloudflare
CF-RAY: 4abb85916a2f35c0-LHR
1f
<p>digninja.org.uk on HTTP</p>
0Jackpot. By manually setting the server name in the SNI extension to match the Host header, the required conditions are met and the request is made to my site. The only problem is that the SNI field is all clear text which leaks the fact that I'm talking to digininja.org.uk and not to cloudflare.com:
From more research on this, I found that leaking this information over a connection which is designed to be encrypted, is a known issue and one that privacy advocates are not happy with. To get round this, Encrypted SNI, or ESNI for short, was created and is being included as part of the TLS 1.3 spec. It is currently in the draft state, so things may change, I'm sure the main concept will stay the same. I've not attempted to understand the crypto that goes on with this, but the basics are that some form of cryptography is used between the client and server to encrypt the SNI details. Cloudflare themselves champion this and have a good write up on how it works on their blog: Encrypt it or lose it: how encrypted SNI works.
ESNI is a very new technology and so has very little client side support and there is currently no support in any of the main branches of OpenSSL. Luckily, an OpenSSL developer called Stephen Farrell recently decided to have a go at implementing it, his design is set out in his paper OpenSSL Encrypted SNI Design and his code is available in his GitHub fork.
Building the new code was simple, there no bugs, no extra libraries were required, it just worked which was really good. Once built, all that is needed to run it is to set the shared library path, LD_LIBRARY_PATH, to point to the libraries that were just built and then it can be ran from its build directory, meaning you don't have to do any installation, perfect for this type of experimentation.
Rather than try to re-write Stephen's notes on how to run it, I'll just say that the command below works and that if you want to know what all the parameters mean, and why you set the ESNIRR variable, then refer to the design notes. In the output, I've highlighted in red the important lines which show ESNI has been correctly setup and used.
$ export LD_LIBRARY_PATH=/home/robin/src/openssl
$ ESNIRR=`dig +short txt _esni.www.cloudflare.com | sed -e 's/"//g'`
$ (cat get_digininja.org;sleep 5) | /home/robin/src/openssl/apps/openssl s_client \
-CApath /etc/ssl/certs/ -tls1_3 -connect www.cloudflare.com:443 -esni digininja.org.uk \
-esnirr $ESNIRR -esni_strict -servername www.cloudflare.com
<Lots of verbose setup stuff...>
---
SSL handshake has read 2530 bytes and written 1125 bytes
Verification: OK
Verified peername: digininja.org.uk
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 256 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
ESNI: success: cover: www.cloudflare.com, hidden: digininja.org.uk
---
read R BLOCK
read R BLOCK
HTTP/1.1 200 OK
Date: Tue, 19 Feb 2019 20:11:10 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: __cfduid=de2af9e14e18dd04cca2de65e241513ae1550607070; expires=Wed, 19-Feb-20 20:11:10 GMT; path=/; domain=.digininja.org.uk; HttpOnly
X-Content-Type-Options: nosniff
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Server: cloudflare
CF-RAY: 4abb618f7fc86a25-LHR
1f
<p>digninja.org.uk on HTTP</p>
0
DONESo, it returned content from my site, and claims to have used ESNI, lets see what the packet trace shows. First, the SNI field:
That is good, it matches the server name given in the servername parameter, next the ESNI. Looking through the list of extensions in Wireshark, there isn't one for ESNI but at the end of the list there is this, "Unknown", extension which was not there before:
A quick check in the OpenSSL source confirms that extension 65486 (0xFFCE) is indeed the ESNI extension:
./include/openssl/tls1.h:#define TLSEXT_TYPE_esni 0xffceAs you can see in the screenshot, there is no visible text in there that mentions digininja.org.uk which seems like a win. While this does not mean that it isn't in there, it could be hidden with a simple XOR for all I know, I'm going to trust the developers on both sides of the handshake and be happy that the data is indeed encrypted in a way that makes in unreadable to any observers.
So this is it, by manually setting the ESNI server name to match the value in the Host header, it is possible to request content from a "bad" domain while all visible signs are pointing at a "good" domain. Domain fronting or not, it is enough for me to hide my C2 traffic.
One last thing tweaked my curiosity, with ESNI enabled, does Cloudflare check the SNI field any more? Lets try with this juvenile example:
$ (cat get_digininja.org;sleep 5) | /home/robin/src/openssl/apps/openssl s_client \
-CApath /etc/ssl/certs/ -tls1_3 -connect www.cloudflare.com:443 -esni digininja.org.uk \
-esnirr $ESNIRR -esni_strict -servername "8=====D"
<Lots of verbose setup stuff...>
<p>digninja.org.uk on HTTP</p>No, they don't seem to care about it at all, you can pass whatever you want and they will still return the data for you. Lets do a quick check in Wireshark to make sure the data is actually being sent, and yes, there it is:
A serious side to this, if you are a defender, and all you've got is the hostname from the DNS lookup and the SNI hostname, you cannot trust them. They are controlled by the user/attacker and, as can be seen, are completely under their control.
Conclusions
Whether you call it domain fronting, domain hiding, kitten hugging or something different, what I've shown in this post, is that it is possible to use ESNI and a custom Host header to hide a "bad" HTTP request inside what looks like a "good" one. As ESNI is being endorsed by privacy groups, I don't see it going away any time soon so I have a feeling that this will work for quite a while. The ability to drop false flags in the logs of monitoring systems which still trust the SNI field, is a really nice bonus.
