Meeting your security requirements for authenticated scanning with a vulnerability scanner can be a challenge. Think about it: as a best practice, we are required to rotate credentials on a specified basis. When creating a static scan, we must populate those credentials and need to update them while ensuring they are in sync with the rotation schedule. Doing this by hand is difficult when you have dozens or hundreds of targets, so automating the process is key. AWS Secrets Manager and a Lambda function, combined with an API key, will allow you to set-and-forget this important, yet tedious process.

Requirements:

• AWS IAM Role with the ability to view Secrets Manager secrets
• Security tool credentials stored in Secrets Manager
• Security tool API to update credentials
• AWS Lambda function written in your favorite language

Optional (the suspense!):

• Role with the ability to query AWS EC2

Once you’ve got these in place, the process is very straightforward. Your Lambda will pull the credentials used to authenticate with your security tool and call the relevant API for updating the password. Assuming you have more than one scan credential to rotate, create a dictionary of Secrets Manager secrets to the name of the credential in your security tool, loop through this dictionary and enjoy the time savings.

One additional step we take is updating our scan targets prior to scans. If you have frequent cloud releases, that can increase the likelihood of omitting newly spun up instances, thus leaving a blind spot. While we are updating rotated credentials in our security tool, we update our scan targets as well. This is where the “Optional” step from above comes into play. In your lambda, query for running EC2 instances, and with the appropriate API, push those hostnames to your security tool.

With one simple Lambda you can rid yourself of tedious, manual, password and scan target updates. We made the switch and scans are no longer something I loathe, because I’ve automated the process. As Ron Popeil, R.I.P., so famously said, “Set it and forget it!”

Simple Python PW rotation example:

import boto3
from tenable.io import TenableIO
import json

# Boto3 Resources
secrets_config = boto3.client('secretsmanager')

# Get ScanTool credentials
# Authenticate to ScanTool prior to taking further actions
scantool_sm_arn = 'arn:aws:secretsmanager:${Region}:${Account}:secret:${SecretId}'
scantool_secrets = secrets_config.get_secret_value(SecretId=scantool_sm_arn)
scantool_secrets = json.loads(scantool_secrets['SecretString'])
scantool_key = scantool_secrets['scantool_key']
scantool_secret = scantools_secrets['scantool_secret']
tio = TenableIO(scantool_key, scantool_secret)

# Get Authenticated Scan Credentials to Rotate
auth_scan_params = 'arn:aws:secretsmanager:${Region}:${Account}:secret:${SecretId}'
# Authenticated Scan UUID in your Scan Tool, which we want to rotate
auth_scan_uuid = '${Auth_Scan_UUID}'
auth_scan_secrets = secrets_config.get_secret_value(SecretId=auth_scan_params)
auth_scan_secrets = json.loads(auth_scan_secrets['SecretString'])
auth_scan_rotate = auth_scan_secrets['password']
# Gather and print description of configured scan to be rotated
# This can be used to troubleshoot is something goes awry
scantool_details = tio.credentials.details(auth_scan_uuid)
print(eps_details)

# Rotate PWs
tio.credentials.edit(auth_scan_uuid,password=auth_scan_rotate)
print('complete')

Simple Python scan target example:

import boto3
from tenable.io import TenableIO
import json

ec2_scan_list = []
auth_params_arn = 'arn:aws:secretsmanager:${Region}:${Account}:secret:${SecretId}'

# Boto Resources
ec2 = boto3.resource('ec2')
secrets_config = boto3.client('secretsmanager')

# Get ScanTool credentials
# Authenticate to ScanTool prior to taking further actions
secrets = secrets_config.get_secret_value(SecretId=auth_params_arn)
secrets = json.loads(secrets['SecretString'])
scan_tool_key = secrets['key']
scan_tool_secret = secrets['secret']
tio = TenableIO(scan_tool_key, scan_tool_secret, url='https://${Scan_Tool_URL_Here}')

# Get information for all EC2 Instances
running_instances = ec2.instances.filter(Filters=[{
    'Name': 'instance-state-name',
    'Values': ['running']}])

# Get private_ip for instances and add to tenable_list
for instance in running_instances:
    private_ip = instance.private_ip_address
    ec2_scan_list.append(private_ip)

# Get the target_group_id
# Scan Target Group to be updated
for tg in tio.target_groups.list():
    if "${Resource_Scan_Group}" in tg['name']:
        target_group = tg['id']

# Update Scan Target Group with running EC2 list
tio.target_groups.edit(target_group, members=ec2_scan_list)
print('complete')

This topic, again? Yeah, egg on my face, I believe I ended the last blog with something like “constant pursuit of better.” The truth of the matter is, I deployed the code from the original post and, upon review of the results, it became clear that something was amiss. We were seeing results which we knew were false positives and thus knew that we needed to dig deeper.

The way we originally tackled dangling resources, via Route 53, was by looking for a cloudfront.net entry and then attempting to locate an S3 bucket matching that entry’s alias. While this logic may work in circumstances where a user tells S3 to host the contents of the bucket, it doesn’t necessarily hold true when the CloudFront service is properly utilized. I reached out to the team who handles this type of configuration to get a better understanding of how it’s properly configured and to make sure the change in logic would both eliminate the false positives and also report actual dangling resources.

CloudFront allows you to name entries whatever you’d like, they don’t need to match an S3 bucket name; the “Origin domain” in the entry is the associated S3 bucket. So how do we go about building logic to make sure S3 buckets associated with CloudFront distributions still exist? Much like before, we are still looking for S3 distributions and as we iterate through CloudFront entries we are gathering:

• Origin domain
• CloudFront ID
• CloudFront domain

If the Origin domain contains “s3”, we drop the tail end of the domain (ex: s3.us-east-1.amazonaws.com) and check to see if an S3 bucket exists that matches that domain. If it does exist, awesome. If not, we report that we have a dangling resource which needs to be addressed.

We still check Route 53 for entries pointed at CloudFront domains to validate if a distribution is configured properly. This becomes a bit more straightforward since we already have a dictionary containing all CloudFront defined entries. We then iterate through Route 53 records looking for “cloudfront” in the DNS name and then compare the AliasTarget DNS name to our dictionary of CloudFront distributions. If the AliasTarget DNS name is not in the CloudFront distribution dictionary, we report the useful information as a dangling resource!

“Why do we fall, sir? So we can learn to pick ourselves up.”* It’s easy to get discouraged when things you’ve spent time architecting and building fail but it’s important to learn from such failures. Rather than telling me my results were wrong, the teams I support helped me better understand the deployment architecture and data. Through that learning I was able to accurately identify misconfigurations and detect dangling resources. I hope you too can accept this blog update for what it is and see the positive side of failure.

* Alfred Pennyworth, Batman Begins (2005)

Do you know what lurks behind your unattached resources that were thought to have been destroyed? Not only are there potential cost implications, paying for resources you are no longer using, there are also risks that threat actors may try to exploit these resources. In our cloud journey, we found this to be a common problem which we had to deal with. Here is how we resolved dangling resources at Code42. 

We utilize Terraform to deploy resources to AWS, and by that same method, Terraform should tear down those resources when they are no longer needed. At the end of last year we found that, despite deploying resources as code, we occasionally had bug bounty researchers notifying us of resources upon which they could squat. 

One easy example of how you could get into this situation: 

• Create an S3 bucket
• Host the contents of that bucket with CloudFront
• Delete your bucket but not the CloudFront entry 

The way AWS handles S3 bucket hosting via CloudFront allows for someone to create an S3 bucket in their account with the same name as one you were hosting, thus potentially hosting a malicious site posing as your company. Their S3 bucket will get attached to your CloudFront entry and the malicious hacker could claim traffic going to your domain. AWS doesn’t appear to have an offering to look for and notify account owners of these dangling resources (interestingly enough, Azure does notify when dangling resources are found within an account). Without an AWS service to handle this we decided to build an in-house solution.

So what are we looking for? As mentioned above, we focused on CloudFront entries which don’t have an associated S3 bucket, or Elastic IPs which aren’t associated with an EC2 instance. I built a pretty simple script in Python, utilizing the AWS SDK Boto3. Since EIPs and CloudFront entries are configured via Route53, we start by listing all zones which are configured for the account and we pull all records within that zone. We are interested in any records which point at CloudFront, since those will be tied to an S3 bucket, and ‘A’ records which will contain EIPs. For records with CloudFront, we compare the name in the record to a list of all S3 buckets within the account. If there isn’t an S3 bucket containing that name, we’ve found a dangling resource. And now, we report it. Similarly with Elastic IPs, we call describe_instances and provide the EIP we found in Route53. If an EC2 instance doesn’t exist with that IP, we’ve found a dangling resource. Again, report it. It is important to note that you can have ‘A’ records porting to internal resources, which won’t have an associated EC2. I’ve added logic to filter those internal resources out based on matching an IP string.

The linked code is a modified version of what we have deployed; this is a small section of a much larger Python script used for info gathering. This code is meant to be run locally, when authenticated to an account, or from within an AWS account. We have our script deployed as an AWS Lambda. Amazon EventBridge handles the scheduling, running every 24 hours, with the results being output to Amazon S3. We have configured a SumoLogic S3 collector which allows us to review and alert on findings.

Deploying infrastructure as code allows for consistent deployments and easy validation that things are correct. While this should add reassurance that resources are torn down appropriately when no longer needed, manual deletions can let resources slip through the cracks. It’s important to verify you’re not leaving resources out there for opportunists to take advantage of. Even if you think your policies and procedures should prevent dangling resources from occurring, review the script and scan your environments.

Associated code: aws_dangling_resources.py
https://github.com/code42/redblue42/blob/master/aws_dangling_resources.py

In a blog post I wrote not too long ago, I dove into the reasons why security teams need to learn how to code. In short, as the lines between security practitioner and developer continue to blur due to the rise of infastructure-as-code, automation, and security tools becoming more developer-friendly, it is very hard to be a top-performing security team without coding. Today, I’ll take this topic a bit further and talk about why security teams need to use Agile as well.

What is Agile? There are many different interpretations of Agile and countless books, videos, and blogs on the subject. Since I don’t come from a formal development background, I’ve never done software development following the Agile methodology, but inexperience doesn’t need to be a barrier to using it. The way our security team uses Agile at Code42 encapsulates several key concepts::

• Iterative development using small chunks of work effort
• Ongoing and timely feedback from stakeholders
• A willingness to experiment that focuses on outcomes rather than jumping to solutions

One of the core tenets of Agile is to iterate and break work up into small pieces in order to rapidly develop, test and deploy. In this regard, it is the opposite of the Waterfall method, with its structured project plans and multi-month timelines. Agile doesn’t mean that complex projects or features can’t be implemented, but that each iteration in the journey to the end state should be able to stand on its own.

It is fairly common in security to see a lot of major projects that have a waterfall-like mentality. Think of implementing a SIEM tool, or maybe rolling out a new endpoint security tool across the organization. Those kinds of projects typically rely on everything coming together perfectly over the entire timeline, and when things inevitably go wrong, the result is slipped deadlines, finger pointing, and angry stakeholders.

With Agile, you can still get to the end, but the path is much more flexible and reactive — it is called “Agile” after all! Instead of rigidly defining the exact order of steps in a project and creating a lot of dependencies between large work efforts, which are inherently fragile, Agile’s goal is to break it up. That massive SIEM project timeline becomes smaller, more independent work efforts like stand up a collector, send one kind of log and validate it, and so forth. That way, if the project priorities or needs change, there isn’t a massive amount of planning that needs to be redone.

That brings me to my next concept: Ongoing, rapid feedback from stakeholders. One of the promises of Agile is that it can prevent the divergence between what stakeholders want and what practitioners deliver, which often happens when large projects run into problems. Flexibility is a must in today’s rapidly-changing world. Take our SIEM project example: what if a SIEM project that started in February had a timeline for onboarding VPN logs in June with a lot of dependencies beforehand? The realities on the ground, and the need to reprioritize that VPN work, would have resulted in a lot of rework and wasted effort put into that original timeline.

With Agile, the goal is to finish your work quickly, then sit back down with your stakeholders and ask questions like “Did this deliver? What is your next priority? What has changed since we last spoke?” This ensures that security teams are delivering real value to stakeholders, instead of going down rabbit holes for months at a time and ending up with a solution that doesn’t improve the security posture of the organization.

The last concept is probably the hardest for security teams to be comfortable with: having a willingness to experiment and fail, while focusing on the outcome instead of the solution. Too often security teams jump to the solution (SIEM, EDR, firewall) before identifying the desired outcome (alerting on events of interest, identifying and acting on C2 traffic). This narrow focus can result in poor resource allocation, or even worse a fundamental misunderstanding of the problem that results in a tool that doesn’t solve it.

There are many, many ways to solve security problems, and experimenting with alternate solutions or questioning the status quo may result in less complex and more efficient ways of achieving goals. These simpler solutions may not have the cachet of a major project, but if your desired outcome is to alert on certain events from a security tool, a one-week effort at leveraging a script, an API, and a cron job may be much more effective than a long SIEM implementation.

How does a security team start to leverage Agile? If you work in an environment where you have Agile development teams, you are in luck: you can reach out to them and start learning the tricks of the trade. Even if you don’t, however, you can become more Agile by breaking your work efforts into small pieces, setting up ongoing feedback both internally and externally, and focusing on the outcome rather than jumping to the solution. Do all of this and you’ll find that you are able to deliver more security value to your organization.