Keeping VMs on Private Networks on Azure, AWS, and GCP | by Teri Radichel | Cloud Security | Dec, 2022
Nuances in multi-cloud networking and how it could contribute to cloud data breaches and security incidents
This is one of my posts on Azure security and AWS Security.
As I wrote in one of my first slides in a presentation for the IANS security forums earlier this year for multi-cloud security says: There is no easy button. I wrote that because many products advertise that they can help you protect your multi-cloud environments or use the same code to deploy to multiple clouds, but it’s not quite that simple when you get into the actual implementation details. It is important to dig into the details and learn how each of the platforms implements different features.
This post is about one of those details where the cloud providers handle private IP addresses differently. They may even treat private IP addresses different on the same cloud provider with different versions of the IP protocol (to test when I have more time.)
Have you ever seen this warning in the Microsoft Azure documentation?
Let’s go through why this matters and why some organizations might not have the private network they think they do.
Private IPs on AWS and GCP
If you gave ever set up a virtual machine (VM) with a private IPv4 IP address to a VM on Amazon Web Services (AWS) or Google Cloud Platform (GCP) you might know that when you add a private IP but no public IP that those VMs won’t have Internet access. You will also need a route to the Internet or some kind of proxy or NAT to enable that access.
This confuses people who are new to cloud and networking sometimes because they don’t understand how the cloud platform or networking in general works. They create a VM and can’t figure out why they cannot SSH or RDP to the host or connect to the Internet. But those properties exist for a reason.
Leveraging public IP restrictions for cloud security
Cybersecurity and network professionals can leverage the fact that lack of a public IP assignment restricts Internet access to improve security of their cloud environment. By disallowing the ability to assign a public IP address, people won’t be able to configure unwanted Internet access that allows malware on a host to exfiltrate data or connect to a C2 channel.
In fact, the initial design of networks and the cloud environment at Capital One did not give developers permission to assign a public IP address to hosts. Developers were constantly asking why they could not connect to the Internet and we had to explain why. They were supposed to use designated systems on private networks within the cloud for software development activities and deployments.
But why? It just makes my job difficult…
If you already believe you need a private network jump to the bottom. If you want to know why what I am about to explain matters, read on.
Why private networks and network security exists
Yes, I’ve written about this before.
Why you need a VPC. Yes, you really do need a VPC.
But I keep trying to explain it in different ways in relation to different concepts and attacks like I did in this post on serverless networking that explains how an AWS Lambda function can bypass deployment systems with outbound access and you might not be able to tell.
By the way, I have been explaining these concepts since day one in my cloud security classes available through 2nd Sight Lab (available virtually now, with CPE credits.) I wish more people had taken them or read my book at the bottom of this post prior to the breaches and security incidents below. Maybe it would have helped, but who knows.
How private networks, VPCs, and bastion hosts would have limited the damage in the worst cyber breach of the US government in history
Why do cybersecurity and network professionals want private networks at all? Isn’t this just making things complicated and harder to write and develop software and deploy applications? No. Security professionals are trying to limit the chance an attacker can breach a system and spot if they do.
I explained how malware leveraged a C2 channel to connect to command and control servers in cloud environments in my explanation of the Solar Winds attack. Once the malware got onto a host, it communicated with a command and control server which sent it additional commands to carry out its attack, and in some cases, attackers were essentially able to take over Azure cloud accounts.
If the infected hosts where the malware got deployed had no Interest access, the malware would have had no way to reach the C2 servers. If you set up a private network on AWS and GCP and only use private IP addresses you can prevent access to command and control servers (at least directly.)
If organizations had deployed Solar Winds on hosts in a private network and used a bastion host (otherwise known as a jump box) to connect from public networks and then from there connected to the Solar Winds host from the bastion host, they would have made it harder for the Solar Winds malware to “call home.”
Private networks would have made network monitoring easier
If the companies also had alerts when hosts in private networks are trying to reach the Internet, then they would have been immediately alerted to suspicious activity and may have been able to more easily spot the breach. Instead, it existed for months, and turned out to be the worst breach of US government systems in history.
How a VPN would have helped
Of course, the attackers may have tried to find a way to leverage the bastion host but as I just explained to my Azure class, making someone first authenticate to a VM and then the bastion host would allow you to keep the bastion host private as well.
Someone wanted me to validate no one could access their bastion host on a penetration test but the only way to access that was through their VPN. They had locked down and hardened their VPN so I couldn’t even get to the bastion host.
A VPN is single purpose with a smaller attack surface, but make sure you keep it up to date and monitor that network closely as it then becomes the critical point in your network security architecture most vulnerable to attack. But closely monitoring the VPN endpoints is easier that monitoring every host on your network connected directly to the Internet for C2 traffic and data exfiltration.
Zero Trust Networks to thwart proxies
At a minimum, a zero trust network on a bastion host makes it harder for an attacker to leverage that host to proxy to the system it is trying to attack — like Solar Winds. If the attacker can install certain types of proxy software onto the bastion host they can leverage the Solar Winds software from the bastion host as if they were connected directly to the Solar Winds host from the Internet.
Typically those connections leverage TCP, and the attacker will need to be able to establish connections using various ports on the system. If no ports are accessible to connect or locked down to specific IPs, they have to try something else. If the ports are already in use the malware cannot bind to them.
If you don’t allow extraneous protocols like full ICMP access then you can prevent things like the ICMP tunnels I wrote about in my white paper on the Target Breach.
If you also set up alerts on unexpected traffic generating rejects as well, you’ve made it much harder to spew malware onto random systems like the Solar Winds attackers dud and not get caught.
Private Networks and Log4J
I also explain how zero trust networks for software updates would have helped in this post — or better yet use a proxy to download and inspect updates prior to installing them. If hosts running Java applications that run Log4J leveraged zero trust networks, then an attack would have failed on the outbound access. I wrote about that here:
People running systems in private networks would have been less concerned about systems running log4j when that vulnerability came to light. Instead of frantically rushing to update and patch systems connected directly to the Internet, if someone tried to leverage log4j from a host in a private network, the outbound connection attempts would fail.
Some systems do need direct Internet access but probably far fewer than exist in many cloud accounts. I also analyze internal networks on penetration tests and security assessments to limit lateral movements within accounts but that is a separate topic.
Private Networks and Ransomware
Enabling Internet access facilitated ransomware to attack a host where one of my coworkers installed a demo of an identity product the company was seeking to acquire. The identity software exposed certain ports typically associated with Active Directory and RDP directly to the Internet. Many brute force attempts to crack the RDP password existed in the logs since it was fully exposed to the Internet. The malware got in, installed XMRig and some ransomware software (which I was easily able to bypass thankfully.) The malware turned off the host-based firewall and security systems. I had to leave to company before I could finish my analysis of that incident.
But the point of that last example is that if the person had used a properly configured network for that product assessment, that host would not have been accessible for attack. In addition, the person opened a network port that the product he was testing required. But that made no difference because the default network rules he left in place had fully inbound and outbound access to and from the Internet.
Learn to implement proper cloud networking — even if you are just testing something out!
Keeping networks private on AWS, Azure, and GCP
As I’ve already explained, you can easily disallow public IP assignment in AWS and GCP to prevent VMs from communicating with the Internet. If and when you want these hosts to obtain software updates, one of your options is to set up a NAT. This is not the best option for software development but it allows systems to reach out to the Internet to get updates.
In Azure, when you assign only a private IP to a host, Azure assigns what I call a magic outbound only IP you can’t control to essentially have outbound internet access through a NAT they set up according to this documentation:
Azure recommends turning it off. But how can you do that? Well one way is to assign a public IP address. Brilliant. That’s exactly what we were trying to avoid. You can also pay for additional Azure resources like a NAT or a load balancer — just so you can create a true private network.
Did this functionality contribute to the Solar Winds breach? I have no idea but I can see how IT professionals coming from other clouds or networks might presume that a private IP address restricts Internet access.
At any rate, be aware that this exists and architect accordingly.
Also note that the documentation for IPv6 on AWS says you only need an address from the subnet to communicate with the Internet. I haven’t tested this yet.
GCP, which used to lag on networking, says in the documentation that both IPv4 and IPv6 private addresses cannot route to the Internet.
As always, test your network to ensure it behaves as expected!
More thoughts on cloud security like this post on the way.
Follow for updates.
Teri Radichel
If you liked this story please clap and follow:
******************************************************************
Medium: Teri Radichel or Email List: Teri Radichel
Twitter: @teriradichel or @2ndSightLab
Requests services via LinkedIn: Teri Radichel or IANS Research
******************************************************************
© 2nd Sight Lab 2022
____________________________________________
Author:
Cybersecurity for Executives in the Age of Cloud on Amazon
Need Cloud Security Training? 2nd Sight Lab Cloud Security Training
Is your cloud secure? Hire 2nd Sight Lab for a penetration test or security assessment.
Have a Cybersecurity or Cloud Security Question? Ask Teri Radichel by scheduling a call with IANS Research.
Cybersecurity & Cloud Security Resources by Teri Radichel: Cybersecurity and Cloud security classes, articles, white papers, presentations, and podcasts
