jonathanquail.com

Cloud architecture, Chef, AWS, and Python stuff

Restricting Access to Servers Behind an Elastic Load Balancer

We’re building a new system at work and I’ve been doing much of the deployment work. This new system will be deployed completely in AWS.

Part of this new system is a RESTful API that we only want to expose to our existing Application servers hosted in a datacenter elsewhere.

Attempt 1 – EC2 Security Groups

The security groups in AWS are great – and you should use them to restrict access to all of your servers so only the specific ports you need are open to the specific addresses (or other security groups) that need them.

Naturally my first stop was to create a nice restrictive security group to allow only SSH and HTTP access from our office for testing. Perfect – problem solved! No one outside the office could access the machine.

Now being the smart engineer that I am deploying to AWS – I started a few instances in more than 1 availability zone and then went and created an Elastic Load Balancer.

The ELB uses a health check to determine when an EC2 instance is healthy – I set it up to hit a part of our API that would prove to me the server was available.

Immediately all EC2 servers reported as un-healthy.

Off to google I go – and after far more searching than I should have had to do I discovered you need to add a special “amazon-elb” security group to your EC2 security group to allow the ELB to communicate with it.

(Note: the name of this group should be “amazon-elb”, but it also appears in the Elastic Load Balancer tab in the AWS Management Console).

So I amended the SG and health-checks were all green. My scalable application was ready. So I sent the URL to my wife at her office and expected her to say “Nope – doesn’t work”.

But she didn’t – it works for her.

See – Elastic Load Balancers forward all traffic to your EC2 instance as if its coming from them. Thus bypassing your perfectly crafted SG on your EC2.

Second Try – Security Group on your ELB!

This one is short. Elastic Load Balancers don’t allow you to specify a security group (except in a VPC – more on that later). The ELB is designed to be completely open and available to the world. Not great if you want to use them for internal-facing applications.

Third time’s a charm – Apache Deny/Allow

Elastic Load Balancers do set the X-Forwarded-For header on every request. So my next thought was to check this header and if the IP address was in my white-list, let the user through.

(Note: The X-Forwarded-For header could be faked, so its not a perfect solution).

The way ELBs work – you may see multiple ELB IP addresses in the header if the request is routed to us-east-1a and you had no healthy instances in that AZ, it would get sent to an ELB in us-east-1b where you may have a healthy instance. Therefore the header would look like:

1
  X-Forwarded-For: 251.21.25.42,81.212.52.12,81.214.55.16

Where 81.212.52.12 and 81.214.55.16 are ELB IPs, and 251.21.25.42 is the client’s IP.

So using some RegEx expressions to extract the client’s IP:

1
2
SetEnvIf REMOTE_ADDR "(.+)" CLIENTIP=$1
SetEnvIf X-Forwarded-For "^([0-9.]+)" CLIENTIP=$1

Now you can check that IP against your white list like:

1
SetEnvIf CLIENTIP "192.152.52.12" allowed_in

And use the Apache Allow/Deny like this:

1
2
3
Order deny,allow
Deny from all
Allow from env=allowed_in

The problem with this approach is that when deployed, traffic from non-whitelist IPs still comes to your Apache server and could still overload it. And of course this doesn’t play nice with Elastic Load Balancers. Their health check won’t be X-Forwarded-For any client (let alone one on the white-list).

Virtual Private Cloud

Using a VPC is one way to go. Not only does it let you assign a security group to your ELB (!) but you can also control both incoming AND outgoing network traffic. Regular EC2 security groups can only limit incoming traffic.

VPCs don’t cost anything above regular EC2 costs (unless you use a VPN device). I have some concerns over the robustness of using a VPC however. All traffic that routes outside the VPC (such as to AWS resources – S3/SimpleDB/Dynamo etc) – must go through a single NAT instance. You can only have a single NAT for your VPC, and it is located in one of the AZs of the region. If you lose that NAT instance, all traffic in all AZs that must reach the outside world will fail until you can bring a new NAT instance online.

My solution

A combination of the ideas above. Use security groups on your EC2 instances that only allow traffic from an ELB. Then use Apache to only allow traffic that was X-Forwarded-For a white-listed IP. You also need to configure Apache to allow any traffic through that was not X-Forwarded-For at all. This allows the health check through only. If someone tries to access the instance directly (not though the ELB) – the EC2 Security Group will stop them.

The Apache configuration would look like this:

1
2
3
4
5
6
SetEnvIf REMOTE_ADDR "(.+)" CLIENTIP=$1
SetEnvIf X-Forwarded-For "^([0-9.]+)" CLIENTIP=$1
SetEnvIf X-Forwarded-For "^$" is_not_forwarded
...
# Whitelist
SetEnvIf CLIENTIP "192.152.52.12" allowed_in

And the Apache Allow/Deny:

1
2
3
4
Order deny,allow
Deny from all
Allow from env=allowed_in
Allow from env=is_not_forwarded

Comments