Security is an essential part of software development and if we have infrastructure on AWS, it's our shared responsibility to make our infrastructure secure. AWS provides security out of the box but still, some components require us to do some security measurements for our infrastructure. We can read more about in AWS Shared Responsibility Model.

In this article we will be looking at how we can secure our subnets in a VPC. Below are the steps we are going to perform and you can do hands on along with this article as this will be more of a tutorial. Make sure you have an AWS account if you're intended to do hands on.

The high level scenario that we'll be covering is that we have our own VPC and we'll create 2 subnets. One is public and second one is private. We'll be having EC2 instances in each of the subnets. The instance which is in public subnet act as a Bastion host for accessing the instance on private subnet.

We may encounter this scenario if you're working for a three tier architecture application and one of our layer i.e Database layer resides in private instance, so obviously we don't want to give inbound access to that database or private instance. So we will create a Bastion Host to access private server in order to do maintenance or updating OS dependencies or patches.

Steps We'll Perform

Create VPC
Create an Internet gateway
Create a Public & Private subnet with Route Table
Create a Bastion Host
Create Network Access Control List (NACL) for Private Subnet
Adding Rules to a Private Network Access Control List
Launch an EC2 Instance on a Private Subnet
Launching a Network Address Translation (NAT) Gateway
Testing access of our Private Subnet Instance

Create VPC

AWS Virtual Private Cloud VPC is an essential component of Amazon Web Services that enables us to launch AWS resources into a virtual network we have defined. It is logically isolated from other virtual networks in the AWS cloud.

Go to AWS console and search VPC in search bar. Every availability zone in a region has at-least one VPC by default. Just Click create a new VPC and Create VPC with CIDR

10.0.0.0/16

(This is a CIDR block from the private (non-publicly routable) IP address ranges as specified in RFC 1918.)

Create an Internet Gateway

An Internet Gateway is a horizontally scaled, redundant, and highly available VPC component that allows communication between instances in our VPC and the Internet. Thin of it as a network node that connects two different networks that use different protocols (rules) for communicating.

An Internet gateway serves two purposes:

Provide a target in our VPC route tables for Internet-routable traffic.
Perform network address translation (NAT) for instances that have been assigned public IP addresses. (Note: It does not do this for instances with private IP addresses.

Search for Internet Gateway in VPC section and create one named IGW-lab. (name could be anything, i am using this just for this article)

Now we have created internet gateway, next up is to create one public and one private subnet along with their route tables.

Create a Public & Private subnet with Route Tables

First let's create Public Subnet , as the name implies this subnet will allow its resources to communicate with the internet. A common use case for this is a DNS server, or a load balancer sitting in front of front-end web servers or web applications.

After creating public subnet Public A create a Route table that contains a set of rules, called routes, that are used to determine where network traffic is directed.
Go to Route Table option in VPC and create one named PublicRouteTable with the VPC we created earlier.

Now, we have to configure the route setting for communicating with the Internet

Click Edit routes to the Routes tab
Click Add route:
Destination: Enter 0.0.0.0/0
Target: Select Internet Gateway we created, then IGW-lab

Up next, we will change the default route table of the public subnet to include the new route table.

Select the Public-A subnet and click the Route table tab
Click the Edit route table associationbutton
Select PublicRouteTable from the Route table ID drop-down menu and confirm the following routes

Now we have to create a Private Subnet. A common use case for private subnets is to configure resources for a back-end tier, such as database servers that should not be accessible from the internet. However, we may eventually want these back-end database servers to access the internet for operating system updates or to be accessible by administrators via a bastion host.
Just repeat previous steps to create private subnet named Private A in the same VPC we created. Enter 10.0.10.0/24 as the CIDR block of our subnet.

Note: If a subnet does not have a route to the Internet (0.0.0.0/0) through a gateway, the subnet is known as a private subnet.

After creating private subnet Private A create a Route table named PrivateRouteTable that contains a set of rules for private subnet. we will add the target of the NAT gateway in a later step. This route will eventually send traffic originating from our private subnet and bound for the public internet, to a NAT device.

Up next, we will change the default route table of the private subnet to include the new route table.

Select the Private-A subnet and click the Route table tab
Click the Edit route table association button
Select PrivateRouteTable from the Route table ID drop-down menu and confirm the following routes

Create a Bastion Host

A bastion host is typically a host that sits inside our public subnet for the purposes of SSH (and/or RDP) access. We can think of it as a host for gaining secure access to resources in our VPC from the public internet. Bastion hosts are sometimes referred to as jump servers, as we jump to one, and then back out of it.
We will be using SSH connectivity to Linux instances & will create an EC2 instance that will serve as both an observer instance that we can run various tests from and a bastion host.
Launch an EC2 server in the Public A subnet as a bastion host with the following security group configurations.

Assign a security group: Select Create a new security group
Security group name: Enter SG-bastion
Description: Enter SG for bastion host. SSH access only
Type: SSH
Protocol: TCP
Port: 22
Source: 0.0.0.0/0

Make sure to download PEM.key file to SSH into bastion host.

Note: It isn't a best practice to set the source to any IP, but i am using it for the sake of this tutorial. If we are in an environment with a static IP, we could set the source field to My IP in the drop-down menu to only allow our IP for improved security.

Create Network Access Control List for Private Subnet

A Network Access Control List (NACL) is an optional layer of security that acts as a firewall for controlling traffic in and out of a subnet. The network access control list is a numbered list of rules, evaluated in order, to determine whether traffic is allowed in or out of any associated subnet. NACLs are stateless which means they cannot tell if a message is in response to a request that started inside the VPC, or if it is a request that started from the internet. Hence, a NACL is better suited for private subnets. For public subnets, using security groups is recommended without NACL. We can Think of it as a Security group for Private subnets.
Now we will create a Network Access Control List for our private subnet.

Configure the following Network ACL settings:

Name: Enter Private-NACL
VPC: Select our VPC from the drop-down menu

By default, a network ACL is not associated with a subnet until we explicitly associate it with one. We need to associate by editing subnet association in NACL.

Select Private-NACL from the Network ACLs list and click the Subnet associations tab
Click Edit subnet associations
Select the check box for the Private A subnet to associate it with the network ACL

Adding Rules to a Private Network ACL

Now we will create inbound and outbound rules for our private Network Access Control List (NACL). The purpose of this is to determine whether traffic is allowed in or out of any subnet associated with the network ACL. As a best practice, we will start by creating rules with rule numbers that are multiples of 100. This can help with the organization if we need to insert new rules later on, as there is room within the numbering scheme.
Add inbound configuration for private subnet NACL.

Click Add a new rule and configure the following:

Rule number: Enter 100
Type: Select SSH
Source: Enter 10.0.20.0/24(The CIDR block of our public subnet)
Allow / Deny: Select Allow from the drop-down menu

For the second rule, click Add a new rule and configure the following:

Rule number: Enter 200
Type: Select Custom TCP Rule
Port Range: Enter 1024-65535
Source: Enter 0.0.0.0/0
Allow / Deny: Select Allow from the drop-down menu

This will allow return traffic for the outbound rules we will add shortly (the range is specified as 1024-65535 because these are the available ports and not reserved). This enables resources inside the subnet to receive responses to their outbound traffic.

Now that we've verified the inbound rules, we will move on to configure the outbound rules. Although the outbound IP addresses can be anything, the ports need to be 80 or 443. In short, operating system updates needed by instances in our private subnet could come from anywhere (0.0.0.0/0), but they will be downloaded over port 80 (HTTP) or 443 (HTTPS). We will need to add rules to account for each port.

With the Private-NACL still selected, switch to the Outbound rules tab and click Edit outbound rules.

Click Add a new rule and configure the following:

Rule number: Enter 100
Type: Select HTTP from the drop-down menu
Destination: Enter 0.0.0.0/0
Allow / Deny: Select Allow from the drop-down menu

For the second outbound rule, click Add a new rule and configure the following:

Rule number: Enter 200
Type: Select HTTPS from the drop-down menu
Destination: Enter 0.0.0.0/0
Allow / Deny: Select Allow from the drop-down menu

For the third outbound rule, click Add a new rule and configure the following:

Rule number: Enter 300
Type: Select Custom TCP from the drop-down menu
Port Range: Enter 32768-61000
Destination: Enter 10.0.20.0/24 (The CIDR block of our public subnet)
Allow / Deny: Select Allow from the drop-down menu

Note: When we add or remove rules from a network ACL, the changes are automatically applied to the subnets it is associated with. NACLs may take longer to propagate, as opposed to security groups, which take effect almost immediately.

Launching an EC2 Instance on a Private Subnet

Now we will launch an instance in a private subnet created earlier. Although the instance does not have database server software installed on it, we can think of it as a database server, which is often shielded from direct access to the internet for security reasons. Once the instance is up and running, we will learn how to SSH into it from our local host, via a bastion host in the public subnet. Enter the name of the instance private. When launching EC2 in a private subnet, make sure we select private subnet with the auto-assign-IP-disable option to prevent from getting direct access to the internet.

For the Security group step, we'll create a new one with the following configurations below.

Select Create a new security group
Security group name: Enter SG-Private
Description: Enter the Security group for private subnet instances. Accept SSH inbound requests from Bastion host only.

Add the following rules:

First Rule

Type SSH
Protocol TCP
Port 22
Source Custom (Select the SecurityGroupOfBastionHost)
Tip: If we don't recall the name of our bastion host's security group, leave the Source as Custom, and start typing "bastion". It will find the security group for you. (Example: SG-bastion)

Second Rule

Type HTTPS
Protocol TCP
Port 443
Source Custom 10.0.20.0/24 (Public VPC CIDR)

Note: If we also needed Windows access, we would add another rule:
Third Rule (optional)

Type RDP
Protocol TCP
Port 3389
Source SecurityGroupOfBastionHost

Review the Summary section and click Launch instance.

Switch to the Inbound rules tab of the security group and locate the SSH rule. If the Source is not the security group of our bastion host, click Edit inbound rules and enter bastion for the Source. The lookup will find the SG-Bastion security group. Make sure it is the source and then click Save.

In an earlier Step, we launched a bastion host which used its own security group. However, at the time we configured the security group, there was not a security group for our private instance on the private subnet. Because there is now, we can further tighten down the bastion host's security group. You'll do that next before using SSH to connect to our bastion host and private instance.

From the VPC Dashboard, click Security Groups. Make note of the Group ID of the SG-Private security group. (For example, sg-xxxxxx)

Select the SG-bastion security group, switch to the Outbound rules tab, and click Edit outbound rules. Now that we have a private security group, we can restrict Outbound rules to instances using SG-Private. Configure the following:

Type: SSH
Protocol: TCP
Port: 22
Destination: Select Custom and then enter the security Security group ID of SG-Private Warning: Make sure to delete the existing SG rule, and add a new one. Click Save rules when ready.

Next, we will SSH into our bastion host, and enable ssh-agent forwarding so we can SSH (jump) to the private instance in our private subnet. The instructions below assume our local machine is Linux. However, we could use an SSH client such as PuTTY.

we must have the PEM file key. Make sure the permissions are correct on this file.

chmod 400 PEMfilename.pem

Since copying SSH private keys to a bastion instance is a security risk, we will enable SSH agent forwarding next. The ssh-add command can add private keys to the keychain application. Essentially, the private key will be used without having to copy it to the bastion host.

Note: Realize that attempting to SSH directly from our local host to the private IP of the instance in our private subnet will fail. (If not convinced, take a moment to try to. The connection will be refused.)

ssh-add -k PEMfilename.pem

ssh-add -L

SSH into our bastion host using the authentication agent we just added.

ssh -A ec2-user@BastionHostPublicIP

Important! we must use the -A option shown above to enable the forwarding of the authentication agent. If we copy/paste the ssh command from the EC2 Dashboard's Connect button, we will not be able to connect to an instance on the private subnet.

Note: Our bastion host public IP can be found in the Details tab of the instance, under the Public IPv4 address.

Up next, SSH into the private instance running in our private subnet from our bastion host.

ssh ec2-user@PrivateInstancePrivateIPaddress

Reminder: we can get the private IP address of the instance running in our private subnet from the running instance's Description tab > Private IP field.

Congratulations! we successfully created and configured a bastion host in order to SSH into a private instance on a private subnet.

Try run this below command for updating OS dependencies.

sudo yum update

Although the private instance security group is configured correctly, and we should have outbound access to the internet, it is still timed out. The time-out is caused by the private NACL denying inbound HTTP traffic. we will need Network Address Translation (NAT) to allow our private instance outgoing connectivity to the Internet, while at the same time blocking inbound traffic from the Internet. Once NAT is in place, we should be able to get package updates.

Note: For convenience sake, leave the SSH connection to our private instance open.

Launch a Network Address Translation (NAT) Gateway

When choosing between a NAT Gateway and a NAT instance to handle our network address translations, there are a few key differences to consider. Some examples include maintenance, availability, and performance.
A NAT instance is managed by you, which includes software installation, updates, and patching. It also requires additional scripting to manage its availability and failover between instances. Its performance relies heavily on a generic Amazon Machine Image (AMI) that is configured to perform NAT.

A NAT Gateway, on the other hand, is managed by AWS which means we do not need to perform any maintenance. It's highly available with a NAT gateway in each Availability Zone to improve redundancy. As for performance, the software used by a NAT Gateway is optimized for handling NAT traffic.

Next up, we will create a NAT Gateway that will be used by the EC2 instance in our private subnet to access the public internet. we will also revisit the route table associated with the private subnet and update the target entry to point to this gateway.

Click Create NAT gateway in VPC

Begin configuring the following in the NAT gateway settings form:

Name: Enter NAT-GW
Subnet: Select Public-A
Connectivity type: Ensure Public is selected The Public connectivity type will allow this NAT Gateway the ability to access the public internet.

Click Allocate Elastic IP next to the Elastic IP allocation ID.

We will need to attach an Elastic IP address to our NAT Gateway. This allows it to be referenced by the route table responsible for routing outbound traffic from instances in the private subnet to the public internet.
This allocates an Elastic IP address for the NAT Gateway to use. When the NAT Gateway is created, this IP address will be attached to the NAT Gateway automatically.

After creating the NAT gateway, wait for the NAT Gateway State to display as Available before continuing.
This process can take up to 2 minutes. After a brief waiting period, we may need to refresh the page to view the available status.

Click on Route Tables and select to PrivateRouteTable
Click Edit Routes
Locate the 0.0.0.0/0 Destination route and clear the Target field then select NAT Gateway, then NAT-GW from the drop-down menu that appears
Click Save Changes

Access of Private Subnet Instances

In last step, we configured a bastion host so we could SSH into instances on our private subnet. However, we were unable to access the internet from private instances. A common use case for needing such internet access is operating system package updates. However, the installation failed.
now that we have a network address translation device in our VPC, we will test if the operating system updates will work from our private instances.

If we are not connected to our instances by SSH, we can repeat these steps.

ssh-add -L    # Confirm the keys are in place on our local host. If not, add them with: $ ssh-add -K PEMfilename.pem

ssh -A ec2-user@BastionHostPublicIP    # SSH from our local host to the bastion host in our public subnet

ssh ec2-user@PrivateInstancePrivateIPaddress   # SSH from the bastion host into the private instance on our private subnet.

Enter this command to verify Installation from the internet

sudo yum update -y

It worked!

Note: The exact package updates for our system may vary slightly, for example as the default AWS Linux AMI changes over time. We may also encounter a successful message: No packages marked for the update, indicating the instance accessed the public internet, but no updates were required.

Summary

The VPC has been configured with two subnets, a public subnet, and a private subnet. If a subnet's traffic is routed to an Internet gateway, the subnet is known as a public subnet. If a subnet doesn't have a route to the Internet gateway, the subnet is known as a private subnet.

Both subnets have a route table associated with them. Instances on the public subnet route internet traffic through the internet gateway. The private subnet routes internet traffic through the NAT device (gateway or instance).

Bastion Host has been created in public subnet to access private subnet instance but only allow via SSH.

Note: Instances launched in a private subnet do not have publicly routable internet addresses either.

Secure AWS VPC using Public and Private Subnets