How to create a self-healing web proxy cluster with AWS and Squid in 8 steps

Web proxy has many purposes, to name a few, some people use it for anonymity purposes, and some people want bypass network censorship, and another common usage is for CDN. Here at CoinGecko, we use web proxies to protect ourselves while consuming and sharing large data by staying anonymous. Let’s go through the steps to create high availability proxy servers with zero maintenance using Amazon Web Services and Squid.

Step 1: Launch an EC2 instance 

Step 2: Reserve Elastic IPs (static ip) from AWS

In this example, We reserved 2 static ip(s) from AWS, specifically 3.12.240.78 and 3.12.195.16. You can reserved as many ip(s) you want depending on your need, each ip represents a proxy node.

Step 3: Creating a golden image (AMI) of EC2 with squid installed (EC2 > Instances)

Now, we are going to prepare a golden image for auto scaling. Now ssh into the ec2 instance created in step 1 and do the following:

sudo yum update -y
sudo yum install docker -y
sudo service docker start
sudo usermod -a -G docker ec2-user
sudo systemctl enable docker
sudo amazon-linux-extras install python3 -y
sudo docker run --name squid -d -p 3128:3128 --restart always --volume ~/squid.conf:/etc/squid/squid.conf datadog/squid

vim ~/squid.conf and and copy the following into the file (feel free to use your own configuration)

acl SSL_ports port 443
acl Safe_ports port 80
acl Safe_ports port 21
acl Safe_ports port 443
acl Safe_ports port 70
acl Safe_ports port 210
acl Safe_ports port 1025-65535
acl Safe_ports port 280
acl Safe_ports port 488
acl Safe_ports port 591
acl Safe_ports port 777
acl CONNECT method CONNECT
#http_access deny !Safe_ports
#http_access deny CONNECT !SSL_ports
#http_access allow localhost manager
#http_access deny manager
#http_access allow localhost
#http_access deny all
http_access allow all
http_port 3128
coredump_dir /var/spool/squid3
refresh_pattern ^ftp:           1440 20% 10080
refresh_pattern ^gopher:        1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0     0% 0
refresh_pattern .               0 20% 4320         

Now we have our ec2 instance setup to be a proxy, it is time to create an AMI.

Step 4: Create an appropriate IAM Role to allow address association. (IAM > Roles)

Step 5: Create AWS Security Group (EC2 > Security Group)

In this tutorial, I will be creating a security group that allows ALL inbound and outbound traffic for simplicity sake. Please use the appropriate security group when you are creating your own proxy cluster.

Step 6: Create AWS Launch Template (EC2 > Launch Templates)

Things to take note while creating launch template:

• For AMI, use the AMI created in Step 3
• For Security group, use the security group created in Step 5
• in Advanced tab > User data put in the following (for the valid ips, put in your comma-separated ip(s) created in Step 2

#!/bin/bash
sudo pip3 install aws-ec2-assign-elastic-ip

sudo /usr/local/bin/aws-ec2-assign-elastic-ip --region us-east-2 --valid-ips 3.12.240.78,3.12.195.16

sudo /usr/local/bin/aws-ec2-assign-elastic-ip --region us-east-2 --valid-ips 3.12.240.78,3.12.195.16

sudo docker container prune -f

sudo docker run --name squid -d -p 3128:3128 --restart unless-stopped --volume /home/ec2-user/squid.conf:/etc/squid/squid.conf datadog/squid

Step 7: Create Auto Scaling Group (EC2 > Auto Scaling Group)

Use the launch template as the recipe to create the Auto Scaling Group. In this example, we configure the desired number of instance to be 2 since we reserved 2 ips in Step 2. Now click done and you should be able to see 2 EC2 instances being created in the dashboard

Step 8: Verify if proxy is working

In order to verify if the proxy is working, you can use https://www.proxy-checker.org/

With the above setup, auto scaling will take spawn new instance(s) using the static ip(s) we reserved and the configuration we setup from the above steps, if the node is down or terminated. Hence zero maintenance required.