Deploy a Flask / MySQL application on AWS - Part 2a: setting up a single, non load balanced EC2 instance

aws linux flask python nginx

November 07, 2020

Even though AWS documentation is pretty comprehensive I found myself running into a few problems while creating an environment for my flask application.
In this article I'll guide you through the required steps to set up a single non load balanced EC2 instance.
Prerequisites
  • An AWS account within the 12 months trial period (not mandatory)
  • A local Python 3 environment (preferably a virtual environment but that's up to you)
  • pip package manager
  • A running local Flask application ready to be deployed
  • Familiarities with the command line
  • macOS or Linux if you want to use the command line interface like I did (I guess this should also work for Windows but the commands are probably slightly different).
Creating a single EC2 instance
If you come from the preceding article you should have a running local flask application and an Elastic Beanstalk environment ready to be deployed. Let's now create a t2.micro EC2 instance running Amazon Linux 2 and Python 3.7. The following command will create a single instance:
(env) $ eb create your-instance -i t2.micro --single --platform "64bit Amazon Linux 2 v3.1.0 running Python 3.7"
Once the instance has been created eb utility will zip our source folder and upload it on the instance. At the same time it will try to install all the dependencies listed in "requirements.txt". Since MySQLdb (our Python mysql client library) requires mysql to be installed first eb should return an error. By looking at the logs of the newly created instance using the following command:
(env) $ eb logs
notice that pip returned the following error:
Collecting mysqlclient ... raise EnvironmentError("%s not found" % (mysql_config.path,)) OSError: mysql_config not found
MySQLdb requires mysql librairies to be installed first so that's what we'll do next.
Installing mysql libraries on the EC2 instance
SSH into your EC2 instance with the key pair you created during environment creation. Using the command:
(env) $ eb ssh
from your application source folder will automatically ssh into your instance without having to manually enter your key pair location.
Note: SSH key pairs are stored in ~/.ssh by default.
You should now be logged into your instance:
INFO: Attempting to open port 22. INFO: SSH port 22 open. INFO: Running ssh -i /Users/guimauve/.ssh/someKey.pem ec2-user@instance-ip _____ _ _ _ ____ _ _ _ | ____| | __ ___| |_(_) ___| __ ) ___ __ _ _ __ ___| |_ __ _| | | __ | _| | |/ _ \/ __| __| |/ __| _ \ / _ \/ _\ | '_ \/ __| __/ _\ | | |/ / | |___| | (_| \__ \ |_| | (__| |_) | __/ (_| | | | \__ \ || (_| | | < |_____|_|\__,_|___/\__|_|\___|____/ \___|\__,_|_| |_|___/\__\__,_|_|_|\_\ Amazon Linux 2 AMI This EC2 instance is managed by AWS Elastic Beanstalk. Changes made via SSH WILL BE LOST if the instance is replaced by auto-scaling. For more information on customizing your Elastic Beanstalk environment, see our documentation here: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-ec2.html [ec2-user@instance-ip ~]$
mysql librairies require gcc to be installed:
[ec2-user@instance-ip ~]$ sudo yum install gcc [ec2-user@instance-ip ~]$ sudo yum install -y mysql-devel
If the installation was successful we can log out from the instance and try to deploy our application again.
(env) $ eb deploy
If everything went fine eb deploy succeeded and you should get the following output:
Creating application version archive "app-7daa-200908_121806". Uploading: [##################################################] 100% Done... 2020-09-08 10:18:13 INFO Environment update is starting. 2020-09-08 10:18:16 INFO Deploying new version to instance(s). 2020-09-08 10:18:22 INFO Instance deployment successfully generated a 'Procfile'. 2020-09-08 10:18:30 INFO Instance deployment completed successfully. 2020-09-08 10:18:34 INFO New application version was deployed to running EC2 instances. 2020-09-08 10:18:34 INFO Environment update completed successfully.
Setting up the instance security group
Since the instance will be accessed directly through its Elastic IP we need to accept all incoming traffic on http and https ports. In the AWS console, go to "Security groups" and select your instance group (or create a new one) and add the following entries into "Inbound rules":
  • Create a http rule and set its source to "Anywhere"
  • Do the same for https
From now on your application should be accessible using its Elastic IP address but we still have a few things to set up.
Routing your domain name to the instance Elastic IP address
We simply need to add an A record containing our Elastic IP address to our registrar DNS zone. Using AWS Route 53, create a new record then select "Simple routing", then "Define simple record".
Having done that your instance should be accessible via your domain name using http. Though for security reasons we should always ensure that users are connected through https.
If Amazon Certificate Manager can be used to generate a SSL certificate at no cost, the certificate can only be used in combination with a load balancer. Indeed there is no way to obtain the whole chain and the private keys of a certificate generated with ACM and install it directly on the server.
Another way to enable TLS on our web application is to obtain a free certificate from Let's Encrypt.
Let’s Encrypt provides basic SSL/TLS certificates for DNS hosts for free that we can use to add encryption-in-transit to our web server. The certificate will be valid for 90 days but will be automatically renewed according to their website. Such certificates only prove domain ownership. If you need a stronger solution for your website you should take a look at commercial SSL certificates or consider using AWS Certificate Manager and Elastic Load Balancing.
Downloading and installing EPEL and certbot
Source: Extending Amazon Linux 2 with EPEL and Let’s Encrypt
Let's Encrypt requires the certbot package that we will install from EPEL. SSH into your instance from your application source folder using the following command:
(env) $ eb ssh
cd into /tmp and run the following command to download EPEL:
[ec2-user@instance-ip tmp]$ wget -O epel.rpm –nv https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
Then install it:
[ec2-user@instance-ip tmp]$ sudo yum install -y ./epel.rpm
Install and configure certbot
Run the following command to install certbot for nginx:
[ec2-user@instance-ip tmp]$ sudo yum install python2-certbot-nginx.noarch
Before running certbot we need to make some modifications in our nginx.conf file so that certbot can properly parse it and update it. cd into /etc/nginx and open nginx.conf:
[ec2-user@instance-ip nginx]$ sudo vim nginx.conf
Replace default_server by your domain name, or create a server block for each of your domains and subdomains. If you only have one domain name and want your users to access your application using both yourdomain.com and www.yourdomain.com you can use the following wild card:
49 server { 50 listen 80; 51 server_name .yourdomain.com; # This will match both yourdomain.com and www.yourdomain.com 52 }
This will be enough for certbot to properly update nginx.conf and add a server block listening on port 443 with properly configured SSL.
Obtaining an SSL certificate from Let's Encrypt using certbot
Run the following command to launch the certbot utility for nginx:
[ec2-user@instance-ip nginx]$ sudo certbot --nginx
You should be prompted with the following options:
Saving debug log to /var/log/letsencrypt/letsencrypt.log Plugins selected: Authenticator nginx, Installer nginx No names were found in your configuration files. Please enter in your domain name(s) (comma and/or space separated) (Enter 'c' to cancel): yourdomain.com www.yourdomain.com
Enter your domain names. If you want to make your website accessible using both its www. and non www names you should enter both.
You will then be prompted to enter your email address, accept the terms of service and some other information. If everything went fine certbot generated your certificates and modified nginx.conf so that they can be used by the server. If it fails make sure that the server_name fields of your server blocks in nginx.conf match the domain names you entered in certbot.
25 server { 26 listen 443 ssl; 27 server_name .yourdomain.com; 28 29 ssl_certificate /etc/letsencrypt/live/www.yourdomain.com/fullchain.pem; 30 ssl_certificate_key /etc/letsencrypt/live/www.yourdomain.com/privkey.pem; 31 include /etc/letsencrypt/options-ssl-nginx.conf; 32 ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
Note: the letsencrypt folder (located in /etc) generated during the process and holding your certificates won't be deleted nor modified on deployment but I would still recommend making a copy of it if your instance happened to be terminated for any reason.
nginx.conf will be overwritten by the one Elastic Beanstalk provides by default on every deployment so we need to replace it with the correct one. Note that we still need to make a few modifications to properly configure our server. Let's copy it to our local machine to make these modifications.
Updating nginx configuration
Note: the following paragraph is not meant to be exhaustive nor provide the absolue best way to configure nginx. There might be some improvements that could be made regarding routing and SSL. Feel free to update it as you need.
The first thing we can do is make sure that our application is only accessible through its domain name on port 80 (http) and 443 (https). But we want to redirect any non secure http request to its https counterpart. Thus we want to have:
  • One server block listening on port 443 having .yourdomain.com as its server_name value (remember that the wild card '.' before the server_name value will catch both yourdomain.com and *.yourdomain.com.)
  • One server block listening on port 80 having also .yourdomain.com as its server_name value that will automatically redirect any traffic on port 443.
  • A final server block listening on ports 443 and 80 catching any traffic not coming from the domain name. It will return 444, a nginx specific http code meaning that the connection was closed without sending a response.
Thus we want to add the following server blocks to ngix.conf:
25 server { 26 listen 443 ssl; 27 server_name .yourdomain.com; 28 29 ssl_certificate /etc/letsencrypt/live/www.yourdomain.com/fullchain.pem; 30 ssl_certificate_key /etc/letsencrypt/live/www.yourdomain.com/privkey.pem; 31 include /etc/letsencrypt/options-ssl-nginx.conf; 32 ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; 33 34 access_log /var/log/nginx/access.log main; 35 36 # Include the Elastic Beanstalk generated locations 37 include conf.d/elasticbeanstalk/*.conf; 38 } 39 40 server { 41 listen 80; 42 server_name .yourdomain.com; 43 44 return 301 https://$host$request_uri; 45 } 46 47 server { 48 listen 80; 49 listen 443 default_server ssl; 50 51 ssl_certificate /etc/letsencrypt/live/www.yourdomain.com/fullchain.pem; 52 ssl_certificate_key /etc/letsencrypt/live/www.yourdomain.com/privkey.pem; 53 54 return 444; 55 }
Using http compression (gzip) over https opens up a security BREACH that doesn't seem to have been completely mitigated yet. To keep it simple I'll juste turn gzip off on our server for the moment but know that the security researchers who discovered BREACH mention other ways to mitigate this attack on their website.
39 gzip off; 40 gzip_comp_level 4; 41 gzip_min_length 1000; 42 gzip_types text/plain text/css application/json application/javascript application/x-javascript text/xml application/xml application/xml+rss text/javascript;
Finally here is what our final nginx.conf should look like:
1 user nginx; 2 error_log /var/log/nginx/error.log warn; 3 pid /var/run/nginx.pid; 4 worker_processes auto; 5 worker_rlimit_nofile 32136; 6 7 events { 8 worker_connections 1024; 9 } 10 11 http { 12 include /etc/nginx/mime.types; 13 default_type application/octet-stream; 14 15 log_format main '$remote_addr - $remote_user [$time_local] "$request" ' 16 '$status $body_bytes_sent "$http_referer" ' 17 '"$http_user_agent" "$http_x_forwarded_for"'; 18 19 include conf.d/*.conf; 20 21 map $http_upgrade $connection_upgrade { 22 default "upgrade"; 23 } 24 25 server { 26 listen 443 ssl; 27 server_name .yourdomain.com; 28 29 ssl_certificate /etc/letsencrypt/live/www.yourdomain.com/fullchain.pem; 30 ssl_certificate_key /etc/letsencrypt/live/www.yourdomain.com/privkey.pem; 31 include /etc/letsencrypt/options-ssl-nginx.conf; 32 ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; 33 34 access_log /var/log/nginx/access.log main; 35 36 client_header_timeout 60; 37 client_body_timeout 60; 38 keepalive_timeout 60; 39 gzip off; 40 gzip_comp_level 4; 41 gzip_min_length 1000; 42 gzip_types text/plain text/css application/json application/javascript application/x-javascript text/xml application/xml application/xml+rss text/javascript; 43 44 # Include the Elastic Beanstalk generated locations 45 include conf.d/elasticbeanstalk/*.conf; 46 } 47 48 server { 49 listen 80; 50 server_name .yourdomain.com; 51 51 # Redirect all http requests to port 443 52 return 301 https://$host$request_uri; 53 } 54 55 server { 56 listen 80; 57 listen 443 default_server ssl; 58 59 ssl_certificate /etc/letsencrypt/live/www.yourdomain.com/fullchain.pem; 60 ssl_certificate_key /etc/letsencrypt/live/www.yourdomain.com/privkey.pem; 61 62 # Close the connection and return no response 63 return 444; 64 } 65 66 }
If the letsencrypt folder containing the certificates should not be removed on application deployment, nginx.conf is automatically overwritten by the default one Elastic Beanstalk provides. To replace nginx.conf with the one properly configured we have to create a .config file in .ebextension in which we will create nginx.conf in the instance /tmp folder. Then we will replace the original nginx.conf file by our custom one using a script located in in .platform.
Source: Extending Elastic Beanstalk Linux platforms
~/flask_app/ |-- application.py |-- static | |-- styles.css |-- templates | |-- index.html |-- readme.md |-- .ebextensions/ | |-- 01_write_custom_ng_conf.config |-- .platform/ |-- hooks |-- postdeploy |-- 01_replace_ng_conf.sh # Executed post deployment
Create a .config file at the root of .ebextensions: 01_write_custom_ng_conf.config
Create your modified nginx.conf file and place it in /tmp.
1 files: 2 "/tmp/custom_nginx.conf": 3 mode: "000755" 4 owner: root 5 group: root 6 content: | 7 # nginx.conf content goes here
01_replace_ng_conf.sh
Create a small script in .platform/hooks/postdeploy and change permissions to 755.
(env) $ touch .platform/hooks/postdeploy/01_replace_ng_conf.sh (env) $ chmod 755 .platform/hooks/postdeploy/01_replace_ng_conf.sh
Add the following lines:
1 #!/usr/bin/bash 2 3 # Replace the original nginx.conf by our custom one 4 sudo mv /tmp/custom_nginx.conf /etc/nginx/nginx.conf 5 6 # Restart nginx to apply modifications 7 sudo service nginx restart
Make sure that your .config files indentation is perfect as errors are not always detected by the parser and the code won't be executed.
If everything went fine our web application should be accessible only via https AND through its domain name. Note that some improvements could still be made regarding SSL and nginx configuration but this will probably be discussed in another article.