My private notes in testing the AWS resources

Setting up initial settings

Create a root account that as all permissions

  1. It has permission to everything so its better to not use this user and instead create another sets of users in groups with limited permissions
  2. Create a dev group and use its credential
    1. Created a Group with Admin access “developer”
    2. Create a user ddd1
      1. Users with AWS Management Console access can sign-in at: https://0000000000.signin.aws.amazon.com/console
      2. Name : ddd1 | Pass : XXXXXX
    3. Use above link to login and reset the password automatically
    4. In the dashboard after logging in select security credentials

Create an access Key

  1. In Account -> security credentials -> Access keys for CLI, SDK &
  2. Create an access key. And make a note of a credentials in a secure password store like one password, etc
  3. Took screenshot
  4. Do not create an access key for your root account.

Set up billing alarm

  1. Search for billing and then billing preferences check the required boxes
  2. Click in manage billing link to go to CloudWatch where you can monitor a lot of settings and create billing alarms as needed
  3. Create an alarm based on usage/or cost and set it up to send a notification to the user email when the metric exceeds the marked value in the monitoring SNS
  4. Once alarm created need to confirm subscription in the email.

History

  1. EC2 (Elastic Cloud Compute) and S3 (Simple Storage Service) launched initially to offer Infrastructure ax a Service : Rent infrastructure from cloud provider and pay only for the usage
  2. Dropbox used S3 and EC2 internally initially.
  3. Shared Responsibility model. Both AWS and provisioner have responsibilities to keep the server running

Availability zones

  1. In order to maintain availability high AWS maintains several redundant data centers. Recommend to have multi AZ

EC2

  1. Search and Select EC2 from the search option
  2. Launch Instance -> select Ubuntu Free Tier eligible AMI (Amazon Machine Image)
  3. Create a new key-pair for connecting to the launched VM instance
  4. Then launch the instance
  5. View instance -> rename the EC2 instance to any name of your choice ( I named it webserver1)
  6. Select the instance and click on the connect button at the top.

Setup the credentials for connecting

  1. Move the downloaded .cer file to within your .ssh/ folder
  2. Run the command : chmod 400 ~/.ssh/awsdemo.cer
  3. This set permission for the file such that it is readable only by the local user
  4. Then run: ssh -i “~/.ssh/awsdemo.cer” ubuntu@ec2-13-114-188-11.ap-northeast-1.compute.amazonaws.com
  5. After cross checking the location of the cer file and the instance the connection was successful
  6. There are different types of CPUs for the Ec2 instance you can choose
    1. t type CPU’s accrue credits when below use that they can consume during burst use
    2. For consistent loads use c for CPU load and m for memory intensive usage
    3. For gpu intensive loads use accelerated computing
  7. After running the ssh command in the terminal your terminal must have ssh into the instance , now run the update command : sudo apt-get update
  8. Run : sudo apt-get install apache2. This is the first step to installing a Wordpress instance should you need it. Look for digital ocean for a great tutorial on how to proceed if needed
  9. Tried to connect to the exposed public DNS setting, but it seems that it is blocked : ec2-54-199-194-212.ap-northeast-1.compute.amazonaws.com . Seems we need to fix the security group setting in the AWS

EC2Console

Security Groups

  • Allows amazon to expose only those parts of server that is configured to be exposed to external traffic
  • Currently by default on port 22 (SSH) traffic is allowed by default and to connect from web we would need to expose traffic over port 80 
  • Set the Security Group and edit the inbound rule to edit the inbound rules to allow SSH traffic from your current IP only. Need to do this each time the IP changes

roles

  • Also add a new rule to allow Http traffic from anywhere in the same page.
  • Can verify the exposed http endpoint to show the web server page

Autoscaling

  • AMI is a great way to take snapshot of a correctly working image and deploy the same image again in a possibly different region, etc with updated or exact resources, etc
  • Note: remember to terminate any active or stopped instances to stop running up charges on provisioned storage

IaaS Storage

Elastic Block Storage(EBS)

Usually attached to one EC2 and created by default with EC2. Sits closer to the EC2 instance.

Elastic File System

  • Amazon’s version of NAS (Network Attached Storage). Allows multiple EC2 instances to mount and use this drive simultaneously.
  • Create a new EFS system with default options
  • SSH into an EC2 instance and install the needed library for mounting EFS storage. Sudo apt-get install nfs-common
  • Now create a mount point in the EC2 instance
    • cd / - go to root
    • mkdir efs
    • Run the command to attach from the EFS Attack button popup dialog
    • sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport fs-0fd4b3b45e63255f1.efs.ap-northeast-1.amazonaws.com:/ efs
    • Realize the security role is blocking traffic from the EFS drive
    • Added the default security group that adds rule to stack on top of the existing security group. Thus you can add access to anything that’s using a security roles rather than having to look through IP addresses or ranges

S3 (Simple storage Service)

  • Create a new bucket
  • Create a folder and upload a file.
  • However the default permissions is that all files are private and hence cannot be accessed publicly. Need to update the policy for the bucket

Installing AWS CLI

  • Install from UI : https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html
  • Check status with was —version
  • Run: aws configure
  • Set you access key info and region name and output format type
  • Run : aws s3api list-buckets
    • To check details information about the buckets
  • Run: aws s3 ls
    • To list all buckets in a line
  • Run : aws s3 cp ~/Downloads/amz-logo.jpg s3://bbucket07/
    • To copy a file to the root of the bucket
    • It takes some time to propagate, but it worked.
  • IT also comes with an aws sync command to sync the on premise folders with the online s3 bucket

s3bucket 

Access the S3 bucket from the EC2 instance

IAM roles for EC2

IAMEC2

  • Give a name for this role and create the role
  • Attach this role to any of the EC2 instance you want to grant permission to.
S3 Glacier
  • Cheapest storage class in S3. Great for server log files , etc that you don’t access frequently or that you keep for auditing purposes.
  • Can automatically configure lifecycle to move unacgessed file to Glacier storage automatically

CloudFront

  • CDN for serve s3 files from for the closest servers to the client requestors.
  • Very useful for static websites / react websites that need to load the initial data.

IaaS Networking

VPC

  • A simple house router creates a private network similar to how
  • Internal network that allows agents to communicate within a network environment without the request having to traverse through the public internet.
  • Subnets allow you to divide the VPC into distinct black with different access permissions
  • Network Address translation : Routes traffic from public network to private devices. A Nat Gateway in AWS performs this function

Elastic IPs

  • Comes from a pool of AWS IP addresses that are bound to your account
  • Public IP are a limited resource hence release them when not in use
  • Provision/Attach and release the resource from the console.

VPN’s

  • Used to securely access/transfer data in private subnets

Elastic Load Balancer (ELB)

  • Balances incoming traffic among several instances for horizontal scaling
  • Configured through the EC2 console.
  • Are of two types
    • Network Load Balancers
      • Faster, less features as they operate on the 4rth layer of OSI model
    • Application Load Balancer
      • HTTP routing rules and operates in the 7th layer
      • Mostly used
  • An ALB is used for HTTP traffic, and an NLB is used for traffic that requires speed, like low-latency streaming services

Route53

  • AWS DNS hosting service
  • Named because all internet DNS communications occur over port 53.
  • Similar to Google domains

Database as a Service( DaaS )

  • Database Migration service : provides a live relational DB migration service for smooth migration
  • Relational Database Service (RDS) - AWS managed RDMS. It supports the most common relational databases like
    • PostgreSQL
    • MySQL
    • SQL Server
    • Oracle
  • Amazon Aurora provides more managed tools and features for Relational Database management in AWS and even runs a server less managed so that you don’t have to manage instances , etc its a great choice
  • DynamoDB : NoSQL AWS offering
  • DocumentDB : similar to MongoDB
  • Elasticache : In-memory cache, helps you deploy two types of caches
    • Redis : store as a datatype in an object and thus can update value in place without replacing the entire value
    • Memcached : store value as string
  • Redshift : Big data store as a data warehouse
    • Data lake : stash of unorganized/unsorted data. Eg : GB’s of raw testifies stored in S3 buckets
    • Data warehouse : organized data with fixed data type usually stored in a relational database
    • Has a heavily modded army of PostGreSQL servers
  • EMR : Is like Elasticache for datastores in that it makes it easier to manage big data clusters. Similar to Apache Spark, Hadoop.
  • Athena : Suitable for querying huge collections of text files in S3 buckets.

Messaging Services

  • Similar to Google Pub/Sub offerings
  • Two main offerings in AWS are
    • Kinesis : Can handle large stream of incoming data. Can connect to other service to perform real-time reporting
    • SQS (simple Queue service) : Simple but can get very expensive if the number of request become huge.
  • Simple Notification Service (SNS) : Can send a simple message, email to a http web hook , etc

Platform as a service (PaaS)

Elastic Beanstalk

  • AWS uses the same basic building blocks for these higher layer abstracted code.
  • Aws takes in the code and manages most of the manual operations needed to run and sclae the system
  • Internally they use the same EC2 instances.

ECS Elastic Container Service

  • Will manage the EC2 instances automatically
  • Elastic Container Register is similar to GCR (Google Container registry)
  • Fargate is like a job that runs only once and exits

Lambda

  • Functions as a service.
  • Runs a single executable code. This form of architecture is called Serverless Architecture. An application that responds to incoming events without the need for always-on servers that you manage.
  • Can run only 1 major task and can run for around 15 min.

Batch Process

  • AWS batch : a simple full job, that can take as long as it want to process data. Its a great way to manage and schedule jobs that can use EC2 spot instance

EC2 spot instance

  • You can manage and bid on Amazon’s unused capacity for a steep discount. By shceduling non-critical jobs you can get some serious discount by having some flexibility on when some jobs get done.

Step Functions

  • Multi step workflows you need to manage a series of tasks, it makes it easier to read and organize a long list of tasks
  • AWS Simple Workflows (SWF) : surprisingly are used to manage a series of complicated workflows with results of one task indicating a different task to be executed.

Software as a Service ( SaaS )

  • Most of the effort in maintaining and running stuff is on AWS and not on you.
  • Cognito : can help you implement sign in integration with other providers. For exmaple allow users to sign in with google or facebook logins.
  • API Gateway : allows you to define a rest API and forwards request to different resources based on those rules
  • AppSync : design GraphQL API backend
  • Amplify : app framework to setup components using CLI commands. Through some small commands it can create boilerplate react code for speeding up web and mobile application development. IT can even help with native iOS development by giving user authentication, online file storage, push notifications, analytics, etc.
  • Sagemaker : build and train machine learning models without having to spend time settting up servers, etc.
  • Comprehend : Performs sentiment analysis
  • Lex : Build a chatbot
  • Personalize : recommend product to a user based on their shopping habits
  • Polly : narrate dynamic text into pretty lifelike voices
  • Rekognition : can find faces/text in an image
  • Textract : Process large number of text document and automate commonly used functions.
  • Translate : detect and translate input text into several languages.
  • Transcribe : translate voice to text

Media services

  • MediaConvert : takes files from input s# bucket and convert and optimize the file size and perform additional actions like adding a watermark.

DevOps with AWS

  • CI/CD : allows you to release early and often.
  • Continous Integration : through CodePipeline by pulling code from CodeCommit and run some automated tests using CodeBuild.
  • Continuous Deployment : tested changes are deployed automatically to Staging environment and can be utomatically sent to production. CodeDeploy can continuously deploy your tested code from container platform (ECS) to EC2.
  • Infrastructure as Code : three most popular open source tools are
    • Puppet^*
    • Chef^*
    • Ansible
  • AWS has tool called AWS OpsWorks to easily manage these tools
  • CloudFormation : is the AWS proprietary offering to match the Open Source alternative.
  • CloudWatch : Monitor and record metrics. It ties into almost all resource in AWS. It cannot dive into code to find the root cause.

Security on AWS

  • Web Application firewall (WAP) : connects to loadbalancer. Can implement features like filtering traffic based on geography.
  • Shield : Mitigate DoS attacks
  • GuardDuty : monitors for suspicious activity or accesses
  • Inspector: monitors for unpatched code or vulnerabilities, that you need to fix. ITs like a full virsu scan.
  • Macie : scans your resources for any publicly shared vulnerabilities and reports to you.
  • CloudTrail : Audit trail of changes made internally.
  • Security hub : Dashboard of security status

REF

  1. https://explore.skillbuilder.aws/learn/course/external/view/elearning/1851/aws-technical-essentials?dt=tile&tile=fdt
  2. Designing Data Intensive applications - by Martin Kleppmann
  3. https://www.linkedin.com/learning/aws-essential-training-for-developers/