DevOps tips & tricks: AWS

Showing posts with label AWS. Show all posts

Sunday, April 26, 2020

AWS Lambda using Python

AWS HOWTO
https://docs.aws.amazon.com/lambda/latest/dg/getting-started.html

AWS repository
https://github.com/awsdocs/aws-lambda-developer-guide

Check prerequisities

[dave@dave ~]$ aws --version

aws-cli/2.0.1 Python/3.7.3 Linux/5.5.16-200.fc31.x86_64 botocore/2.0.0dev5

[dave@dave ~]$ python --version

Python 3.7.6

Python Lambda

import json

 
def lambda_handler(event, context):

    # TODO implement

    return {

        'statusCode': 200,

        'body': json.dumps('Hello from Lambda!')

    }

Get lambda in AWS CLI

[dave@dave aws-lambda-developer-guide]$  aws lambda get-function --function-name dave-test-function

{

    "Configuration": {

        "FunctionName": "dave-test-function",

        "FunctionArn": "arn:aws:lambda:eu-central-1:45454545454:function:dave-test-function",

        "Runtime": "python3.8",

        "Role": "arn:aws:iam::45454545:role/dave-lambda-role-test",

        "Handler": "lambda_function.lambda_handler",

        "CodeSize": 299,

        "Description": "",

        "Timeout": 3,

        "MemorySize": 128,

        "LastModified": "2020-04-26T07:53:06.901+0000",

        "CodeSha256": "45454545454545",

        "Version": "$LATEST",

        "TracingConfig": {

            "Mode": "PassThrough"

        },

        "RevisionId": "454545454545",

        "State": "Active",

        "LastUpdateStatus": "Successful"

    },

    "Code": {

        "RepositoryType": "S3",

        "Location": "https://awslambda-eu-cent-1-tasks.s3.eu-central-1.amazonaws.com/snapshots/454534345656/dave-test-function-456544562342423424"

    }

}

Wednesday, November 2, 2016

Run Docker JBoss Wildfly image on AWS ECS

Install Docker on AWS ECS
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/docker-basics.html

       __|  __|_  )
       _|  (     /   Amazon Linux AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-ami/2016.09-release-notes/
[ec2-user@ip-172-31-31-199 ~]$ sudo yum update -y
Loaded plugins: priorities, update-motd, upgrade-helper
No packages marked for update
[ec2-user@ip-172-31-31-199 ~]$  sudo yum install -y docker
Loaded plugins: priorities, update-motd, upgrade-helper
Resolving Dependencies
--> Running transaction check
---> Package docker.x86_64 0:1.11.2-1.6.amzn1 will be installed
--> Processing Dependency: xfsprogs for package: docker-1.11.2-1.6.amzn1.x86_64
--> Running transaction check
---> Package xfsprogs.x86_64 0:3.2.2-2.20.amzn1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package         Arch          Version                   Repository        Size
================================================================================
Installing:
 docker          x86_64        1.11.2-1.6.amzn1          amzn-main         17 M
Installing for dependencies:
 xfsprogs        x86_64        3.2.2-2.20.amzn1          amzn-main        1.7 M

Transaction Summary
================================================================================
Install  1 Package (+1 Dependent package)

Total download size: 18 M
Installed size: 59 M
Downloading packages:
(1/2): docker-1.11.2-1.6.amzn1.x86_64.rpm                |  17 MB     00:00     
(2/2): xfsprogs-3.2.2-2.20.amzn1.x86_64.rpm              | 1.7 MB     00:00     
--------------------------------------------------------------------------------
Total                                               21 MB/s |  18 MB  00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : xfsprogs-3.2.2-2.20.amzn1.x86_64                             1/2 
  Installing : docker-1.11.2-1.6.amzn1.x86_64                                                        2/2 
  Verifying  : docker-1.11.2-1.6.amzn1.x86_64                                                        1/2 
  Verifying  : xfsprogs-3.2.2-2.20.amzn1.x86_64                                                      2/2 

Installed:
  docker.x86_64 0:1.11.2-1.6.amzn1                                                                       

Dependency Installed:
  xfsprogs.x86_64 0:3.2.2-2.20.amzn1                                                                     

Complete!
[ec2-user@ip-172-31-31-199 ~]$ sudo service docker start
Starting cgconfig service:                                 [  OK  ]
Starting docker:    .                                  [  OK  ]
[ec2-user@ip-172-31-31-199 ~]$ sudo usermod -a -G docker ec2-user
[ec2-user@ip-172-31-31-199 ~]$

Verify Docker status

[ec2-user@ip-172-31-31-199 ~]$ docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 1.11.2
Storage Driver: devicemapper
 Pool Name: docker-202:1-394626-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 11.8 MB
 Data Space Total: 107.4 GB
 Data Space Available: 7.223 GB
 Metadata Space Used: 581.6 kB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.147 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Either use `--storage-opt dm.thinpooldev` or use `--storage-opt dm.no_warn_on_loop_devices=true` to suppress this warning.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.93-RHEL7 (2015-01-28)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: null host bridge
Kernel Version: 4.4.23-31.54.amzn1.x86_64
Operating System: Amazon Linux AMI 2016.09
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 995.4 MiB
Name: ip-172-31-31-199
ID: IEGJ:BLTG:7DGR:PNN7:SDZS:3M7H:L6LR:CB3M:52IV:SG2R:O3KF:NZPI
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
[ec2-user@ip-172-31-31-199 ~]$

Run JBoss Wildfly Docker image

[ec2-user@ip-172-31-31-199 ~]$ docker run -it -p 8080:8080 jboss/wildfly 
=========================================================================

  JBoss Bootstrap Environment

  JBOSS_HOME: /opt/jboss/wildfly

  JAVA: /usr/lib/jvm/java/bin/java

  JAVA_OPTS:  -server -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true

=========================================================================

21:14:22,924 INFO  [org.jboss.modules] (main) JBoss Modules version 1.5.2.Final
21:14:23,279 INFO  [org.jboss.msc] (main) JBoss MSC version 1.2.6.Final
21:14:23,368 INFO  [org.jboss.as] (MSC service thread 1-1) WFLYSRV0049: WildFly Full 10.1.0.Final (WildFly Core 2.2.0.Final) starting

List Docker processes

[ec2-user@ip-172-31-31-199 ~]$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
0a5959e5cd9e        jboss/wildfly       "/opt/jboss/wildfly/b"   9 minutes ago       Up 9 minutes        0.0.0.0:8080->8080/tcp   adoring_allen

Connect to Wildfly with links browser from other shell

Install links and connect to http:localhost:8080


[ec2-user@ip-172-31-31-199 ~]$ yum install links

[ec2-user@ip-172-31-31-199 ~]$ links http://localhost:8080

Saturday, September 24, 2016

AWS IAM setup for CodeDeploy

Create new IAM user using AWS console

https://console.aws.amazon.com/iam/home#users

Create new user awsdave

Download user credentials into file

Create new policies using AWS console

Required role policies samples are here https://github.com/andrewpuch/code_deploy_example/blob/master/IAM.txt

Create new custom policy CodeDeploy and
EC2CodeDeploy

Create new roles using AWS console

https://console.aws.amazon.com/iam/home#roles

Create roles and assign policy as described in AWS tutorial video
https://www.youtube.com/watch?v=qZa5JXmsWZs&list=LLl1AVDEQMsLIiTYI5M-v0ow&index=1

Create role CodeDeploy
Create role EC2CodeDeploy

Edit trust relationship for the created Role if needed

Launch EC2 instance

Select EC2 instance IAM role EC2CodeDeploy

Policy CodeDeploy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "autoscaling:PutLifecycleHook",
                "autoscaling:DeleteLifecycleHook",
                "autoscaling:RecordLifecycleActionHeartbeat",
                "autoscaling:CompleteLifecycleAction",
                "autoscaling:DescribeAutoscalingGroups",
                "autoscaling:PutInstanceInStandby",
                "autoscaling:PutInstanceInService",
                "autoscaling:DescribeLifecycleHooks",
                "ec2:Describe*"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
}

Trust Relationship

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": [
          "codedeploy.us-east-1.amazonaws.com",
          "codedeploy.us-west-2.amazonaws.com",
          "codedeploy.ap-northeast-2.amazonaws.com",
          "codedeploy.ap-southeast-2.amazonaws.com",
          "codedeploy.ap-southeast-1.amazonaws.com",
          "codedeploy.us-west-1.amazonaws.com",
          "codedeploy.eu-west-1.amazonaws.com",
          "codedeploy.ap-south-1.amazonaws.com",
          "codedeploy.eu-central-1.amazonaws.com",
          "codedeploy.ap-northeast-1.amazonaws.com",
          "codedeploy.sa-east-1.amazonaws.com"
        ]
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Thursday, August 4, 2016

Deploy Java application from GitHub to AWS

HOWTO from AWS tutorial

AWS tutorial video Code Deploy Setup (IAM, EC2)

https://www.youtube.com/watch?v=qZa5JXmsWZs

GitHub for AWS tutorial video https://github.com/andrewpuch/code_deploy_example/

AWS CodeDeploy tutorial video https://www.youtube.com/watch?v=jcR9iIWdU7E

Steps to create running AWS EC2 instance

Create AWS free account

Getting started :
https://aws.amazon.com/free/

Create AWS account

Create own Linux instance
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html

Connect to AWS Linux instance via ssh

[dave@localhost aws]$ chmod 400 dave_amazon.pem
[dave@localhost aws]$ ssh -i "dave_amazon.pem" ec2-user@ec2-11-123-765-44.us-west-2.compute.amazonaws.com


       __|  __|_  )
       _|  (     /   Amazon Linux AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-ami/2016.03-release-notes/
8 package(s) needed for security, out of 17 available
Run "sudo yum update" to apply all updates.

Install AWS tools on Fedora

https://fedoramagazine.org/aws-tools-fedora/

[root@localhost ~]# dnf install  awscli
Dependencies resolved.
=================================================================================================
 Package                  Arch         Version                               Repository     Size
=================================================================================================
Installing:
 awscli                   noarch       1.10.45-1.fc23                        updates       868 k
 python3-botocore         noarch       1.4.35-1.fc23                         updates       1.8 M
 python3-colorama         noarch       0.3.2-2.fc23                          fedora         29 k
 python3-docutils         noarch       0.12-0.3.20140510svn7747.fc23         fedora        1.5 M
 python3-jmespath         noarch       0.9.0-1.fc23                          updates        43 k
 python3-pyasn1           noarch       0.1.8-1.fc23                          fedora        108 k
 python3-rsa              noarch       3.4.1-1.fc23                          updates        72 k
 python3-s3transfer       noarch       0.0.1-2.fc23                          updates        29 k

Transaction Summary
=================================================================================================
Install  8 Packages

Total download size: 4.5 M
Installed size: 27 M
Is this ok [y/N]: 
Paste your text here.

[dave@localhost aws]$ aws --version
aws-cli/1.10.45 Python/3.4.3 Linux/4.5.7-200.fc23.x86_64 botocore/1.4.35

Add AWS IAM access keys

[dave@localhost aws]$ aws configure

Check running instances

[dave@localhost aws]$ aws ec2 describe-instances 
{
    "Reservations": [
        {
            "Instances": [
                {
                    "PublicDnsName":

Integrating AWS CodeDeploy with GitHub

https://docs.aws.amazon.com/codedeploy/latest/userguide/github-integ.html

Setup roles in IAM
https://docs.aws.amazon.com/codedeploy/latest/userguide/getting-started-setup.html

Amazon AWS Java samples
https://github.com/aws/aws-sdk-java/tree/master/src/samples

Create GitHub repository with sample application
https://github.com/dveselka/aws-java-sample

User guide GitHub - AWS integration
https://docs.aws.amazon.com/codedeploy/latest/userguide/github-integ-tutorial.html

Create application - see tutorial page

Step 5: Deploy the Application to the Instance

https://docs.aws.amazon.com/codedeploy/latest/userguide/github-integ-tutorial.html#github-integ-tutorial-deploy

Connect with GitHub

Deploy AWS sample Linux
https://github.com/dveselka/aws-sample-linux

Check AWS CodeDeploy console

Deploy fails with HEALTH_CONSTRAINTS

It is necessary to install CodeDeploy on instance
Follow this tutorial :

Install or reinstall the AWS CodeDeploy agent for Amazon Linux or RHEL

https://docs.aws.amazon.com/codedeploy/latest/userguide/how-to-run-agent-install.html#how-to-run-agent-install-linux

[ec2-user@ip-172-31-21-129 ~]$ wget https://aws-codedeploy-eu-central-1.s3.amazonaws.com/latest/install
--2016-08-07 13:45:20--  https://aws-codedeploy-eu-central-1.s3.amazonaws.com/latest/install
Resolving aws-codedeploy-eu-central-1.s3.amazonaws.com (aws-codedeploy-eu-central-1.s3.amazonaws.com)... 54.231.193.44
Connecting to aws-codedeploy-eu-central-1.s3.amazonaws.com (aws-codedeploy-eu-central-1.s3.amazonaws.com)|54.231.193.44|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 13359 (13K) []
Saving to: ‘install’

install                    100%[======================================>]  13.05K  78.6KB/s    in 0.2s    

2016-08-07 13:45:21 (78.6 KB/s) - ‘install’ saved [13359/13359]

Check installation success

[ec2-user@ip-172-31-21-129 ~]$ sudo service codedeploy-agent status
The AWS CodeDeploy agent is running as PID 2660

Deploy application from GithHub using CodeDeploy - Deployments

Access application using public IP address

Check log on EC2 instance

c2-user]# tail -500f /var/log/httpd/access_log 
189.278.189.245 - - [24/Sep/2016:18:03:53 +0000] "GET / HTTP/1.1" 200 911 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0"

Sample app Linux
https://github.com/awslabs/aws-codedeploy-samples/tree/master/applications/SampleApp_Linux

Sample Tomcat application on GitHub
https://github.com/awslabs/aws-codedeploy-sample-tomcat

You can also download sample app from Amazon S3

[dave@localhost aws]$ aws s3 cp s3://aws-codedeploy-eu-central-1/samples/latest/SampleApp_Linux.zip . --region eu-central-1
download: s3://aws-codedeploy-eu-central-1/samples/latest/SampleApp_Linux.zip to ./SampleApp_Linux.zip

[dave@localhost aws]$ unzip -l SampleApp_Linux.zip 
Archive:  SampleApp_Linux.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
       34  11-03-2014 22:42   scripts/install_dependencies
       33  11-03-2014 22:43   scripts/start_server
      105  11-03-2014 22:43   scripts/stop_server
      359  11-03-2014 22:42   appspec.yml
      717  11-03-2014 22:42   index.html
    10884  11-03-2014 22:42   LICENSE.txt
---------                     -------
    12132                     6 files

DevOps tips & tricks