DevOps tips & tricks: June 2021

Monday, June 7, 2021

HOWTO

https://docs.microsoft.com/en-us/azure/azure-functions/create-first-function-cli-java?tabs=bash%2Cazure-cli%2Cbrowser

GitHub

https://github.com/dveselka/devops-terraform/tree/master/azure/function/az-func

Create function using Maven

[dave@dave function]$ mvn archetype:generate -DarchetypeGroupId=com.microsoft.azure -DarchetypeArtifactId=azure-functions-archetype -DjavaVersion=11
[INFO] Scanning for projects...
[INFO] 
[INFO] ------------------< org.apache.maven:standalone-pom >-------------------
[INFO] Building Maven Stub Project (No POM) 1
[INFO] --------------------------------[ pom ]---------------------------------
[INFO] 
[INFO] >>> maven-archetype-plugin:3.1.2:generate (default-cli) > generate-sources @ standalone-pom >>>
[INFO] 
[INFO] <<< maven-archetype-plugin:3.1.2:generate (default-cli) < generate-sources @ standalone-pom <<<
[INFO] 
[INFO] 
[INFO] --- maven-archetype-plugin:3.1.2:generate (default-cli) @ standalone-pom ---
[INFO] Generating project in Interactive mode
[INFO] Archetype [com.microsoft.azure:azure-functions-archetype:1.39] found in catalog remote
Downloading from central: https://repo.maven.apache.org/maven2/com/microsoft/azure/azure-functions-archetype/1.39/azure-functions-archetype-1.39.pom
Downloaded from central: https://repo.maven.apache.org/maven2/com/microsoft/azure/azure-functions-archetype/1.39/azure-functions-archetype-1.39.pom (1.8 kB at 38 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/com/microsoft/azure/azure-functions-archetype/1.39/azure-functions-archetype-1.39.jar
Downloaded from central: https://repo.maven.apache.org/maven2/com/microsoft/azure/azure-functions-archetype/1.39/azure-functions-archetype-1.39.jar (18 kB at 457 kB/s)
Define value for property 'groupId' (should match expression '[A-Za-z0-9_\-\.]+'): com.dave
[INFO] Using property: groupId = com.dave
Define value for property 'artifactId' (should match expression '[A-Za-z0-9_\-\.]+'): az-func
[INFO] Using property: artifactId = az-func
Define value for property 'version' 1.0-SNAPSHOT: : 
Define value for property 'package' com.dave: : 
[INFO] Using property: appName = $(artifactId)-$(timestamp)
[INFO] Using property: appRegion = westus
[INFO] Using property: appServicePlanName = java-functions-app-service-plan
[INFO] Using property: docker = false
[INFO] Using property: javaVersion = 11
[INFO] Using property: resourceGroup = java-functions-group
[INFO] Using property: trigger = HttpTrigger
Confirm properties configuration:
groupId: com.dave
groupId: com.dave
artifactId: az-func
artifactId: az-func
version: 1.0-SNAPSHOT
package: com.dave
appName: $(artifactId)-$(timestamp)
appRegion: westus
appServicePlanName: java-functions-app-service-plan
docker: false
javaVersion: 11
resourceGroup: java-functions-group
trigger: HttpTrigger
 Y: : 
[INFO] ----------------------------------------------------------------------------
[INFO] Using following parameters for creating project from Archetype: azure-functions-archetype:1.39
[INFO] ----------------------------------------------------------------------------
[INFO] Parameter: groupId, Value: com.dave
[INFO] Parameter: artifactId, Value: az-func
[INFO] Parameter: version, Value: 1.0-SNAPSHOT
[INFO] Parameter: package, Value: com.dave
[INFO] Parameter: packageInPathFormat, Value: com/dave
[INFO] Parameter: resourceGroup, Value: java-functions-group
[INFO] Parameter: package, Value: com.dave
[INFO] Parameter: appName, Value: $(artifactId)-$(timestamp)
[INFO] Parameter: javaVersion, Value: 11
[INFO] Parameter: groupId, Value: com.dave
[INFO] Parameter: artifactId, Value: az-func
[INFO] Parameter: appServicePlanName, Value: java-functions-app-service-plan
[INFO] Parameter: trigger, Value: HttpTrigger
[INFO] Parameter: appRegion, Value: westus
[INFO] Parameter: version, Value: 1.0-SNAPSHOT
[INFO] Parameter: docker, Value: false
[INFO] Executing META-INF/archetype-post-generate.groovy post-generation script
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.codehaus.groovy.reflection.CachedClass (file:/home/dave/.m2/repository/org/codehaus/groovy/groovy/2.4.16/groovy-2.4.16.jar) to method java.lang.Object.finalize()
WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.reflection.CachedClass
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[INFO] Project created from Archetype in dir: /git/devops-terraform/azure/function/az-func
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  36.355 s
[INFO] Finished at: 2021-06-07T22:04:15+02:00
[INFO] ------------------------------------------------------------------------
[dave@dave function]$ find
.
./az-func
./az-func/host.json
./az-func/pom.xml
./az-func/local.settings.json
./az-func/src
./az-func/src/main
./az-func/src/main/java
./az-func/src/main/java/com
./az-func/src/main/java/com/dave
./az-func/src/main/java/com/dave/Function.java
./az-func/src/test
./az-func/src/test/java
./az-func/src/test/java/com
./az-func/src/test/java/com/dave
./az-func/src/test/java/com/dave/FunctionTest.java
./az-func/src/test/java/com/dave/HttpResponseMessageMock.java
./az-func/.gitignore

Deploy function to Azure

[dave@dave az-func]$ mvn azure-functions:deploy
[INFO] Scanning for projects...
[INFO] 
[INFO] --------------------------< com.dave:az-func >--------------------------
[INFO] Building Azure Java Functions 1.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- azure-functions-maven-plugin:1.12.0:deploy (default-cli) @ az-func ---
[WARNING] The POM for com.microsoft.azure.applicationinsights.v2015_05_01:azure-mgmt-insights:jar:1.0.0-beta is invalid, transitive dependencies (if any) will not be available, enable debug logging for more details
[INFO] Set function worker runtime to java.
[INFO] Auth type: OAUTH2
Default subscription: Free Trial(3434343434343434343)
Username: some.name@gmail.com
[INFO] Subscription: Free Trial(3434343434343434343)
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.codehaus.groovy.reflection.CachedClass (file:/home/dave/.m2/repository/org/codehaus/groovy/groovy/2.5.11/groovy-2.5.11.jar) to method java.lang.Object.finalize()
WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.reflection.CachedClass
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[INFO] Creating resource group java-functions-group in region westus...
[INFO] Successfully created resource group java-functions-group.
[INFO] Creating app service plan...
[INFO] Successfully created app service plan java-functions-app-service-plan.
[INFO] Creating function app az-func-20210607220415178...
[INFO] Creating application insights...
[INFO] Successfully created the application insights az-func-20210607220415178 for this Function App. You can visit https://ms.portal.azure.com/#@/resource/subscriptions/3434343434343434343/resourceGroups/java-functions-group/providers/microsoft.insights/components/az-func-20210607220415178/overview to view your Application Insights component.
[INFO] Ignoring decoding of null or empty value to:com.azure.resourcemanager.storage.fluent.models.StorageAccountInner
[INFO] Successfully created function app az-func-20210607220415178.
[INFO] Starting deployment...
[INFO] Trying to deploy artifact to az-func-20210607220415178...
[INFO] Successfully deployed the artifact to https://az-func-20210607220415178.azurewebsites.net
[INFO] Deployment done, you may access your resource through https://ms.portal.azure.com/#@/resource/subscriptions/343434343434/resourceGroups/java-functions-group/providers/Microsoft.Web/sites/az-func-20210607220415178
[INFO] Syncing triggers and fetching function information (Attempt 1/3)...
[INFO] Syncing triggers and fetching function information (Attempt 2/3)...
[INFO] HTTP Trigger Urls:
[INFO]      az-func-20210607220415178/HttpExample : https://az-func-20210607220415178.azurewebsites.net/api/httpexample
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  03:17 min
[INFO] Finished at: 2021-06-07T22:15:06+02:00
[INFO] ------------------------------------------------------------------------

Run function

https://az-func-20210607343434.azurewebsites.net/api/httpexample?name=dave

Azure function in Portal

Sunday, June 6, 2021

Azure CLI - upload file to Azure Storage Account

HOWTO

https://danielveselka.blogspot.com/2021/06/azure-networking.html

Get Account Key and upload file

[dave@dave azure]$ export ACCOUNT_KEY=`az storage account keys list --account-name davestorageaccounttf  --query [0].value -o tsv`
[dave@dave azure]$ az storage azcopy blob upload -c davecontainer --account-name davestorageaccounttf -s  /tmp/foo.txt  --account-key $ACCOUNT_KEY

Show container

[dave@dave azure]$ az storage container show --account-key  $ACCOUNT_KEY --account-name davestorageaccounttf --name davecontainer
{
  "metadata": {},
  "name": "davecontainer",
  "properties": {
    "contentLength": 0,
    "etag": "\"34343434\"",
    "hasImmutabilityPolicy": false,
    "hasLegalHold": false,
    "lastModified": "2021-06-06T20:44:21+00:00",
    "lease": {
      "duration": null,
      "state": "available",
      "status": "unlocked"
    },
    "publicAccess": "blob"
  }
}

Blob list

dave@dave azure]$ az storage blob list  --account-key  $ACCOUNT_KEY --account-name davestorageaccounttf --container-name davecontainer
This command has been deprecated and will be removed in future release. Use 'az storage fs file list' instead. For more information go to https://github.com/Azure/azure-cli/blob/dev/src/azure-cli/azure/cli/command_modules/storage/docs/ADLS%20Gen2.md
The behavior of this command has been altered by the following extension: storage-preview
[
  {
    "content": null,
    "deleted": false,
    "metadata": {},
    "name": "foo.txt",
    "properties": {
      "Content-CRC64": null,
      "appendBlobCommittedBlockCount": null,
      "blobTier": "Hot",
      "blobTierChangeTime": null,
      "blobTierInferred": true,
      "blobType": "BlockBlob",
      "contentLength": 6,
      "contentRange": null,
      "contentSettings": {
        "cacheControl": null,
        "contentDisposition": null,
        "contentEncoding": null,
        "contentLanguage": null,
        "contentMd5": "4343434==",
        "contentType": "text/plain; charset=utf-8"
      },
      "copy": {
        "completionTime": null,
        "id": null,
        "progress": null,
        "source": null,
        "status": null,
        "statusDescription": null
      },
      "creationTime": "2021-06-06T20:53:16+00:00",
      "deletedTime": null,
      "encryptionKeySha256": null,
      "etag": "0x8D9292D1B7AE62E",
      "lastModified": "2021-06-06T20:53:16+00:00",
      "lease": {
        "duration": null,
        "state": "available",
        "status": "unlocked"
      },
      "pageBlobSequenceNumber": null,
      "remainingRetentionDays": null,
      "serverEncrypted": true
    },
    "snapshot": null
  }
]

Azure networking and storage accounts

HOWTO

Github

Create Storage account with Network Rules

Terraform plan

resource "azurerm_storage_account" "davestorageaccounttf" {
  name                = "davestorageaccounttf"
  resource_group_name = azurerm_resource_group.daveterraformgroup.name

  location                 = "westeurope"
  account_tier             = "Standard"
  account_replication_type = "LRS"

  network_rules {
    default_action             = "Deny"
    ip_rules                   = ["100.0.0.1"]
    virtual_network_subnet_ids = [azurerm_subnet.daveterraformsubnet.id]
    bypass                     = ["Metrics", "AzureServices"]
  }

  tags = {
    environment = "staging"
  }
}

Add container

resource "azurerm_storage_container" "example" {
  name                  = "${var.prefix}storagecontainer"
  storage_account_name  = azurerm_storage_account.davestorageaccounttf.name
  container_access_type = "blob"
}

# azurerm_storage_account.daveterraformsa will be created
  + resource "azurerm_storage_account" "daveterraformsa" {
      + access_tier                      = (known after apply)
      + account_kind                     = "StorageV2"
      + account_replication_type         = "LRS"
      + account_tier                     = "Standard"
      + allow_blob_public_access         = false
      + enable_https_traffic_only        = true
      + id                               = (known after apply)
      + is_hns_enabled                   = false
      + large_file_share_enabled         = (known after apply)
      + location                         = "westeurope"
      + min_tls_version                  = "TLS1_0"
      + name                             = "davestorageaccount"
      + nfsv3_enabled                    = false
      + primary_access_key               = (sensitive value)
      + primary_blob_connection_string   = (sensitive value)
      + primary_blob_endpoint            = (known after apply)
      + primary_blob_host                = (known after apply)
      + primary_connection_string        = (sensitive value)
      + primary_dfs_endpoint             = (known after apply)
      + primary_dfs_host                 = (known after apply)
      + primary_file_endpoint            = (known after apply)
      + primary_file_host                = (known after apply)
      + primary_location                 = (known after apply)
      + primary_queue_endpoint           = (known after apply)
      + primary_queue_host               = (known after apply)
      + primary_table_endpoint           = (known after apply)
      + primary_table_host               = (known after apply)
      + primary_web_endpoint             = (known after apply)
      + primary_web_host                 = (known after apply)
      + resource_group_name              = "rg-dave-terraform-test"
      + secondary_access_key             = (sensitive value)
      + secondary_blob_connection_string = (sensitive value)
      + secondary_blob_endpoint          = (known after apply)
      + secondary_blob_host              = (known after apply)
      + secondary_connection_string      = (sensitive value)
      + secondary_dfs_endpoint           = (known after apply)
      + secondary_dfs_host               = (known after apply)
      + secondary_file_endpoint          = (known after apply)
      + secondary_file_host              = (known after apply)
      + secondary_location               = (known after apply)
      + secondary_queue_endpoint         = (known after apply)
      + secondary_queue_host             = (known after apply)
      + secondary_table_endpoint         = (known after apply)
      + secondary_table_host             = (known after apply)
      + secondary_web_endpoint           = (known after apply)
      + secondary_web_host               = (known after apply)
      + tags                             = {
          + "environment" = "Development"
        }

      + blob_properties {
          + change_feed_enabled      = (known after apply)
          + default_service_version  = (known after apply)
          + last_access_time_enabled = (known after apply)
          + versioning_enabled       = (known after apply)

          + container_delete_retention_policy {
              + days = (known after apply)
            }

          + cors_rule {
              + allowed_headers    = (known after apply)
              + allowed_methods    = (known after apply)
              + allowed_origins    = (known after apply)
              + exposed_headers    = (known after apply)
              + max_age_in_seconds = (known after apply)
            }

          + delete_retention_policy {
              + days = (known after apply)
            }
        }

      + identity {
          + principal_id = (known after apply)
          + tenant_id    = (known after apply)
          + type         = (known after apply)
        }

      + network_rules {
          + bypass                     = (known after apply)
          + default_action             = "Deny"
          + ip_rules                   = [
              + "100.0.0.1",
            ]
          + virtual_network_subnet_ids = [
              + "/subscriptions/3434434-343-4/resourceGroups/rg-dave-terraform-test/providers/Microsoft.Network/virtualNetworks/daveVnet/subnets/daveSubnet",
            ]
        }

      + queue_properties {
          + cors_rule {
              + allowed_headers    = (known after apply)
              + allowed_methods    = (known after apply)
              + allowed_origins    = (known after apply)
              + exposed_headers    = (known after apply)
              + max_age_in_seconds = (known after apply)
            }

          + hour_metrics {
              + enabled               = (known after apply)
              + include_apis          = (known after apply)
              + retention_policy_days = (known after apply)
              + version               = (known after apply)
            }

          + logging {
              + delete                = (known after apply)
              + read                  = (known after apply)
              + retention_policy_days = (known after apply)
              + version               = (known after apply)
              + write                 = (known after apply)
            }

          + minute_metrics {
              + enabled               = (known after apply)
              + include_apis          = (known after apply)
              + retention_policy_days = (known after apply)
              + version               = (known after apply)
            }
        }

      + routing {
          + choice                      = (known after apply)
          + publish_internet_endpoints  = (known after apply)
          + publish_microsoft_endpoints = (known after apply)
        }
    }

Plan: 1 to add, 1 to change, 0 to destroy.

Terraform apply

[dave@dave azure]$ terraform apply
azurerm_resource_group.daveterraformgroup: Refreshing state... [id=/subscriptions/3434434-343-4/resourceGroups/rg-dave-terraform-test]
azurerm_public_ip.daveterraformpublicip: Refreshing state... [id=/subscriptions/3434434-343-4/resourceGroups/rg-dave-terraform-test/providers/Microsoft.Network/publicIPAddresses/davePublicIP]
azurerm_virtual_network.daveterraformnetwork: Refreshing state... [id=/subscriptions/3434434-343-4/resourceGroups/rg-dave-terraform-test/providers/Microsoft.Network/virtualNetworks/daveVnet]
azurerm_network_security_group.daveterraformnsg: Refreshing state... [id=/subscriptions/3434434-343-4/resourceGroups/rg-dave-terraform-test/providers/Microsoft.Network/networkSecurityGroups/daveNetworkSecurityGroup]
azurerm_subnet.daveterraformsubnet: Refreshing state... [id=/subscriptions/3434434-343-40/resourceGroups/rg-dave-terraform-test/providers/Microsoft.Network/virtualNetworks/daveVnet/subnets/daveSubnet]
azurerm_network_interface.daveterraformnic: Refreshing state... [id=/subscriptions/3434434-343-4/resourceGroups/rg-dave-terraform-test/providers/Microsoft.Network/networkInterfaces/daveNIC]
azurerm_network_interface_security_group_association.dave: Refreshing state... [id=/subscriptions/3434434-343-4/resourceGroups/rg-dave-terraform-test/providers/Microsoft.Network/networkInterfaces/daveNIC|/subscriptions/3434434-343-4/resourceGroups/rg-dave-terraform-test/providers/Microsoft.Network/networkSecurityGroups/daveNetworkSecurityGroup]
azurerm_linux_virtual_machine.daveterraformvm: Refreshing state... [id=/subscriptions/3434434-343-4/resourceGroups/rg-dave-terraform-test/providers/Microsoft.Compute/virtualMachines/dave-terraform-test]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create
  ~ update in-place

Terraform will perform the following actions:

  # azurerm_storage_account.davestorageaccounttf will be created
  + resource "azurerm_storage_account" "davestorageaccounttf" {
      + access_tier                      = (known after apply)
      + account_kind                     = "StorageV2"
      + account_replication_type         = "LRS"
      + account_tier                     = "Standard"
      + allow_blob_public_access         = false
      + enable_https_traffic_only        = true
      + id                               = (known after apply)
      + is_hns_enabled                   = false
      + large_file_share_enabled         = (known after apply)
      + location                         = "westeurope"
      + min_tls_version                  = "TLS1_0"
      + name                             = "davestorageaccounttf"
      + nfsv3_enabled                    = false
      + primary_access_key               = (sensitive value)
      + primary_blob_connection_string   = (sensitive value)
      + primary_blob_endpoint            = (known after apply)
      + primary_blob_host                = (known after apply)
      + primary_connection_string        = (sensitive value)
      + primary_dfs_endpoint             = (known after apply)
      + primary_dfs_host                 = (known after apply)
      + primary_file_endpoint            = (known after apply)
      + primary_file_host                = (known after apply)
      + primary_location                 = (known after apply)
      + primary_queue_endpoint           = (known after apply)
      + primary_queue_host               = (known after apply)
      + primary_table_endpoint           = (known after apply)
      + primary_table_host               = (known after apply)
      + primary_web_endpoint             = (known after apply)
      + primary_web_host                 = (known after apply)
      + resource_group_name              = "rg-dave-terraform-test"
      + secondary_access_key             = (sensitive value)
      + secondary_blob_connection_string = (sensitive value)
      + secondary_blob_endpoint          = (known after apply)
      + secondary_blob_host              = (known after apply)
      + secondary_connection_string      = (sensitive value)
      + secondary_dfs_endpoint           = (known after apply)
      + secondary_dfs_host               = (known after apply)
      + secondary_file_endpoint          = (known after apply)
      + secondary_file_host              = (known after apply)
      + secondary_location               = (known after apply)
      + secondary_queue_endpoint         = (known after apply)
      + secondary_queue_host             = (known after apply)
      + secondary_table_endpoint         = (known after apply)
      + secondary_table_host             = (known after apply)
      + secondary_web_endpoint           = (known after apply)
      + secondary_web_host               = (known after apply)
      + tags                             = {
          + "environment" = "staging"
        }

      + blob_properties {
          + change_feed_enabled      = (known after apply)
          + default_service_version  = (known after apply)
          + last_access_time_enabled = (known after apply)
          + versioning_enabled       = (known after apply)

          + container_delete_retention_policy {
              + days = (known after apply)
            }

          + cors_rule {
              + allowed_headers    = (known after apply)
              + allowed_methods    = (known after apply)
              + allowed_origins    = (known after apply)
              + exposed_headers    = (known after apply)
              + max_age_in_seconds = (known after apply)
            }

          + delete_retention_policy {
              + days = (known after apply)
            }
        }

      + identity {
          + principal_id = (known after apply)
          + tenant_id    = (known after apply)
          + type         = (known after apply)
        }

      + network_rules {
          + bypass                     = [
              + "AzureServices",
              + "Metrics",
            ]
          + default_action             = "Deny"
          + ip_rules                   = [
              + "100.0.0.1",
            ]
          + virtual_network_subnet_ids = [
              + "/subscriptions/3434434-343-4/resourceGroups/rg-dave-terraform-test/providers/Microsoft.Network/virtualNetworks/daveVnet/subnets/daveSubnet",
            ]
        }

      + queue_properties {
          + cors_rule {
              + allowed_headers    = (known after apply)
              + allowed_methods    = (known after apply)
              + allowed_origins    = (known after apply)
              + exposed_headers    = (known after apply)
              + max_age_in_seconds = (known after apply)
            }

          + hour_metrics {
              + enabled               = (known after apply)
              + include_apis          = (known after apply)
              + retention_policy_days = (known after apply)
              + version               = (known after apply)
            }

          + logging {
              + delete                = (known after apply)
              + read                  = (known after apply)
              + retention_policy_days = (known after apply)
              + version               = (known after apply)
              + write                 = (known after apply)
            }

          + minute_metrics {
              + enabled               = (known after apply)
              + include_apis          = (known after apply)
              + retention_policy_days = (known after apply)
              + version               = (known after apply)
            }
        }

      + routing {
          + choice                      = (known after apply)
          + publish_internet_endpoints  = (known after apply)
          + publish_microsoft_endpoints = (known after apply)
        }
    }

  # azurerm_storage_account_network_rules.davenetworkrules will be created
  + resource "azurerm_storage_account_network_rules" "davenetworkrules" {
      + bypass                     = [
          + "AzureServices",
          + "Metrics",
        ]
      + default_action             = "Allow"
      + id                         = (known after apply)
      + ip_rules                   = [
          + "127.0.0.1",
        ]
      + resource_group_name        = "rg-dave-terraform-test"
      + storage_account_name       = "davestorageaccounttf"
      + virtual_network_subnet_ids = [
          + "/subscriptions/3434434-343-4/resourceGroups/rg-dave-terraform-test/providers/Microsoft.Network/virtualNetworks/daveVnet/subnets/daveSubnet",
        ]
    }

  # azurerm_subnet.daveterraformsubnet will be updated in-place
  ~ resource "azurerm_subnet" "daveterraformsubnet" {
        id                                             = "/subscriptions/3434434-343-4/resourceGroups/rg-dave-terraform-test/providers/Microsoft.Network/virtualNetworks/daveVnet/subnets/daveSubnet"
        name                                           = "daveSubnet"
      ~ service_endpoints                              = [
          + "Microsoft.Storage",
        ]
        # (7 unchanged attributes hidden)
    }

Plan: 2 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

azurerm_subnet.daveterraformsubnet: Modifying... [id=/subscriptions/3434434-343-4/resourceGroups/rg-dave-terraform-test/providers/Microsoft.Network/virtualNetworks/daveVnet/subnets/daveSubnet]
azurerm_subnet.daveterraformsubnet: Modifications complete after 4s [id=/subscriptions/3434434-343-4/resourceGroups/rg-dave-terraform-test/providers/Microsoft.Network/virtualNetworks/daveVnet/subnets/daveSubnet]
azurerm_storage_account.davestorageaccounttf: Creating...
azurerm_storage_account.davestorageaccounttf: Still creating... [10s elapsed]
azurerm_storage_account.davestorageaccounttf: Still creating... [20s elapsed]
azurerm_storage_account.davestorageaccounttf: Creation complete after 22s [id=/subscriptions/3434434-343-4/resourceGroups/rg-dave-terraform-test/providers/Microsoft.Storage/storageAccounts/davestorageaccounttf]

Azure Storage Account

Azure list of resources

Azure VM network topology

Azure Analytics Services

HOWTO

General HOWTO https://azure.microsoft.com/en-us/product-categories/analytics/
analytics service that brings together data integration, enterprise data warehousing, and big data analytics ahttps://azure.microsoft.com/en-us/services/synapse-analytics/
Apache® Spark™ based analytics https://azure.microsoft.com/en-us/services/databricks/
Apache Hadoop, Spark, Hive, Kafka https://azure.microsoft.com/en-us/services/hdinsight/
data integration service to orchestrate and automate data movement https://azure.microsoft.com/en-us/services/data-factory/
develop and run massively parallel data transformation and processing programs in U-SQL, R, Python https://azure.microsoft.com/en-us/services/data-lake-analytics/

Azure DataBricks

https://docs.microsoft.com/en-us/azure/databricks/

https://docs.microsoft.com/en-us/azure/databricks/scenarios/quickstart-create-databricks-workspace-portal?tabs=azure-portal

Azure HDInsight

https://docs.microsoft.com/en-us/azure/hdinsight/spark/apache-spark-jupyter-spark-sql-use-portal

Friday, June 4, 2021

Create Azure VM using Terraform

sudo dnf install -y dnf-plugins-core

sudo dnf config-manager --add-repo https://rpm.releases.hashicorp.com/fedora/hashicorp.repo

sudo dnf -y install terraform

Install Azure CLI

https://docs.microsoft.com/en-us/cli/azure/install-azure-cli-yum

Verify installation

[dave@dave ~]$ terraform version
Terraform v0.13.5
[dave@dave ~]$ az version
{
  "azure-cli": "2.14.2",
  "azure-cli-core": "2.14.2",
  "azure-cli-telemetry": "1.0.6",
  "extensions": {}
}

Login via Azure CLI using own credentials

`https://docs.microsoft.com/en-us/cli/azure/authenticate-azure-cli`

$ az login

[dave@dave azure]$ az account show
{
  "environmentName": "AzureCloud",
  "homeTenantId": "23432432423-2342423423",
  "id": "3234234234-234234234",
  "isDefault": true,
  "managedByTenants": [],
  "name": "Free Trial",
  "state": "Enabled",
  "tenantId": "234234234-234234234",
  "user": {
    "name": "daniel.veselka",
    "type": "user"
  }
}

Authenticate to Azure using service principal variables

https://docs.microsoft.com/en-us/azure/developer/terraform/get-started-cloud-shell#authenticate-via-azure-service-principal

Get values for service principal and store them into env vars ARM_*

A Service Principal is an application within Azure Active Directory whose authentication tokens can be used as the client_id, client_secret, and tenant_id fields needed by Terraform (subscription_id can be independently recovered from your Azure account details).

Set subscription to use with Terraform

https://docs.microsoft.com/en-us/azure/developer/terraform/get-started-cloud-shell#set-the-current-azure-subscription

az account set --subscription="<subscription_id>"

Export Azure principle secrets variables

https://www.terraform.io/docs/providers/azurerm/guides/service_principal_client_secret.html

$ export ARM_CLIENT_ID="00000000-0000-0000-0000-000000000000"
$ export ARM_CLIENT_SECRET="00000000-0000-0000-0000-000000000000"
$ export ARM_SUBSCRIPTION_ID="00000000-0000-0000-0000-000000000000"
$ export ARM_TENANT_ID="00000000-0000-0000-0000-000000000000"

Find Centos OS Azure image

https://docs.microsoft.com/en-us/azure/virtual-machines/linux/cli-ps-findimage

 az vm image list -l westeurope --publisher openlogic --offer centos --sku 8 --all -o table 
Offer                      Publisher    Sku              Urn                                                                 Version
-------------------------  -----------  ---------------  ------------------------------------------------------------------  --------------
CentOS                     OpenLogic    8_3              OpenLogic:CentOS:8_3:8.3.2020120900                                 8.3.2020120900
CentOS                     OpenLogic    8_3              OpenLogic:CentOS:8_3:8.3.2021020400                                 8.3.2021020400
CentOS                     OpenLogic    8_3-gen2         OpenLogic:CentOS:8_3-gen2:8.3.2020120901                            8.3.2020120901
CentOS                     OpenLogic    8_3-gen2         OpenLogic:CentOS:8_3-gen2:8.3.2021020401                            8.3.2021020401

Install VSCode with Terraform extension

For VSCode installation see here https://danielveselka.blogspot.com/2020/11/google-cloud-code-ide-extension.html

https://github.com/hashicorp/vscode-terraform

Choose VM size

https://docs.microsoft.com/en-us/azure/virtual-machines/sizes-b-series-burstable

Create VM

https://docs.microsoft.com/en-us/azure/developer/terraform/create-linux-virtual-machine-with-infrastructure

https://github.com/terraform-providers/terraform-provider-azurerm/blob/master/examples/virtual-machines/linux/basic-ssh/main.tf

https://github.com/terraform-providers/terraform-provider-azurerm/blob/master/examples/virtual-machines/linux/custom-data/main.tf

[dave@dave azure]$ terraform init

Initializing the backend...

Initializing provider plugins...
- Finding latest version of hashicorp/tls...
- Finding latest version of hashicorp/azurerm...
- Installing hashicorp/tls v3.0.0...
- Installed hashicorp/tls v3.0.0 (signed by HashiCorp)
- Installing hashicorp/azurerm v2.36.0...
- Installed hashicorp/azurerm v2.36.0 (signed by HashiCorp)

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, we recommend adding version constraints in a required_providers block
in your configuration, with the constraint strings suggested below.

* hashicorp/azurerm: version = "~> 2.36.0"
* hashicorp/tls: version = "~> 3.0.0"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Terraform plan

[dave@dave azure]$ terraform plan

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # azurerm_linux_virtual_machine.daveterraformvm will be created
  + resource "azurerm_linux_virtual_machine" "daveterraformvm" {
      + admin_username                  = "centos"
      + allow_extension_operations      = true
      + computer_name                   = (known after apply)
      + disable_password_authentication = true
      + extensions_time_budget          = "PT1H30M"
      + id                              = (known after apply)
      + location                        = "westeurope"
      + max_bid_price                   = -1
      + name                            = "dave_terraform_test"
      + network_interface_ids           = (known after apply)
      + platform_fault_domain           = -1
      + priority                        = "Regular"
      + private_ip_address              = (known after apply)
      + private_ip_addresses            = (known after apply)
      + provision_vm_agent              = true
      + public_ip_address               = (known after apply)
      + public_ip_addresses             = (known after apply)
      + resource_group_name             = "rg-dave-terraform-test"
      + size                            = "Standard_DS3_v2"
      + virtual_machine_id              = (known after apply)
      + zone                            = (known after apply)

      + admin_ssh_key {
          + public_key = <<-EOT
                ssh-rsa ererererererer dave@dave
            EOT
          + username   = "centos"
        }

      + os_disk {
          + caching                   = "ReadWrite"
          + disk_size_gb              = (known after apply)
          + name                      = (known after apply)
          + storage_account_type      = "Premium_LRS"
          + write_accelerator_enabled = false
        }

      + source_image_reference {
          + offer     = "CentOS"
          + publisher = "OpenLogic"
          + sku       = "8_3"
          + version   = "latest"
        }
    }

  # azurerm_network_interface.daveterraformnic will be created
  + resource "azurerm_network_interface" "daveterraformnic" {
      + applied_dns_servers           = (known after apply)
      + dns_servers                   = (known after apply)
      + enable_accelerated_networking = false
      + enable_ip_forwarding          = false
      + id                            = (known after apply)
      + internal_dns_name_label       = (known after apply)
      + internal_domain_name_suffix   = (known after apply)
      + location                      = "westeurope"
      + mac_address                   = (known after apply)
      + name                          = "daveNIC"
      + private_ip_address            = (known after apply)
      + private_ip_addresses          = (known after apply)
      + resource_group_name           = "rg-dave-terraform-test"
      + tags                          = {
          + "environment" = "Terraform Demo"
        }
      + virtual_machine_id            = (known after apply)

      + ip_configuration {
          + name                          = "daveNicConfiguration"
          + primary                       = (known after apply)
          + private_ip_address            = (known after apply)
          + private_ip_address_allocation = "dynamic"
          + private_ip_address_version    = "IPv4"
          + public_ip_address_id          = (known after apply)
          + subnet_id                     = (known after apply)
        }
    }

  # azurerm_network_interface_security_group_association.dave will be created
  + resource "azurerm_network_interface_security_group_association" "dave" {
      + id                        = (known after apply)
      + network_interface_id      = (known after apply)
      + network_security_group_id = (known after apply)
    }

  # azurerm_network_security_group.daveterraformnsg will be created
  + resource "azurerm_network_security_group" "daveterraformnsg" {
      + id                  = (known after apply)
      + location            = "westeurope"
      + name                = "daveNetworkSecurityGroup"
      + resource_group_name = "rg-dave-terraform-test"
      + security_rule       = [
          + {
              + access                                     = "Allow"
              + description                                = ""
              + destination_address_prefix                 = "*"
              + destination_address_prefixes               = []
              + destination_application_security_group_ids = []
              + destination_port_range                     = "22"
              + destination_port_ranges                    = []
              + direction                                  = "Inbound"
              + name                                       = "SSH"
              + priority                                   = 1001
              + protocol                                   = "Tcp"
              + source_address_prefix                      = "*"
              + source_address_prefixes                    = []
              + source_application_security_group_ids      = []
              + source_port_range                          = "*"
              + source_port_ranges                         = []
            },
        ]
      + tags                = {
          + "environment" = "Terraform Demo"
        }
    }

  # azurerm_public_ip.daveterraformpublicip will be created
  + resource "azurerm_public_ip" "daveterraformpublicip" {
      + allocation_method       = "Dynamic"
      + fqdn                    = (known after apply)
      + id                      = (known after apply)
      + idle_timeout_in_minutes = 4
      + ip_address              = (known after apply)
      + ip_version              = "IPv4"
      + location                = "westeurope"
      + name                    = "davePublicIP"
      + resource_group_name     = "rg-dave-terraform-test"
      + sku                     = "Basic"
      + tags                    = {
          + "environment" = "Terraform Demo"
        }
    }

  # azurerm_resource_group.daveterraformgroup will be created
  + resource "azurerm_resource_group" "daveterraformgroup" {
      + id       = (known after apply)
      + location = "westeurope"
      + name     = "rg-dave-terraform-test"
      + tags     = {
          + "environment" = "Terraform Demo"
        }
    }

  # azurerm_subnet.daveterraformsubnet will be created
  + resource "azurerm_subnet" "daveterraformsubnet" {
      + address_prefix                                 = (known after apply)
      + address_prefixes                               = [
          + "10.0.1.0/24",
        ]
      + enforce_private_link_endpoint_network_policies = false
      + enforce_private_link_service_network_policies  = false
      + id                                             = (known after apply)
      + name                                           = "daveSubnet"
      + resource_group_name                            = "rg-dave-terraform-test"
      + virtual_network_name                           = "daveVnet"
    }

  # azurerm_virtual_network.daveterraformnetwork will be created
  + resource "azurerm_virtual_network" "daveterraformnetwork" {
      + address_space         = [
          + "10.0.0.0/16",
        ]
      + guid                  = (known after apply)
      + id                    = (known after apply)
      + location              = "westeurope"
      + name                  = "daveVnet"
      + resource_group_name   = "rg-dave-terraform-test"
      + subnet                = (known after apply)
      + tags                  = {
          + "environment" = "Terraform Demo"
        }
      + vm_protection_enabled = false
    }

Plan: 8 to add, 0 to change, 0 to destroy.

Terraform apply

Create VM using https://github.com/dveselka/devops-terraform/blob/master/azure/main.tf

$ terraform apply
azurerm_resource_group.daveterraformgroup: Refreshing state... [id=/subscriptions/3434343-334/resourceGroups/rg-dave-terraform-test]
azurerm_virtual_network.daveterraformnetwork: Refreshing state... [id=/subscriptions/43443-434-3434/resourceGroups/rg-dave-terraform-test/providers/Microsoft.Network/virtualNetworks/daveVnet]
azurerm_network_security_group.daveterraformnsg: Refreshing state... [id=/subscriptions/434343-43-43/resourceGroups/rg-dave-terraform-test/providers/Microsoft.Network/networkSecurityGroups/daveNetworkSecurityGroup]
azurerm_public_ip.daveterraformpublicip: Refreshing state... [id=/subscriptions/343434-43434/resourceGroups/rg-dave-terraform-test/providers/Microsoft.Network/publicIPAddresses/davePublicIP]
azurerm_subnet.daveterraformsubnet: Refreshing state... [id=/subscriptions/343434-43434/resourceGroups/rg-dave-terraform-test/providers/Microsoft.Network/virtualNetworks/daveVnet/subnets/daveSubnet]
azurerm_network_interface.daveterraformnic: Refreshing state... [id=/subscriptions/3434-434343/resourceGroups/rg-dave-terraform-test/providers/Microsoft.Network/networkInterfaces/daveNIC]
azurerm_network_interface_security_group_association.dave: Refreshing state... [id=/subscriptions/3434-434343/resourceGroups/rg-dave-terraform-test/providers/Microsoft.Network/....

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the last "terraform apply":


Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan may include actions to undo or respond
to these changes.

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # azurerm_linux_virtual_machine.daveterraformvm will be created
  + resource "azurerm_linux_virtual_machine" "daveterraformvm" {
      + admin_username                  = "centos"
      + allow_extension_operations      = true
      + computer_name                   = (known after apply)
      + disable_password_authentication = true
      + extensions_time_budget          = "PT1H30M"
      + id                              = (known after apply)
      + location                        = "westeurope"
      + max_bid_price                   = -1
      + name                            = "dave-terraform-test"
      + network_interface_ids           = [
          + "/subscriptions/45454-54-54-545/resourceGroups/rg-dave-terraform-test/providers/Microsoft.Network/networkInterfaces/daveNIC",
        ]
      + platform_fault_domain           = -1
      + priority                        = "Regular"
      + private_ip_address              = (known after apply)
      + private_ip_addresses            = (known after apply)
      + provision_vm_agent              = true
      + public_ip_address               = (known after apply)
      + public_ip_addresses             = (known after apply)
      + resource_group_name             = "rg-dave-terraform-test"
      + size                            = "Standard_A2_v2"
      + virtual_machine_id              = (known after apply)
      + zone                            = (known after apply)

      + admin_ssh_key {
          + public_key = <<-eot 0="" 1="" 2m49s="" above.="" accepted="" actions="" add="" added="" after="" apply="" approve.="" artrterteteteterte="" azure="" azurerm_linux_virtual_machine.daveterraformvm:="" be="" caching="ReadWrite" change="" changed="" code="" complete="" creating...="" creation="" dave="" described="" destroy.="" destroyed.="" disk_size_gb="(known" do="" elapsed="" eot="" id="/subscriptions/e535613a-7524-4b38-8915-ad04b02fe3d0/resourceGroups/rg-dave-terraform-test/providers/Microsoft.Compute/virtualMachines/dave-terraform-test]" m0s="" m10s="" m20s="" m30s="" m40s="" m50s="" name="(known" offer="CentOS" only="" os_disk="" perform="" plan:="" publisher="OpenLogic" resources:="" s="" sku="8_3" source_image_reference="" ssh-rsa="" still="" storage_account_type="Standard_LRS" terraform="" the="" these="" to="" username="centos" version="latest" want="" will="" write_accelerator_enabled="false" yes="" you="">
azurerm_linux_virtual_machine.daveterraformvm: Creating...
azurerm_linux_virtual_machine.daveterraformvm: Still creating... [10s elapsed]
azurerm_linux_virtual_machine.daveterraformvm: Still creating... [20s elapsed]
azurerm_linux_virtual_machine.daveterraformvm: Still creating... [30s elapsed]
azurerm_linux_virtual_machine.daveterraformvm: Still creating... [40s elapsed]
azurerm_linux_virtual_machine.daveterraformvm: Still creating... [50s elapsed]
azurerm_linux_virtual_machine.daveterraformvm: Still creating... [1m0s elapsed]
azurerm_linux_virtual_machine.daveterraformvm: Still creating... [1m10s elapsed]
azurerm_linux_virtual_machine.daveterraformvm: Still creating... [1m20s elapsed]
azurerm_linux_virtual_machine.daveterraformvm: Still creating... [1m30s elapsed]
azurerm_linux_virtual_machine.daveterraformvm: Still creating... [1m40s elapsed]
azurerm_linux_virtual_machine.daveterraformvm: Still creating... [1m50s elapsed]
azurerm_linux_virtual_machine.daveterraformvm: Still creating... [2m0s elapsed]
azurerm_linux_virtual_machine.daveterraformvm: Still creating... [2m10s elapsed]
azurerm_linux_virtual_machine.daveterraformvm: Still creating... [2m20s elapsed]
azurerm_linux_virtual_machine.daveterraformvm: Still creating... [2m30s elapsed]
azurerm_linux_virtual_machine.daveterraformvm: Still creating... [2m40s elapsed]
azurerm_linux_virtual_machine.daveterraformvm: Creation complete after 2m49s [id=/subscriptions/34343-4343/resourceGroups/rg-dave-terraform-test/providers/Microsoft.Compute/virtualMachines/dave-terraform-test]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
[dave@dave azure]$

Check VM in Azure Console

https://portal.azure.com/

SSH into VM

[dave@dave azure]$ ssh centos@20.73.164.213
The authenticity of host '20.73.164.213 (20.73.164.213)' can't be established.
ECDSA key fingerprint is SHA256:ovtXmI52ndfouGE97PX4foS099J2lj91xie4cRQz8w8.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '20.73.164.213' (ECDSA) to the list of known hosts.
Activate the web console with: systemctl enable --now cockpit.socket

[centos@dave-terraform-test ~]$ more /etc/redhat-release
CentOS Linux release 8.3.2011

Monday, June 7, 2021

HOWTO

GitHub

Sunday, June 6, 2021

HOWTO

HOWTO

Github

Create Storage account with Network Rules

Azure DataBricks

Azure HDInsight

Friday, June 4, 2021

HOWTO

Differences Between On-Premise, SaaS, PaaS, IaaS

GitHub

Install Terraform on Fedora

Install Azure CLI

Login via Azure CLI using own credentials

Authenticate to Azure using service principal variables

Find Centos OS Azure image

Install VSCode with Terraform extension

Create VM

Terraform plan

Terraform apply