DevOps tips & tricks: 2024

Sunday, December 8, 2024

Argo CD - k8s GitOps on minikube

HOWTO

Argo CD architecture https://argo-cd.readthedocs.io/en/stable/operator-manual/architecture/
Getting started https://argo-cd.readthedocs.io/en/stable/getting_started/
Install Argo CD CLI https://argo-cd.readthedocs.io/en/stable/cli_installation/
Login to Argo CD UI https://argo-cd.readthedocs.io/en/stable/getting_started/#4-login-using-the-cli
Declarative setup on Argo CD https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/
Argo CD tutorial video https://youtu.be/MeU5_k9ssrs?si=KORabth8rx980I_f
Install Minikube https://danielveselka.blogspot.com/2020/11/install-kubectl-on-fedora.html

Git

https://github.com/dveselka/devops-k8s/blob/main/argocd/application.yaml

Argo CD architecture

Start minikube

dave@fedora:~$ minikube start --driver=docker
😄  minikube v1.32.0 on Fedora 40
✨  Using the docker driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🎉  minikube 1.34.0 is available! Download it: https://github.com/kubernetes/minikube/releases/tag/v1.34.0
💡  To disable this notice, run: 'minikube config set WantUpdateNotification false'

🔄  Restarting existing docker container for "minikube" ...
🐳  Preparing Kubernetes v1.28.3 on Docker 24.0.7 ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/minikube-ingress-dns:0.0.2
    ▪ Using image docker.io/kubernetesui/metrics-scraper:v1.0.8
    ▪ Using image docker.io/kubernetesui/dashboard:v2.7.0
    ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
    ▪ Using image registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20231011-8b53cabe0
    ▪ Using image registry.k8s.io/ingress-nginx/controller:v1.9.4
🔎  Verifying ingress addon...
💡  Some dashboard features require the metrics-server addon. To enable all features please run:

    minikube addons enable metrics-server    


🌟  Enabled addons: ingress-dns, storage-provisioner, dashboard, ingress, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

Install Argo CD


dave@fedora:~$ kubectl create namespace argocd
namespace/argocd created
dave@fedora:~$ kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
customresourcedefinition.apiextensions.k8s.io/applications.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/applicationsets.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/appprojects.argoproj.io created
serviceaccount/argocd-application-controller created
serviceaccount/argocd-applicationset-controller created
serviceaccount/argocd-dex-server created
serviceaccount/argocd-notifications-controller created
serviceaccount/argocd-redis created
serviceaccount/argocd-repo-server created
serviceaccount/argocd-server created
role.rbac.authorization.k8s.io/argocd-application-controller created
role.rbac.authorization.k8s.io/argocd-applicationset-controller created
role.rbac.authorization.k8s.io/argocd-dex-server created
role.rbac.authorization.k8s.io/argocd-notifications-controller created
role.rbac.authorization.k8s.io/argocd-redis created
role.rbac.authorization.k8s.io/argocd-server created
clusterrole.rbac.authorization.k8s.io/argocd-application-controller created
clusterrole.rbac.authorization.k8s.io/argocd-applicationset-controller created
clusterrole.rbac.authorization.k8s.io/argocd-server created
rolebinding.rbac.authorization.k8s.io/argocd-application-controller created
rolebinding.rbac.authorization.k8s.io/argocd-applicationset-controller created
rolebinding.rbac.authorization.k8s.io/argocd-dex-server created
rolebinding.rbac.authorization.k8s.io/argocd-notifications-controller created
rolebinding.rbac.authorization.k8s.io/argocd-redis created
rolebinding.rbac.authorization.k8s.io/argocd-server created
clusterrolebinding.rbac.authorization.k8s.io/argocd-application-controller created
clusterrolebinding.rbac.authorization.k8s.io/argocd-applicationset-controller created
clusterrolebinding.rbac.authorization.k8s.io/argocd-server created
configmap/argocd-cm created
configmap/argocd-cmd-params-cm created
configmap/argocd-gpg-keys-cm created
configmap/argocd-notifications-cm created
configmap/argocd-rbac-cm created
configmap/argocd-ssh-known-hosts-cm created
configmap/argocd-tls-certs-cm created
secret/argocd-notifications-secret created
secret/argocd-secret created
service/argocd-applicationset-controller created
service/argocd-dex-server created
service/argocd-metrics created
service/argocd-notifications-controller-metrics created
service/argocd-redis created
service/argocd-repo-server created
service/argocd-server created
service/argocd-server-metrics created
deployment.apps/argocd-applicationset-controller created
deployment.apps/argocd-dex-server created
deployment.apps/argocd-notifications-controller created
deployment.apps/argocd-redis created
deployment.apps/argocd-repo-server created
deployment.apps/argocd-server created
statefulset.apps/argocd-application-controller created
networkpolicy.networking.k8s.io/argocd-application-controller-network-policy created
networkpolicy.networking.k8s.io/argocd-applicationset-controller-network-policy created
networkpolicy.networking.k8s.io/argocd-dex-server-network-policy created
networkpolicy.networking.k8s.io/argocd-notifications-controller-network-policy created
networkpolicy.networking.k8s.io/argocd-redis-network-policy created
networkpolicy.networking.k8s.io/argocd-repo-server-network-policy created
networkpolicy.networking.k8s.io/argocd-server-network-policy created

Get pods in argocd namespace

$ kubectl get pod -n argocd
NAME                                                READY   STATUS              RESTARTS   AGE
argocd-application-controller-0                     0/1     ContainerCreating   0          20s
argocd-applicationset-controller-5c787df94f-npjn9   0/1     ContainerCreating   0          22s
argocd-dex-server-6bb9b5fc75-x4vt8                  0/1     Init:0/1            0          22s
argocd-notifications-controller-7ccbd7fb6-dm8bq     0/1     ContainerCreating   0          22s
argocd-redis-c5c567495-cgtrl                        0/1     Init:0/1            0          22s
argocd-repo-server-799b498d8b-pc2l8                 0/1     Init:0/1            0          22s
argocd-server-f6d4d8775-9hlcj                       0/1     ContainerCreating   0          21s

Get services in argocd namespace

$ kubectl get svc -n argocd
NAME                                      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
argocd-applicationset-controller          ClusterIP   10.101.144.51   <none>        7000/TCP,8080/TCP            105s
argocd-dex-server                         ClusterIP   10.111.165.14   <none>        5556/TCP,5557/TCP,5558/TCP   105s
argocd-metrics                            ClusterIP   10.100.56.238   <none>        8082/TCP                     105s
argocd-notifications-controller-metrics   ClusterIP   10.111.151.85   <none>        9001/TCP                     105s
argocd-redis                              ClusterIP   10.97.35.44     <none>        6379/TCP                     104s
argocd-repo-server                        ClusterIP   10.109.34.145   <none>        8081/TCP,8084/TCP            104s
argocd-server                             ClusterIP   10.99.111.141   <none>        80/TCP,443/TCP               104s
argocd-server-metrics                     ClusterIP   10.106.186.75   <none>        8083/TCP                     104s

Port forward to access Argo CD UI

$ kubectl port-forward svc/argocd-server -n argocd 8080:443
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080

Get admin password

$ kubectl get secret argocd-initial-admin-secret -n argocd -o yaml
apiVersion: v1
data:
  password: ab123==
kind: Secret
metadata:
  creationTimestamp: "2024-12-15T07:58:19Z"
  name: argocd-initial-admin-secret
  namespace: argocd
  resourceVersion: "190652"
  uid: a017dc3e-3232-40f6-a2e6-374872963888
type: Opaque

$ echo ab123|base64 --decode

Open UI at https://127.0.0.1:8080/applications

Apply application.yaml on k8s cluster via kubectl

https://github.com/dveselka/devops-k8s/blob/main/argocd/application.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: guestbook
  namespace: argocd
spec:
  project: default

  source:
    repoURL: https://github.com/dveselka/devops-k8s.git
    targetRevision: HEAD
    path: argocd
    
  destination:
    server: https://kubernetes.default.svc
    namespace: myapp

  syncPolicy:
    syncOptions:
    - CreateNamespace = true  


    automated:
      selfHeal: true
      prune: true

Apply

$ cp * /git/devops-k8s/argocd
dave@dave:/git/argocd-app-config/dev$ ls
application.yaml  deployment.yaml  service.yaml
dave@dave:/git/argocd-app-config/dev$ kubectl apply -f application.yaml 
application.argoproj.io/guestbook created

Argo CD UI - unsynced

Argo CD UI with synced application

Argo CD descriptors

Get pods in myapp namespace

$  kubectl get pods -n myapp
NAME                     READY   STATUS    RESTARTS   AGE
myapp-55c645d9b5-62llh   1/1     Running   0          6m29s
myapp-55c645d9b5-vmtmb   1/1     Running   0          6m29s

Get services in myapp namespace

$  kubectl get svc  -n myapp
NAME            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
myapp-service   ClusterIP   10.104.101.233   <none>        8080/TCP   6m35s

Minikube dashboard

Deployments

Thursday, October 10, 2024

Create GCP VM via Terraform

HOWTO

See also

https://danielveselka.blogspot.com/2021/09/gcp-create-vm-using-terraform.html

GitHub

https://github.com/dveselka/devops-terraform/tree/master/gcp

Initialize Terraform

dave@dave:/git/devops-terraform/gcp$ terraform init
Initializing the backend...
Initializing modules...
- mynet-eu-vm in instance
- mynet-us-vm in instance
Initializing provider plugins...
- Finding latest version of hashicorp/google...
- Installing hashicorp/google v6.6.0...
- Installed hashicorp/google v6.6.0 (signed by HashiCorp)
Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Provider

provider "google" {}

Variables

variable "instance_name" {}
variable "instance_zone" {}
variable "instance_type" {
  default = "e2-micro"
}
variable "instance_network" {}

Network

$ cat mynetwork.tf
# Create the mynetwork network
resource "google_compute_network" "mynetwork" {
  name = "mynetwork"
  # RESOURCE properties go here
  auto_create_subnetworks = "true"
}
# Add a firewall rule to allow HTTP, SSH, RDP and ICMP traffic on mynetwork
resource "google_compute_firewall" "mynetwork-allow-http-ssh-rdp-icmp" {
  name = "mynetwork-allow-http-ssh-rdp-icmp"
  # RESOURCE properties go here
  network = google_compute_network.mynetwork.self_link
  allow {
    protocol = "tcp"
    ports    = ["22", "80", "3389"]
  }
  allow {
    protocol = "icmp"
  }
  source_ranges = ["0.0.0.0/0"]
}
# Create the mynet-us-vm instance
module "mynet-us-vm" {
  source           = "./instance"
  instance_name    = "mynet-us-vm"
  instance_zone    = "us-central1-a"
  instance_network = google_compute_network.mynetwork.self_link
}
# Create the mynet-eu-vm" instance
module "mynet-eu-vm" {
  source           = "./instance"
  instance_name    = "mynet-eu-vm"
  instance_zone    = "europe-west1-d"
  instance_network = google_compute_network.mynetwork.self_link
}

Instance

resource "google_compute_instance" "vm_instance" {
  name = "${var.instance_name}"
  # RESOURCE properties go here
  zone         = "${var.instance_zone}"
  machine_type = "${var.instance_type}"
  boot_disk {
    initialize_params {
      image = "debian-cloud/debian-11"
      }
  }
    network_interface {
    network = "${var.instance_network}"
    access_config {
      # Allocate a one-to-one NAT IP to the instance
    }
  }
}

Project structure

dave@dave:/git/devops-terraform/gcp$ find
.
./provider.tf
./.terraform
./.terraform/modules
./.terraform/modules/modules.json
./.terraform/providers
./.terraform/providers/registry.terraform.io
./.terraform/providers/registry.terraform.io/hashicorp
./.terraform/providers/registry.terraform.io/hashicorp/google
./.terraform/providers/registry.terraform.io/hashicorp/google/6.6.0
./.terraform/providers/registry.terraform.io/hashicorp/google/6.6.0/linux_amd64
./.terraform/providers/registry.terraform.io/hashicorp/google/6.6.0/linux_amd64/LICENSE.txt
./.terraform/providers/registry.terraform.io/hashicorp/google/6.6.0/linux_amd64/terraform-provider-google_v6.6.0_x5
./instance
./instance/main.tf
./instance/variables.tf
./.terraform.lock.hcl
./mynetwork.tf

1st attempt terraform plan

p$ terraform plan

Planning failed. Terraform encountered an error while generating this plan.

╷
│ Error: Attempted to load application default credentials since neither `credentials` nor `access_token` was set in the provider block.  No credentials loaded. To use your gcloud credentials, run 'gcloud auth application-default login'
│ 
│   with provider["registry.terraform.io/hashicorp/google"],
│   on provider.tf line 1, in provider "google":
│    1: provider "google" {}
│ 
│ google: could not find default credentials. See https://cloud.google.com/docs/authentication/external/set-up-adc for more information

Add GCP credentials

$ gcloud auth application-default login
Your browser has been opened to visit:

    https://accounts.google.com/o/oauth2/auth?....

Credentials saved to file: [/home/dave/.config/gcloud/application_default_credentials.json]

These credentials will be used by any library that requests Application Default Credentials (ADC).

Quota project "dave-terraform" was added to ADC which can be used by Google client libraries for billing and quota. Note that some services may still bill the project owning the resource.

2nd attempt to run terraform plan

$ terraform plan

Planning failed. Terraform encountered an error while generating this plan.

╷
│ Error: Failed to retrieve project, pid: , err: project: required field is not set
│ 
│   with google_compute_network.mynetwork,
│   on mynetwork.tf line 2, in resource "google_compute_network" "mynetwork":
│    2: resource "google_compute_network" "mynetwork" {

Add project id into provider.tf

provider "google" {
    project = "dave-terraform"
}

3rd attempt terraform plan

$ terraform plan

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # google_compute_firewall.mynetwork-allow-http-ssh-rdp-icmp will be created
  + resource "google_compute_firewall" "mynetwork-allow-http-ssh-rdp-icmp" {
      + creation_timestamp = (known after apply)
      + destination_ranges = (known after apply)
      + direction          = (known after apply)
      + enable_logging     = (known after apply)
      + id                 = (known after apply)
      + name               = "mynetwork-allow-http-ssh-rdp-icmp"
      + network            = (known after apply)
      + priority           = 1000
      + project            = "dave-terraform"
      + self_link          = (known after apply)
      + source_ranges      = [
          + "0.0.0.0/0",
        ]

      + allow {
          + ports    = [
              + "22",
              + "80",
              + "3389",
            ]
          + protocol = "tcp"
        }
      + allow {
          + ports    = []
          + protocol = "icmp"
        }
    }

  # google_compute_network.mynetwork will be created
  + resource "google_compute_network" "mynetwork" {
      + auto_create_subnetworks                   = true
      + delete_default_routes_on_create           = false
      + gateway_ipv4                              = (known after apply)
      + id                                        = (known after apply)
      + internal_ipv6_range                       = (known after apply)
      + mtu                                       = (known after apply)
      + name                                      = "mynetwork"
      + network_firewall_policy_enforcement_order = "AFTER_CLASSIC_FIREWALL"
      + numeric_id                                = (known after apply)
      + project                                   = "dave-terraform"
      + routing_mode                              = (known after apply)
      + self_link                                 = (known after apply)
    }

  # module.mynet-eu-vm.google_compute_instance.vm_instance will be created
  + resource "google_compute_instance" "vm_instance" {
      + can_ip_forward       = false
      + cpu_platform         = (known after apply)
      + current_status       = (known after apply)
      + deletion_protection  = false
      + effective_labels     = {
          + "goog-terraform-provisioned" = "true"
        }
      + id                   = (known after apply)
      + instance_id          = (known after apply)
      + label_fingerprint    = (known after apply)
      + machine_type         = "e2-micro"
      + metadata_fingerprint = (known after apply)
      + min_cpu_platform     = (known after apply)
      + name                 = "mynet-eu-vm"
      + project              = "dave-terraform"
      + self_link            = (known after apply)
      + tags_fingerprint     = (known after apply)
      + terraform_labels     = {
          + "goog-terraform-provisioned" = "true"
        }
      + zone                 = "europe-west1-d"

      + boot_disk {
          + auto_delete                = true
          + device_name                = (known after apply)
          + disk_encryption_key_sha256 = (known after apply)
          + kms_key_self_link          = (known after apply)
          + mode                       = "READ_WRITE"
          + source                     = (known after apply)

          + initialize_params {
              + image                  = "debian-cloud/debian-11"
              + labels                 = (known after apply)
              + provisioned_iops       = (known after apply)
              + provisioned_throughput = (known after apply)
              + resource_policies      = (known after apply)
              + size                   = (known after apply)
              + type                   = (known after apply)
            }
        }

      + confidential_instance_config (known after apply)

      + guest_accelerator (known after apply)

      + network_interface {
          + internal_ipv6_prefix_length = (known after apply)
          + ipv6_access_type            = (known after apply)
          + ipv6_address                = (known after apply)
          + name                        = (known after apply)
          + network                     = (known after apply)
          + network_ip                  = (known after apply)
          + stack_type                  = (known after apply)
          + subnetwork                  = (known after apply)
          + subnetwork_project          = (known after apply)

          + access_config {
              + nat_ip       = (known after apply)
              + network_tier = (known after apply)
            }
        }

      + reservation_affinity (known after apply)

      + scheduling (known after apply)
    }

  # module.mynet-us-vm.google_compute_instance.vm_instance will be created
  + resource "google_compute_instance" "vm_instance" {
      + can_ip_forward       = false
      + cpu_platform         = (known after apply)
      + current_status       = (known after apply)
      + deletion_protection  = false
      + effective_labels     = {
          + "goog-terraform-provisioned" = "true"
        }
      + id                   = (known after apply)
      + instance_id          = (known after apply)
      + label_fingerprint    = (known after apply)
      + machine_type         = "e2-micro"
      + metadata_fingerprint = (known after apply)
      + min_cpu_platform     = (known after apply)
      + name                 = "mynet-us-vm"
      + project              = "dave-terraform"
      + self_link            = (known after apply)
      + tags_fingerprint     = (known after apply)
      + terraform_labels     = {
          + "goog-terraform-provisioned" = "true"
        }
      + zone                 = "us-central1-a"

      + boot_disk {
          + auto_delete                = true
          + device_name                = (known after apply)
          + disk_encryption_key_sha256 = (known after apply)
          + kms_key_self_link          = (known after apply)
          + mode                       = "READ_WRITE"
          + source                     = (known after apply)

          + initialize_params {
              + image                  = "debian-cloud/debian-11"
              + labels                 = (known after apply)
              + provisioned_iops       = (known after apply)
              + provisioned_throughput = (known after apply)
              + resource_policies      = (known after apply)
              + size                   = (known after apply)
              + type                   = (known after apply)
            }
        }

      + confidential_instance_config (known after apply)

      + guest_accelerator (known after apply)

      + network_interface {
          + internal_ipv6_prefix_length = (known after apply)
          + ipv6_access_type            = (known after apply)
          + ipv6_address                = (known after apply)
          + name                        = (known after apply)
          + network                     = (known after apply)
          + network_ip                  = (known after apply)
          + stack_type                  = (known after apply)
          + subnetwork                  = (known after apply)
          + subnetwork_project          = (known after apply)

          + access_config {
              + nat_ip       = (known after apply)
              + network_tier = (known after apply)
            }
        }

      + reservation_affinity (known after apply)

      + scheduling (known after apply)
    }

Plan: 4 to add, 0 to change, 0 to destroy.

────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.

Terraform apply


$ terraform apply


Plan: 4 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

google_compute_network.mynetwork: Creating...
google_compute_network.mynetwork: Still creating... [10s elapsed]
google_compute_network.mynetwork: Still creating... [20s elapsed]
google_compute_network.mynetwork: Still creating... [30s elapsed]
google_compute_network.mynetwork: Still creating... [40s elapsed]
google_compute_network.mynetwork: Still creating... [50s elapsed]
google_compute_network.mynetwork: Creation complete after 53s [id=projects/dave-terraform/global/networks/mynetwork]
google_compute_firewall.mynetwork-allow-http-ssh-rdp-icmp: Creating...
module.mynet-eu-vm.google_compute_instance.vm_instance: Creating...
module.mynet-us-vm.google_compute_instance.vm_instance: Creating...
google_compute_firewall.mynetwork-allow-http-ssh-rdp-icmp: Still creating... [10s elapsed]
module.mynet-eu-vm.google_compute_instance.vm_instance: Still creating... [10s elapsed]
module.mynet-us-vm.google_compute_instance.vm_instance: Still creating... [10s elapsed]
google_compute_firewall.mynetwork-allow-http-ssh-rdp-icmp: Creation complete after 12s [id=projects/dave-terraform/global/firewalls/mynetwork-allow-http-ssh-rdp-icmp]
module.mynet-us-vm.google_compute_instance.vm_instance: Creation complete after 18s [id=projects/dave-terraform/zones/us-central1-a/instances/mynet-us-vm]
module.mynet-eu-vm.google_compute_instance.vm_instance: Still creating... [20s elapsed]
module.mynet-eu-vm.google_compute_instance.vm_instance: Creation complete after 26s [id=projects/dave-terraform/zones/europe-west1-d/instances/mynet-eu-vm]

Check created resources

VPC network

Firewall

VMs

Destroy created resources


  $ terraform destroy
  
  Terraform will perform the following actions:

  # google_compute_firewall.mynetwork-allow-http-ssh-rdp-icmp will be destroyed
  - resource "google_compute_firewall" "mynetwork-allow-http-ssh-rdp-icmp" {
      - creation_timestamp      = "2024-10-10T09:46:54.415-07:00" -> null
      - destination_ranges      = [] -> null
      - direction               = "INGRESS" -> null
      - disabled                = false -> null
      - id                      = "projects/dave-terraform/global/firewalls/mynetwork-allow-http-ssh-rdp-icmp" -> null
      - name                    = "mynetwork-allow-http-ssh-rdp-icmp" -> null
      - network                 = "https://www.googleapis.com/compute/v1/projects/dave-terraform/global/networks/mynetwork" -> null
      - priority                = 1000 -> null
      - project                 = "dave-terraform" -> null
      - self_link               = "https://www.googleapis.com/compute/v1/projects/dave-terraform/global/firewalls/mynetwork-allow-http-ssh-rdp-icmp" -> null
      - source_ranges           = [
          - "0.0.0.0/0",
        ] -> null
      - source_service_accounts = [] -> null
      - source_tags             = [] -> null
      - target_service_accounts = [] -> null
      - target_tags             = [] -> null
        # (1 unchanged attribute hidden)

      - allow {
          - ports    = [
              - "22",
              - "80",
              - "3389",
            ] -> null
          - protocol = "tcp" -> null
        }
      - allow {
          - ports    = [] -> null
          - protocol = "icmp" -> null
        }
    }

  # google_compute_network.mynetwork will be destroyed
  - resource "google_compute_network" "mynetwork" {
      - auto_create_subnetworks                   = true -> null
      - delete_default_routes_on_create           = false -> null
      - enable_ula_internal_ipv6                  = false -> null
      - id                                        = "projects/dave-terraform/global/networks/mynetwork" -> null
      - mtu                                       = 0 -> null
      - name                                      = "mynetwork" -> null
      - network_firewall_policy_enforcement_order = "AFTER_CLASSIC_FIREWALL" -> null
      - numeric_id                                = "8609352241044109862" -> null
      - project                                   = "dave-terraform" -> null
      - routing_mode                              = "REGIONAL" -> null
      - self_link                                 = "https://www.googleapis.com/compute/v1/projects/dave-terraform/global/networks/mynetwork" -> null
        # (3 unchanged attributes hidden)
    }

Plan: 0 to add, 0 to change, 2 to destroy.

Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value:

Sunday, September 15, 2024

Convert Java properties file to sed commands using ant

HOWTO

GitHub

Get antcontrib.jar https://github.com/dveselka/java-tutorial/blob/master/tools/pom.xml
ant file to produce sed commands from property file https://github.com/dveselka/java-tutorial/blob/master/tools/build.xml

Install antcontrib

Get via Maven https://github.com/dveselka/java-tutorial/blob/master/tools/pom.xml

To install ant-contrib:

Copy ant-contrib-0.3.jar to the lib directory of your Ant installation. If you want to use one of the tasks in your own project, add the lines
```
<taskdef resource="net/sf/antcontrib/antcontrib.properties"/>
```
to your build file.

Keep ant-contrib-0.3.jar in a separate location. You now have to tell Ant explicitly where to find it (say in /usr/share/java/lib):

<taskdef resource="net/sf/antcontrib/antcontrib.properties">
  <classpath>
    <pathelement location="/usr/share/java/lib/ant-contrib-0.3.jar"/>
  </classpath>
</taskdef>

Get antcontrib.jar via Maven

<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>dave</groupId>
    <artifactId>tools</artifactId>
    <version>1.0.0-SNAPSHOT</version>
    <packaging>jar</packaging>

    <parent>
        <groupId>dave</groupId>
        <artifactId>parent</artifactId>
        <version>1.0.0-SNAPSHOT</version>
    </parent>

    <dependencies>
        <!-- https://mvnrepository.com/artifact/ant-contrib/ant-contrib -->
        <dependency>
            <groupId>ant-contrib</groupId>
            <artifactId>ant-contrib</artifactId>
            <version>1.0b3</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <artifactId>maven-dependency-plugin</artifactId>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>copy-dependencies</goal>
                        </goals>
                        <configuration>
                            <outputDirectory>${project.build.directory}/lib</outputDirectory>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>


$ mvn clean install


[INFO] --- maven-dependency-plugin:2.8:copy-dependencies (default) @ tools ---
[INFO] Copying ant-contrib-1.0b3.jar to /git/java-tutorial/tools/target/lib/ant-contrib-1.0b3.jar
[INFO] Copying ant-1.5.jar to /git/java-tutorial/tools/target/lib/ant-1.5.jar

Properties file https://github.com/dveselka/java-tutorial/blob/master/tools/dave.properties

a=1
jdbc.url=jdbc:oracle:thin:dave@dave:1521/DAVE_DB
url=https://dave.com/dave
date=Sun Sep 15 08:43:48 AM CEST 2024

Build.xml to produce sed file from properties file https://github.com/dveselka/java-tutorial/blob/master/tools/build.xml

Notice that @ gives value of for cycle variable , $ gives value of ant property

<project name="properties-to-sed">

    <taskdef resource="net/sf/antcontrib/antlib.xml">
        <classpath>
            <pathelement location="target/lib/ant-contrib-1.0b3.jar"/>
        </classpath>
    </taskdef>

    <target name="convert">
        
        <property file="dave.properties" prefix="dave."/>
        <propertyselector property="dave" match="dave\.(.*)" select="\1"/>
        
        <for param="key" list="${dave}">
            <sequential>
                <echo>@{key}=${dave.@{key}}</echo>
                <echo message="s#@{key}#${dave.@{key}}#g${line.separator}" file="replace-dave-props.sed" append="true"/>
            </sequential>
        </for>
    </target>
</project>

Run ant

$ ant convert
Buildfile: /git/java-tutorial/tools/build.xml

convert:
     [echo] url=https://dave.com/dave
     [echo] a=1
     [echo] date=Sun Sep 15 08:43:48 AM CEST 2024
     [echo] jdbc.url=jdbc:oracle:thin:dave@dave:1521/DAVE_DB

BUILD SUCCESSFUL
Total time: 0 seconds

Sed file produced from properties file

$ cat replace-dave-props.sed
s#url#https://dave.com/dave#g
s#a#1#g
s#date#Sun Sep 15 08:43:48 AM CEST 2024#g
s#jdbc.url#jdbc:oracle:thin:dave@dave:1521/DAVE_DB#g

Asking Gemimi for hints - useful, provided idea with antcontrib for cycle, needs fixing - add antcontrib install, fix for cycle

Approach:

Create a build.xml file:
- Write the Ant build script in a build.xml file.
- Define a target to perform the conversion.
Load properties file:
- Use the property task to load the properties file.
Generate sed commands:
- Use the for task to iterate over the properties entries.
- For each entry, create a sed command using the echo task.
Write sed commands to file:
- Use the echo task to write the generated sed commands to a file.

build.xml example:

XML
<project name="properties-to-sed">
    <target name="convert">
        <property file="replacements.properties"/>

        <for param="key" elements="${replacements.keys()}">
            <sequential>
                <echo message="s/${replacements.${key}}/${replacements.${key}}/g" file="sed_script.sed" append="true"/>
            </sequential>
        </for>
    </target>
</project>
 Use code with caution.

Explanation:

The replacements.properties file should contain key-value pairs where the key is the search pattern and the value is the replacement string. For example:

search1=replacement1
search2=replacement2

The build.xml file defines a convert target.
The property task loads the properties file.
The for task iterates over the keys in the properties file.
For each key, an echo task creates a sed command using the key and value as the search pattern and replacement string.
The echo task writes the sed command to the sed_script.sed file, appending it to the end of the file.

Additional considerations:

You can customize the sed command format based on your specific requirements.
If you need to escape special characters in the search patterns or replacement strings, use appropriate escape sequences.
For more complex scenarios or performance optimization, consider using a dedicated scripting language like Python or Perl.

Asking ChatGPT for comparison

Daniel Veselka

convert properties file to sed commands using antcontrib for cycle

ChatGPT

To convert a properties file to sed commands using ant-contrib's for loop functionality, you'll need to follow a similar approach as previously mentioned, with some adjustments to implement the loop correctly. Below, I’ll provide a complete and detailed example, including how to set up your environment to use ant-contrib for iterating through properties.

Example Properties File

Assuming you have a properties file named config.properties:

properties

# config.properties
app.name=MyApp
app.version=1.0.0
app.author=John Doe

Ant Build File

<project name="PropertiesToSed" default="convert" basedir=".">
    
    <property file="config.properties"/>
    
    <target name="convert">
        <echo message="Converting properties to sed commands..." />

        <!-- Loop through the properties defined in config.properties -->
        <property name="properties.list" value="app.name,app.version,app.author"/>

        <foreach list="${properties.list}" param="property">
            <sequential>
                <echo message="sed -i 's/${property}=.*/${property}=${property}:${property}/g' target_file.txt" />
            </sequential>
        </foreach>
        
    </target>
    
</project>

Explanation of the Ant Build File

Task Definition for Ant-Contrib: The line <taskdef resource="antcontrib.properties" /> defines the ant-contrib functions so you can use them in your Ant script. Make sure you have the ant-contrib.jar in your Ant classpath.
Loading Properties: The properties file is loaded into the project with <property file="config.properties"/>.
Defining Properties List: A property properties.list is defined, listing all the keys in the properties file that you want to generate sed commands for.
Using <for> Loop: The <for> loop is used to iterate over the properties list. Each property will be accessible via the variable ${property} in the loop.
Generating sed Commands:
- The echo command inside the loop outputs the sed command that will perform the desired replacement in target_file.txt.
- If you want to replace the value of each property in the file, you can use ${property} to get the key and ${property.value} to retrieve the corresponding value.

Step to Execute the Ant Script

Ensure you have ant-contrib: Download the ant-contrib.jar and place it in the lib directory of your Ant installation or add it to your classpath.
Run the Ant command:

bash

ant convert

Final Notes

The generated sed command will replace the line in target_file.txt that matches the pattern property=, updating it with the value from config.properties.
Adjust the target_file.txt with your actual target file; this is the file where the properties should be replaced.
Remember that you can customize the sed command based on how you want to implement the replacements based on your specific requirements.

Saturday, May 25, 2024

Java Microprofile GraphQL client using Open Liberty

HOWTO

https://openliberty.io/guides/graphql-client.html#what-youll-learn

GitHub

https://github.com/openliberty/guide-graphql-client

Build client

[INFO] -------------&lt; io.openliberty.guides:guide-graphql-intro &gt;--------------
[INFO] Building guide-graphql-intro 1.0-SNAPSHOT                          [5/5]
[INFO] --------------------------------[ pom ]---------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for guide-graphql-intro 1.0-SNAPSHOT:
[INFO] 
[INFO] guide-graphql-client-models ........................ SUCCESS [  0.786 s]
[INFO] guide-graphql-client-query ......................... SUCCESS [  1.080 s]
[INFO] guide-graphql-client-graphql ....................... SUCCESS [  0.252 s]
[INFO] guide-graphql-client-system ........................ SUCCESS [  0.189 s]
[INFO] guide-graphql-intro ................................ SUCCESS [  0.001 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS

Access client http://localhost:9084/openapi/ui/

Run query

Java Microprofile GraphQL server using Open Liberty

HOWTO

GitHub

https://github.com/OpenLiberty/guide-microprofile-graphql

Build models

$ mvn -pl models install
[INFO] Scanning for projects...
[INFO] 
[INFO] ------< io.openliberty.guides:guide-microprofile-graphql-models >-------
[INFO] Building guide-microprofile-graphql-models 1.0-SNAPSHOT

[INFO] Installing /git/guide-microprofile-graphql/finish/models/target/guide-microprofile-graphql-models-1.0-SNAPSHOT.jar to /home/dave/.m2/repository/io/openliberty/guides/guide-microprofile-graphql-models/1.0-SNAPSHOT/guide-microprofile-graphql-models-1.0-SNAPSHOT.jar
[INFO] Installing /git/guide-microprofile-graphql/finish/models/pom.xml to /home/dave/.m2/repository/io/openliberty/guides/guide-microprofile-graphql-models/1.0-SNAPSHOT/guide-microprofile-graphql-models-1.0-SNAPSHOT.pom
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS

Package WAR


$ mvn package


[INFO] Reactor Summary for guide-graphql-intro 1.0-SNAPSHOT:
[INFO] 
[INFO] guide-microprofile-graphql-models .................. SUCCESS [  0.794 s]
[INFO] guide-microprofile-graphql-system .................. SUCCESS [ 10.233 s]
[INFO] guide-microprofile-graphql-graphql ................. SUCCESS [  0.254 s]
[INFO] guide-graphql-intro ................................ SUCCESS [  0.001 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS

Enable GraphQL UI in server.xml

<server description="GraphQL service">
    <featureManager>
        <feature>restfulWS-3.1</feature>
        <feature>jsonb-3.0</feature>
        <feature>jsonp-2.1</feature>
        <feature>cdi-4.0</feature>
        <feature>mpConfig-3.1</feature>
        <feature>mpRestClient-3.0</feature>
        <feature>mpGraphQL-2.0</feature>
    </featureManager>

    <variable name="http.port" defaultValue="9082"/>
    <variable name="https.port" defaultValue="9445"/>

    <variable name="io.openliberty.enableGraphQLUI" value="true" />

    <webApplication location="guide-microprofile-graphql-graphql.war" contextRoot="/" />
    <httpEndpoint host="*" httpPort="${http.port}" 
        httpsPort="${https.port}" id="defaultHttpEndpoint"/>
</server>

GraphQL WAR

$ jar tvf ./graphql/target/guide-microprofile-graphql-graphql.war
     0 Sat May 25 07:38:34 CEST 2024 META-INF/
    81 Sat May 25 07:38:34 CEST 2024 META-INF/MANIFEST.MF
     0 Sat May 25 07:38:34 CEST 2024 WEB-INF/
     0 Sat May 25 07:38:34 CEST 2024 WEB-INF/classes/
     0 Sat May 25 07:38:34 CEST 2024 WEB-INF/classes/io/
     0 Sat May 25 07:38:34 CEST 2024 WEB-INF/classes/io/openliberty/
     0 Sat May 25 07:38:34 CEST 2024 WEB-INF/classes/io/openliberty/guides/
     0 Sat May 25 07:38:34 CEST 2024 WEB-INF/classes/io/openliberty/guides/graphql/
     0 Sat May 25 07:38:34 CEST 2024 WEB-INF/classes/io/openliberty/guides/graphql/client/
     0 Sat May 25 07:38:34 CEST 2024 WEB-INF/lib/
   433 Sat May 25 07:27:24 CEST 2024 WEB-INF/web.xml
  6366 Sat May 25 07:38:34 CEST 2024 WEB-INF/classes/io/openliberty/guides/graphql/GraphQLService.class
  1415 Sat May 25 07:38:34 CEST 2024 WEB-INF/classes/io/openliberty/guides/graphql/client/SystemClient.class
   568 Sat May 25 07:38:34 CEST 2024 WEB-INF/classes/io/openliberty/guides/graphql/client/UnknownUriException.class
  2268 Sat May 25 07:38:34 CEST 2024 WEB-INF/classes/io/openliberty/guides/graphql/client/UnknownUriExceptionMapper.class
  6693 Sat May 25 07:37:56 CEST 2024 WEB-INF/lib/guide-microprofile-graphql-models-1.0-SNAPSHOT.jar
 15086 Sat May 25 07:27:24 CEST 2024 favicon.ico
  3150 Sat May 25 07:27:24 CEST 2024 index.html
  4503 Sat May 25 07:27:24 CEST 2024 META-INF/maven/io.openliberty.guides/guide-microprofile-graphql-graphql/pom.xml
    97 Sat May 25 07:38:34 CEST 2024 META-INF/maven/io.openliberty.guides/guide-microprofile-graphql-graphql/pom.properties

SystemWAR

dave@fedora:/git/guide-microprofile-graphql/finish$ jar tvf ./system/target/guide-microprofile-graphql-system.war
     0 Sat May 25 07:38:34 CEST 2024 META-INF/
    81 Sat May 25 07:38:34 CEST 2024 META-INF/MANIFEST.MF
     0 Sat May 25 07:38:34 CEST 2024 WEB-INF/
     0 Sat May 25 07:38:34 CEST 2024 WEB-INF/classes/
     0 Sat May 25 07:38:34 CEST 2024 WEB-INF/classes/io/
     0 Sat May 25 07:38:34 CEST 2024 WEB-INF/classes/io/openliberty/
     0 Sat May 25 07:38:34 CEST 2024 WEB-INF/classes/io/openliberty/guides/
     0 Sat May 25 07:38:34 CEST 2024 WEB-INF/classes/io/openliberty/guides/system/
     0 Sat May 25 07:38:34 CEST 2024 WEB-INF/lib/
   432 Sat May 25 07:27:24 CEST 2024 WEB-INF/web.xml
  1942 Sat May 25 07:38:30 CEST 2024 WEB-INF/classes/io/openliberty/guides/system/SystemPropertiesResource.class
   444 Sat May 25 07:38:30 CEST 2024 WEB-INF/classes/io/openliberty/guides/system/SystemApplication.class
  2398 Sat May 25 07:38:30 CEST 2024 WEB-INF/classes/io/openliberty/guides/system/SystemMetricsResource.class
  6693 Sat May 25 07:37:56 CEST 2024 WEB-INF/lib/guide-microprofile-graphql-models-1.0-SNAPSHOT.jar
  2461 Sat May 25 07:27:24 CEST 2024 index.html
  4007 Sat May 25 07:27:24 CEST 2024 META-INF/maven/io.openliberty.guides/guide-microprofile-graphql-system/pom.xml
    96 Sat May 25 07:38:34 CEST 2024 META-INF/maven/io.openliberty.guides/guide-microprofile-graphql-system/pom.properties

Build Docker images

dave@fedora:/git/guide-microprofile-graphql/finish$ docker images
REPOSITORY                    TAG                   IMAGE ID       CREATED          SIZE
graphql                       1.0-SNAPSHOT          727f633aa833   19 seconds ago   753MB
system                        1.0-java17-SNAPSHOT   6812f8c9c7c0   45 seconds ago   765MB
system                        1.0-java11-SNAPSHOT   915f228505ad   2 minutes ago    747MB

Start containers

$ ./scripts/startContainers.sh
fec411c8ccc1e46a593b48399907c93284a2b4b2db82b027d876fd53b1bbf10d
53de0a3383d827a43f6bafe185143f92f885a8a99d36e9c60bb167730399b44e
d175e2f72bfd024c4179659893e92dd566b1120948de122fd7d714d668e403cc
1b7b58a43e02e84c750137e416e55b800b36acac4d6f0c481ccde5b184ef6d8f
dave@fedora:/git/guide-microprofile-graphql/finish$ docker ps 
CONTAINER ID   IMAGE                        COMMAND                  CREATED         STATUS         PORTS                                                           NAMES
d175e2f72bfd   graphql:1.0-SNAPSHOT         "/opt/ol/helpers/run…"   7 seconds ago   Up 6 seconds   9080/tcp, 9443/tcp, 0.0.0.0:9082->9082/tcp, :::9082->9082/tcp   graphql
53de0a3383d8   system:1.0-java17-SNAPSHOT   "/opt/ol/helpers/run…"   7 seconds ago   Up 6 seconds   9080/tcp, 9443/tcp                                              system-java17
1b7b58a43e02   system:1.0-java11-SNAPSHOT   "/opt/ol/helpers/run…"   7 seconds ago   Up 6 seconds   9080/tcp, 9443/tcp                                              system-java11

Call GraphQL endpoint http://localhost:9082/graphql/schema.graphql

"Mutation root"
type Mutation {
  "Changes the note set for the system"
  editNote(hostname: String, note: String): Boolean!
}

"Query root"
type Query {
  "Gets information about the system"
  system(hostname: String): system!
  "Gets system load data from the systems"
  systemLoad(hostnames: [String]): [systemLoad]
}

"Information about a Java installation"
type java {
  vendorName: String
  version: String!
}

"System usage data"
type loadData {
  heapUsed: BigInteger!
  loadAverage: Float!
  nonHeapUsed: BigInteger!
}

"Information about a single system"
type system {
  hostname: String!
  java: java!
  note: String
  osArch: String
  osName: String
  osVersion: String
  systemMetrics: systemMetrics!
  username: String!
}

"Information of system usage"
type systemLoad {
  hostname: String!
  loadData: loadData
}

"System metrics"
type systemMetrics {
  heapSize: BigInteger!
  nonHeapSize: BigInteger!
  processors: Int!
}

Access GraphiQL at the http://localhost:9082/graphql-ui

Query structure

# An example GraphQL query might look like:
#
#     {
#       field(arg: "value") {
#         subField
#       }
#     }

Query

query {
  system(hostname: "system-java11") {
    hostname
    username
    osArch
    osName
    osVersion
    systemMetrics {
      processors
      heapSize
      nonHeapSize
    }
    java {
      vendorName
      version
    }
  }
}

Response

{
  "data": {
    "system": {
      "hostname": "system-java11",
      "username": "default",
      "osArch": "amd64",
      "osName": "Linux",
      "osVersion": "6.8.10-200.fc39.x86_64",
      "systemMetrics": {
        "processors": 12,
        "heapSize": 4110352384,
        "nonHeapSize": -1
      },
      "java": {
        "vendorName": "IBM Corporation",
        "version": "11.0.22"
      }
    }
  }
}