DevOps tips & tricks: pipeline

Showing posts with label pipeline. Show all posts

Thursday, December 31, 2020

Call another job in Jenkins declarative pipeline with parameters

HOWTO

Parameters on the pipeline job

pipeline {
    agent any
    parameters {
        string(name: 'PERSON', defaultValue: 'Mr Jenkins', description: 'Who should I say hello to?')

        text(name: 'BIOGRAPHY', defaultValue: '', description: 'Enter some information about the person')

        booleanParam(name: 'TOGGLE', defaultValue: true, description: 'Toggle this value')

        choice(name: 'CHOICE', choices: ['One', 'Two', 'Three'], description: 'Pick something')

        password(name: 'PASSWORD', defaultValue: 'SECRET', description: 'Enter a password')
    }
    stages {
        stage('Example') {
            steps {
                echo "Hello ${params.PERSON}"

                echo "Biography: ${params.BIOGRAPHY}"

                echo "Toggle: ${params.TOGGLE}"

                echo "Choice: ${params.CHOICE}"

                echo "Password: ${params.PASSWORD}"
            }
        }
    }
}

Called job


pipeline {
    agent { 
       label 'dave_node'
    }
    parameters {
        string(name: 'branch', defaultValue: 'main', description: 'Some branch')
    }
    stages {
        stage('CalledJob') {
        
            steps {
            
                 echo "Build CalledJob branch ${params.branch}"  
                 
                 script {         
                     params.each {param ->
                       println "${param.key} -> ${param.value} "
                     }
                 }
                         
            
                git url: 'https://github.com/dveselka/calledjob/', credentialsId: 'daveCredentials', branch: "${params.branch}"     
            }
        }
    }
}

Calling job

   
       stage('BuildOtherJob') {
            steps {   
            
            echo "Call other job job"        
                script {           
                      build job: 'jobtocall', parameters: [
                             string(name: 'branch', value: "some_branch")
                        ]
                }
            }
       }

Dynamic agent

pipeline {
    agent { 
       label params.agent == null || params.agent.isEmpty() ? "default-agent" : params.agent
    }

Sunday, December 27, 2020

Deploy JBoss Wildfly Java EE app with Maven using Jenkins pipeline and Docker

HOWTO

GitHub projects

https://github.com/dveselka/java-ee-8/tree/main/java-ee8-minimal

Pipeline file https://github.com/dveselka/java-ee-8/blob/main/java-ee8-minimal/Jenkinsfile

Add jenkins user into docker group

sudo usermod -aG docker jenkins
sudo systemctl restart jenkins

Checkout app from GitHub

        stage('Checkout') {
          git url: 'https://github.com/dveselka/java-ee-8/', credentialsId: 'dave-devops', branch: 'main'
        }

Build application using Maven

        stage('Build') {
          
          dir('java-ee8-minimal'){
             withMaven(maven:'Maven'
             ) {
                sh 'pwd'
                sh 'env'
                sh 'mvn clean package'
            
                def pom = readMavenPom file:'pom.xml'
                print pom.version
                env.version = pom.version
             }
          }
        }

Build Docker image

https://www.jenkins.io/doc/book/pipeline/docker/

        stage("BuildDockerImage"){
          dir('java-ee8-minimal'){
            def customImage = docker.build("my-image:${env.BUILD_ID}")
          }
        }

Jenkins log - Docker image build

[Pipeline] { (Image)
[Pipeline] dir
Running in /var/lib/jenkins/workspace/java-ee-8/java-ee8-minimal
[Pipeline] {
[Pipeline] isUnix
[Pipeline] sh
+ docker build -t com.dave/java-ee8-minimal:4 .
Sending build context to Docker daemon  46.59kB

Step 1/4 : FROM jboss/wildfly
 ---> 8d9094a2468d
Step 2/4 : RUN /opt/jboss/wildfly/bin/add-user.sh admin Admin#123 --silent
 ---> Using cache
 ---> 8b8272a09327
Step 3/4 : COPY ./target/java-ee8-minimal.war /opt/jboss/wildfly/standalone/deployments/
 ---> 229ba08da529
Step 4/4 : CMD ["/opt/jboss/wildfly/bin/standalone.sh", "-b", "0.0.0.0", "-bmanagement", "0.0.0.0"]
 ---> Running in fdea8bab71ee
Removing intermediate container fdea8bab71ee
 ---> c165d87af41e
Successfully built c165d87af41e
Successfully tagged com.dave/java-ee8-minimal:4

Push Docker image to repository (optional)

Run container

        stage ('Run') {
          dir('java-ee8-minimal'){
            docker.image("com.dave/java-ee8-minimal:${env.BUILD_ID}").run('-it --name java-ee8-minimal com.dave/java-ee8-minimal')
          }
        }

Install Jenkins Docker plugin to remove this error

groovy.lang.MissingPropertyException: No such property: docker for class: groovy.lang.Binding
    at groovy.lang.Binding.getVariable(Binding.java:63)
    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onGetProperty(SandboxInterceptor.java:270)

Jenkins log - run Docker container

[Pipeline] { (Run)
[Pipeline] dir
Running in /var/lib/jenkins/workspace/java-ee-8/java-ee8-minimal
[Pipeline] {
[Pipeline] isUnix
[Pipeline] sh
+ docker run -d -it --name java-ee8-minimal com.dave/java-ee8-minimal:6
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

Docker process

docker ps -a
CONTAINER ID        IMAGE                         COMMAND                  CREATED             STATUS              PORTS               NAMES
0eb3d2588c6b        com.dave/java-ee8-minimal:6   "/opt/jboss/wildfly/…"   2 minutes ago       Up 2 minutes        8080/tcp            java-ee8-minimal

Monday, November 30, 2020

Configure Jenkins agent using declarative pipeline

HOWTO

Configure agents https://www.howtoforge.com/tutorial/ubuntu-jenkins-master-slave/
Agent x node https://stackoverflow.com/questions/42050626/jenkins-pipeline-agent-vs-node
Declarative pipeline syntax https://www.jenkins.io/doc/book/pipeline/syntax/
Pipeline example https://www.jenkins.io/doc/pipeline/examples/
https://www.jenkins.io/doc/book/pipeline/jenkinsfile/#using-multiple-agents

Sample projects

Configure Jenkins agent

agent

The agent section specifies where the entire Pipeline, or a specific stage, will execute in the Jenkins environment depending on where the agent section is placed. The section must be defined at the top-level inside the pipeline block, but stage-level usage is optional.

label

Execute the Pipeline, or stage, on an agent available in the Jenkins environment with the provided label. For example: agent { label 'my-defined-label' }

node: agent { node { label 'labelName' } } behaves the same as agent { label 'labelName' }, but node allows for additional options (such as customWorkspace).

Multiple agents

pipeline {
    agent none
    stages {
        stage('Build') {
            agent any
            steps {
                checkout scm
                sh 'make'
                stash includes: '**/target/*.jar', name: 'app' 
            }
        }
        stage('Test on Linux') {
            agent { 
                label 'linux'
            }
            steps {
                unstash 'app' 
                sh 'make check'
            }
            post {
                always {
                    junit '**/target/*.xml'
                }
            }
        }
        stage('Test on Windows') {
            agent {
                label 'windows'
            }
            steps {
                unstash 'app'
                bat 'make check' 
            }
            post {
                always {
                    junit '**/target/*.xml'
                }
            }
        }
    }
}

Configure GitHub credentials for Jenkins

Configure pipeline

GitHub credentials must be username/password ( use GH token)

https://stackoverflow.com/questions/38461705/checkout-jenkins-pipeline-git-scm-with-credentials?rq=1

If you're using the ssh url then your credentials must be username + private key. If you're using the https clone url instead of the ssh one, then your credentials should be username + password.

To change working dir use dir ('somedir'){ ... }

https://github.com/dveselka/weblogic/blob/master/Jenkinsfile

node {

        stage('Checkout') {
          git url: 'https://github.com/dveselka/weblogic/', credentialsId: 'dave-devops', branch: 'weblogic-14.1.1'
        }

        stage('Build') {
            dir ('dave-basic-webapp-ejb-project') {
                withMaven(maven:'local') {
                   sh 'mvn clean package'

                   def pom = readMavenPom file:'pom.xml'
                   print pom.version
                   env.version = pom.version
                }
            }     
        }

}

Install Maven pipeline plugin

https://www.jenkins.io/doc/pipeline/steps/pipeline-maven/

https://plugins.jenkins.io/pipeline-maven/

Install Pipeline utility steps plugin

https://plugins.jenkins.io/pipeline-utility-steps/

Configure Maven

dave@dave ~]$ which mvn
/opt/maven/bin/mvn
[dave@dave ~]$ mvn --version
Apache Maven 3.6.2 (40f52333136460af0dc0d7232c0dc0bcf0d9e117; 2019-08-27T17:06:16+02:00)
Maven home: /opt/maven
Java version: 11.0.9, vendor: Oracle Corporation, runtime: /usr/java/jdk-11.0.9
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "5.9.15-100.fc32.x86_64", arch: "amd64", family: "unix"

https://docs.oracle.com/en/middleware/fusion-middleware/weblogic-server/12.2.1.4/wlprg/maven.html#GUID-C6FC7582-2D1C-4EA5-B000-71AE9A2F2B05

Configure Oracle Maven repository for Weblogic

[dave@dave 14.1.1]$ cd /app/weblogic/oracle_common/plugins/maven/com/oracle/maven/oracle-maven-sync/14.1.1/^C
[dave@dave 14.1.1]$ mvn install:install-file -DpomFile=oracle-maven-sync-14.1.1.pom -Dfile=oracle-maven-sync-14.1.1.jar
[INFO] Scanning for projects...
[INFO] 
[INFO] ------------------< org.apache.maven:standalone-pom >-------------------
[INFO] Building Maven Stub Project (No POM) 1
[INFO] --------------------------------[ pom ]---------------------------------
[INFO] 
[INFO] --- maven-install-plugin:2.4:install-file (default-cli) @ standalone-pom ---
[INFO] Installing /app/weblogic/oracle_common/plugins/maven/com/oracle/maven/oracle-maven-sync/14.1.1/oracle-maven-sync-14.1.1.jar to /home/dave/.m2/repository/com/oracle/maven/oracle-maven-sync/14.1.1-0-0/oracle-maven-sync-14.1.1-0-0.jar
[INFO] Installing /app/weblogic/oracle_common/plugins/maven/com/oracle/maven/oracle-maven-sync/14.1.1/oracle-maven-sync-14.1.1.pom to /home/dave/.m2/repository/com/oracle/maven/oracle-maven-sync/14.1.1-0-0/oracle-maven-sync-14.1.1-0-0.pom
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS

Install Oracle artifacts into local repo

[dave@dave 14.1.1]$ mvn com.oracle.maven:oracle-maven-sync:push -DoracleHome=/app/weblogic

Build pipeline

Started by user DaVe
Obtained Jenkinsfile from git https://github.com/dveselka/weblogic/
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/lib/jenkins/workspace/weblogic-pipeline
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Checkout)
[Pipeline] git
The recommended git tool is: NONE
using credential dave-devops
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/dveselka/weblogic/ # timeout=10
Fetching upstream changes from https://github.com/dveselka/weblogic/
 > git --version # timeout=10
 > git --version # 'git version 2.26.2'
using GIT_ASKPASS to set credentials dave-devops
 > git fetch --tags --force --progress -- https://github.com/dveselka/weblogic/ +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git rev-parse refs/remotes/origin/weblogic-14.1.1^{commit} # timeout=10
Checking out Revision d880e4d74b8302545401074d68fd449115fdca1b (refs/remotes/origin/weblogic-14.1.1)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d880e4d74b8302545401074d68fd449115fdca1b # timeout=10
 > git branch -a -v --no-abbrev # timeout=10
 > git branch -D weblogic-14.1.1 # timeout=10
 > git checkout -b weblogic-14.1.1 d880e4d74b8302545401074d68fd449115fdca1b # timeout=10
Commit message: "Merge pull request #2 from dveselka/master"
 > git rev-list --no-walk 7fe92410b7aab93803b8dbd39200a48c23331709 # timeout=10
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Build)
[Pipeline] dir
Running in /var/lib/jenkins/workspace/weblogic-pipeline/dave-basic-webapp-ejb-project
[Pipeline] {
[Pipeline] withMaven
[withMaven] Options: []
[withMaven] Available options: 
[withMaven] using JDK installation provided by the build agent
[withMaven] using Maven installation 'local'
[Pipeline] {
[Pipeline] sh
+ mvn clean package
----- withMaven Wrapper script -----
Picked up JAVA_TOOL_OPTIONS: -Dmaven.ext.class.path="/var/lib/jenkins/workspace/weblogic-pipeline/dave-basic-webapp-ejb-project@tmp/withMaven8d13bb8c/pipeline-maven-spy.jar" -Dorg.jenkinsci.plugins.pipeline.maven.reportsFolder="/var/lib/jenkins/workspace/weblogic-pipeline/dave-basic-webapp-ejb-project@tmp/withMaven8d13bb8c" 
Apache Maven 3.6.2 (40f52333136460af0dc0d7232c0dc0bcf0d9e117; 2019-08-27T17:06:16+02:00)
Maven home: /opt/maven
Java version: 11.0.9, vendor: Oracle Corporation, runtime: /usr/java/jdk-11.0.9
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "5.9.15-100.fc32.x86_64", arch: "amd64", family: "unix"
[INFO] [jenkins-event-spy] Generate /var/lib/jenkins/workspace/weblogic-pipeline/dave-basic-webapp-ejb-project@tmp/withMaven8d13bb8c/maven-spy-20201227-095826-57316330863830296583212.log.tmp ...
[INFO] Scanning for projects...
[INFO] 
[INFO] -----------------< dave:dave-basic-webapp-ejb-project >-----------------
[INFO] Building basicWebappEjb 1.0-SNAPSHOT
[INFO] --------------------------------[ war ]---------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ dave-basic-webapp-ejb-project ---
[INFO] Deleting /var/lib/jenkins/workspace/weblogic-pipeline/dave-basic-webapp-ejb-project/target
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ dave-basic-webapp-ejb-project ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 1 resource
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ dave-basic-webapp-ejb-project ---
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
[INFO] Compiling 6 source files to /var/lib/jenkins/workspace/weblogic-pipeline/dave-basic-webapp-ejb-project/target/classes
[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ dave-basic-webapp-ejb-project ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory /var/lib/jenkins/workspace/weblogic-pipeline/dave-basic-webapp-ejb-project/src/test/resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ dave-basic-webapp-ejb-project ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ dave-basic-webapp-ejb-project ---
[INFO] No tests to run.
[INFO] 
[INFO] --- maven-war-plugin:2.1.1:war (default-war) @ dave-basic-webapp-ejb-project ---
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.thoughtworks.xstream.core.util.Fields (file:/var/lib/jenkins/.m2/repository/com/thoughtworks/xstream/xstream/1.3.1/xstream-1.3.1.jar) to field java.util.Properties.defaults
WARNING: Please consider reporting this to the maintainers of com.thoughtworks.xstream.core.util.Fields
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[INFO] Packaging webapp
[INFO] Assembling webapp [dave-basic-webapp-ejb-project] in [/var/lib/jenkins/workspace/weblogic-pipeline/dave-basic-webapp-ejb-project/target/basicWebappEjb]
[INFO] Processing war project
[INFO] Copying webapp resources [/var/lib/jenkins/workspace/weblogic-pipeline/dave-basic-webapp-ejb-project/src/main/webapp]

[INFO] Webapp assembled in [33 msecs]
[INFO] Building war: /var/lib/jenkins/workspace/weblogic-pipeline/dave-basic-webapp-ejb-project/target/basicWebappEjb.war
[INFO] WEB-INF/web.xml already added, skipping
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  2.763 s
[INFO] Finished at: 2020-12-27T09:58:29+01:00
[INFO] ------------------------------------------------------------------------
[INFO] [jenkins-event-spy] Generated /var/lib/jenkins/workspace/weblogic-pipeline/dave-basic-webapp-ejb-project@tmp/withMaven8d13bb8c/maven-spy-20201227-095826-57316330863830296583212.log
[Pipeline] readMavenPom
[Pipeline] echo
1.0-SNAPSHOT
[Pipeline] }
[withMaven] artifactsPublisher - Archive artifact pom.xml under dave/dave-basic-webapp-ejb-project/1.0-SNAPSHOT/dave-basic-webapp-ejb-project-1.0-SNAPSHOT.pom
[withMaven] artifactsPublisher - Archive artifact target/basicWebappEjb.war under dave/dave-basic-webapp-ejb-project/1.0-SNAPSHOT/dave-basic-webapp-ejb-project-1.0-SNAPSHOT.war
[withMaven] junitPublisher - Archive test results for Maven artifact dave:dave-basic-webapp-ejb-project:war:1.0-SNAPSHOT generated by maven-surefire-plugin:test (default-test): target/surefire-reports/*.xml
[withMaven] junitPublisher - Jenkins JUnit Attachments Plugin not found, can't publish test attachments.Recording test results
None of the test reports contained any result
[withMaven] Jenkins Task Scanner Plugin not found, don't display results of source code scanning for 'TODO' and 'FIXME' in pipeline screen.
[withMaven] Publishers: Pipeline Graph Publisher: 3 ms, Generated Artifacts Publisher: 17 ms, Junit Publisher: 31 ms
[Pipeline] // withMaven
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS