Setup and Configure CI/CD pipeline for Jenkins Distributed Builds (Jenkins Master-Slaves Architecture)

Launch 3 AWS Ubuntu Server 18.04 EC2 instance as follows

Setup and Configure Jenkins Master

Update system
$ sudo apt update

Install and Configure Java
1. Download the Java version https://www.oracle.com/technetwork/java/javase/downloads/index.html
2. Extract the downloaded file jdk-8u221-linux-x64.tar.gz
    $ tar -zxvf jdk-8u221-linux-x64.tar.gz
3. Configure JAVA_HOME and PATH
    $ sudo nano ~/.bash_profile
       export JAVA_HOME=/home/ubuntu/bin/jdk1.8.0_221
       export PATH=$PATH:$JAVA_HOME/bin
    $ . ~/.bash_profile

Install and Configure Maven
1. Download the Maven version
    $ wget https://www-eu.apache.org/dist/maven/maven-3/3.6.2/binaries/apache-maven-3.6.2-bin.tar.gz
2. Extract the downloaded file apache-maven-3.6.2-bin.tar.gz
    $ tar -zxvf apache-maven-3.6.2-bin.tar.gz

Install and Configure Jenkins
    $ wget -q -O - https://pkg.jenkins.io/debian/jenkins-ci.org.key | sudo apt-key add -
    $ sudo sh -c 'echo deb http://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list'
    $ sudo apt-get update
    $ sudo apt-get install jenkins
Update JAVA path for Jenkins
    $ sudo nano /etc/init.d/jenkins
       PATH=/bin:/usr/bin:/sbin:/usr/sbin:/home/ubuntu/bin/jdk1.8.0_221/bin
    $ sudo systemctl daemon-reload
    $ sudo service jenkins start

Configure JAVA and Maven path in Jenkins --> Gobal Tool Configuration

Setup and Configure Jenkins Slave 1

Update system
    $ sudo apt update

Install and Configure Java
    $ sudo apt update
    $ sudo apt install default-jre
    $ sudo apt install default-jdk

Install and Configure Tomcat
    $ wget https://www-us.apache.org/dist/tomcat/tomcat-8/v8.5.46/bin/apache-tomcat-8.5.46.tar.gz
    $ tar -zxvf apache-tomcat-8.5.46.tar.gz
    $ sudo chown -R ubuntu apache-tomcat-8.5.46
Configure tomact to allow traffic on custom port 8082
    $ sudo nano ./apache-tomcat-8.5.46/conf/server.xml
       <Connector port="8082" protocol="HTTP/1.1"
       connectionTimeout="20000"
       redirectPort="8443" />
Configure tomcat user role and credentials
    $ sudo nano ./apache-tomcat-8.5.46/conf/tomcat-users.xml
       <role rolename="manager-script"/>
       <role rolename="manager-gui"/>
       <user username="xdk" password="*****" roles="manager-script,manager-gui"/>
Configure tomcat to allow deployment from other source
    $ sudo nano ./apache-tomcat-8.5.46/webapps/manager/META-INF/context.xml
       Comment
       <!-- <Valve className="org.apache.catalina.valves.RemoteAddrValve"
       allow="127\.\d+\.\d+\.\d+|::1|0:0:0:0:0:0:0:1" /> -->
    $ ./apache-tomcat-8.5.46/bin/startup.sh

Setup and Configure Jenkins Slave 1

Update system
    $ sudo apt update

Install and Configure Java 
    $ sudo apt update
    $ sudo apt install default-jre
    $ sudo apt install default-jdk

Install and Configure Tomcat
    $ wget https://www-us.apache.org/dist/tomcat/tomcat-8/v8.5.46/bin/apache-tomcat-8.5.46.tar.gz
    $ tar -zxvf apache-tomcat-8.5.46.tar.gz
    $ sudo chown -R ubuntu apache-tomcat-8.5.46
Configure tomcat user role and credentials
    $ sudo nano ./apache-tomcat-8.5.46/conf/tomcat-users.xml
       <role rolename="manager-script"/>
       <role rolename="manager-gui"/>
       <user username="xdk" password="*****" roles="manager-script,manager-gui"/>
Configure tomcat to allow deployment from other source
    $ sudo nano ./apache-tomcat-8.5.46/webapps/manager/META-INF/context.xml
       Comment
       <!-- <Valve className="org.apache.catalina.valves.RemoteAddrValve"
       allow="127\.\d+\.\d+\.\d+|::1|0:0:0:0:0:0:0:1" /> -->
    $ ./apache-tomcat-8.5.46/bin/startup.sh

Setup and Configure Master-Slaves Machines in Jenkins (Setup Auto SSH Login)

SSH to Jenkins Master
    $ sudo -iu jenkins
Generate Public & Private RSA Key.
    $ ssh-keygen -t rsa
       public key location : /var/lib/jenkins/.ssh/id_rsa.pub
       private key location : /var/lib/jenkins/.ssh/id_rsa
Create .ssh directory on Jenkins Slave 1 from Jenkins Master
    $ ssh -i /home/ubuntu/bin/myaccountkey.pem ubuntu@3.91.65.12 mkdir -p .ssh
Attach master public key with slave authorized directory
    $ cat .ssh/id_rsa.pub | ssh -i /home/ubuntu/bin/myaccountkey.pem ubuntu@3.91.65.12 'cat >> .ssh/authorized_keys'
Verify connection
    $ ssh ubuntu@3.91.65.12
Create .ssh directory on Jenkins Slave 2 from Jenkins Master
    $ ssh -i /home/ubuntu/bin/myaccountkey.pem ubuntu@3.95.20.115 mkdir -p .ssh
Attach master public key with slave authorized directory
    $ cat .ssh/id_rsa.pub | ssh -i /home/ubuntu/bin/myaccountkey.pem ubuntu@3.95.20.115 'cat >> .ssh/authorized_keys'

Download Jenkins Slave 1 Agent Program on Jenkins Slave 1 Machine
SSH to Jenkins Slave 1
Create bin directory.
    $ mkdir ~/bin
Goto bin and download slave.jar from master
    $ cd bin
    $ wget http://54.81.205.161:8080/jnlpJars/slave.jar

Add Jenkins Slave 1 Machine to Jenkins Master
Login to Jenkins site
Click --> -->

 

Launch Command
    $ ssh ubuntu@3.91.65.12 java -jar ./bin/slave.jar

Download Jenkins Slave 2 Agent Program on Jenkins Slave 2 Machine
SSH to Jenkins Slave 2
Create bin directory.
    $ mkdir ~/bin
Goto bin and download slave.jar from master
    $ cd bin 
    $ wget http://54.81.205.161:8080/jnlpJars/slave.jar

Add Jenkins Slave 2 Machine to Jenkins Master
Login to Jenkins site
Click --> -->

Launch Command
    $ ssh ubuntu@3.95.20.115 java -jar ./bin/slave.jar

Create Jenkins Project for CI/CD pipeline

Create Jenkins job to build and archive artifacts of Maven-Java Project

Examples :
0 0 * * * Everyday at Midnight.
0 2-4 * * * 2am, 3am, 4am Everyday.


Create Jenkins job to perform static code analysis for Maven-Java Project
Install the CheckStyle plug-in.


Create Jenkins job to deploy the Maven-Java Project build artifacts remote servers
Install "Copy Artifact" and "Deploy to container" Plugins

Create upstream and down stream projects for CI/CD pipeline
build maven-java project Job


     

Create build pipeline view
Install build pipeline plug in

 

About the author

My name is Xavier Dilip Kumar Jayaraj having 16+ years of IT experience which includes solid experience and depth Knowledge in Application Life Cycle Management, Configuration Management, Implementation and Support using TFS on-premises and Azure DevOps. I have invested in gaining DevOps knowledege to expertise with Cloud Computing providers namely Microsoft Azure and Amazon Web Services in recent years. I am very positive to learn and adapt emerging technologies to client’s environment.

Microsoft Certified: Azure Administrator Associate

Microsoft Certified: Azure DevOps Engineer Expert

DevOps Engineer Certificate Program | Transcript 

OTP-AWSD5: AWS IoT: Developing and Deploying an Internet of Things

[PCEP-30-01] PCEP – Certified Entry-Level Python Programmer

LinkedIn Profile

Knowledge Base Articles

Quotes I Like

"Failure will never overtake me if my determination to succeed is strong enough."  - Dr. APJ. Abdul Kalam

"Always be yourself, express yourself, have faith in yourself, do not go out and look for a successful personality and duplicate it." - Bruce Lee

"Technology is just a tool. In terms of getting the kids working together and motivating them, the teacher is the most important." - Bill Gates

"Innovation distinguishes between a leader and a follower." - Steve Jobs

CategoryList

Disclaimer

The information provided here is based on my expreriences, troubleshooting and online/offline findings. It can be used as is on your own risk without any warranties and I impose no rights.