This is the first article in the IBM Cloud Pak for Integration (CP4I) series. It describes how to deploy OpenShift 4.3 cluster as a platform to run IBM Cloud Pak for Integration. This article does not consider many important production things and can be used for PoC, MVP or other demo cases. This article is not covering VMs deployment, therefore I consider you have your machines ran on a hypervisor with OS installed. I would recommend to create snapshots for prepared VMs in case you need to rollback to the first step.

The Architecture of the installation

The installation includes 10 machines ran in VmWare: 3 Masters based on RHCOS; 5 Workers based on RHEL7.6; 1 Bootstrap; 1 Helpernode for cluster management.

[TBD: Details]

Prepare the Helpernode

Based on the quickstart: https://github.com/RedHatOfficial/ocp4-helpernode/blob/master/docs/quickstart.md#prepare-the-helper-node

Install EPEL
yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-$(rpm -E %rhel).noarch.rpm

Install ansible and git and clone this repo
yum -y install ansible git
git clone https://github.com/RedHatOfficial/ocp4-helpernode
cd ocp4-helpernode

Get the Mac addresses for the cluster machines and write them down somewhere

  bootstrap: 00:50:56:XX:XX:XX
  master1: 00:50:56:XX:XX:XX
  master2: 00:50:56:XX:XX:XX
  master3: 00:50:56:XX:XX:XX
  worker1: 00:50:56:XX:XX:XX
  worker2: 00:50:56:XX:XX:XX
  worker3: 00:50:56:XX:XX:XX
  worker4: 00:50:56:XX:XX:XX
  worker5: 00:50:56:XX:XX:XX

Edit the vars.yaml file. Use the example below.
cp docs/examples/vars.yaml .
vi vars.yaml

---
disk: sda
helper:
  name: "helpernode"
  ipaddr: "192.168.28.50"
  networkifacename: "ens192"
dns:
  domain: "marukhno.com"
  clusterid: "ocp01"
  forwarder1: "8.8.8.8"
  forwarder2: "8.8.4.4"
dhcp:
  router: "192.168.31.254"
  bcast: "192.168.31.255"
  netmask: "255.255.255.0"
  poolstart: "192.168.28.50"
  poolend: "192.168.28.69"
  ipid: "192.168.24.0"
  netmaskid: "255.255.248.0"
bootstrap:
  name: "bootstrap"
  ipaddr: "192.168.28.59"
  macaddr: "00:50:56:XX:XX:XX"
masters:
  - name: "master1"
    ipaddr: "192.168.28.51"
    macaddr: "00:50:56:XX:XX:XX"
  - name: "master2"
    ipaddr: "192.168.28.52"
    macaddr: "00:50:56:XX:XX:XX"
  - name: "master3"
    ipaddr: "192.168.28.53"
    macaddr: "00:50:56:XX:XX:XX"
workers:
  - name: "worker1"
    ipaddr: "192.168.28.54"
    macaddr: "00:50:56:XX:XX:XX"
  - name: "worker2"
    ipaddr: "192.168.28.55"
    macaddr: "00:50:56:XX:XX:XX"
  - name: "worker3"
    ipaddr: "192.168.28.56"
    macaddr: "00:50:56:XX:XX:XX"
  - name: "worker4"
    ipaddr: "192.168.28.57"
    macaddr: "00:50:56:XX:XX:XX"
  - name: "worker5"
    ipaddr: "192.168.28.58"
    macaddr: "00:50:56:XX:XX:XX"
vars.yaml

Run the main.yaml playbook
ansible-playbook -e @vars.yaml tasks/main.yml

Run the following to get info about your environment. 'haproxy' is just one of the arguments you can use with this command. For now, what we need is the address of a console which shows if the cluster is ready.
/usr/local/bin/helpernodecheck haproxy

Create Ignition config files

Create an install dir
mkdir ~/ocp4
cd ~/ocp4

Create a place to store your pull-secret
mkdir -p ~/.openshift

Visit https://cloud.redhat.com/openshift/install and select "Bare Metal" or "vSphere", depends on what you are using. Download your pull secret and save it under ~/.openshift/pull-secret.

The playbook run before creates an sshkey for you; it's under ~/.ssh/helper_rsa. You can use this key or create/use another one if you wish.

Create install-config.yaml (I used id_rsa here)

cat <<EOF > install-config.yaml
apiVersion: v1
baseDomain: marukhno.com
compute:
- hyperthreading: Enabled
  name: worker
  replicas: 5
controlPlane:
  hyperthreading: Enabled
  name: master
  replicas: 3
metadata:
  name: ocp01
networking:
  clusterNetworks:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  networkType: OpenShiftSDN
  serviceNetwork:
  - 172.30.0.0/16
platform:
  none: {}
pullSecret: '$(< ~/.openshift/pull-secret)'
sshKey: '$(< ~/.ssh/id_rsa.pub)'
EOF
install-config.yaml creation

Create installation manifests
openshift-install create manifests

If you need masters to be schedulable enable this:
Edit the manifests/cluster-scheduler-02-config.yml Kubernetes manifest file to prevent Pods from being scheduled on the control plane machines by setting mastersSchedulable to false.
sed -i 's/mastersSchedulable: true/mastersSchedulable: false/g' manifests/cluster-scheduler-02-config.yml
I didn't do this for my installation as I didn't want any pods be scheduled on my masters.

Generate the ignition configs
openshift-install create ignition-configs

Copy the ignition files in the ignition directory for the websever
cp ~/ocp4/*.ign /var/www/html/ignition/
restorecon -vR /var/www/html/
chmod o+r /var/www/html/ignition/*.ign

Adding Nodes

Launch vSphere, and boot the VMs into the boot menu; and select PXE. The vms should boot into the proper PXE profile, based on their IP address. I used Fn+F12 while RHEL 7.6 loaded to load PXE mode and chose Bootstrap and Master installation on VMs. I didn't install Workers on this step, because I planned to run them on RHEL.

Boot/install the VMs in the following order

  1. Bootstrap
  2. Masters
  3. Workers

On your laptop/workstation visit the status page for haproxy. Use this command to get the URL: /usr/local/bin/helpernodecheck haproxy

You'll see the bootstrap turn "green" and then the masters turn "green", then the bootstrap turn "red". This is your indication that you can continue.

Wait for install. The boostrap VM actually does the install for you; you can track it with the following command.

openshift-install wait-for bootstrap-complete --log-level debug

Once you see this message below...

DEBUG OpenShift Installer v4.2.0-201905212232-dirty
DEBUG Built from commit 71d8978039726046929729ad15302973e3da18ce
INFO Waiting up to 30m0s for the Kubernetes API at https://api.ocp4.example.com:6443...
INFO API v1.13.4+838b4fa up
INFO Waiting up to 30m0s for bootstrapping to complete...
DEBUG Bootstrap status: complete
INFO It is now safe to remove the bootstrap resources
...you can continue....at this point you can delete the bootstrap server.

Add workers which are on RHEL7.6 in my case
https://docs.openshift.com/container-platform/4.3/machine_management/adding-rhel-compute.html
Keep in mind that you have less than 24 hours to add Worker nodes without additional steps. After that time you will have to update worker.ign because of expired certificate. You can find details on this link: https://access.redhat.com/solutions/4799921

Verify that you have access to the console. Your login will be kubeadmin and the password can be found under the file ocp4/auth/kubeadmin-password
Login to OCP
oc login -u kubeadmin
Get your console URL
oc whoami --show-console=true

Open the link in the browser and use the same credentials to login

Process Restart
If something went wrong and you want to start the process from the step one be sure to clean up before trying again

rm -rf /var/www/cgi-bin/ /var/www/html/
rm -rf ocp4 ocp4-helpernode/ ansible/ openshift-ansible.log install-config.yaml.orig
rm -rf .openshift .openshift_install.log .ansible/
rm -rf .ssh/known_hosts .ssh/helper_rsa.pub .ssh/helper_rsa .ssh/config
rm -rf /usr/local/src/openshift-install-linux.tar.gz
rm -rf /var/www/html/install/bios.raw.gz
rm -rf /var/lib/tftpboot/rhcos/initramfs.img
rm -rf /var/lib/tftpboot/rhcos/kernel