AWS High Availability Incorta Cluster Upgrade Guide
Guide: Upgrade a High Availability Incorta Cluster in AWS
Whereas the AWS High Availability Incorta Cluster Guide describes how to install, configure, and deploy a high availability (HA) Incorta Cluster in Amazon Web Services (AWS), this guide describes how to upgrade your AWS High Availability Incorta Cluster so that Incorta as a Unified Data Analytics Platform is continuously available.
High level summary of the Incorta Cluster Upgrade Procedure
Here is a high level summary of the this upgrade procedure:
- Pause all scheduled jobs for all cluster tenants
- Upgrade the Cluster Management Console (CMC)
- Upgrade LoaderService_1
- Upgrade AnalyticsService_1
- Upgrade LoaderService_2
- Upgrade AnalyticsService_2
- Upgrade Apache Spark
- Resume all scheduled jobs for all cluster tenants
Assumptions
As this guide is based on the AWS High Availability Incorta Cluster Guide, it contains assumptions about your cluster typology in AWS.
EC2 Host Configuration Details
Only 5 of the 7 EC2 hosts run Incorta HA Nodes and 1 EC2 runs the Cluster Management Console. Two EC2 hosts run Apache ZooKeeper and are not part of this cluster upgrade guide.
EC2 Host IP and DNS Addresses
Each EC2 Host in this public facing cluster has a:
- Private DNS
- Private IP
- Public DNS (IPv4)
- IPv4 Public IP
EC2 Host | Private DNS | Private IP | Public DNS (IPv4) | IPv4 Public IP |
---|---|---|---|---|
Host_1 | <HOST_1_Private_DNS> | <HOST_1_Private_IP> | <HOST_1_Public_DNS_IPv4> | <HOST_1_IPv4_Public_IP> |
Host_2 | <HOST_2_Private_DNS> | <HOST_2_Private_IP> | <HOST_2_Public_DNS_IPv4> | <HOST_2_IPv4_Public_IP> |
Host_3 | <HOST_3_Private_DNS> | <HOST_3_Private_IP> | <HOST_3_Public_DNS_IPv4> | <HOST_3_IPv4_Public_IP> |
Host_4 | <HOST_4_Private_DNS> | <HOST_4_Private_IP> | <HOST_4_Public_DNS_IPv4> | <HOST_4_IPv4_Public_IP> |
Host_5 | <HOST_5_Private_DNS> | <HOST_5_Private_IP> | <HOST_5_Public_DNS_IPv4> | <HOST_5_IPv4_Public_IP> |
EC2 Hosts, Nodes, Users, Applications, and Services
Here is a list of the EC2 Hosts, Nodes, Applications, and Services:
EC2_Host | Node Name | Application / Service Name |
---|---|---|
Host_1 | n/a | ● Cluster Management Console ● MySQL 8 ● Apache Spark |
Host_2 | LoaderNode_1 | LoaderService_1 |
Host_3 | LoaderNode_2 | LoaderService_2 |
Host_4 | AnalyticsNode_1 | AnalyticsService_1 |
Host_5 | AnalyticsNode_2 | AnalyticsService_2 |
Secure Shell Access to the AWS EC2 Hosts
This guide assumes that you can readily access 5 of the 7 EC2 hosts using a bash shell terminal that supports both Secure Shell (SSH) and Secure Copy File (SCP) with a shared Key Pair (PEM) file.
In PuTTY or another Windows SSH client, you can create and save a SSH connection for each of EC2 Hosts.
Create Shell Variables for Mac OS or Linux
To expedite secure shell access from either Mac OS or Linux and the EC2 Hosts, create the following variables to store the IPv4_Public_IP values for each host. The Public IPs in the following listing are for illustration.
HOST_1=34.155.100.1HOST_2=34.155.100.2HOST_3=34.155.100.3HOST_4=34.155.100.4HOST_5=34.155.100.5
Copy the Incorta Package Zip
This guide assumes that you have already downloaded the incorta-package-<new-version>.zip
file and can securely copy this file using scp
. To copy the incorta-package-<new-version>.zip
from your local Documents
directory to the tmp
directory on each EC2 Host, here are example scp
shell commands:
cd ~/Documentsscp -i ~/.ssh/<ssh-auth-file>.pem incorta-package-<new-version>.zip \ incorta@${HOST_1}:/tmp/incorta-package-<new-version>.zipscp -i ~/.ssh/<ssh-auth-file>.pem incorta-package-<new-version>.zip \ incorta@${HOST_2}:/tmp/incorta-package-<new-version>.zipscp -i ~/.ssh/<ssh-auth-file>.pem incorta-package-<new-version>.zip \ incorta@${HOST_3}:/tmp/incorta-package-<new-version>.zipscp -i ~/.ssh/<ssh-auth-file>.pem incorta-package-<new-version>.zip \ incorta@${HOST_4}:/tmp/incorta-package-<new-version>.zipscp -i ~/.ssh/<ssh-auth-file>.pem incorta-package-<new-version>.zip \ incorta@${HOST_5}:/tmp/incorta-package-<new-version>.zip
Upgrade Requirements
Prior to upgrade, your Incorta Cluster must be operating in good health. You must have already securely copied the incorta-package-<new-version>.zip
file to all 5 EC2 hosts that run Incorta (CMC and Incorta Nodes). In order to backup your existing installation, you must also have adequate disk space on each EC2 host.
Prior to upgrade, it is also highly recommended that you first review the Release Notes for the new version.
Currently, the Notebook Add-on does not support an Incorta High Availability Cluster. The Notebook Add-on is an Incorta Labs feature. To learn more, review the Notebook Add-on documentation.
Ability to sign in to the Cluster Management Console (CMC)
Using either Google Chrome or Mozilla Firefox, open a web browser with two tabs. In the first browser tab, navigate to the CMC http://<HOST_1_IPv4_Public_IP>:6060/cmc/
and sign in with the Administrator username and password.
Ability to sign in to the Incorta Direct Data Platform via the Classic Load Balancer DNS
In the second browser tab, navigate to the Incorta Direct Data Platform through the Classic Load Balancer (<ClassicLB_DNS>:8080/incorta/#/login) URL and sign in to the exampleTenant as the Tenant administrator user with the administrator username and password.
Ability to verify high availability throughout the upgrade
This guide assumes that you have the exampleTenant
with installed Sample Data. During the cluster upgrade procedure, you will verify the high availability of the cluster by loading the SALES schema and viewing the Sales Executive Dashboard.
Pause all scheduled jobs for all cluster tenants
Prior to upgrading the Cluster Management Console, be sure to pause all scheduled jobs for all cluster tenants. Here are the steps to pause all scheduled jobs for each cluster in the CMC:
- Sign in to the CMC as the CMC Administrator user..
- In the Navigation bar, select Clusters.
- In the cluster list, select a cluster name.
- Select the Tenants tab.
- For each tenant in the cluster list...
- Select Configure.
- In the left pane, select Data Loading.
- In the right pane, toggle Pause Scheduled Jobs to enabled.
- Select Save.
- Sign out of the CMC.
Upgrade the Cluster Management Console (CMC)
This procedure uses the CMC as configured in AWS High Availability Cluster as an illustration. Incorta Administrators use the CMC to create and manage one or more Incorta Clusters, including:
- Incorta HA Nodes
- Incorta Services
- Server Configurations
- Default Tenant Configurations
- Tenants
Activities not to perform during CMC upgrade
While upgrading the CMC, do not perform the following CMC activities:
- Create or delete a Cluster
- Add or remove Incorta Nodes
- Add or remove an Incorta Service to an Incorta Node
- Start, stop, or restart Incorta Services
- Edit Server Configurations
- Edit Default Tenant Configurations
- Edit Tenant Configurations for a specific tenant
- Create or delete a Tenant
- Enable or disable a Tenant
- View or download log files
CMC shell scripts
To manage the CMC from the command line interface, use the following bash shell scripts. The default directory for these scripts is /home/incorta/IncortaAnalytics/cmc
. Here are the related scripts:
stop-cmc.sh
— stops the CMCstart-cmc.sh
— starts the CMC
Summary of CMC upgrade procedure
Here is a summary of the steps to upgrade the CMC:
- Secure shell in to the EC2 host, Host_1
- Use the CLI to stop the CMC
- Create a backup of the CMC directory
- Create a backup of the MySQL metadata database
- Unzip the installation package
- Run the Incorta Installer
- Use the Incorta Installer to upgrade and start the CMC
- Verify the CMC upgrade
- Upgrade the Incorta Metadata Database
- Exit the EC2 host
Secure shell in to the EC2 host, Host_1
Secure Shell into the EC2 host, HOST_1, as the incorta user and change to the CMC directory:
ssh -i ~/.ssh/<ssh-auth-file>.pem incorta@${HOST_1}cd ~/IncortaAnalytics/cmc
Use the CLI to stop the CMC
Stop the CMC with the stop-cmc.sh
shell script:
./stop-cmc.sh
Create a backup of the CMC directory
- Change to the IncortaAnalytics directory and create a backup of the
cmc
directory:
cd ..tar -cvf /tmp/cmc-<old-version>.tar.gz cmc
Create a backup of the MySQL metadata database
- To create a backup of the incorta_metadata database, use mysqldump command line utility. Here is example with the MySql user as root with the password incorta_root:
mysqldump -uroot -pincorta_root incorta_metadata > /tmp/incorta_metadata.bak
Unzip the installation package
- Change to the /tmp directory:
cd /tmp
- Create the incorta directory if it is not already present:
mkdir incorta-package-<new-version>
- Unzip the incorta package:
unzip incorta-package-<new-version>.zip -d incorta-package-<new-version>
Run the Incorta Installer
- Run the Incorta Incorta Installer:
cd incorta-package-<new-version>java -jar incorta-installer.jar -i console
Use the Incorta Installer to upgrade and start the CMC
- In the Installation Wizard, enter the following values:
Welcome : EnterCopyright : EnterCopyright/License : YInstallation Type : 2- UpgradeInstallation Set : 2- CustomIncorta HA components : 1- Central Management Console (CMC)Choose Installation Folder : Enter (default)Start CMC : 1- Start CMCInstallation Status : Enter
Verify the CMC upgrade
- Return to your web browser in the CMC tab of your web browser.
- In the Sign-in page of the CMC, verify the upgrade version.
Upgrade the Incorta Metadata Database
- Sign in to the CMC using the CMC Administrator username and password, for example:
USERNAME : adminPASSWORD : Incorta#1
- In the Navigation bar, select Clusters.
- For a Cluster in the CMC, select Upgrade Cluster Metadata.
A dialog indicates to restart the Incorta Services. You will manually stop and start the Incorta Services using scripts. In the dialog, select OK.
Exit the EC2 host
- Now that the CMC is running and the Incorta Metadata Database is upgraded, you can exit EC2 Host_1:
exit
Exiting the EC2 host, Host_1, completes the upgrade of the CMC.
Upgrade the Incorta Nodes
An Incorta HA Node shows a status of being online when the Incorta Node Agent and its associated Export Server are running. An Incorta HA Node runs at least one Incorta Node Service, such as a Analytics Service or Loader Service Add-on.
To manage Incorta HA Nodes and Incorta Services from the command line interface, use the following bash shell scripts in the default directory, /home/incorta/IncortaAnalytics/IncortaNode
:
listServices.sh -all
— lists services associated with the current NodestopService.sh
<service_name> — stops the named servicestartService.sh
<service_name> — starts the named servicenodeAgent/agent.sh status
— view the status of an Incorta HA NodestopNode.sh
— takes a Node offline, stopping the Node Agent and the export serverstartNode.sh
— brings a Node online, starting the Node Agent and the export server
Summary of Incorta HA Node Upgrades
There are two parts of this summary: the upgrade order of the Incorta HA Nodes in the cluster, and the steps to upgrade an Incorta HA Node. Here is the upgrade order of the Incorta HA Nodes:
- LoaderNode_1
- AnalyticsNode_1
- LoaderNode_2
- AnalyticsNode_2
Here is a summary of the steps to upgrade an Incorta HA Node:
- Secure shell in to the EC2 Host that is running an Incorta HA Node (Host_2, Host_3, Host_4, Host_5)
- Use the CLI to stop the Incorta Service
- Use the CLI to stop the Incorta Node
- Verify high availability
- Create a backup of the IncortaNode directory
- Unzip the installation package
- Run the Incorta Installer
- Use the Incorta Installer to upgrade the Incorta HA Node
- Verify that the Incorta Node is online
- Use the CLI to start the Incorta Service
- Verify that the Incorta Service is running
- Exit the EC2 host
Upgrade LoaderNode_1
In this cluster typology, EC2 Host_2 hosts LoaderNode_1. LoaderNode_1 manages the LoaderService_1.
Secure shell in to the HOST_2 as the incorta user and change to the IncortaNode directory:
ssh -i ~/.ssh/<ssh-auth-file>.pem incorta@${HOST_2}cd ~/IncortaAnalytics/IncortaNode
Use the CLI to stop the Incorta Service
- Stop LoaderService_1:
./stopService.sh LoaderService_1
- Confirm that Tomcat is stopped in the output.
Using CATALINA_BASE: /home/incorta/IncortaAnalytics/IncortaNode/services/f27a2ffe-193c-45c7-a831-0aad52cc0d66Using CATALINA_HOME: /home/incorta/IncortaAnalytics/IncortaNode/runtimeUsing CATALINA_TMPDIR: /home/incorta/IncortaAnalytics/IncortaNode/services/f27a2ffe-193c-45c7-a831-0aad52cc0d66/tempUsing JRE_HOME: /usr/lib/jvm/java-11-openjdk-11.0.5.10-0.amzn2.x86_64/Using CLASSPATH: /home/incorta/IncortaAnalytics/IncortaNode/runtime/bin/bootstrap.jar:/home/incorta/IncortaAnalytics/IncortaNode/runtime/bin/tomcat-juli.jarUsing CATALINA_PID: /home/incorta/IncortaAnalytics/IncortaNode/services/f27a2ffe-193c-45c7-a831-0aad52cc0d66/tomcat.pidNOTE: Picked up JDK_JAVA_OPTIONS: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMEDTomcat stopped.
- Verify LoaderService_1 has stopped:
./listServices.sh -all
- To view the Service Status, review the console output.
Listing Services:================Service Name = LoaderService_1Service Type = LOADERservice Location = /home/incorta/IncortaAnalytics/IncortaNode/services/f27a2ffe-193c-45c7-a831-0aad52cc0d66Service Status = Not Running
Use the CLI to stop the Incorta Node
- Stop the Node Agent and Export Service:
./stopNode.sh
- Review the console output.
Node type is customStopping Node in /home/incorta/IncortaAnalytics/IncortaNode :================================Stopping export-serverStopping export-server in /home/incorta/IncortaAnalytics/IncortaNode with PID:13710Stopping NodeAgentStopping Agent in /home/incorta/IncortaAnalytics/IncortaNode/nodeAgent
- Verify Node Agent is stopped:
./nodeAgent/agent.sh status
- Review the console output.
Agent is not running in /home/incorta/IncortaAnalytics/IncortaNode/nodeAgent
Verify high availability
The goal of this verification process is to confirm the operation of the cluster while LoaderNode_1 is offline.
Verify that you are able to load the SALES schema:
- In the browser tab for the Incorta Direct Data Platform, sign in if not logged in already via the Classic Load Balancer DNS.
- In the Navigation bar, select Schema.
- In Schema Manager, search for and then select, the SALES schema.
- In the Schema Designer, in the Action bar, select Load → Load Now → Full.
- In the Data Loading dialog, select Load.
- In Schema summary section, in Last Schema Load, select the timestamp.
- For the schema job, verify the successful load of the SALES Schema.
Verify that you are able to view the Sales Executive Dashboard:
- In the Navigation bar, select Content.
- Search for and then select Sales Executive Dashboard.
- Review the Dashboard.
Create a backup of the IncortaNode directory
- Create a backup file of the Incorta Node directory:
cd ~/IncortaAnalyticstar -czvf /tmp/IncortaNode-<old-version>.tar.gz IncortaNode
Unzip the installation package
- Change to the
tmp
directory:
cd /tmp
- Create the
incorta-package-<new-version>
directory:
mkdir incorta-package-<new-version>
- Unzip the
incorta-package-<new-version>.zip
file:
unzip incorta-package-<new-version>.zip -d incorta-package-<new-version>/cd incorta-package-<new-version>
Run the Incorta Installer
- From the unzip directory, you can now run the Java incorta-installer JAR file:
java -jar incorta-installer.jar -i console
Use the Incorta Installer to upgrade the Incorta HA Node
- In the Incorta Installer console, enter these exact values:
Welcome : EnterLicense Agreement/Copyright : EnterLicense Agreement/Copyright : YInstallation Type : 2- UpgradeInstallation Set : 2- CustomIncorta HA components : 2- Incorta HA NodeChoose Installation Folder : Enter- DefaultInstallation Status : Enter
Verify that the Incorta Node is online
- The Incorta Installer automatically starts the Incorta Node Agent. Verify that the Node Agent is running:
cd ~/IncortaAnalytics/IncortaNode./nodeAgent/agent.sh status
- Review the console output.
Agent is already running in /home/incorta/IncortaAnalytics/IncortaNode/nodeAgent. PID: 21122
- If the Incorta Node Agent is not running, execute the
startNode.sh
shell script:
./startNode.sh
Use the CLI to start the Incorta Service
To start the LoaderService_1, execute the startService.sh
shell script:
./startService.sh LoaderService_1
Verify that the Incorta Service is running
- List the services on the Incorta HA Node:
./listServices.sh -all
- To view the Service Status, review the console output.
Listing full details about services:===================================Service Name = LoaderService_1Service Type = LOADERservice Location = /home/incorta/IncortaAnalytics/IncortaNode/services/f27a2ffe-193c-45c7-a831-0aad52cc0d66Service Cluster Name = exampleClusterService Memory Size = 4GService Status = Running
Exit the EC2 host
You can now exit the terminal for EC2 Host_2:
exit
Upgrade AnalyticsNode_1
In this cluster typology, EC2 Host_4 hosts AnalyticsNode_1. AnalyticsNode_1 manages the AnalyticsService_1.
Secure shell in to the HOST_4 as the incorta user and change to the IncortaNode directory:
ssh -i ~/.ssh/<ssh-auth-file>.pem incorta@${HOST_4}cd ~/IncortaAnalytics/IncortaNode
Use the CLI to stop the Incorta Service
- Stop AnalyticsService_1:
./stopService.sh AnalyticsService_1
- Confirm that Tomcat is stopped in the output.
Using CATALINA_BASE: /home/incorta/IncortaAnalytics/IncortaNode/services/a27a2ffe-193c-45c7-a831-0aad52cc0d44Using CATALINA_HOME: /home/incorta/IncortaAnalytics/IncortaNode/runtimeUsing CATALINA_TMPDIR: /home/incorta/IncortaAnalytics/IncortaNode/services/a27a2ffe-193c-45c7-a831-0aad52cc0d44/tempUsing JRE_HOME: /usr/lib/jvm/java-11-openjdk-11.0.5.10-0.amzn2.x86_64/Using CLASSPATH: /home/incorta/IncortaAnalytics/IncortaNode/runtime/bin/bootstrap.jar:/home/incorta/IncortaAnalytics/IncortaNode/runtime/bin/tomcat-juli.jarUsing CATALINA_PID: /home/incorta/IncortaAnalytics/IncortaNode/services/a27a2ffe-193c-45c7-a831-0aad52cc0d44/tomcat.pidNOTE: Picked up JDK_JAVA_OPTIONS: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMEDTomcat stopped.
- Verify AnalyticsService_1 has stopped:
./listServices.sh -all
- To view the Service Status, review the console output.
Listing Services:================Service Name = AnalyticsService_1Service Type = Analyticsservice Location = /home/incorta/IncortaAnalytics/IncortaNode/services/a27a2ffe-193c-45c7-a831-0aad52cc0d44Service Status = Not Running
- Verify Node Agent is stopped:
./nodeAgent/agent.sh status
- Review the console output.
Agent is not running in /home/incorta/IncortaAnalytics/IncortaNode/nodeAgent
Verify high availability
The goal of this verification process is to confirm the operation of the cluster while AnalyticsNode_1 is offline.
Verify that you are able to load the SALES schema:
- In the browser tab for the Incorta Direct Data Platform, sign in if not logged in already via the Classic Load Balancer DNS.
- In the Navigation bar, select Schema.
- In Schema Manager, search for and then select, the SALES schema.
- In the Schema Designer, in the Action bar, select Load → Load Now → Full.
- In the Data Loading dialog, select Load.
- In Schema summary section, in Last Schema Load, select the timestamp.
- For the schema job, verify the successful load of the SALES Schema.
Verify that you are able to view the Sales Executive Dashboard:
- In the Navigation bar, select Content.
- Search for and then select Sales Executive Dashboard.
- Review the Dashboard.
Create a backup of the IncortaNode directory
- Create a backup file of the Incorta Node directory:
cd ~/IncortaAnalyticstar -czvf /tmp/IncortaNode-<old-version>.tar.gz IncortaNode
Unzip the installation package
- Change to the
tmp
directory:
cd /tmp
- Create the
incorta-package-<new-version>
directory:
mkdir incorta-package-<new-version>
- Unzip the
incorta-package-<new-version>.zip
file:
unzip incorta-package-<new-version>.zip -d incorta-package-<new-version>/cd incorta-package-<new-version>
Run the Incorta Installer
- From the unzip directory, you can now run the Java incorta-installer JAR file:
java -jar incorta-installer.jar -i console
Use the Incorta Installer to upgrade the Incorta HA Node
- In the Incorta Installer console, enter these exact values:
Welcome : EnterLicense Agreement/Copyright : EnterLicense Agreement/Copyright : YInstallation Type : 2- UpgradeInstallation Set : 2- CustomIncorta HA components : 2- Incorta HA NodeChoose Installation Folder : Enter- DefaultInstallation Status : Enter
Verify that the Incorta Node is online
- The Wizard automatically starts the Incorta Node Agent. Verify that the Node Agent is running:
cd ~/IncortaAnalytics/IncortaNode./nodeAgent/agent.sh status
- Review the console output.
Agent is already running in /home/incorta/IncortaAnalytics/IncortaNode/nodeAgent. PID: 21144
- If the Incorta Node Agent is not running, execute the
startNode.sh
shell script:
./startNode.sh
Use the CLI to start the Incorta Service
To start the AnalyticsService_1, execute the startService.sh
shell script:
./startService.sh AnalyticsService_1
Verify that the Incorta Service is running
- List the services on the Incorta HA Node:
./listServices.sh -all
- To view the Service Status, review the console output.
Listing full details about services:===================================Service Name = AnalyticsService_1Service Type = ANALYTICSservice Location = /home/incorta/IncortaAnalytics/IncortaNode/services/a27a2ffe-193c-45c7-a831-0aad52cc0d44Service Cluster Name = exampleClusterService Memory Size = 4GService Status = Running
Exit the EC2 host
You can now exit the terminal for EC2 Host_4:
exit
Upgrade LoaderNode_2
In this cluster typology, EC2 Host_3 hosts LoaderNode_2. LoaderNode_2 manages the LoaderService_2.
Secure shell in to the HOST_3 as the incorta user and change to the IncortaNode directory:
ssh -i ~/.ssh/<ssh-auth-file>.pem incorta@${HOST_3}cd ~/IncortaAnalytics/IncortaNode
Use the CLI to stop the Incorta Service
- Stop LoaderService_2 with the
stopService.sh
shell script:
./stopService.sh LoaderService_2
- Confirm that Tomcat is stopped in the output.
Using CATALINA_BASE: /home/incorta/IncortaAnalytics/IncortaNode/services/d27a2ffe-193c-45c7-a831-0aad52cc0d33Using CATALINA_HOME: /home/incorta/IncortaAnalytics/IncortaNode/runtimeUsing CATALINA_TMPDIR: /home/incorta/IncortaAnalytics/IncortaNode/services/d27a2ffe-193c-45c7-a831-0aad52cc0d33/tempUsing JRE_HOME: /usr/lib/jvm/java-11-openjdk-11.0.5.10-0.amzn2.x86_64/Using CLASSPATH: /home/incorta/IncortaAnalytics/IncortaNode/runtime/bin/bootstrap.jar:/home/incorta/IncortaAnalytics/IncortaNode/runtime/bin/tomcat-juli.jarUsing CATALINA_PID: /home/incorta/IncortaAnalytics/IncortaNode/services/d27a2ffe-193c-45c7-a831-0aad52cc0d33/tomcat.pidNOTE: Picked up JDK_JAVA_OPTIONS: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMEDTomcat stopped.
- Verify LoaderService_2 has stopped:
./listServices.sh -all
- To view the Service Status, review the console output.
Listing Services:================Service Name = LoaderService_2Service Type = LOADERservice Location = /home/incorta/IncortaAnalytics/IncortaNode/services/d27a2ffe-193c-45c7-a831-0aad52cc0d33Service Status = Not Running
Use the CLI to stop the Incorta Node
- Stop the Node Agent and Export Service with the
stopNode.sh
shell script:
./stopNode.sh
- Review the console output.
Node type is customStopping Node in /home/incorta/IncortaAnalytics/IncortaNode :================================Stopping export-serverStopping export-server in /home/incorta/IncortaAnalytics/IncortaNode with PID:13733Stopping NodeAgentStopping Agent in /home/incorta/IncortaAnalytics/IncortaNode/nodeAgent
- Verify Node Agent is stopped:
./nodeAgent/agent.sh status
- Review the console output.
Agent is not running in /home/incorta/IncortaAnalytics/IncortaNode/nodeAgent
Verify high availability
The goal of this verification process is to confirm the operation of the cluster while LoaderNode_2 is offline.
Verify that you are able to load the SALES schema:
- In the browser tab for the Incorta Direct Data Platform, sign in if not logged in already via the Classic Load Balancer DNS.
- In the Navigation bar, select Schema.
- In Schema Manager, search for and then select, the SALES schema.
- In the Schema Designer, in the Action bar, select Load → Load Now → Full.
- In the Data Loading dialog, select Load.
- In Schema summary section, in Last Schema Load, select the timestamp.
- For the schema job, verify the successful load of the SALES Schema.
Verify that you are able to view the Sales Executive Dashboard:
- In the Navigation bar, select Content.
- Search for and then select Sales Executive Dashboard.
- Review the Dashboard.
Create a backup of the IncortaNode directory
- Create a backup file of the Incorta Node directory:
cd ~/IncortaAnalyticstar -czvf /tmp/IncortaNode-<old-version>.tar.gz IncortaNode
Unzip the installation package
- Change to the
tmp
directory:
cd /tmp
- Create the
incorta-package-<new-version>
directory:
mkdir incorta-package-<new-version>
- Unzip the
incorta-package-<new-version>.zip
file:
unzip incorta-package-<new-version>.zip -d incorta-package-<new-version>/cd incorta-package-<new-version>
Run the Incorta Installer
- From the unzip directory, you can now run the Java incorta-installer JAR file:
java -jar incorta-installer.jar -i console
Use the Incorta Installer to upgrade the Incorta HA Node
- In the Incorta Installer console, enter these exact values:
Welcome : EnterLicense Agreement/Copyright : EnterLicense Agreement/Copyright : YInstallation Type : 2- UpgradeInstallation Set : 2- CustomIncorta HA components : 2- Incorta HA NodeChoose Installation Folder : Enter- DefaultInstallation Status : Enter
Verify that the Incorta Node is online
- The Incorta Installer automatically starts the Incorta Node Agent. Verify that the Node Agent is running:
cd ~/IncortaAnalytics/IncortaNode./nodeAgent/agent.sh status
- Review the console output.
Agent is already running in /home/incorta/IncortaAnalytics/IncortaNode/nodeAgent. PID: 21133
- If the Incorta Node Agent is not running, execute the
startNode.sh
shell script:
./startNode.sh
Use the CLI to start the Incorta Service
To start the LoaderService_2, execute the startService.sh
shell script:
./startService.sh LoaderService_2
Verify that the Incorta Service is running
- List the services on the Incorta HA Node:
./listServices.sh -all
- To view the Service Status, review the console output.
Listing full details about services:===================================Service Name = LoaderService_2Service Type = LOADERservice Location = /home/incorta/IncortaAnalytics/IncortaNode/services/d27a2ffe-193c-45c7-a831-0aad52cc0d33Service Cluster Name = exampleClusterService Memory Size = 4GService Status = Running
Exit the EC2 host
You can now exit the terminal for EC2 Host_4:
exit
Upgrade AnalyticsNode_2
In this cluster typology, EC2 Host_5 hosts AnalyticsNode_2. AnalyticsNode_2 manages the AnalyticsService_2.
Secure shell in to the EC2 Host_5
Secure shell in to EC2 Host_5 as the incorta user and change to the IncortaNode directory:
ssh -i ~/.ssh/<ssh-auth-file>.pem incorta@${HOST_5}cd ~/IncortaAnalytics/IncortaNode
Use the CLI to stop the Incorta Service
- Stop AnalyticsService_2:
./stopService.sh AnalyticsService_2
- Confirm that Tomcat is stopped in the output.
Using CATALINA_BASE: /home/incorta/IncortaAnalytics/IncortaNode/services/e27a2ffe-193c-45c7-a831-0aad52cc0d55Using CATALINA_HOME: /home/incorta/IncortaAnalytics/IncortaNode/runtimeUsing CATALINA_TMPDIR: /home/incorta/IncortaAnalytics/IncortaNode/services/e27a2ffe-193c-45c7-a831-0aad52cc0d55/tempUsing JRE_HOME: /usr/lib/jvm/java-11-openjdk-11.0.5.10-0.amzn2.x86_64/Using CLASSPATH: /home/incorta/IncortaAnalytics/IncortaNode/runtime/bin/bootstrap.jar:/home/incorta/IncortaAnalytics/IncortaNode/runtime/bin/tomcat-juli.jarUsing CATALINA_PID: /home/incorta/IncortaAnalytics/IncortaNode/services/e27a2ffe-193c-45c7-a831-0aad52cc0d55/tomcat.pidNOTE: Picked up JDK_JAVA_OPTIONS: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMEDTomcat stopped.
- Verify AnalyticsService_2 has stopped:
./listServices.sh -all
- To view the Service Status, review the console output.
Listing Services:================Service Name = AnalyticsService_2Service Type = Analyticsservice Location = /home/incorta/IncortaAnalytics/IncortaNode/services/e27a2ffe-193c-45c7-a831-0aad52cc0d55Service Status = Not Running
- Verify Node Agent is stopped:
./nodeAgent/agent.sh status
- Review the console output.
Agent is not running in /home/incorta/IncortaAnalytics/IncortaNode/nodeAgent
Verify high availability
The goal of this verification process is to confirm the operation of the cluster while AnalyticsNode_2 is offline.
Verify that you are able to load the SALES schema:
- In the browser tab for the Incorta Direct Data Platform, sign in if not logged in already via the Classic Load Balancer DNS.
- In the Navigation bar, select Schema.
- In Schema Manager, search for and then select, the SALES schema.
- In the Schema Designer, in the Action bar, select Load → Load Now → Full.
- In the Data Loading dialog, select Load.
- In Schema summary section, in Last Schema Load, select the timestamp.
- For the schema job, verify the successful load of the SALES Schema.
Verify that you are able to view the Sales Executive Dashboard:
- In the Navigation bar, select Content.
- Search for and then select Sales Executive Dashboard.
- Review the Dashboard.
Create a backup of the IncortaNode directory
- Create a backup file of the Incorta Node directory:
cd ~/IncortaAnalyticstar -czvf /tmp/IncortaNode-<old-version>.tar.gz IncortaNode
Unzip the installation package
- Change to the
tmp
directory:
cd /tmp
- Create the
incorta-package-<new-version>
directory:
mkdir incorta-package-<new-version>
- Unzip the
incorta-package-<new-version>.zip
file:
unzip incorta-package-<new-version>.zip -d incorta-package-<new-version>/cd incorta-package-<new-version>
Run the Incorta Installer
- From the unzip directory, you can now run the Java incorta-installer JAR file:
java -jar incorta-installer.jar -i console
Use the Incorta Installer to upgrade the Incorta HA Node
- In the Incorta Installer console, enter these exact values:
Welcome : EnterLicense Agreement/Copyright : EnterLicense Agreement/Copyright : YInstallation Type : 2- UpgradeInstallation Set : 2- CustomIncorta HA components : 2- Incorta HA NodeChoose Installation Folder : Enter- DefaultInstallation Status : Enter
Verify that the Incorta Node is online
- The Wizard automatically starts the Incorta Node Agent. Verify that the Node Agent is running:
cd ~/IncortaAnalytics/IncortaNode./nodeAgent/agent.sh status
- Review the console output.
Agent is already running in /home/incorta/IncortaAnalytics/IncortaNode/nodeAgent. PID: 21155
- If the Incorta Node Agent is not running, execute the
startNode.sh
shell script:
./startNode.sh
Use the CLI to start the Incorta Service
To start the AnalyticsService_2, execute the startService.sh
shell script:
./startService.sh AnalyticsService_2
Verify that the Incorta Service is running
- List the services on the Incorta HA Node:
./listServices.sh -all
- To view the Service Status, review the console output.
Listing full details about services:===================================Service Name = AnalyticsService_2Service Type = ANALYTICSservice Location = /home/incorta/IncortaAnalytics/IncortaNode/services/e27a2ffe-193c-45c7-a831-0aad52cc0d55Service Cluster Name = exampleClusterService Memory Size = 4GService Status = Running
Exit the EC2 host
You can now exit the terminal for EC2 Host_5:
exit
Upgrade Apache Spark
In this cluster typology, in addition to the CMC and MySQL Server, EC2 Host_1 also runs Apache Spark as a standalone instance.
Incorta requires a specific build version of Apache Spark. The easiest way to upgrade a standalone version of Apache Spark is to follow the procedure for installing an Incorta HA Node as the required Spark files reside in the directory, IncortaNode/spark
.
Because the CMC was already upgraded on Host_1, the Incorta Installer is already available to run in console mode. Here is a summary of the steps to upgrade Apache Spark on Host_1:
- Secure shell in to the EC2 Host that is running the CMC (Host_1)
- Stop Apache Spark
- Create a backup of the IncortaNode directory
- Run the Incorta Installer
- Use the Incorta Installer to upgrade the Incorta HA Node
- Stop the Incorta Node Agent
- Update the Apache Spark configuration files
- Use the CLI to start Apache Spark
- Verify that the Apache Spark Master and Worker nodes are running
- Exit the EC2 host
Secure shell in to the EC2 Host_1 as the incorta user and change to the IncortaNode directory:
ssh -i ~/.ssh/<ssh-auth-file>.pem incorta@${HOST_1}cd ~/IncortaAnalytics/IncortaNode
Use the CLI to stop Apache Spark
- Stop Apache Spark with the
stopSpark.sh
shell script.
./stopSpark.sh
Create a backup of the IncortaNode directory
- Create a backup file of the Incorta Node directory:
cd ~/IncortaAnalyticstar -czvf /tmp/IncortaNode-<old-version>.tar.gz IncortaNode
Run the Incorta Installer
- From the unzip directory, you can now run the Java incorta-installer JAR file:
cd /tmp/incorta-package-<new-version>/java -jar incorta-installer.jar -i console
Use the Incorta Installer to upgrade the Incorta HA Node
- In the Incorta Installer console, enter these exact values:
Welcome : EnterLicense Agreement/Copyright : EnterLicense Agreement/Copyright : YInstallation Type : 2- UpgradeInstallation Set : 2- CustomIncorta HA components : 2- Incorta HA NodeChoose Installation Folder : Enter- DefaultInstallation Status : Enter
Stop the Incorta Node Agent
The Incorta Installer automatically starts the Incorta Node Agent. The Incorta Node Agent does not need to run on this EC2 Host and needs to be stopped:
- To stop the Incorta Node Agent, execute the
stopNode.sh
shell script:
cd ~/IncortaAnalytics/IncortaNode/./stopNode.sh
- Verify that the Node Agent is stopped:
./nodeAgent/agent.sh status
- Review the console output.
Agent is not running in /home/incorta/IncortaAnalytics/IncortaNode/nodeAgent
- Update the Apache Spark configuration files
Both the spark-env.sh
and spark-default.conf
files need to update the EC2 Host IP addresses.
- To update
spark-env.sh
, execute the following bash shell commands:
HOST_1_Public_DNS_IPv4=<HOST_1_Public_DNS_IPv4>HOST_1_Private_DNS=<HOST_1_Private_DNS>echo $HOST_1_Public_DNS_IPv4echo $HOST_1_Private_DNScd ~/IncortaAnalytics/IncortaNode/spark/sed -i "s/SPARK_PUBLIC_DNS=.*/SPARK_PUBLIC_DNS=${HOST_1_Public_DNS_IPv4}/g" conf/spark-env.shsed -i "s/SPARK_MASTER_IP=.*/SPARK_MASTER_IP=${HOST_1_Private_DNS}/g" conf/spark-env.sh
- Verify the updated values for SPARK_PUBLIC_DNS and SPARK_MASTER_IP
cat conf/spark-env.sh
- To update
spark-defaults.conf
, edit the file directly usingvi
orvim
:
vim conf/spark-env.sh
- Update
spark.master
to use the value of <*HOST_1_Private_IP**>. Also update the following settings with the appropriate values:- spark.cores.max
- spark.executor.cores
- spark.sql.shuffle.partitions
- spark.driver.memory
spark.master spark://<HOST_1_Private_IP>:7077spark.eventLog.enabled truespark.eventLog.dir /home/incorta/IncortaAnalytics/IncortaNode/spark/eventlogsspark.local.dir /home/incorta/IncortaAnalytics/IncortaNode/spark/tmpspark.executor.extraJavaOptions-Djava.io.tmpdir=/home/incorta/IncortaAnalytics/IncortaNode/spark/tmpspark.driver.extraJavaOptions-Djava.io.tmpdir=/home/incorta/IncortaAnalytics/IncortaNode/spark/tmpspark.cores.max 2spark.executor.cores 2spark.sql.shuffle.partitions 2spark.driver.memory 6gspark.port.maxRetries 100
Use the CLI to start Apache Spark
To start the both the Apache Spark Master and Worker nodes, execute the startSpark.sh
shell script:
cd ~/IncortaAnalytics/IncortaNode./startSpark.sh
Verify that Apache Spark is running
- In your web browser, enter the URL for Apache Spark such as
http://<HOST_1_IPv4_Public_IP>:9091
This completes the Apache Spark upgrade.
Resume all scheduled jobs for all cluster tenants
Now that you have upgrade all Incorta Nodes, the Incorta CMC, and Apache Spark, you can resume all scheduled jobs that were paused earlier. To resume the scheduled jobs, follow these steps:
- Sign in to the CMC as the CMC Administrator user.
- In the Navigation bar, select Clusters.
- In the cluster list, select a cluster name.
- Select the Tenants tab.
- For each tenant in the cluster list...
- Select Configure.
- In the left pane, select Data Loading.
- In the right pane, toggle Pause Scheduled Jobs to disabled.
- Select Save.
- Sign out of the CMC.
Conclusion
Throughout the upgrade process of your AWS High Availability Incorta Cluster, it is possible to ensure that Incorta as a Unified Data Analytics Platform is continuously available by following these steps:
- Pause all scheduled jobs for all cluster tenants
- Upgrade the CMC
- Upgrade LoaderService_1
- Upgrade the AnalyticsService_1
- Upgrade LoaderService_2
- Upgrade the AnalyticsService_2
- Upgrade Apache Spark
- Resume all scheduled jobs for all cluster tenants