Installation instructions for hot fix B3I005 on Linux for x64

Installation Instructions for Hot Fix B3I005

Linux for x64

Hot fix B3I005 addresses the issue(s) in SAS Data Loader Mid-Tier 3.1 on Linux for x64 as documented in the Issue(s) Addressed section of the hot fix download page:

https://tshf.sas.com/techsup/download/hotfix/HF2/B3I.html#B3I005

The hot fix downloaded, B3I005pt.zip, contains the updated files required to address the documented issues.
Do NOT extract the contents of B3I005pt.zip. The hot fix installation process will extract the contents as needed.

IMPORTANT NOTES

You must have SAS Data Loader Mid-Tier 3.1 installed on your system before applying this hot fix. Refer to SN-35968 for instructions on how to determine which product releases you have installed.
Files delivered in this hot fix will be backed up during the installation process. However, it is good general practice to back up your system before applying updates to software.
You must have Administrator Privileges on your CLIENT or SERVER machine.
All currently active SAS sessions, daemons, spawners and servers must be terminated before applying this hot fix.
This hot fix should be installed using the same userid who performed the initial software installation.
CONFIGURATION: No automatic configuration scripting is included for this hot fix. If you have previously configured software installed, the SAS Deployment Manager may present a screen where you will see "Apply SAS Hot Fixes" and "Configure SAS Hot Fixes" options. On this screen, you must ensure that the "Configure SAS Hot Fix" option is *not* selected. If this option is automatically selected, please de-select it prior to proceeding with the SAS Deployment Manager Screens. Failure to do so could have unintended consequences when applying this hot fix.

INSTALLATION

The B3I005 hot fix for SAS Data Loader Mid-Tier 3.1 will be installed using the SAS Deployment Manager. By default, the SAS Deployment Manager will search in the <SASHOME>/InstallMisc/HotFixes/New directory for hot fixes to be applied, but will also prompt for a location if you have downloaded hot fixes to a different directory.

After downloading B3I005pt.zip, follow the instructions for applying hot fixes in the SAS Deployment Wizard and SAS Deployment Manager 9.4: User's Guide.

Please review the CONFIGURATION Important Note above concerning proper selection of the "Configure SAS Hot Fix" option in the SAS Deployment Manager.

The content of this hot fix is listed in the hot fix manifest.

POST-INSTALLATION INSTRUCTIONS

This hot fix requires post-installation steps that will update the web application deployed to your application server.
Follow the steps documented in the rebuild_redeploy_webapp doc to update and deploy the SAS Data Loader Mid-Tier 3.1 war.
Create a "B3I005" directory on your machine and unzip the contents of B3I005pt.zip into that directory.
The sas.svcs.dm.sparktools.jar file is found in the Miscellaneous subdirectory.
Follow the steps documented here to complete the configuration.
Follow these instructions to configure and enable Oozie Dispatch with the Hadoop EP.
1. HDP2.X
  - Client Side Instructions
    1. Manually pull the oozie-site.xml from the node running the oozie service from:
      /usr/hdp/current/oozie-server/conf/oozie-site.xml
    2. Include the oozie-site.xml to your client side SAS_HADOOP_CONFIG_PATH folder
      Note: The SAS_HADOOP_CONFIG_PATH can only be a single directory
  - Cluster Side Changes to publish spark jars to oozie sharelib
    This step is required to enable Oozie Dispatch to use the Spark EP.
    1. Create the spark2 oozie sharelib directory on the node running the oozie service:
      mkdir /usr/hdp//oozie/share/lib/spark2
    2. Copy contents in from the spark2 jars folder to the oozie sharelib spark2 folder:
      cp /usr/hdp/<version>/spark2/jars/* /usr/hdp/<version>/oozie/share/lib/spark2
    3. Copy the oozie-sharelib-spark-<version>.jar located under the spark ShareLib folder to the spark2 ShareLib folder:
      cp /usr/hdp/<version>/oozie/share/lib/spark/oozie-sharelib-spark-<version>.jar /usr/hdp/<version>/oozie/share/lib/spark2
    4. Logged in as oozie user, run the oozie-setup.sh sharelib create command on the node running the oozie service:
      /usr/hdp/current/oozie-client/bin/oozie-setup.sh sharelib create -fs hdfs://hdp26k1 -locallib /usr/hdp/current/oozie-server/share
    5. Logged in as oozie user, run the Oozie admin sharelibupdate command:
      oozie admin --sharelibupdate
    6. Verify the configuration by running the Oozie admin shareliblist command:
      oozie admin --shareliblist
      [Available ShareLib]
      hive
      spark2
      distcp
      mapreduce-streaming
      spark
      oozie
      hcatalog
      hive2
      sqoop
      pig
2. CDH5.X
  - Client Side Instructions
    1. Manually pull the oozie-site.xml from the node running the oozie service from:
      /var/run/cloudera-scm-agent/process/*oozie*/oozie-site.xml
    2. Include the oozie-site.xml to your client side SAS_HADOOP_CONFIG_PATH folder
      Note: The SAS_HADOOP_CONFIG_PATH can only be a single directory
  - Client Side Workarounds for Spark EP
    1. Edit the spark-defaults.conf file under SAS_HADOOP_CONFIG_PATH folder and add the following property:
      spark.executor.extraClassPath /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/*
    2. Edit the oozie-site.xml file under SAS_HADOOP_CONFIG_PATH folder and add the following property:
      Note: Do NOT apply this workaround if you are running the Hadoop EP with MapRed. This is only for the Hadoop EP with Spark.
      <property>
      <name>oozie.launcher.mapreduce.user.classpath.first</name>
      <value>true</value>
      </property>
  - Cluster Side Changes to publish spark jars to oozie sharelib
    This step is required to enable Oozie Dispatch to use the Spark EP.
    1. Create the spark2 oozie sharelib directory on the node running the oozie service:
      mkdir /opt/cloudera/parcels/CDH/lib/oozie/oozie-sharelib-yarn/lib/spark2
    2. Copy contents in from the spark2 jars folder to the oozie sharelib spark2 folder:
      cp /opt/cloudera/parcels/SPARK2/lib/spark2/jars/* /opt/cloudera/parcels/CDH/lib/oozie/oozie-sharelib-yarn/lib/spark2/.
    3. Log into Cloudera Manager and redeploy the Oozie ShareLib
      Go to the Oozie service.
      Select Actions > Install Oozie ShareLib.
3. CDH6.X
  - Client Side Instructions
    1. Manually pull the oozie-site.xml from the node running the oozie service from:
      /run/cloudera-scm-agent/process/*oozie*/oozie-site.xml
    2. Include the oozie-site.xml to your client side SAS_HADOOP_CONFIG_PATH folder
      Note: The SAS_HADOOP_CONFIG_PATH can only be a single directory
  - Cluster Side Changes to publish spark jars to oozie sharelib
    This step is required to enable Oozie Dispatch to use the Spark EP.
    1. Create the spark2 oozie sharelib directory on the node running the oozie service:
      mkdir /opt/cloudera/parcels/CDH/lib/oozie/oozie-sharelib-yarn/lib/spark2
    2. Copy contents in from the spark folder to the new spark2 folder:
      cp /opt/cloudera/parcels/CDH/lib/oozie/oozie-sharelib-yarn/lib/spark/* /opt/cloudera/parcels/CDH/lib/oozie/oozie-sharelib-yarn/lib/spark2/.
    3. Log into Cloudera Manager and redeploy the Oozie ShareLib
      Go to the Oozie service.
      Select Actions > Install Oozie ShareLib.
4. Verify Spark2 Jars in Oozie ShareLib
  How to validate the spark2 jars were published correctly to the Oozie ShareLib to enable Oozie Dispatch to run with the Spark EP:
  - When you republish oozie, a new directory with the current timestamp will be generated on HDFS:
    $ hadoop fs -ls /user/oozie/share/lib/
    Found 2 items
    drwxrwxr-x - oozie oozie 0 2019-02-08 17:03 /user/oozie/share/lib/lib_20190208170256
    drwxrwxr-x - oozie oozie 0 2019-09-27 13:52 /user/oozie/share/lib/lib_20190927135206
  - The new oozie sharelib directory will now have a new spark2 folder with the contents you added:
    $ hadoop fs -ls /user/oozie/share/lib/lib_20190927135206
    Found 11 items
    drwxrwxr-x - oozie oozie 0 2019-09-27 13:52 /user/oozie/share/lib/lib_20190927135206/distcp
    drwxrwxr-x - oozie oozie 0 2019-09-27 13:52 /user/oozie/share/lib/lib_20190927135206/hcatalog
    drwxrwxr-x - oozie oozie 0 2019-09-27 13:52 /user/oozie/share/lib/lib_20190927135206/hive
    drwxrwxr-x - oozie oozie 0 2019-09-27 13:52 /user/oozie/share/lib/lib_20190927135206/hive2
    drwxrwxr-x - oozie oozie 0 2019-09-27 13:52 /user/oozie/share/lib/lib_20190927135206/mapreduce-streaming
    drwxrwxr-x - oozie oozie 0 2019-09-27 13:52 /user/oozie/share/lib/lib_20190927135206/oozie
    drwxrwxr-x - oozie oozie 0 2019-09-27 13:52 /user/oozie/share/lib/lib_20190927135206/pig
    -rwxrwxr-x 3 oozie oozie 1365 2019-09-27 13:52 /user/oozie/share/lib/lib_20190927135206/sharelib.properties
    drwxrwxr-x - oozie oozie 0 2019-09-27 13:52 /user/oozie/share/lib/lib_20190927135206/spark
    drwxrwxr-x - oozie oozie 0 2019-09-27 13:52 /user/oozie/share/lib/lib_20190927135206/spark2
    drwxrwxr-x - oozie oozie 0 2019-09-27 13:52 /user/oozie/share/lib/lib_20190927135206/sqoop