Ibm Infosphere Datastage On The Aws Cloud

IBM InfoSphere DataStage on the
AWS Cloud
Quick Start Reference Deployment
August 2019
Shrumit Mehta and Shashidhar Yellareddy, IBM

Vinod Shukla and Jim McConnell, AWS Quick Start team
Visit our GitHub repository for source files and to post feedback,
report bugs, or submit feature ideas for this Quick Start.
Contents
Overview .................................................................................................................................... 3
IBM InfoSphere DataStage on AWS ...................................................................................... 3
Cost and licenses .................................................................................................................... 4
Architecture ............................................................................................................................... 5
Planning the deployment ..........................................................................................................6
Specialized knowledge ...........................................................................................................6
AWS account .......................................................................................................................... 7
Technical requirements ......................................................................................................... 7
Deployment options ...............................................................................................................9
Deployment steps ......................................................................................................................9
Step 1. Sign in to your AWS account ......................................................................................9
Step 2. Retrieve IBM InfoSphere DataStage license information .........................................9
Step 3. Launch the Quick Start ............................................................................................ 10
Option 1: Parameters for deploying InfoSphere DataStage into a new VPC ....................11
Option 2: Parameters for deploying InfoSphere DataStage into an existing VPC .......... 15
Page 1 of 33
Amazon Web Services – IBM InfoSphere DataStage on the AWS Cloud August 2019
Step 4. Test the deployment ................................................................................................ 19

Accessing the IIS Launchpad from a browser .................................................................. 19
Accessing the DataStage Windows Client ....................................................................... 20
Logging in using Remote Desktop Protocol (RDP) ............................................................ 20
Using InfoSphere DataStage and QualityStage Designer ................................................... 21
Using the IIS Launchpad ..................................................................................................... 21
Accessing the cluster nodes through the Linux bastion host........................................... 21
Transferring files from your local computer to the cluster ..............................................22
Bash ...............................................................................................................................22
WinSCP ..........................................................................................................................22
Manual cleanup ....................................................................................................................23
Best practices for using InfoSphere DataStage on AWS ........................................................23
Enabling backups .................................................................................................................23
Security ....................................................................................................................................24
Scaling compute ...................................................................................................................... 25
Horizontal scaling ................................................................................................................ 25
Changing the number of is-engine-compute pods .......................................................... 25
Vertical scaling .....................................................................................................................26
Scaling back down ................................................................................................................26
Provisioning more DataStage Client instances ....................................................................... 27
Performance considerations for Amazon EFS ........................................................................ 27
Troubleshooting ..................................................................................................................... 28
IBM support ............................................................................................................................ 31
Create a case ......................................................................................................................... 31
Support forum ...................................................................................................................... 31
Send us feedback ..................................................................................................................... 31
Additional resources ............................................................................................................... 31
Document revisions.................................................................................................................32
Page 2 of 33
This Quick Start was created by IBM in collaboration with Amazon Web Services (AWS).
Quick Starts are automated reference deployments that use AWS CloudFormation
templates to deploy key technologies on AWS, following AWS best practices.
Overview
This Quick Start reference deployment guide provides step-by-step instructions for
deploying IBM InfoSphere DataStage 11.7.0.2 (fix pack 2) on the AWS Cloud.
This Quick Start is for users who want to deploy InfoSphere DataStage on the AWS Cloud to
integrate data from multiple sources and prepare data for insights.
IBM InfoSphere DataStage on AWS

IBM InfoSphere DataStage is a data integration, extract, transform, and load (ETL) tool
that enables users to move and transform data between operational, transactional, and
analytical target systems.
Data transformation and movement is the process by which source data is selected,
converted, and mapped to the format required by target systems. The process manipulates
data to bring it into compliance with business, domain, and integrity rules, and with other
data in the target environment.
This reference deployment provides AWS CloudFormation templates to deploy a

Kubernetes-based InfoSphere DataStage cluster. This cluster includes:
 Red Hat Enterprise Linux (RHEL) 7.5 running a highly available Kubernetes control
plane and Docker image registry.
 Scalable worker nodes running InfoSphere DataStage.
 A Microsoft Windows-based DataStage Client machine.
 A Linux bastion host instance for Secure Shell (SSH) access to the cluster instances.
Worker nodes recover automatically from instance failures and are distributed across two
Availability Zones. Data is stored on an encrypted Amazon Elastic File System (Amazon
EFS) volume, which is highly available across all selected Availability Zones. The Amazon
Elastic Compute Cloud (Amazon EC2) instances use encrypted secondary Amazon Elastic
Block Store (Amazon EBS) volumes for storing ephemeral application data.
Page 3 of 33
Cost and licenses

You are responsible for the cost of AWS services used while running this Quick Start
reference deployment.
The AWS CloudFormation template for this Quick Start includes configuration parameters
that you can customize. Some of these settings, such as instance type, will affect the cost of
deployment. For cost estimates, see the pricing pages for each AWS service you will be
using. Prices are subject to change.
Tip After you deploy the Quick Start, we recommend that you enable the AWS Cost
and Usage Report to track costs associated with the Quick Start. This report delivers
billing metrics to an S3 bucket in your account. It provides cost estimates based on
usage throughout each month, and finalizes the data at the end of the month. For
more information about the report, see the AWS documentation.
This Quick Start requires licenses for IBM InfoSphere DataStage and IBM InfoSphere
DataStage and QualityStage Designer. You can purchase licenses from Passport Advantage
or an IBM representative. For general assistance with Passport Advantage, see the Passport
Advantage Online for customers or the eCustom care webpages.
After you purchase a license, IBM will email a Proof of Entitlement (PoE) certificate to the
primary contact person on the order form. The PoE confirms the eligible software and level
of use for which you are authorized, and contains your IBM Customer Number (ICN). To
use this Quick Start, you must provide the ICN and part numbers listed in your PoE.
If you’re an existing IBM client, please contact your IBM representative for additional
information about using your entitlements with this Quick Start.
When you launch the Quick Start, read the following software license agreements, and
agree to the terms:
 Software license agreement for IBM InfoSphere DataStage v11.7.0.2.

 Software license agreement for IBM InfoSphere DataStage and QualityStage Designer
v11.7.
In addition, prior to using this Quick Start, please review IBM’s Eligible Public Cloud
BYOSL Policy.
Page 4 of 33
Architecture
Deploying this Quick Start for a new virtual private cloud (VPC) with default parameters
builds the following InfoSphere DataStage environment in the AWS Cloud.
Figure 1: Quick Start architecture for IBM InfoSphere DataStage on AWS
The Quick Start sets up the following:

 A highly available architecture that spans three Availability Zones.*
 A VPC configured with public and private subnets according to AWS best practices, to
provide you with your own virtual network on AWS.*
 In the public subnets, managed NAT gateways to allow outbound internet access for
resources in the private subnets.*
 In a public subnet, a Linux bastion host instance (Amazon Linux 2) to allow inbound
Secure Shell (SSH) access to EC2 instances in public and private subnets.*
 In a public subnet, an EC2 instance (Windows Server 2012 R2) running the InfoSphere
DataStage thick client. Inbound SSH to EC2 instances in the public and private subnets
are also possible from this instance using PuTTY.
Page 5 of 33
 In the private subnets, three EC2 instances (RHEL 7.5) running a highly available
Kubernetes control plane and Docker registry.
 In a private subnet, a repository-type EC2 instance (RHEL 7.5) in an Auto Scaling
group. This instance runs the services tier (including IBM WebSphere Liberty) and
repository tier components, and one engine-compute pod. Ephemeral application data
is stored on an encrypted secondary Amazon Elastic Block Store (EBS) volume mounted
to the instance.
 In a private subnet, an engine-type EC2 instance (RHEL 7.5) in an Auto Scaling group.
This instance runs the engine conductor and one engine-compute pod. Ephemeral
application data is stored on an encrypted secondary Amazon Elastic Block Store (EBS)
volume mounted to the instance. The parallel processing capability of the cluster can be
scaled by increasing the number of instances in this group. (See Scaling compute, later
in this guide.)
 An Amazon EFS volume with a mount target in all selected Availability Zones.
 An Elastic Load Balancing (ELB) load balancer spanning the public subnets, for
proxying the browser-based DataStage thin client service.
 An ELB load balancer that spans the private subnets, for connecting the Kubernetes
worker instances to the Kubernetes control plane (master nodes).
 An Amazon Route 53 private hosted zone for resolving internal Domain Name System
(DNS) queries.
* The template that deploys the Quick Start into an existing VPC skips the components
marked by asterisks and prompts you for your existing VPC configuration.
Planning the deployment

Specialized knowledge
This Quick Start assumes basic familiarity with the use of the InfoSphere DataStage
application, including a browser-based Designer (thin client), a Windows-based Designer
(thick client), and a basic awareness of the components of a DataStage installation. If you’re
new to InfoSphere DataStage, see the Additional resources section.
This Quick Start also assumes familiarity with Kubernetes, kubectl, and Linux.
Lastly, this deployment guide requires a moderate level of familiarity with AWS services. If
you’re new to AWS, visit the Getting Started Resource Center and the AWS Training and
Page 6 of 33
Certification website for materials and programs that can help you develop the skills to
design, deploy, and operate your infrastructure and applications on the AWS Cloud.
AWS account
If you don’t already have an AWS account, create one at https://aws.amazon.com by
following the on-screen instructions. Part of the sign-up process involves receiving a phone
call and entering a PIN using the phone keypad.
Your AWS account is automatically signed up for all AWS services. You are charged only for
the services you use.
Technical requirements
You must provide your IBM Customer Number (ICN) and the part numbers of the software
licenses purchased, as noted in your Proof of Entitlement (PoE) certificate.
Red Hat Enterprise Linux (RHEL) 7.5 is used for the Kubernetes nodes in this deployment.
Other distributions aren’t currently supported. The DataStage Windows Client instance is
deployed from a private Amazon Machine Image (AMI) based on Windows Server 2012 R2,
and the bastion host instance runs Amazon Linux. Your AWS account is given launch
permission for the private AMI when the Quick Start is deployed.
Before you launch the Quick Start, your account must be configured as specified in the
following table. Otherwise, deployment might fail.
Page 7 of 33
Resources If necessary, request service limit increases for the following resources. You might need
to do this if you already have an existing deployment that uses these resources, and you
think you might exceed the default limits with this deployment. For default limits, see
the AWS documentation.
AWS Trusted Advisor offers a service limits check that displays your usage and limits
for some aspects of some services.
Resource This deployment uses
VPCs 1
Elastic IP addresses 5
IAM roles 2
Auto Scaling groups 3
Network Load
2
Balancers
t3.micro instances
1
(Linux bastion)
t3.xlarge instances
1
(Windows Client)
t3.large instances
(Kubernetes master 3
nodes)
2
m5.xlarge instances Consists of one Engine-type and
(IS worker nodes) one Repository-type worker
node instance.
Route 53 hosted zones 1
Regions This deployment includes Amazon EFS, which isn’t currently supported in all AWS
Regions. It is recommended that this Quick Start be deployed in a region with three
Availability Zones. For a current list of supported regions, see AWS Regions and
Endpoints in the AWS documentation.
Key pair Make sure that at least one Amazon EC2 key pair exists in your AWS account in the
region where you are planning to deploy the Quick Start. Make note of the key pair
name. You’ll be prompted for this information during deployment. To create a key pair,
follow the instructions in the AWS documentation.
If you’re deploying the Quick Start for testing or proof-of-concept purposes, we
recommend that you create a new key pair instead of specifying a key pair that’s already
being used by a production instance.
IAM permissions To deploy the Quick Start, you must log in to the AWS Management Console with IAM
permissions for the resources and actions the templates will deploy. The
AdministratorAccess managed policy within IAM provides sufficient permissions,
although your organization may choose to use a custom policy with more restrictions.
Page 8 of 33
Deployment options
This Quick Start provides two deployment options:
 Deploy InfoSphere DataStage into a new VPC (end-to-end deployment). This
option builds a new AWS environment consisting of the VPC, subnets, NAT gateways,
security groups, bastion hosts, and other infrastructure components, and then deploys
InfoSphere DataStage into this new VPC.
 Deploy InfoSphere DataStage into an existing VPC. This option provisions
InfoSphere DataStage in your existing AWS infrastructure.
The Quick Start provides separate templates for these options. It also lets you configure
CIDR blocks, instance types, and InfoSphere DataStage settings, as discussed later in this
guide.
Deployment steps
Step 1. Sign in to your AWS account
1. Sign in to your AWS account at https://aws.amazon.com with an IAM user role that has
the necessary permissions. For details, see Planning the deployment earlier in this
guide.
2. Make sure that your AWS account is configured correctly, as discussed in the Technical
requirements section.
Step 2. Retrieve IBM InfoSphere DataStage license information

1. Purchase a license from Passport Advantage or your IBM representative.
2. After the purchase is complete, IBM will email a Proof of Entitlement (PoE) certificate
to the primary contact person on the order form. The PoE certificate contains your IBM
customer number (ICN). To use this Quick Start for IBM InfoSphere DataStage, you will
need to provide your ICN and the part numbers you used to purchase the licenses for
the product.
Page 9 of 33
Step 3. Launch the Quick Start
Notes The instructions in this section reflect the older version of the AWS
CloudFormation console. If you’re using the redesigned console, some of the user
interface elements might be different.
You are responsible for the cost of the AWS services used while running this Quick
Start reference deployment. There is no additional cost for using this Quick Start.
For full details, see the pricing pages for each AWS service you will be using in this
Quick Start. Prices are subject to change.
1. Sign in to your AWS account, and choose one of the following options to launch the
AWS CloudFormation template. For help choosing an option, see deployment options
earlier in this guide.
• new VPC
Deploy Deploy • workload only
• workload
Deploy InfoSphere DataStage into a Deploy InfoSphere DataStage into an

new VPC on AWS existing VPC on AWS
Important If you’re deploying InfoSphere DataStage into an existing VPC, make

sure that your VPC has three private subnets in different Availability Zones for the
workload instances, and that the subnets aren’t shared. This Quick Start doesn’t
support shared subnets. These subnets require NAT gateways in their route tables, to
allow the instances to download packages and software without exposing them to the
internet. You will also need the domain name option configured in the DHCP options
as explained in the Amazon VPC documentation. You will be prompted for your VPC
settings when you launch the Quick Start.
Each deployment takes about 2 hours to complete.

2. Check the region that’s displayed in the upper-right corner of the navigation bar and
change it if necessary. This is where the network infrastructure for InfoSphere
DataStage will be built. The template is launched in the US East (Ohio) Region by
default.
Page 10 of 33
Note This deployment includes Amazon EFS, which is only supported in some
AWS regions. For a current list of supported regions, see the AWS Regions and
Endpoints webpage.
3. On the Select Template page, keep the default setting for the template URL, and then
choose Next.
4. On the Specify Details page, change the stack name if needed. Review the parameters
for the template. Provide values for the parameters that require input. For all other
parameters, review the default settings and customize them as necessary.
In the following tables, parameters are listed by category and described separately for
the two deployment options:
– Parameters for deploying InfoSphere DataStage into a new VPC
– Parameters for deploying InfoSphere DataStage into an existing VPC
When you finish reviewing and customizing the parameters, choose Next.
OPTION 1: PARAMETERS FOR DEPLOYING INFOSPHERE DATASTAGE INTO A NEW VPC

View template
VPC network configuration:
Parameter label
Default Description
(name)
Availability Zones Requires input List of Availability Zones to use for the subnets in the VPC.
(AvailabilityZones) Three Availability Zones are required for this deployment, and
the logical order of your selections is preserved.
Allowed external Requires input The CIDR IP range that is permitted to access InfoSphere
access CIDR DataStage. We recommend that you set this value to a trusted
(AllowedPublicCIDR) IP range. For example, you might want to grant only your
corporate network access to the software.
VPC CIDR 10.0.0.0/16 The CIDR block for the VPC to be created.
(VPCCIDR)
Private subnet 1 CIDR 10.0.0.0/19 The CIDR block for the private subnet located in Availability
(PrivateSubnet1CIDR) Zone 1.
Page 11 of 33
Parameter label
Default Description
(name)
Public subnet 1 CIDR 10.0.128.0/20 The CIDR block for the public subnet located in Availability
(PublicSubnet1CIDR) Zone 1.
Amazon EC2 instance configuration:

Parameter label
Default Description
(name)
Key pair name Requires input A public/private key pair, which allows you to connect
(KeyPairName) securely to your instance after it launches. This is the key pair
you created in your preferred region; see the Technical
InfoSphere DataStage t3.xlarge The type of EC2 instance for the DataStage Windows Client
Windows Client instance.
instance type
(DSClientInstanceType)
InfoSphere DataStage m5.xlarge The type of EC2 instance for the Repository-type worker node
Repo worker instance instances.
type
(ISRepoInstanceType)
InfoSphere DataStage m5.xlarge The type of EC2 instance for the Engine-type worker node
Engine worker instances.
instance type
(ISEngineInstanceType)
Kubernetes master t3.large The type of EC2 instance for the Kubernetes master node
instance type instances.
(K8sMasterInstance
Type)
Page 12 of 33
Licensing configuration:
Parameter label Default Description
(name)
License agreement — Choose Accept to acknowledge that you have read and agree
(LicenseAgreement) to the license terms for IBM InfoSphere DataStage
(https://www14.software.ibm.com/cgi-
bin/weblap/lap.pl?popup=Y&li_formnum=L-RDOS-
B5K4RY).
IBM Customer Number Requires input The IBM Customer Number (ICN) listed in your Proof of
for InfoSphere Entitlement for InfoSphere DataStage.
DataStage
(IBMDataStageICN)
InfoSphere DataStage Requires input The IBM part number associated with your InfoSphere
part number DataStage license.
(IBMDataStagePart
Number)
IBM Customer Number Requires input The IBM Customer Number (ICN) listed in your Proof of
for InfoSphere Entitlement for InfoSphere DataStage and QualityStage client.
DataStage and
QualityStage client
(IBMDataStageClientICN)
and QualityStage client DataStage and QualityStage client license.
part number
(IBMDataStageClient
PartNumber)
DataStage application configuration:

Parameter label
Default Description
(name)
InfoSphere Requires input The password to be set on the DataStage application for the
DataStage password user name "isadmin".
(IISPassword)
Resource tag ds1 This will be used to label AWS resources and the DataStage
(ResourceTag) Kubernetes namespace. Ensure that every InfoSphere
DataStage deployment on your AWS account uses a unique
resource tag.
Page 13 of 33
AWS Quick Start configuration:
Note We recommend that you keep the default settings for the following two
parameters, unless you are customizing the Quick Start templates for your own
deployment projects. Changing the settings of these parameters will automatically
update code references to point to a new Quick Start location. For additional details,
see the AWS Quick Start Contributor’s Guide.
Parameter label
Default Description
(name)
Quick Start S3 bucket aws-quickstart The S3 bucket you created for your copy of Quick Start assets,
name if you decide to customize or extend the Quick Start for your
(QSS3BucketName) own use. The bucket name can include numbers, lowercase
letters, uppercase letters, and hyphens, but should not start or
end with a hyphen.
Quick Start S3 key quickstart-ibm- The S3 key name prefix used to simulate a folder for your copy
prefix infosphere- of Quick Start assets, if you decide to customize or extend the
(QSS3KeyPrefix) datastage/ Quick Start for your own use. This prefix can include numbers,
lowercase letters, uppercase letters, hyphens, and forward
slashes.
Page 14 of 33
OPTION 2: PARAMETERS FOR DEPLOYING INFOSPHERE DATASTAGE INTO AN EXISTING

VPC
View template
Network configuration:
Parameter label
Default Description
(name)
Allowed external Requires input The CIDR IP range that is permitted to access InfoSphere
access CIDR DataStage. We recommend that you set this value to a trusted
(AllowedPublicCIDR) IP range. For example, you might want to grant only your
corporate network access to the software.
VPC ID Requires input The ID of your existing VPC for deployment.

(VPCID)
VPC CIDR Requires input The CIDR block of your existing VPC.
(VPCCIDR)
Bastion security group Requires input The ID of the security group attached to the bastion instance
ID of the VPC.
(BastionSecurityGroup
ID)
Private subnet 1 ID Requires input The ID of the private subnet in Availability Zone 1.
(PrivateSubnet1ID)
(PrivateSubnet2ID)
(PrivateSubnet3ID)
Public subnet 1 ID Requires input The ID of the public subnet in Availability Zone 1.
(PublicSubnet1ID)
(PublicSubnet2ID)
(PublicSubnet3ID)
Private subnet 1 CIDR Requires input The CIDR block for the private subnet located in Availability
Page 15 of 33
Amazon EC2 instance configuration:

Parameter label
Default Description
(name)
Key pair name Requires input A public/private key pair, which allows you to connect
(KeyPairName) securely to your instance after it launches. This is the key
pair you created in your preferred region; see the Technical
InfoSphere DataStage t3.xlarge The type of EC2 instance for the DataStage Windows Client
Windows Client instance.
instance type
(DSClientInstanceType)
InfoSphere DataStage m5.xlarge The type of EC2 instance for the Repository-type worker
Repo worker instance node instances.
type
(ISRepoInstanceType)
InfoSphere DataStage m5.xlarge The type of EC2 instance for the Engine-type worker node
Engine worker instances.
instance type
(ISEngineInstanceType)
Kubernetes master t3.large The type of EC2 instance for the Kubernetes master node
instance type instances.
(K8sMasterInstance
Type)
Licensing configuration:
(name)
License agreement — Choose Accept to acknowledge that you have read and
(LicenseAgreement) agree to the license terms for IBM InfoSphere DataStage
(https://www14.software.ibm.com/cgi-
bin/weblap/lap.pl?popup=Y&li_formnum=L-RDOS-
B5K4RY).
IBM Customer Requires input The IBM Customer Number (ICN) listed in your Proof of
Number for Entitlement for InfoSphere DataStage.
InfoSphere DataStage
(IBMDataStageICN)
part number DataStage license.
(IBMDataStagePart
Number)
Page 16 of 33

(name)
IBM Customer Requires input The IBM Customer Number (ICN) listed in your Proof of
Number for Entitlement for InfoSphere DataStage and QualityStage
InfoSphere DataStage client.
and QualityStage
client
(IBMDataStageClient
ICN)
and QualityStage DataStage and QualityStage client license.
client part number
(IBMDataStageClient
PartNumber)
DataStage application configuration:

Parameter label
Default Description
(name)
InfoSphere DataStage Requires input The password to be set on the DataStage application for the
password user name "isadmin".
(IISPassword)
Resource tag ds1 This will be used to label AWS resources and the DataStage
(ResourceTag) Kubernetes namespace. Ensure that every InfoSphere
DataStage deployment on your AWS account uses a unique
resource tag.
AWS Quick Start configuration:
Note We recommend that you keep the default settings for the following two
parameters, unless you are customizing the Quick Start templates for your own
deployment projects. Changing the settings of these parameters will automatically
update code references to point to a new Quick Start location. For additional details,
see the AWS Quick Start Contributor’s Guide.
Parameter label
Default Description
(name)
Quick Start S3 bucket aws-quickstart The S3 bucket name for the Quick Start assets. This bucket
name name can include numbers, lowercase letters, uppercase
(QSS3BucketName) letters, and hyphens (-), but should not start or end with a
hyphen. You can specify your own bucket if you copy all of the
assets and submodules into it, if you want to override the
Quick Start behavior for your specific implementation.
Page 17 of 33
Parameter label
Default Description
(name)
Quick Start S3 key quickstart-ibm- The S3 key name prefix used to simulate a folder for your copy
prefix infosphere- of Quick Start assets, if you decide to customize or extend the
(QSS3KeyPrefix) datastage/ Quick Start for your own use. This prefix can include numbers,
lowercase letters, uppercase letters, hyphens, and forward
slashes.
5. On the Options page, you can specify tags (key-value pairs) for resources in your stack
and set advanced options. When you’re done, choose Next.
6. On the Review page, review and confirm the template settings. Under Capabilities,
select the two check boxes to acknowledge that the template will create IAM resources
and that it might require the capability to auto-expand macros.
7. Choose Create to deploy the stack.
8. Monitor the status of the stack. When the status is CREATE_COMPLETE, the
InfoSphere DataStage cluster is ready.
9. Use the URLs displayed in the Outputs tab for the stack to view the resources that were
created.
Figure 2: InfoSphere DataStage outputs after successful deployment
Page 18 of 33
Step 4. Test the deployment

You can access InfoSphere DataStage from the browser by using the URL for the IBM
InfoSphere Information Server (IIS) Launchpad, and from the DataStage Windows Client
instance that’s provisioned with the cluster.
ACCESSING THE IIS LAUNCHPAD FROM A BROWSER

After stack creation has finished, use the link from the Outputs section of the root stack
shown in Figure 2 to open the Launchpad. On password-protected pages, the user name
will be “isadmin” and the password will be what you entered for the parameter
IISPassword.
Note The Launchpad link will only work when accessed from an IP address in the
AllowedPublicCIDR range.
The Launchpad is a standard, single web interface for opening the various clients or
consoles for IBM InfoSphere Information Server. After deploying this Quick Start, the
Launchpad gives you access to the following services, shown in Figure 3:
 IBM DataStage Flow Designer (you might need to purchase additional licenses to use
this)
 IBM InfoSphere DataStage Operations Console
 Subscription Manager
 IBM InfoSphere Metadata Asset Manager
 IBM InfoSphere Information Server Administration Console
Page 19 of 33
Figure 3: InfoSphere Information Server Launchpad
ACCESSING THE DATASTAGE WINDOWS CLIENT
Logging in using Remote Desktop Protocol (RDP)

1. Retrieve the initial password from the Amazon EC2 console:
a. Select the instance named “<ResourceTag>-DS-Client” in the Amazon EC2 console.
b. Choose Actions, and then choose Get Windows Password. It can take a few
minutes for this option to become available.
c. Upload your key pair file, and then choose Decrypt Password.
2. Log in to the DS-Client instance by using RDP, with the user name “Administrator” and
the password retrieved in step 1.
Page 20 of 33
Using InfoSphere DataStage and QualityStage Designer

1. Double-click the Designer Client icon on the desktop.
2. Enter the following values. You can also find these values in README.txt on the
desktop. Before entering the Project value, select the dropdown button, accept the
security alert, and edit the pre-populated value (do not copy and paste).
Host name is-en-conductor-0.en-cond:32500
User name isadmin
Password <value entered for the IISPassword parameter>
Project IS-EN-CONDUCTOR-0.EN-COND:32501/dstage
3. Choose Login.
4. If you encounter a timeout error the first time that you try to log in, please try again in a
couple minutes.
Using the IIS Launchpad
Note For compatibility, we recommend that you install Mozilla Firefox (version
54 and later) or Google Chrome (version 63 and later).
1. On the desktop, double-click the IIS Launchpad Web icon.

2. Choose Continue to this website.
3. On the Security Alert dialog box, choose Yes.
ACCESSING THE CLUSTER NODES THROUGH THE LINUX BASTION HOST
Note Amazon Linux instances are launched with the latest yum critical security
updates installed. We recommend that you periodically run the command
sudo yum update -y --security; sudo reboot
or configure yum-cron to keep the instance hardened with the latest security patches.
The recommended method of SSH access to the Kubernetes cluster instances via the
bastion host is by using SSH agent forwarding, as in the following Bash instructions:
1. Run the command ssh-add -K <your-key.pem> to store the key in your keychain.
On Linux, you might need to omit the -K flag.
Page 21 of 33
2. Retrieve the IP address of the Linux bastion host from the Amazon EC2 console.
3. To log in to the bastion host, run ssh -A ec2-user@<bastion-ip>.
4. To log in to private subnet instances, run ssh <instance-ip> from the bastion host.
For details and Windows instructions, see the blog post Securely Connect to Linux
Instances Running in a Private Amazon VPC.
TRANSFERRING FILES FROM YOUR LOCAL COMPUTER TO THE CLUSTER

Files can be manually transferred to the Kubernetes cluster by tunneling via SSH through
the Linux bastion host. Files copied to the /mnt directory in any of the Kubernetes EC2
instances (i.e., not the Linux bastion) will be visible to the DataStage application.
Bash
1. Run the command ssh-add -K <your-key.pem> to store the key in your keychain.
On Linux, you might need to omit the -K flag.
2. In one terminal window, run the following command to establish an SSH tunnel. Keep
this terminal open for the duration of your file transfer.
ssh -L 9999:<k8s-instance-ip>:22 ec2-user@<bastion-public-ip>
3. In another terminal, you may SFTP through the tunnel bound to port 9999 at localhost.
sftp -P 9999 ec2-user@localhost
WinSCP
Open an SCP connection using the following site configuration:
 File protocol: SCP

 Host name: <k8s-instance-ip>
 User name: ec2-user
 Advanced > SCP/Shell:
– Shell: sudo su
 Advanced > Tunnel:
– Host name: <bastion-public-ip>
Page 22 of 33
– Private key file: <your-key.ppk>

If the key pair file is in .pem format, you may still select it, and WinSCP will
offer to convert it to .ppk format.
 Advanced > Authentication:
Private key file: <your-key.ppk>
Manual cleanup
When you delete the stack created by this Quick Start, the DataStage cluster will be deleted
automatically. However, certain cluster information is stored in AWS Systems Manager
Parameter Store, which may persist following a stack deletion. Follow these steps to
manually clean values by using the AWS Management Console:
1. Navigate to the Parameter Store page in the Amazon EC2 console.
2. Delete all the parameters that begin with /<ResourceTag>/.
Alternatively, you can run the following commands in a Bash shell:
$ REGION=<aws-region>
$ TAG=<ResourceTag>
$ aws ssm get-parameters-by-path --path /$TAG/ --query
'Parameters[*].Name' --region $REGION --output text | xargs -t -n 1
aws ssm delete-parameter --region $REGION --name
Best practices for using InfoSphere DataStage on AWS
Enabling backups
The cluster comes pre-loaded with scripts that you can use to enable backup. These backup
scripts copy relevant data and application configuration files to another directory on
Amazon EFS. Because Amazon EFS is highly available, this backup provides a replica in
case data is corrupted or deleted.
To enable backups, run the following on any of the master instances:
/disk1/quickstart/install/DS-Kube-Installer/BackupScripts/CronJobs.sh
Before running this, you can modify the cron schedule. By default, a full IBM Db2 backup is
set to run once a week, and incremental Db2 and config file backups run every day.
Page 23 of 33
Note If this master instance is terminated, the cron job must be re-enabled on
another master instance. Backups are only available for 30 days and are then
deleted.
Security
When you build systems on the AWS infrastructure, security responsibilities are shared
between you and AWS. This shared model can reduce your operational burden. AWS
operates, manages, and controls the components from the host operating system and
virtualization layer down to the physical security of the facilities in which the services
operate.
In turn, you assume responsibility and management of the guest operating system
(including updates and security patches), other associated applications, and configuration
of the AWS-provided security group firewall. For more information about security on AWS,
visit the AWS Security Center.
This Quick Start creates an IAM role that’s attached to all instances of Kubernetes
configured with a minimum set of access policies, and restricted to tagged resources when
possible. The bastion host and DataStage Client instances do not come with an IAM role
attached, and therefore can’t manipulate AWS resources.
Encrypted EFS file systems and EBS drives are used for storing application files and data.
The password of the isadmin user is set to the value entered for the parameter IISPassword.
This password is also used for Db2 native encryption and the LWAS signer certificate. The
default password for other internal-facing users is set to “isadmin”.
Security groups are used to restrict network ingress traffic for instances in this Quick Start.
Inbound access from the internet is only permitted from IP addresses in the
AllowedPublicCIDR to specific ports. Security groups are also used to control traffic flow
within the instances and subnets of the VPC and for the EFS mount targets. Security
Groups are created from ibm-datastage-security.template. The following summarizes the
role of the various security groups:
 BastionSecurityGroup
This is created by the Linux Bastion Hosts on AWS Quick Start and invoked as a nested
stack in this Quick Start. It allows SSH access to the bastion host instance from the IP
range that you specify in the AllowedPublicCIDR parameter.
Page 24 of 33
 DataStageClientSecurityGroup
Allows RDP access to the DataStage Client instance from the AllowedPublicCIDR IP
range.
 MountTargetSecurityGroup
Allows TCP port 2049 access from instances in the ClusterNodeSecurityGroup.
 ClusterNodeSecurityGroup
Attached to all Kubernetes cluster instances. Allows SSH access from the bastion host
instance, the DataStage Client instance, and all private subnet traffic.
 MasterNodeAddonSecurityGroup
Attached only to the Kubernetes master instances. Exposes the DataStage NodePorts to
the internet, through the Network Load Balancer.
Scaling compute
At the time of deploying the Quick Start, two is-engine-compute pods are provisioned on
the Kubernetes cluster (one each in the ISRepo and ISEngine instances). The parallel
compute capability of the cluster can be scaled up by provisioning more of these pods.
To ensure proper distribution of additional compute pods across all the instances, first
horizontally scale the ISEngine instances, as outlined in Horizontal scaling. Then, scale the
number of is-engine-compute pods. Depending on your use case and the instance type, you
might also want to have multiple compute pods on the same worker instance.
Horizontal scaling
1. Navigate to the “ISEngine-ASG” Auto Scaling group in the Amazon EC2 console.
2. Increase the Desired capacity value as needed.
3. Wait 15 minutes for Ansible to prepare the instance and attach it to the cluster.
Changing the number of is-engine-compute pods

1. Connect via SSH into one of the master nodes, and navigate to the directory
/disk1/quickstart/installer/DS-Kube-Installer/.
2. Run the following command as root:
./scaleCompute.sh -ns <resource-tag> -replicas <number-of-replicas>
Page 25 of 33
Vertical scaling
Changing the EC2 instance type requires the instance to be stopped. Before doing this, the
Auto Scaling activity of the group must be suspended (otherwise, a new instance will be
created inadvertently).
1. Select the ISEngine-ASG Auto Scaling group in the Amazon EC2 console.
2. Choose Actions > Edit, and then choose ReplaceUnhealthy in the Suspended
Processes box.
3. Stop the K8s-Worker-ISEngine instance.
4. Choose Actions > Instance Settings > Change Instance Type and choose a larger
instance as desired.
5. Start the instance again. Wait 10-15 minutes to allow the instance to become healthy.
6. Once the instance is healthy, undo step 2 to re-enable Auto Scaling.
Note Vertical scaling won’t alter the instance type in the launch template that was
originally associated with the Auto Scaling group. Therefore, future Auto Scaling
action will use the originally specified instance type.
Scaling back down

If the number of worker instances in the cluster is reduced by decreasing the desired
capacity of the Auto Scaling group, you might need to manually deregister the terminated
nodes from Kubernetes. You might also want to run scaleCompute.sh as shown earlier, to
reduce the number of engine-compute pods.
1. Log in to one of the master instances.

2. Run the following command as root:
/disk1/quickstart/installer/DS-Kube-Installer/deleteDeadNodes.sh
3. Ensure that the output shows the names of the instances that were terminated by the
Auto Scaling group. If the output is empty, Kubernetes hasn’t yet detected that the node
is unhealthy. Try again after a minute.
Page 26 of 33
Provisioning more DataStage Client instances

More DataStage Client instances can be provisioned as required.
1. Open the Amazon EC2 console, and then navigate to the Launch Templates page.
2. Select the template named <ResourceTag>-DSClientLT. Choose Actions, and then
choose Launch instance from template.
3. You can use the default values or change any parameters as desired. Ensure that the
selected subnet is one of the public subnets originally created by the Quick Start.
4. You can add an Elastic IP address to the new instance, so that the public IP persists
between reboots. Select the new instance on the Instances page, choose Actions >
Networking > Manage IP Addresses, and then choose Allocate an Elastic IP.
5. The public IP of the new Client instance must be added as an inbound rule in the
<ResourceTag>-MasterNodeAddonSG security group, allowing TCP ports 32500-32502.
Performance considerations for Amazon EFS

The InfoSphere DataStage application is installed on an encrypted Amazon Elastic File
System (Amazon EFS) file system for high availability and scalability.
Amazon EFS offers two throughput modes: Bursting and Provisioned. The EFS file system
used in this Quick Start is set to Bursting by default. In Bursting mode, the baseline
throughput scales with the size of the file system (@ 50 KB/s/GB), with 100 MB/s bursting
available for limited periods of time based on a credit system. Bursting credits are earned
over time, based on the size of the file system. New EFS file systems are provisioned with a
large initial grant of burst credits. You can view the balance of burst credits for your EFS
file system in Amazon CloudWatch, as the metric BurstCreditBalance.
Typical DataStage workloads are CPU- and RAM-intensive, rather than disk-intensive.
Because of this, leaving the system on Bursting mode will provide adequate disk
performance. In the following scenarios, we recommend changing to the Provisioned mode,
where a constant throughput can be set (at a cost):
 Your workloads are disk throughput–intensive, and burst credits are running out
frequently.
 You anticipate a large data transfer into or out of the cluster.
 You want better performance than the 100 MB/s burst throughput.
 Your use case can’t tolerate unpredictable performance.
Page 27 of 33
Modes may be changed once in a 24-hour period. For details, see Throughput nodes in the
AWS documentation.
Troubleshooting
Q. I encountered a CREATE_FAILED error when I launched the Quick Start.
A. If AWS CloudFormation fails to create the stack, we recommend that you relaunch the
template with Rollback on failure set to No. (This setting is under Advanced in the
AWS CloudFormation console, Options page.) With this setting, the stack’s state will be
retained and the instance will be left running, so you can troubleshoot the issue. (For
Windows, look at the log files in %ProgramFiles%\Amazon\EC2ConfigService and
C:\cfn\log.)
Important When you set Rollback on failure to No, you will continue to incur
AWS charges for this stack. Please make sure to delete the stack when you finish
troubleshooting.
For additional information, see Troubleshooting AWS CloudFormation on the AWS

website.
Q. I encountered a size limitation error when I deployed the AWS CloudFormation

templates.
A. We recommend that you launch the Quick Start templates from the links in this guide or
from another S3 bucket. If you deploy the templates from a local copy on your computer or
from a non-S3 location, you might encounter template size limitations when you create the
stack. For more information about AWS CloudFormation limits, see the AWS
documentation.
Q. A CREATE_FAILED error occurred at the Custom::AMIInfo resource in the

DataStageClientStack.
A. You may have entered an invalid IBM Customer Number (ICN) and/or part number.
Please provide the ICN and part numbers noted in the Proof of Entitlement (PoE) that IBM
provided when you purchased your entitlements to IBM InfoSphere DataStage and
InfoSphere DataStage and QualityStage Designer. Please note that your entitlements might
be in different PoEs, if you purchased licenses for InfoSphere DataStage and for InfoSphere
DataStage and QualityStage Designer separately.
Page 28 of 33
You can obtain your PoE from IBM’s Passport Advantage Online portal by following these
steps. For general assistance with Passport Advantage, see the Passport Advantage Online
for customers or the Customer eCare team webpages.
If you’re unable to locate your PoE or don’t have one, please contact your IBM
representative or IBM support. Please do not post your ICN and part number in a GitHub
Issue.
Q. A CREATE_FAILED error occurred at the WaitCondition resource in the

DataStageStack.
A. This error usually indicates that a problem has occurred when running bootstrap scripts
to set up the Kubernetes cluster. Please use the following checklist:
 Ensure that you have entered a valid IBM Customer Number (ICN) and/or part number.
It may be helpful to know that if the deployment didn’t fail at DataStageClientStack, you
entered a valid ICN and part number for the DataStage client. Refer to the preceding
troubleshooting question.
 If you’re deploying to your own VPC, ensure that the VPC is correctly configured with
three distinct private and public subnets. Each private-public subnet pair must reside in
a different Availability Zone.
 Ensure that the Parameter Store doesn’t have values left over from previous
deployments of this Quick Start. Refer to Manual cleanup earlier in this guide.
 Ensure that an adequately-sized EC2 instance type is selected for the
K8sMasterInstanceType parameter. It is recommended that an instance type similar or
better than the default type be selected.
 Run the stack again with Rollback on failure set to No. (See How do I prevent my
AWS CloudFormation stack from rolling back on failure? for instructions). Connect via
SSH into the K8s-Master-1 instance and inspect the contents of the following files:
/var/log/cfn-init.log, /var/log/cfn-init-cmd.log,
/disk1/quickstart/installer/DS-Kube-Installer/install_log. If you open an IBM
Support ticket or GitHub Issue for this error, attach these logs.
Q. I encountered a java.lang.NoClassDefFoundError: com.ibm.iis.isf.admin.Config
(initialization failure) error message when I tried to access the Launchpad URL.
A. After the stack has been created, the pods might require some time to finish initializing
the InfoSphere DataStage application. If the error persists for longer than 60 minutes after
stack creation has finished, please contact IBM support, or open an Issue in the Quick
Start’s GitHub repository.
Page 29 of 33
Q. In the Launchpad Operations Console, one or more service status monitors is in an error
state.
A. If this occurs right after the stack has been deployed, all the containers might not have
finished initializing. Check back after a few minutes. In other cases, try the following
command to restart the monitoring services. (Note that the snippet is a single command.)
kubectl exec -i is-en-conductor-0 -n <resource-tag> -- bash -c

'/opt/IBM/InformationServer/Server/DSODB/bin/DSAppWatcher.sh -stop
ResMonApp;
/opt/IBM/InformationServer/Server/DSODB/bin/DSAppWatcher.sh -start
ResMonApp;
/opt/IBM/InformationServer/Server/DSODB/bin/DSAppWatcher.sh -stop
EngMonApp;
/opt/IBM/InformationServer/Server/DSODB/bin/DSAppWatcher.sh -start
EngMonApp'
Q. After the Auto Scaling group adds worker instances, the application is in an error state.
A. After an ISRepo instance replacement, it takes 15-20 minutes for the pods to finish
deploying again. After an ISEngine instance replacement or scaling action, it takes 5-10
minutes for the application to get to a working state. If the cluster isn’t functional 30
minutes following an instance launch, log in to one of the Kubernetes master instances and
run kubectl get nodes as root. If one of the EC2 worker instances in the cluster is absent
from the list, delete it from the Amazon EC2 console. The Auto Scaling group will replace it.
Q. Trying to connect with the DataStage Windows Client instance gives the error: “Failed to
authenticate the current user against the selected Service Tier.”
A. Ensure that you are using the correct credentials. Also, make sure that you are following
the instructions in Accessing the DataStage Windows Client or the README.txt file. Modify
the pre-populated “Project” value instead of copying-and-pasting it into the form.
Q. Disk performance is poor.

A. Refer to the EFS performance considerations section.
Q. Disk performance was good initially, but has suddenly dropped.

A. The default throughput option for the Amazon EFS file system in this Quick Start is set
to bursting mode. In this mode, a burst credit system provides bursting capability above
baseline speeds for a limited period of time. Amazon EFS provisions new file systems with a
large grant of initial credits. When they run out, throughput falls to a baseline speed that
Page 30 of 33
might be significantly slower, depending on the size of the file system. Refer to the EFS
performance considerations section, earlier in this guide.
IBM support
IBM support is available to users of the InfoSphere DataStage application and InfoSphere
DataStage and QualityStage Designer.
Create a case
1. Log in to the Cloud Service Portal using your IBMid. If you don’t have an IBMid, please
sign up for one.
2. In Need more help, choose Create a case.
3. In What type of support do you need, select Technical.
4. In the Category drop-down menu, select Analytics.
5. Under Offering, enter dsoncloud.
6. Fill in the Subject and Description fields, and include as much information about the
issue as possible. State that you’re an AWS Quick Start user.
7. Submit the case, and note the case number.
Support forum
Look for answers in our support forum.
Send us feedback
To post feedback, submit feature ideas, or report bugs, use the Issues section of the
GitHub repository for this Quick Start. If you’d like to submit code, please review the Quick
Start Contributor’s Guide.
Additional resources
AWS resources
 Getting Started Resource Center
 AWS General Reference
 AWS Glossary
Page 31 of 33
AWS services
 AWS CloudFormation
 Amazon EBS
 Amazon EC2
 Amazon EFS
 IAM
 Amazon VPC
IBM InfoSphere DataStage documentation

 InfoSphere Information Server documentation
 Getting started with InfoSphere DataStage and InfoSphere QualityStage
 IBM DataStage Flow Designer (web-based; thin client)
 IBM Support home page
 Topology for DataStage with Kubernetes
Other Quick Start reference deployments

 AWS Quick Start home page
Document revisions
Date Change In sections
August 2019 Added information about manually transferring Transferring files from your
files to the Kubernetes cluster; local computer to the cluster;
Added troubleshooting information about how to Troubleshooting
resolve a CREATE_FAILED error that occurs at
the WaitCondition resource in the
DataStageStack
May 2019 Initial publication —
Page 32 of 33
© 2019, Amazon Web Services, Inc. or its affiliates, and IBM. All rights reserved.
Notices
This document is provided for informational purposes only. It represents AWS’s current product offerings
and practices as of the date of issue of this document, which are subject to change without notice. Customers
are responsible for making their own independent assessment of the information in this document and any
use of AWS’s products or services, each of which is provided “as is” without warranty of any kind, whether
express or implied. This document does not create any warranties, representations, contractual
commitments, conditions or assurances from AWS, its affiliates, suppliers or licensors. The responsibilities
and liabilities of AWS to its customers are controlled by AWS agreements, and this document is not part of,
nor does it modify, any agreement between AWS and its customers.
The software included with this paper is licensed under the Apache License, Version 2.0 (the "License"). You
may not use this file except in compliance with the License. A copy of the License is located at
http://aws.amazon.com/apache2.0/ or in the "license" file accompanying this file. This code is distributed on
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and limitations under the License.
Page 33 of 33

Ibm Infosphere Datastage On The Aws Cloud

Hochgeladen von

Dokumentinformationen

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Ibm Infosphere Datastage On The Aws Cloud

Hochgeladen von

Copyright:

Verfügbare Formate

IBM InfoSphere DataStage on the

Shrumit Mehta and Shashidhar Yellareddy, IBM

Step 4. Test the deployment ................................................................................................ 19

IBM InfoSphere DataStage on AWS

This reference deployment provides AWS CloudFormation templates to deploy a

Cost and licenses

 Software license agreement for IBM InfoSphere DataStage v11.7.0.2.

Figure 1: Quick Start architecture for IBM InfoSphere DataStage on AWS

The Quick Start sets up the following:

Planning the deployment

Resource This deployment uses

Auto Scaling groups 3

Route 53 hosted zones 1

Step 2. Retrieve IBM InfoSphere DataStage license information

Step 3. Launch the Quick Start

Deploy InfoSphere DataStage into a Deploy InfoSphere DataStage into an

Important If you’re deploying InfoSphere DataStage into an existing VPC, make

Each deployment takes about 2 hours to complete.

OPTION 1: PARAMETERS FOR DEPLOYING INFOSPHERE DATASTAGE INTO A NEW VPC

Amazon EC2 instance configuration:

DataStage application configuration:

AWS Quick Start configuration:

OPTION 2: PARAMETERS FOR DEPLOYING INFOSPHERE DATASTAGE INTO AN EXISTING

VPC ID Requires input The ID of your existing VPC for deployment.

Amazon EC2 instance configuration:

Parameter label Default Description

DataStage application configuration:

AWS Quick Start configuration:

Figure 2: InfoSphere DataStage outputs after successful deployment

Step 4. Test the deployment

ACCESSING THE IIS LAUNCHPAD FROM A BROWSER

Figure 3: InfoSphere Information Server Launchpad

ACCESSING THE DATASTAGE WINDOWS CLIENT

Logging in using Remote Desktop Protocol (RDP)

Using InfoSphere DataStage and QualityStage Designer

Host name is-en-conductor-0.en-cond:32500

User name isadmin

Password <value entered for the IISPassword parameter>

Using the IIS Launchpad

1. On the desktop, double-click the IIS Launchpad Web icon.

ACCESSING THE CLUSTER NODES THROUGH THE LINUX BASTION HOST

TRANSFERRING FILES FROM YOUR LOCAL COMPUTER TO THE CLUSTER

ssh -L 9999:<k8s-instance-ip>:22 ec2-user@<bastion-public-ip>

sftp -P 9999 ec2-user@localhost

 File protocol: SCP

– Private key file: <your-key.ppk>

Best practices for using InfoSphere DataStage on AWS

To enable backups, run the following on any of the master instances:

Changing the number of is-engine-compute pods

2. Run the following command as root:

./scaleCompute.sh -ns <resource-tag> -replicas <number-of-replicas>

Scaling back down

1. Log in to one of the master instances.

Provisioning more DataStage Client instances

Performance considerations for Amazon EFS

For additional information, see Troubleshooting AWS CloudFormation on the AWS

Q. I encountered a size limitation error when I deployed the AWS CloudFormation

Q. A CREATE_FAILED error occurred at the Custom::AMIInfo resource in the

Q. A CREATE_FAILED error occurred at the WaitCondition resource in the

kubectl exec -i is-en-conductor-0 -n <resource-tag> -- bash -c

Q. Disk performance is poor.

Q. Disk performance was good initially, but has suddenly dropped.

IBM InfoSphere DataStage documentation