Click here to

Join the DSXchange

Home • Forum • Search • Favorites • Register • Log in

Usergroups • Contact • FAQ • Privacy Policy • Log in to check your private messages

View next topic

Grid storage NAS vs SAN View previous topic
Add To Favorites

Post new topic Reply to topic DSXchange Forum Index » IBM® DataStage Enterprise Edition (Formerly Parallel Extender/PX)

Author Message

thompsonp Posted: Mon Jul 03, 2017 7:44 am Reply with quote Back to top

DataStage® Release: 11x

Job Type: Parallel
OS: Unix
Group memberships: Additional info: RHEL
Premium Members I have seen a few posts about the choice between NAS or SAN in a grid environment and some dsx members appear to prefer NAS, mostly
for simplicity.
Joined: 01 Mar 2005
Posts: 204
In one of the redbooks it states:
Points: 1920 •In a SAN configuration with more than one head node, the compute nodes cannot identify the specific head node /home directory to use
without GFS. With GFS, however, the compute nodes can still only share the one /home directory between both head nodes. With a NAS
configuration, each user has only one /home directory. Different users can access different /home directories. Software mount points are
defined based on a specific head node, avoiding any confusion about the specific /home directory to be used. As mentioned previously, you
can support different uses such as development and regression testing from the same compute nodes.

Please can someone elaborate on the statement above?

Assuming that the head nodes are for different environments, say DEV and TEST, what is the issue with the SAN where something like GFS
or GPFS is used?
Is it saying that the compute nodes can still only see either the dev or test user home directories? If that is the case what is the solution
(without reverting to NAS)?

Thanks

View user's profile Send private message

PaulVL Posted: Mon Jul 03, 2017 8:23 am Reply with quote Back to top

A users HOME directory does not need to be a shared mount.

It does need SSH keys set up (assuming you are using ssh rather than rsh) between HN and CNs.
Group memberships:
Premium Members

Joined: 17 Dec 2010 NAS vs SAN... as long as the data mounts, project mounts, engine mounts are shared and visible to each host in question. Doesn't matter.
Posts: 1272

Points: 8350 I can speak from experience that I prefer NAS over shared SAN.

View user's profile Send private message Rate this response: 0 1 2 3 4 5 Not yet rated

thompsonp Posted: Tue Jul 25, 2017 8:56 am Reply with quote Back to top

Group memberships:
Premium Members
Paul,
I'm still not clear on what is required of the file system and what the issues with having a SAN might be.
We have two head nodes sharing the compute nodes and some parts of the file system use GPFS e.g. for datasets. I thought this was to
allow all the nodes access to specific parts of the file system.

Joined: 01 Mar 2005 Is it possible to set up an Information Server grid using a SAN without having a clustered file system?
Posts: 204 Are there any downsides or issues in not having a clustered file system in a setup where multiple head nodes (for different environments)
Points: 1920 share compute nodes?

View user's profile Send private message Rate this response: 0 1 2 3 4 5 Not yet rated

asorrell Posted: Tue Jul 25, 2017 11:56 am Reply with quote Back to top
Site Admin
The file system must support files being shared between servers with locking. This basically means NFS, IBM GPFS (rebranded as Spectrum
Group memberships:
Premium Members,
Scale) or Red Hat GFS2 with the Resilient Storage add-on. If you don't put in a locking file system you are going to have corruption.
DSXchange Team, Inner
Circle, Server to Parallel _________________
Transition Group Andy Sorrell
Certified DataStage Consultant
Joined: 04 Apr 2003 IBM Analytics Champion 2009 - 2017
Posts: 1694
Location: Colleyville, Texas
Points: 23058 View user's profile Send private message Rate this response: 0 1 2 3 4 5 Not yet rated

PaulVL Posted: Tue Jul 25, 2017 12:13 pm Reply with quote Back to top

The DataStage binaries (install path) must be shared between HN and CNs.
The Project Directories must be shared between HN and CNs.
The Data must be shared (for obvious reasons as well).
Group memberships:
Premium Members
The Batch IDs Home path, do NOT have to be shared, but it's handy if they are.
Joined: 17 Dec 2010
Posts: 1272 The GRid Job Dir path (used to store the dynamically created APT file) must be shared.
Points: 8350 The grid Fifo directory must be local to the HN.

TMPDIR should be local to the HN but not mandatory, it can be a shared mount but the shared aspect is not a factor. Speed is important on
this mount and IBM will advise to make it local.

asorrell
Site Admin
Group memberships:
Premium Members,
DSXchange Team, Inner
Circle, Server to Parallel
Transition Group
Joined: 04 Apr 2003
Posts: 1694
Location: Colleyville, Texas
Points: 23058
PaulVL
Group memberships:
Premium Members
Joined: 17 Dec 2010
Posts: 1272
Points: 8350
Post new topic Reply to topic
Jump to: IBM® DataStage Enterprise Edition (Formerly Parallel Extender/PX)
"Shared" can be done in many ways, so as long as you poke a file on HN and it's the same physical file on the CNs, mission accomplished.
SCRATCH disk should be local (and fast) to the HN and each CN. It CAN be a shared mount, but typically the SHARED aspect slows down
the data access speed so given that you want scratch as fast as you can, make it local to each host.
Websphere does not need to be exposed to the compute nodes.
View user's profile Send private message Rate this response: 0 1 2 3 4 5 Not yet rated
Posted: Tue Jul 25, 2017 12:18 pm Reply with quote Back to top
PaulVL wrote:
Websphere does not need to be exposed to the compute nodes.
My personal opinion is nothing should EVER be exposed to WebSphere...
_________________
Andy Sorrell
Certified DataStage Consultant
IBM Analytics Champion 2009 - 2017
View user's profile Send private message Rate this response: 0 1 2 3 4 5 Not yet rated
Posted: Tue Jul 25, 2017 4:54 pm Reply with quote Back to top
There are some stages that use some Java JAR files where are only present on Websphere path, so IBM made some "assumptions" and
never tested the IRULE or ILOG stuff on a cluster/grid environment. I ran into that and basically had to copy a JAR file over in order to
make the environment work. The support PMR basically said "So you have it working now... can we close the ticket?" instead of ensuring
that the JAR file would be on a compute nodes in a grid setting.
View user's profile Send private message Rate this response: 0 1 2 3 4 5 Not yet rated
Di s p l ay p os ts f r o m p r ev io us: All Posts Oldest First Go
DSXchange Forum Index » IBM® DataStage Enterprise Edition (Formerly Parallel Extender/PX)
Add To Favorites
View next topic
Go
View previous topic
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
Powered by phpBB © 2001, 2002 phpBB Group
Theme & Graphics by Daz :: Portal by Smartor
All times are GMT - 6 Hours