Under the hood of GlassFish v2 Clustering

Tuesday Jun 05, 2007 from Nazrul's Blog

GlassFish V2 is a production-ready Java EE 5 application server currently in beta. There was a
lot of interest about GlassFish this year during JavaOne. I had a chance to talk to many
customers interested in GlassFish. Some were already using Sun's distribution and dealing with
very large deployments. In this blog, I will share few tricks and tips that may come in handy
while dealing with GlassFish V2.

First, a quick re-cap. GlassFish V2 has support for profiles (OnePager,

blog - [1], [2]) where the server installation is configured for different
usage. Three profiles are available by default: developer, cluster, and
enterprise. Developers would typically work with a single Java EE server
instance and use the developer profile. Cluster and enterprise profiles are
targeted for deployments.

A typical GlassFish V2 installation [in cluster or enterprise profile] consists of one or more
administration domain. A domain provides a common authentication and administration point for
a collection of Java EE server instances. Domain Administration Server (DAS) handles all the
administrative operations in a domain.

Each remote machine [in cluster or enterprise profile] has a light weight process called Node
Agent. DAS interacts with the Node Agent to manage the life-cycle operations (create, delete,
start, stop) in a remote machine. Node Agents and remote Java EE server instances synchronizes
with DAS during startup to get the latest configuration from central repository managed by DAS.

GlassFish V2 supports the concept of cluster where multiple Java EE server instances can be
grouped for horizontal scalability. GlassFish V2 also supports in-memory replication feature
where state of a deployed application is replicated to a peer in the cluster and thus enables the
application to be highly available. Each Java EE server instance in a cluster shares the same set
of applications and configuration information.

To summarize, GlassFish V2 offers enterprise features such as clustering (screencast, blog), high
availability (screencast), and load balancing (GF page, blog) to support highly scalable, volume
enterprise deployments for service-oriented architecture (SOA) and Web 2.0 applications. Refer
to GlassFish V2 architecture wiki and JavaOne session TS-4436: Technical Overview of
GlassFish v2 (abstract, blog) for more details.

Q: Does GlassFish V2 support hot deployment to a cluster?

Yes, GlassFish V2 offers multi-phase deployment to cluster and minimizes deployment related
failures to distributed machines. Use the --target option in asadmin deploy command (example,
%asadmin deploy --target cluster1 foo.war) or the Administration Console. Deployment is
dynamic; i.e., deployed application bits are transferred automatically to the remote server
instances. You don't need to restart the cluster after deploying an application. V2 also offers
other deployment features such as auto-deploy, directory deploy, ant integration (blog),
NetBeans Integration, etc. Refer to deployment documentation for more details.

Q: Is it possible to deploy the same application to

multiple clusters?

Yes. Use asadmin create-application-ref (example,

%asadmin create-application-ref --target cluster2
myApp ) and asadmin delete-application-ref commands
with --target option in CLI or manage target for the
deployed application in Administration Console.

Q: Is there a way to start a remote server instance when DAS is not running?

Yes, you may use the startserv/stopserv scripts under the bin directory (blog) or re-start the node
agent. asadmin start-node-agent command will automatically start the remote server instances
without synchronizing with DAS.

Q: How do I force a full synchronization of a remote server instance?

A hidden file named .com_sun_appserv_timestamp keeps track of the last successful

synchronization time of DAS. If you remove these files manually (not an official interface) under
the remote server instance(s), a full synchronization will happen during the next server re-start. If
synchronization fails for any reason, these files are automatically removed to force a full
synchronization. In GlassFish V2, significant improvement was made in synchronization
performance (~40% - your mileage may vary).

The configuration under remote instances are treated as cache (for example, all files under
nodeagents/na1/server1). In extreme cases, if you remove all files of a remote server instance and
restart the node agent, the remote server instance (for example, server1) will be recreated and all
necessary files will be synchronized. Refer to synchronization documentation for more details.

Q: Is there a way to backup/clone DAS so that it is not a single point of failure for
management?

DAS has no role in application runtime functionality. So, when DAS is un-available, the cluster
will continue to run. You may use %asadmin backup-domain command to create a backup of
DAS. After that, please backup the zip from backup-domain command in a highly available
storage. When /if DAS's machine crashes, you may use %asadmin restore-domain command
with the previously backed-up zip to restore the domain in another machine. During DAS's
startup, it will ping all the node agents to notify the new co-ordinates of DAS. Solaris [Sun]
Cluster has support for Sun Java System Application Server. You may use it to provide
automatic failover. Refer to this documentation for more details.
Q: Is there a way to specify machine specific JVM options for a server instance in a cluster?

Yes. You may use tokens (${token}) in the config element for a cluster and define the token
property with different values for different server instance. Refer to administration reference
guide for more details. Here is a quick example where we are setting max heap size for a server
instance that is part of a cluster.

<domain ...>
<applications>...</applications>
<resources>...</resources>

<configs>
<config ... name="cluster1-config">
...
<java-config ...>
<jvm-options>${MAX_HEAP_SIZE}</jvm-options>
</java-config>
</config>
</configs>

<servers>
<server ... config-ref="cluster1-config" name="server1">
<system-property name="MAX_HEAP_SIZE" value="-
Xmx512m"/>
</server>
</servers>

<clusters>...</clusters>
<node-agents>...</node-agents>

<lb-configs>...</lb-configs>
<load-balancers>...</load-balancers>

</domain>

Q: Is it possible to specify JNDI name at cluster scope? For example, is it possible to use two
different connection pools for an application that is deployed to two clusters?

Yes. You may use the same tokenization strategy described above. Create a token for JDBC
resource pool name and then specify the token property with desired (different) values for the
two clusters. When application code looks up the JNDI name, it will end up connecting to two
different database pools for the two different clusters. For example,

<jdbc-resource ... jndi-name="jdbc/db" pool-name="$

{POOL_NAME}">
<cluster ... name="cluster1">
<system-property name="POOL_NAME" value="DerbyPool"/>
</cluster>
<cluster ... name="cluster2">
<system-property name="POOL_NAME" value="MySQLPool"/>
 </cluster>

Q: Say, I am dealing with a large

deployment involving many clusters
with multiple server instances (> 30). Is
there a way to optimize DAS's foot-
print?

During the cluster creation, instead of cloning the config for each cluster, use a reference to an
existing config (example, cluster1-config). If you need to customize a value, use tokens and
define them in each cluster. This will reduce the overall memory foot-print. Also, avoid
deploying the applications to DAS (on Administration Console, target named server)
unnecessarily. You may consider creating new domains (%asadmin create-domain) when
appropriate.

Q: Is there a way share a library across a cluster?

Yes. If you copy the library jar under the cluster's config/lib directory (example,
domains/domain1/config/cluster1-config/lib), the library will be available cluster wide. You will
need to update the classpath-prefix or classpath-suffix. Refer to Sivakumar's blog for more
classloader related issues. According to Siva's blog, library jars can be shared at domain and
application scope also in GlassFish V2.

 domains/domain1/lib - domain wide scope, common classloader add the

jars automatically
 domains/domain1/config/cluster1-config/lib - config wide, update
classpath-prefix or classpath-suffix
 domains/domain1/lib/applibs - application scope, added to application
class loader automatically
 domains/domain1/config/cluster1-config/lib/ext - adds to java.ext.dirs
automatically

Q: Do you need special packaging of applications to use them in cluster?

No. Application code level changes are not needed. GlassFish V2 offers in-memory replication
for high-availability with almost zero configuration also.

Q: Is it possible to install different versions on the same machine?

For GlassFish, this is not an issue. For SJS AS 9.1, you may also use the filebased installer. For
native package based installation, SJS AS 8.x will be upgraded to SJS AS 9.1 on the same
machine.

Q: Do you have support for monitoring?

GlassFish V2 offers a wide range of monitoring statistics. Statistics on server log, server runtime,
deployed applications, resources and transactions are available. Refer to the following blogs ([1],
[2]) for more details. Call Flow (tech tip) allows you to find out how a particular application
response is being processed across different containers of the Application Server. Web services
monitoring is also available (article) along with new improvements in CLI (blog) for
mornitoring.

Q: Do you have any support for large data center deployments?

Yes, you may use N1 Service Provisioning System with SJS AS. We are working on a new
plugin for SJS AS 9.1 (Sun's distribution of GlassFish V2) also.