Beruflich Dokumente
Kultur Dokumente
JBoss Clustering
High Availability and Load Balancing
Clustering Topics
Professional Open Source™
– Features
– How to use it
– Design overview
– Future
1
Load Balancing != Fault Tolerance
Professional Open Source™
State Replication
2
Clustering Has a Cost
Professional Open Source™
NetBoot
– Microkernel installed on every machine
– JBoss configuration & applications on a central web server
– Supply URL to micro-kernel
• JBoss boots up from micro-kernel
3
Cluster Manager: Net Boot
Professional Open Source™
NetBoot Client
NetBoot Client
NetBoot Server
Configuration
Files
Deployment
Packages
Web Server
Administrator
4
Cluster Manager: Farming
Professional Open Source™
Package Replication
Deploy Packages
on Single Cluster Node
Behaviour
– On boot up, new cluster node detects existing group members
– Group members are notified on node start/stop/failures
– No configuration of topology needed.
5
Cluster Fault Tolerance – Server Side
Professional Open Source™
6
Cluster Load Balancing
Professional Open Source™
HTTP LoadBalancing
– Extension of Failover mechanism
• Hardware solution: Cisco, etc.
• Software solutions: Apache, etc.
What to Cluster?
Professional Open Source™
Apache
Apache
AJP13
AJP13
HTTP
HTTP Session
Session HTTP
HTTP Session
Session
SLSB
SLSB SLSB
SLSB
EB
EB EB
EB
7
Specific Optimisations – SFSB
Professional Open Source™
With SFSB, the bean state must be replicated to other nodes after
each invocation
– Replication requires serialization which has a high cost
Net
Net Boot
Boot
Apache
Apache Load
Load Balancing
Balancing
AJP13
AJP13 HTTP
HTTP Failover
Failover
Replicated
Replicated HTTP
HTTP Session
Session
Global
Global JNDI
JNDI Context
Context
Tomcat
Tomcat Tomcat
Tomcat Tomcat
Tomcat
EJB
EJB EJB
EJB EJB
EJB
Database
Database Commit
Commit Option
Option ‘B’/’C’
‘B’/’C’
8
Using Clustering – Basic Steps
Professional Open Source™
Using Clustering
Professional Open Source™
deploy/cluster-service.xml
deploy/cluster-service.xml
<?xml
<?xmlversion="1.0"
version="1.0"encoding="UTF-8"?>
encoding="UTF-8"?>
<server>
<server>
<classpath
<classpatharchives="jbossha.jar"/>
archives="jbossha.jar"/>
<depends>JBOSS-SYSTEM:service=Naming</depends>
<depends>JBOSS-SYSTEM:service=Naming</depends>
<mbean
<mbeancode="org.jboss.ha.framework.server.ClusterPartition”
code="org.jboss.ha.framework.server.ClusterPartition”
name="JBOSS-SYSTEM:service=DefaultPartition"/>
name="JBOSS-SYSTEM:service=DefaultPartition"/>
<mbean
<mbeancode="org.jboss.ha.jndi.HANamingService"
code="org.jboss.ha.jndi.HANamingService"name="JBOSS-SYSTEM:service=HAJNDI">
name="JBOSS-SYSTEM:service=HAJNDI">
<attribute
<attributename="Port">1100</attribute>
name="Port">1100</attribute>
</mbean>
</mbean>
<mbean
<mbeancode="org.jboss.ha.framework.server.FarmMemberService“
code="org.jboss.ha.framework.server.FarmMemberService“
name="jboss:service=FarmMember,partition=DefaultPartition"
name="jboss:service=FarmMember,partition=DefaultPartition">>
<depends>jboss:service=DefaultPartition</depends>
<depends>jboss:service=DefaultPartition</depends>
<attribute
<attributename="PartitionName">DefaultPartition</attribute>
name="PartitionName">DefaultPartition</attribute>
<attribute
<attributename="FarmDeployDirectory">./farm</attribute>
name="FarmDeployDirectory">./farm</attribute>
<attribute
<attributename="ScannerName">jboss.deployment:type=DeploymentScanner,flavor=URL</attribute>
name="ScannerName">jboss.deployment:type=DeploymentScanner,flavor=URL</attribute>
</mbean>
</mbean>
</server>
</server>
9
Using Clustering – Protocol Stack
Professional Open Source™
deploy/cluster-service.xml
deploy/cluster-service.xml
<attribute
<attributename="PartitionConfig">
name="PartitionConfig">
<Config>
<Config>
<!--
<!--UDP:
UDP:ififyou
youhave
haveaamultihomed
multihomedmachine,
machine,
set
setthe
thebind_addr
bind_addrattribute
attributeto
tothe
theappropriate
appropriateNICNICIP
IPaddress
address-->
-->
<!--
<!-- UDP: On Windows machines, becauseof
UDP: On Windows machines, because ofthe
themedia
mediasense
sensefeature
feature
being broken with multicast (even after disabling media
being broken with multicast (even after disabling media sense) sense)
set
setthe
theloopback
loopbackattribute
attributetototrue
true-->
-->
<UDP
<UDPmcast_addr="228.1.2.3"
mcast_addr="228.1.2.3"bind_addr="192.168.0.20"
bind_addr="192.168.0.20"mcast_port="45566"
mcast_port="45566"
ip_ttl=”32"
ip_ttl=”32"ip_mcast="true"
ip_mcast="true"
mcast_send_buf_size="150000"
mcast_send_buf_size="150000"mcast_recv_buf_size="80000"
mcast_recv_buf_size="80000"
ucast_send_buf_size="150000"
ucast_send_buf_size="150000"ucast_recv_buf_size="80000"
ucast_recv_buf_size="80000"loopback="false"
loopback="false"/>/>
<PING
<PINGtimeout="2000"
timeout="2000"num_initial_members="3"
num_initial_members="3" up_thread="true"
up_thread="true"down_thread="true"
down_thread="true"/>/>
<MERGE2
<MERGE2min_interval="5000"
min_interval="5000"max_interval="10000"
max_interval="10000"/>/>
<FD
<FDshun="true"
shun="true"up_thread="true"
up_thread="true"down_thread="true"
down_thread="true"timeout="2500"
timeout="2500"max_tries="5"
max_tries="5"/>/>
<VERIFY_SUSPECT
<VERIFY_SUSPECTtimeout="3000"
timeout="3000"num_msgs="3”
num_msgs="3”up_thread="true"
up_thread="true"down_thread="true"
down_thread="true"/>/>
<pbcast.NAKACK
<pbcast.NAKACKgc_lag="50"
gc_lag="50"retransmit_timeout="300,600,1200,2400,4800"
retransmit_timeout="300,600,1200,2400,4800"
max_xmit_size="8192"
max_xmit_size="8192"use_mcast_xmit="true"
use_mcast_xmit="true"up_thread="true"
up_thread="true"down_thread="true"
down_thread="true"/>/>
<UNICAST
<UNICASTtimeout=”600,1200,2400"
timeout=”600,1200,2400"window_size="100"
window_size="100"min_threshold="10"
min_threshold="10"down_thread="true"
down_thread="true"/>/>
<pbcast.STABLE
<pbcast.STABLEdesired_avg_gossip="20000”
desired_avg_gossip="20000”up_thread="true"
up_thread="true"down_thread="true"
down_thread="true"/>/>
<FRAG
<FRAGfrag_size="8192"
frag_size="8192"down_thread="true"
down_thread="true"up_thread="true"
up_thread="true"/>/>
<pbcast.GMS
<pbcast.GMSjoin_timeout="5000"
join_timeout="5000"join_retry_timeout="2000"
join_retry_timeout="2000"shun="true"
shun="true"print_local_addr="true"
print_local_addr="true"/>/>
<pbcast.STATE_TRANSFER
<pbcast.STATE_TRANSFER up_thread="true" down_thread="true"/>/>
up_thread="true" down_thread="true"
</Config>
</Config>
</attribute>
</attribute>
EJB Configuration
jboss.xml
jboss.xml
<jboss>
<jboss>
<secure>false</secure>
<secure>false</secure>
<enterprise-beans>
<enterprise-beans>
<session>
<session>
<ejb-name>SLSB</ejb-name>
<ejb-name>SLSB</ejb-name>
<jndi-name>SLSB</jndi-name>
<jndi-name>SLSB</jndi-name>
<clustered>True</clustered>
<clustered>True</clustered>
</session>
</session>
</enterprise-beans>
</enterprise-beans>
</jboss>
</jboss>
10
JBossCluster – Using It…
Professional Open Source™
EJB Configuration
jboss.xml
jboss.xml
<enterprise-beans>
<enterprise-beans>
<session>
<session>
<ejb-name>SFSB</ejb-name>
<ejb-name>SFSB</ejb-name>
<jndi-name>SFSB</jndi-name>
<jndi-name>SFSB</jndi-name>
<clustered>True</clustered>
<clustered>True</clustered>
<cluster-config>
<cluster-config>
<partition-name>MyPartition</partition-name>
<partition-name>MyPartition</partition-name>
<home-load-balance-policy>
<home-load-balance-policy>
org.jboss.ha.framework.interfaces.RoundRobin
org.jboss.ha.framework.interfaces.RoundRobin
</home-load-balance-policy>
</home-load-balance-policy>
<bean-load-balance-policy>
<bean-load-balance-policy>
org.jboss.ha.framework.interfaces.FirstAvailable
org.jboss.ha.framework.interfaces.FirstAvailable
</bean-load-balance-policy>
</bean-load-balance-policy>
</cluster-config>
</cluster-config>
</session>
</session>
HTTPSession Replication
Professional Open Source™
<jboss-web>
<jboss-web>
<replication-config>
<replication-config>
<replication-trigger>SET</replication-trigger>
<replication-trigger>SET</replication-trigger>
<replication-type>ASYNCH</replication-type>
<replication-type>ASYNCH</replication-type>
</replication-config>
</replication-config>
</jboss-web>
</jboss-web>
11
HTTP Session Replication
Professional Open Source™
Replication Trigger
– SET_AND_GET
– SET_AND_NON_PRIMITIVE_GET
– SET
Replication Type
– SYNCH – synchronous
– ASYNCH - asynchronous
HAInvokerProxy
– Proxy behavior is enhanced to deal with clustered invokers
– Interceptor contains list of available nodes
– Interceptor chooses server to invoke on through load-balance policy
List
List of
of
nodes
nodes
HA
RMI JMX
Cluster
Client
Container EJB Tx Security Client
Interceptor HA
RMI JMX
12
JBossCluster Uses Client Side Proxy
Professional Open Source™
HARMIResponse
– Client-side cluster interceptor maintains a list of nodes the MBean (EJB,
JNDI, etc…) is available.
– If cluster topology changes, new list is piggy-backed with invocation
response to client.
– HA invoker manages this.
List
List of
of nodes
nodes
HA
RMI JMX
Cluster
Client
Container EJB Tx Security Client
Interceptor HA
RMI JMX
package
package org.jboss.invocation.jrmp.interfaces;
org.jboss.invocation.jrmp.interfaces;
/**
/**
** JRMPInvokerProxy,
JRMPInvokerProxy, local local toto the
the proxy
proxyandand is is capable
capable of
ofdelegating
delegating to
to local
local and
and JRMP
JRMP
implementations
implementations
*/*/
public
public classclassJRMPInvokerProxyHA
JRMPInvokerProxyHAextends extends JRMPInvokerProxy
JRMPInvokerProxyimplements
implements Externalizable
Externalizable
{{
//// Public --------------------------------------------------------
Public --------------------------------------------------------
protected
protected ArrayList
ArrayListtargets
targets ==null;
null;
protected
protected LoadBalancePolicy
LoadBalancePolicyloadBalancePolicy;
loadBalancePolicy; The
The HA
HA version
version
contains
contains aa list
list of
of
protected
protected transient
transientlong
long currentViewId
currentViewId ==0;
0; invoker
invoker proxies
proxies
public
public static
static final
final HashSet
HashSetcolocation
colocation == new
new HashSet();
HashSet();
13
Client Side Proxy for Clustering
Professional Open Source™
public
publicObject
ObjectgetRemoteTarget()
getRemoteTarget(){{
ifif(targets.size()
(targets.size()==
==0)
0)
return
returnnull;
null;
synchronized
synchronized(targets)
(targets)
Pluggable
Pluggable load
load
return balance
balance policies
policies are
are
returnloadBalancePolicy.chooseTarget(targets);
loadBalancePolicy.chooseTarget(targets);
}} responsible
responsible forfor
picking
picking the
the target
target
/**
/**
**Returns
Returnswether
wetherweweare
arelocal
localto
tothe
theoriginating
originatingcontainer
containeror
ornot.
not.
*/*/
public
publicboolean
booleanisLocal(Invocation
isLocal(Invocationinvocation)
invocation) {{
return
returncolocation.contains(invocation.getObjectName());
colocation.contains(invocation.getObjectName());
}}
public
publicObject
Objectinvoke(Invocation
invoke(Invocationinvocation)
invocation)throws throwsException
Exception
{{
////we
wegive
givethe
theopportunity,
opportunity,to
toany
anyserver
serverinterceptor,
interceptor,to
toknow
knowififthis
thisaa
////first
firstinvocation
invocationtotoaanode
nodeor
orififititis
isaafailovered
failoveredcall
call
int
intfailoverCounter
failoverCounter==0; 0;
invocation.setValue
invocation.setValue("FAILOVER_COUNTER",
("FAILOVER_COUNTER",new newInteger(failoverCounter),
Integer(failoverCounter),invocation.AS_IS);
invocation.AS_IS);
////optimize
optimizeififcalling
callinganother
anotherbean
beanin
insame
sameEJB-application
EJB-application
ifif(isLocal(invocation))
(isLocal(invocation)){{
return
returnInvokerInterceptor.getLocal().invoke(invocation); We
InvokerInterceptor.getLocal().invoke(invocation); We still
still collocate
collocate ifif we
we are
are
}} local
local
else
else
{{
////We
Weare
aregoing
goingto togo
gothrough
throughaaRemote
Remoteinvocation,
invocation,switch
switchto
toaaMarshalled
MarshalledInvocation
Invocation
MarshalledInvocation
MarshalledInvocationmi mi==new
newMarshalledInvocation(invocation);
MarshalledInvocation(invocation);
////Set
Setthe
thetransaction
transactionpropagation
propagationcontext
context
mi.setTransactionPropagationContext(getTransactionPropagationContext());
mi.setTransactionPropagationContext(getTransactionPropagationContext());
mi.setValue("CLUSTER_VIEW_ID",
mi.setValue("CLUSTER_VIEW_ID",new newLong(currentViewId));
Long(currentViewId));
Invoker
Invokertarget
target==(Invoker)getRemoteTarget();
(Invoker)getRemoteTarget();
14
Client Side Proxy for Clustering
Professional Open Source™
while
while(target
(target!= !=null)
null){{
try
try{{
HARMIResponse
HARMIResponsersp rsp==(HARMIResponse)((MarshalledObject)target.invoke(mi)).get();
(HARMIResponse)((MarshalledObject)target.invoke(mi)).get();
ifif(rsp.newReplicants
(rsp.newReplicants!= !=null)
null){{
setTargets(rsp.newReplicants);
setTargets(rsp.newReplicants);
currentViewId
currentViewId==rsp.currentViewId;
rsp.currentViewId; Server
Server piggy-backs
piggy-backs
}} information
information about
about the
the
return rsp.response;
return rsp.response; topology
topology
}}
////ALL
ALLCATCH
CATCHJUST JUSTGOES
GOESON ON
catch
catch(java.rmi.ConnectException
(java.rmi.ConnectExceptionce){}ce){}
////IfIfwe
We fail over here by taking
We fail over here by taking
wereach
reachhere,
here,this
thismeans
meansthat
thatwe
wemust
mustfail-over
fail-over aa new
remoteTargetHasFailed(target); new target
target and
and looping
looping on
on
remoteTargetHasFailed(target);
target the
the while
while (target
(target !=null)
!=null)
target==(Invoker)getRemoteTarget();
(Invoker)getRemoteTarget();
failoverCounter++;
failoverCounter++;
mi.setValue
mi.setValue("FAILOVER_COUNTER",
("FAILOVER_COUNTER",new newInteger(failoverCounter),
Integer(failoverCounter),invocation.AS_IS);
invocation.AS_IS);
}}
////ififwe
weget
gethere
herethis
thismeans
meanslist
listwas
wasexhausted
exhausted
throw
thrownew newjava.rmi.RemoteException("Service
java.rmi.RemoteException("Serviceunavailable.");
unavailable.");
}}
}}
Conclusion
Professional Open Source™
15