tomcat native with user-specified open ssl libraries

quiet

looks like that tomcat native can not work without the openssl support. so we are
going to build ssl also as part of our apr package. as usual we will start with
apr.
download the apr 1.2.7 bundle
(1)
./configure --prefix=/oad/x/apr
make
make install
(2)
then go a grab a copy of openssl sources from www.openssl.org. i get 0.9.7f
tarball, the version that corresponds to my fedora core 4 install.

./config --prefix=/oad/x/apr --openssldir=/home/rajeevj01/x/tmp shared

make
make test
make install

the documents and tests would be installed in x/tmp location and libraries are
installed in /oad/x/apr location.
(3)

next, we un-tar and build the tomcat-nativve1.1.2 source ball with the following
options.

[rajeevj01@bdc31035e native]$ ./configure --prefix=/oad/x/apr --with-

ssl=/oad/x/apr --with-apr=/oad/x/apr --with-java-home=/home/rajeevj01/sw/jdk1.5
--with-java-platform=2

(4) now, set up the tomcat server to use this native library , add the following
in $tomcat_home/bin/setenv.sh script (or equivalent)
----------------------------------------------------------------------------------
--------
ld_library_path=/oad/x/apr/lib:$ld_library_path
export ld_library_path
----------------------------------------------------------------------------------
---------

rajeevj01 at 2:41:57 am edt link to this entry | blog about this entry | notify
aol
this entry has 0 comments: add your own

friday, june 16, 2006

building apr library on feodra core 4
worried

the libtcnative (the native tomcat connector) requires the openssl and apr
libraries headers to be built. so the first step is : download and install the
apr. i head over to http://apr.apache.org and grab the latest tarball. untar it
and just run configure with the required prefix
1051 ./configure --prefix=/oad/x/apr
 1052 make
1053 make install

apr is now installed. we are planning to use ajp connector and so we build the
non-ssl version of tomcat ajp/13 connector.

[rajeevj01@bdc31035e native]$ ./configure --prefix=/home/rajeevj01/x/tcnative126

--with-apr=/home/rajeevj01/x/apr126 --without-ssl --with-java-
home=/home/rajeevj01/sw/jdk1.5 --with-java-platform=2

if we do not want to use ssl, we can go ahead and remove the -lssl flag from the
makefile also. but if you do that, the tomcat can not find the apr installed. ssl
must be enabled for apr (why?)
there are some version issues with tomcat 1.1.1(bundled with tomcat 5.5.16) and
apr 1.2.7 , so download the tomcat 1.1.2 sources from
http://tomcat.heanet.ie/native/1.1.2/source/

1100 make
1101 make install

this builds the native tc library. once we have the library, the next big question
is how to use it.

rajeevj01 at 4:18:18 am edt link to this entry | blog about this entry | notify
aol
this entry has 0 comments: add your own

wednesday, june 14, 2006

ethereal for http packet captures
silly

we need some packet capture tool to analyze the server headers and what the
browser is sending to the server. after using couple of useless freeware tool, now
i have settled on ethereal. it can do a lot of stuff and options seem quite
daunting but the good thing is i can make it work to my purpose with minimal fuss.

download and install ethereal and read the user guide anyway. to get up to speed
very quickly,
1) start ethereal , to start capture , go to menu | capture |options
2) on this screen, select the interface (like vpn adapter or intel ethernet card
etc)
3) in filter box, type tcp port 5400
one more filter that you can try is tcp port 5400 and host 10.141.11.189
etc.

this will capture all traffic to and from that host going through your designated
interface.

4) i do not like to see lots of colors , so menu | view | colorize packet list
this lets you switch on/off the packet colors in display.

5) you can save the packets captured during the session with menu | file | export

6) in the capture window, each packet can be probed further for frame details etc.
all in all a really useful tool and w/o any freeware shit!!

rajeevj01 at 7:29:51 am edt link to this entry | blog about this entry | notify
aol
this entry has 0 comments: add your own

monday, june 12, 2006

webchat gc tuning
loopy

okay, so we have done one round of our jvm garbage collection data capture and
this is what i can infer:

+ the eden space size is small , so minor collections are happening every now and
then.
+ survivor space is not big, so after minor collections overflow is happening to
tenured space at a very young age

so this means, i should start with a bigger young generation. plus i am running a
server application, so i will go with same -xms and -xmx options.

here are my gc parameters

==============================================================
-xss256k -xms512m -xmx512m -xx:newratio=2 -xx:survivorratio=6 -xx:+useparallelgc
=============================================================
i may want to try with -xx:newsize=256m -xx:maxnewsize=256m

so that young generation size is half the heap size to start with. right now, i
will settle for young:tenure ratio as 1:2.

with young generation = half the heap size we get

young = 256 mb, eden = 6 * (256/8) == 192 mb. survivor space is 32 mb each.

=================================================================
to check the jvm stats, we use
$jstat -gcutil <vmid> <delay in millis> > jstat.log
like:
$jstat -gcutil <vmid> 200 > jstat.log

we can get the vmid using the $jps command.

rajeevj01 at 7:22:48 am edt link to this entry | blog about this entry | notify
aol
this entry has 0 comments: add your own

tuesday, june 6, 2006

how to make the eclipse profiler work for you
silly
first things first. we want to do remote profiling and we do not want lots of over
heads. the option was there to run one of the standard tools like optimizeit or
jboss profiler etc. however, i could not set up jboss profiler quickly enough and
optimizeit would have required tomcat to start with its own audit system.

so finally i settled for eclipse plug-in profiler. this plugin can attach to
remote servers. it just needs one library file and few command line
options on server.

required server changes

a) catalina.sh script or the script that starts you server will require the
following chages

this one is for cpu profiling, see the link for more details on memory profiling
etc. http://eclipsecolorer.sourceforge.net/index_profiler.html

java_opts="-xrunprofilerdll:1 -xbootclasspath/a:jakarta-
regexp.jar:profiler_trace.jar:commons-lang.jar \
-d__profiler_package_filter=__a__%
mainclass%:__m__sun.:__m__com.sun.:__m__java.:__m__javax.:__m__org.apache. \
-d__profiler_timing_method=1"

for our setup, we need to include these options in post.tcl file along with other
java command line options . this is a jvmpi plugin so some of the gc options you
require may not be supported.

b) copy common-lang.jar, jakarta-regex.jar and profiler_trace.jar to

$tomcat_home/bin folder.
in our case, this may mean the folder from where we bootstrap the tomcat
application. so copy the 3 jar files to the port folder like to 48800/ folder.

c) compile the agent using the following script ( better to compile on target
server ). the default agent is compiled with gcc 3.2 and it may not work on all
systems.

g++ -o0 -dlinux -shared -wall -i. -i/home/rajeevj01/sw/jdk1.5/include

-i/home/rajeevj01/sw/jdk1.5/include/linux profilerdll.cpp -o libprofilerdll.so

d)copy the agent file to jdk_home/jre/lib/i386 folder

or you can put it in ld_library_path also.

after making the changes, start the server. check logs for any errors. once server
startup is successful, we are ready to attach to this server to get profiling
data.

eclipse changes :

you need to do few changes to your eclipse before you can open the profiler
window.

a) we require a new plugin. download it from the web link

http://eclipsecolorer.sourceforge.net/index_profiler.html and copy to your
eclipse plug-ins folder. then from menu | windows | open perspective | profiler
b) you need to create a launcher configuration to connect to remote server. to do
that , open menu | run and choose run .. option. under remote profiler node,
create a new configuration by clicking new button. add the server name and click
on run button. thats it!!!!

c) you have to patch the plugin for eclipse 3.1 version.

link is
http://sourceforge.net/tracker/index.php?func=detail&aid=1202373&group_id=48823&at
id=454283

d) you may require unrar on windoze machines. see unxutils zip on sourceforge.
http://unxutils.sourceforge.net/

rajeevj01 at 8:15:30 am edt link to this entry | blog about this entry | notify
aol
this entry has 0 comments: add your own

monday, june 5, 2006

roadmap : for doing server performance tuning
silly

1) chat is working okay for so-called 400 simultaneous users. we do not know when
we get the apache on chdl-m05 blue box. so in the mean-while this is the plan i
have for performance testing

a) we need to profile the application and see what is taking time. right now,
there are too many synchronization blocks around!! for profiling , we can take a
look at jprobe/optimizeit or jboss profiler, whatever works for us.

b) shut down apache debugging and dns lookup etc. point is you run a very lean
server. should we compile the mod_status module at this point ? linux server data
should give enough indications at this point!

c) we need to do tomcat and jvm monitoring. we can do the monitoring using mc4j.
d) we capture the linux box stats for different combinations.

we need snapshots of following data

0) remove all the debugging options and make a lean server.
1) active thread count

2) open sockets
this number should be less than my $ulimit -n output.
3) cpu utilization
4) memory ( i should not be swapping )

5) jvm heap status

we can use the jstat tool. what we want to understand is how much of the heap we
are actually using under projected loads. this will give us a better idea of our
heap usage. also, i would like to know if i am doing lots of gc around.

the link for jstat is

http://java.sun.com/j2se/1.5.0/docs/tooldocs/share/jstat.html
see the effect of using -server option etc.

6) apache status
we can use mod_status for this purpose (?)

the parameters to vary are

1) apache server threads + number of child processes

2) jvm heap size, -server option, -xincgc option
3) thread stack size for apache as well as jvm threads

this entry has 3 comments: hide recent | add your own

kernel-imposed limits on the system

-------------------------------------------
1.fd-max can be got at: cat /proc/sys/fs/file-max
2. threads-max can be got at: cat /proc/sys/kernel/threads-max

comment from shrutikbdc - 6/5/06 7:56 am

top command
-----------------

top -b -d 2 > top_stats

this redirects the output of top to a file in plain text format every 2
seconds

comment from shrutikbdc - 6/5/06 7:45 am

memory

vmstat <seconds> ; say "vmstat 2" gives stats every 2 seconds

the following can be captured from vmstat output:

procs
r: the number of processes waiting for run time.
b: the number of processes in uninterruptable sleep.
w: the number of processes swapped out but otherwise runnable. this
field is calculated, but linux never desperation swaps.

memory
swpd: the amount of virtual memory used (kb).
free: the amount of idle memory (kb).
buff: the amount of memory used as buffers (kb).

swap
si: amount of memory swapped in from disk (kb/s).
so: amount of memory swapped to disk (kb/s).

io
bi: blocks sent to a block device (blocks/s).
bo: blocks received from a block device (blocks/s).

system
in: the number of interrupts per second, including the clock.
cs: the number of context switches per second.
 cpu
these are percentages of total cpu time.
us: user time
sy: system time
id: idle time

comment from shrutikbdc - 6/5/06 7:26 am

mod_jk connector to plug jetty/tomcat in apache

loopy

we are going to use mod_jk to connect apache and jetty.

step1) download and copy mod_jk.so to /usr/lib/httpd/modules
download link is:
http://tomcat.apache.org/download-connectors.cgi
beware : do not download jk2 module, download jk module. this is very confusing!
jk is the one that we need not jk2. you may want to rename the jk .so file.

step2) add loadmodule directive to the apache httpd.conf file

[ add where other loadmodule directives are ]
loadmodule jk_module modules/mod_jk.so

beware : the way to add modules may differ in apache 1.3 and apache 2. i dont
think you require anything more than above one line with apache 2 that has been
enabled with dso support.

question: how do i check if my apache has been compiled with dso support?
[rajeevj01@bdc31035e installs]$ /usr/sbin/httpd -l
should list mod_so.c

step3) go to /etc/httpd/conf.d directory

[ comment: this may be specific to fc4 , if such a folder is not there then you
have to specify the path to mod_jk.conf in apache httpd.conf file using an include
<path-to-conf-file> directive, like,
include mod_jk.conf
]

create a mod_jk.conf file there with the following content

--------------------------x------------------------------------------------
# rjha: adding the mod_jk options
jkworkersfile /etc/httpd/conf/workers.properties
jklogfile /home/rajeevj01/logs/mod_jk.log
jkloglevel debug
jklogstampformat "[%a %b %d %h:%m:%s %y] "
jkmount /servlet/* ajp13

------------------------------x-----------------------------------------------

replace /servlet/* with your dymanic scripts context path.

step 4) create the logs dir if that does not exist

step5) create the workers.properties file with following info
-------------------------x-----------------------------------------------------
ps=/

# rjha: sample worker.properties lifted from geronimo

worker.list=ajp13
worker.ajp13.port=8009
worker.ajp13.host=localhost
worker.ajp13.type=ajp13

step 6) now configure your servlet container to listen on ajp13 port.

a) for jetty do the changes in $jetty_home/etc/jetty.xml

rjha : uncommenting the ajp13 listener -->

<call name="addlistener">
<arg>
<new class="org.mortbay.http.ajp.ajp13listener">
<set name="port">8009</set>
<set name="minthreads">5</set>
<set name="maxthreads">20</set>
<set name="maxidletimems">0</set>
<set name="confidentialport">443</set>
</new>
</arg>
</call>
<!-- rjha : uncommented -->

b) for tomcat , do the changes in server.xml file. comment out the http connector
and uncomment the ajp13 connector portions.

thats it. now next set of things to do :

1) know more about jetty logging options
2) know more about mod_jk logging options
3) how servlets are deployed on jetty, the context mappings

then we can start changing and debugging the jetty code.

deploying the dit dot servlet

happy

the dit dot servlet just prints one dot every 1/2 seconds. point is to replace the
chat pipe servlet with this dit dot servlet on my machine and do the stress
testing. i am going to deploy the dit dot servlet on my development box, with
tomcat (ditch jetty for the moment). so the next steps in my wave of experiments
is

1) download and install the latest tomcat (preferably from sources ?) no ? binary
should do at the moment. download sources as a side-kick.
2) deploy the dit dot servlet with tomcat container
3) deploy the rest of the chat apps
4) now do the jvm stats monitoring , things like, how much of memory i am
munching, how many threads etc. need to find out more about this.

============= web.xml file ==============================

<?xml version="1.0" encoding="iso-8859-1"?>
<!doctype web-app
public "-//sun microsystems, inc.//dtd web application 2.3//en"
"http://java.sun.com/dtd/web-app_2_3.dtd">

<web-app>

<servlet>
<servlet-name>vanillapipeservlet</servlet-name>
<servlet-class>com.bobo.chat.vanillapipeservlet</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>vanillapipeservlet</servlet-name>
<url-pattern>/pipe</url-pattern>
</servlet-mapping>
</web-app>

we want to monitor our tomcat application jvm. the development boxes i have are
linux boxes. on these machines, we just have to add the following 4 lines to
java_opts in catalina.sh script
====================================================
# rjha: supply this argument to jvm on start-up
# we want to monitor tomcat jvm remotely
# development system, no password!
java_opts="-dcom.sun.management.jmxremote \
-dcom.sun.management.jmxremote.port=9421 \
-dcom.sun.management.jmxremote.ssl=false \
-dcom.sun.management.jmxremote.authenticate=false"
===================================================
no authentication, no passwords, i dont give a damn at the moment.
then you should have the jdk 1.5 installed on your machines.
on the monitoring machine, just fire jconsole and connect to target vm.
right now, remote monitoring not working, only local monitoring is working.
i can monitor the number of threads now!!!!

areas of performance monitoring of tomcat

silly

sometimes you would like to know how your tomcat server is doing. how many threads
are open on container etc. there are two ways you can do this

1) use the server status link on tomcat manager application

log into tomcat manager application. you will see a server status link on top of
application's list.

2) use one of the jmx monitoring tools. catalina container properties can be
exposed as jmx beans and we can use one of the jmx monitoring tools. i cover
jmx tools list in a separate document.
mc4j one of the tools that we are going to use (covered in separate blog) . we
need to check out visualgc also. plus there is a tool called hat for heap
analysis. i need to pay attention to jvm tunining next. but for the starters, here
is the points to ponder when tuning tomcat

1) server configuration file. need more info on this from the tomcat user's guide.
http://tomcat.apache.org/tomcat-5.5-doc/config/index.html
the config parameters should tell me about
maximum processes, acceptcount, max-threads etc. need to do rtfm here.

2) jvm parameters may play a big role here, so we need some info on jvm tuning
also
+ thread stack size
+ head min/max size
+ gc (incremental/ how much allocated to generational space)
+ -x arguments to jvm @ startup
-xincgc (for incremental gc)

3) threading support
how do i find out if my vm is using nptl libs or not ? start with the linux kernel
version
$uname -r
if you are running a 2.6.x kernel then you should be ok.
use this small program to print java library paths
================================================
public class printlibrarypath {
public static void main(string[] args) {
system.out.println(system.getproperty("java.library.path"
));
}
}
====================================================
then we can use ldd to see if nptl is linked in our jvm or not.
[rajeevj01@bdc31035e jdk1.5]$ ldd
/home/rajeevj01/bin/jdk1.5/jre/lib/i386/client/libjvm.so
linux-gate.so.1 => (0x00835000)
libm.so.6 => /lib/libm.so.6 (0x00111000)
libdl.so.2 => /lib/libdl.so.2 (0x003e4000)
libpthread.so.0 => /lib/libpthread.so.0 (0x00219000)
libc.so.6 => /lib/libc.so.6 (0x0022b000)
/lib/ld-linux.so.2 (0x00864000)
[rajeevj01@bdc31035e jdk1.5]$ nm /lib/libpthread.so.0 | grep nptl
nptl is linked!!
----------------------------------------------------------------------------------
-----------------

+ switch off servlet reloading

+ compile jsps a-priori in production usage. use jspc compiler.
+ we are not using tomcat in stand-alone mode, so *comment* out that connector.

we need something to monitor the tomcat! something to monitor the jvm.

use jconsole to see how the jvm is performing. use servlet statistics page to see
how the tomcat installation is doing.

tomcat configurations
loopy

+ you should not allow any tracing/debugging options

+ you should disable the servlet re-loading
+ the number of threads should be tuned on the server. is there a way to increase
the maxthreads programatically ?

maxkeepaliverequests

the maximum number of http requests which can be pipelined until the connection is
closed by the server. setting this attribute to 1 will disable http/1.0 keep-
alive, as well as http/1.1 keep-alive and pipelining. setting this to -1 will
allow an unlimited amount of pipelined or keep-alive http requests. if not
specified, this attribute is set to 100.
maxsparethreads

the maximum number of unused request processing threads that will be allowed to
exist until the thread pool starts stopping the unnecessary threads. the default
value is 50.
maxthreads

the maximum number of request processing threads to be created by this connector,

which therefore determines the maximum number of simultaneous requests that can be
handled. if not specified, this attribute is set to 200.
minsparethreads

the number of request processing threads that will be created when this connector
is first started. the connector will also make sure it has the specified number of
idle processing threads available. this attribute should be set to a value smaller
than that set for maxthreads. the default value is 4.

descriptionacceptcount

the maximum queue length for incoming connection requests when all possible
request processing threads are in use. any requests received when the queue is
full will be refused. the default value is 10.

java.sun.com has lots of articles on how to tune jvm settings. crux of the matter
is that you should be able to
+ size up a heap for your purposes
+ thread stack size and numbers for your purpose

right now, we are running with

-xms128m -xmx256m -xincgc -xss128k

you can do the required arithmetic. try tuning with server mode also, the warm-up
may take some time but after that it should be ok!
use some jmx remote management tool like mc4j etc to monitor the health of the jvm
as well as tomcat instance running on top of it.

this is a good reference to see what all options are available with jvm
http://blogs.sun.com/roller/resources/watt/jvm-options-list.html
this one discusses the jvm threads
http://java.sun.com/docs/hotspot/threads/threads.html
and this one tuning the gc
http://developers.sun.com/techtopics/mobility/midp/articles/garbagecollection2/
and finally we have the jvm tuning whitepaper
http://java.sun.com/performance/reference/whitepapers/tuning.html

threads tuning params on linux 2.6

the issues here is to find out the system limits and application limits imposed on
a running program on linux 2.6 kernel series. later we see how we can tune the
parameters to spawn a lot of threads and do fast context switching as well.

* application limit tuning *

==================
1) ulimit -s and ulimit -u
ulimit has options to set thread stack size.
check by typing ulimit -a. looks like this is not system wide setting but just
applies to applications started in that bash shell. you can not go beyond the
hard-limits of a parameter. the hard limits can be checked by typing ulimit -ah.

2) you can set linux thread stack size using

$ulimit -s <thread statck size>

then you make sure that you use the same value with java -xss option when starting
your jvm. that way the thread runs in a limited stack space and more and more
threads can be spawned.

* system limit tuning *

================
1) check the system wide settings in following places
$ cat /proc/sys/kernel/threads-max
$grep totalpages /var/log/dmesg (totalpages/16 is the max # threads possible)
$ cat /proc/sys/kernel/pid_max
$cat /proc/sys/vm/max_map_count (* need to figure out what exactly is this *)

* test program *
=============

3) run the test program to see how many threads are supported on your machine

idea is , final limit on # max threads should only be imposed by the amount of ram
in system.

4) how to do system wide ulimit settings ?

check the contents of file
/etc/security/limits.conf

bumping the ulimit

happy
the ulimit command has two variants. ulimit-ah and ulimit -as. h is for the hard
options (what is set in the config files for the user) and soft is the current
value.
now if you are root its all pretty ok,you can just use ulimit command to increases
whatever you want.

for other logins, we can not bump up the resources more than the hard limits.
for such cases, we have to first increase the hard-limits by changing the
/etc/security/limits.conf on linux 2.6 systems.
=================================================
edit /etc/security/limits.conf and add the lines:rajeevj01 soft nofile 1024
rajeevj01 hard nofile 65535

also check the values in /proc/sys/fs/file-max.

[ on 2.6.x systems, they are pretty ok!]
==================================================

after bumping up the hard limits, use ulimit command for user login

$ ulimit -n unlimited

a: max address space (kb)

c: max core file size (kb)
d: max data size (kb)
f: maximum filesize (kb)
m: max locked-in-memory address space (kb)
n: max number of open files
r: max resident set size (kb)
s: max stack size (kb)
t: max cpu time (min)
u: max number of processes
k: file creation mask, set by umask(2) .
l: max number of logins for this user
p: process priority, set by setpriority(2) .

for example, l2d2048n5 is a valid limits_string. for reading convenience, the

following entries are equivalent:

username l2d2048n5
username l2 d2048 n5

so far not able to locate any remote monitoring tools for linux box. most of the
scripts there can be put as part of cron and flushed to log files. i can use a
user-specific crontab files!!
+ install the redhat sysstat package $yum install sysstat
+ need to do some research on sar and crond to achieve the desired results.
other useful commands are
ps,iostat, vmstat, top ,mpstat [ but the hitch is i have to be on the machine.]

one good link is http://people.redhat.com/alikins/system_tuning.html

apart from that try reading the redhat sysadmin guide also
http://www.redhat.com/docs/manuals/linux/rhl-9-manual/admin-primer/index.html
need to do some research on flushing cron jobs
so its agreed , to use netstat to monitor the number of open sockets and the above
utilities to monitor the system health. since i am going to run all the
experiments on lan, i am not going to monitor the bandwidth usage as such!

i still can not locate any tool that allows for remote monitoring. anyways, i am
not going to break my head over it. mostly, this is what i need to do
a) how to monitor
+ open processes
+ open threads
+ memory
+ i/o, open sockets and open file descriptors.

i think the most important is memory monitoring. when i check my /proc parameters,
i dont think i am going to run out of fds and processes in a hurry! so kernel-
limits are all ok. so we got to watch memory and cpu after bumping the user limits
of fd and thread stack size using ulimit command in a shell.
to see the open sockets on my machine , i use this command lifted from red hat's
site:
=====================x===================================

one of the most twisted, but useful invocations is:

$netstat -a -n|grep -e "^(tcp)"| cut -c 68-|sort|uniq -c|sort -n
this will show you a sorted list of how many sockets are in each connection state.
=====================x====================================
to see the memory and cpu usage , i am relying on top. for a quick snapshot , you
may consider using vmstat also.
$vmstat 1 5
gives 5 readings with 1 sec interval.

b) gnome-system-monitor is a graphic utility that can be installed using yum if it

is not there on your system. i have fc4 and gnome-system-monitor is installed.
this shows the process list and memory usage.

c) top, watch, free , ps etc. are usual unix commands.

the following gives a recipe to bump the number of allowed file descriptors.
1) setting up the hard limit in /etc/limits.conf
rajeevj01 nofile hard 2048
2) then executing ulimit -n 2048 in a shell should increase the number of file
descriptors. on recent systems , /proc/sys/fs/file-max is good enough for all the
purposes (limited by hardware only). on my machine it is ~ 101k.
3) apache or tomcat uses some start-up scripts like apachectl or catalina.sh etc
(or if started from /etc/rc.d/httpd etc)
there, in the beginning of such scripts we can set the soft ulimit. (make sure the
hard-limits have been bumped for apachectl user)
=========================================
ulimit -s -n `ulimit -h -n`
=========================================
if you are using pam, then you also need to do the following to honor the
limits.conf
edit /etc/security/limits.conf and add to any config file in /etc/pam.d/ that
handles logins ("login" and "ssh" example):

session required /lib/security/pam_limits.so

1) packet analyzer/tcp mon

to see what the server is sending and what the browser is receiving. should show
the files/size/transfer time plus all the headers. useful for debugging.
2) bandwidth monitor and throttler
what is my download speed right now, how can i make my speed simulate a modem
speed.
3) network load/stress monitor
how is my network pipe doing. what kinds of stress it is subjected to. is my
webserver saturating my network card?
4) javascript debugger
how to use venkman. microsoft script debugger. get rid of alert debugging.
5) javascript compiler/size muncher
6) web server log analyzer, web server and servlet container tuning
tomcat, apache, weblogic
7) os resources monitoring and tuning.

apache connection tuning

silly

the apache 2 has 256 maxclients limit out-of-box. the compiled-in hard-limit of
maxclients(serverlimit) is 20,000 (way beyond our purpose). first we will look at
some of the apache directives used in httpd.conf to tune the server.

1) serverlimit : this goes hand-in-hand with the maxclients.

for prefork mpm , this is number of maxclients. in case of worker mpm , this
together with threadsperchild determines the maxclients
for worker mpm
maxclients = threadsperchild * serverlimit

2) threadperchild
number of threads created for a apache child process. default is 25. hard limit
can be 64 determined by the threadlimit.

3) maxclients : this is the number of simultaneous requests that apache can

handle.

below are some of the real world setting examples (this is one of the busiest
sourceforge mirrors)
=============================================
<ifmodule worker.c>
serverlimit 900
startservers 10
maxclients 28800
minsparethreads 25
maxsparethreads 75
 threadsperchild 32
maxrequestsperchild 100
</ifmodule>

=================================================
as you can see, maxclients = serverlimit * threadsperchild
=================================================
<ifmodule prefork.c>
startservers 100
minspareservers 10
maxspareservers 10
serverlimit 50000
maxclients 50000
maxrequestsperchild 2000
</ifmodule>
=================================================
i do not know if we can start so many apache processes on a machine, even
theoretically.
on my linux machine,
[rajeevj01@bdc31035e ~]$ cat /proc/sys/kernel/pid_max
32768
so this number needs to be tuned first. plus of course there is

+ this issue of open fds

+ and linux thread stack size
+ and increasing the number of processes limit in shell using the ulimit command.

the next example config is taken from aol search team

=====================================================

## worker mpm settings

# aol: fixed 2 servers and 200 threads each for a total of 400 max
# concurrent requests. can be expanded to 1000 * 2 = 2000 via a
# graceful httpd controlled restart.

serverlimit 2
threadlimit 1000
startservers 2
maxclients 400
minsparethreads 200
maxsparethreads 400
threadsperchild 200

====================================================

after going through all such examples lets find a configuration that will allow us
to run 1k simultaneous connections. this is our first goal. and perspective gained
from here will determine how we go ahead with 2k simultaneous connections.

for our ec 2.0 case lets try this setup:

serverlimit 8
threadlimit 256
startservers 2
maxclients 1024
minsparethreads 128
maxsparethreads 256
threadsperchild 128
to achieve these numbers, we would need to
+ tune the number of file descriptors
+ and linux thread stack size.

number of threads should not be a problem. we can try tuning serverlimit to 4 and
increasing the threadsperchild. but only more data can tells us how to proceed
ahead with tuning.

finally, we have installed apache 2.0 with worker mpm. all you need to do is --
with-mpm=worker switch when using the configuration. using the worker mpm, we can
start apache with 4k threads (maxclients). see the blog about apache tuning to
know more.

idea is to start 4 ajp workers with 1k threads each and interface it with apache.
here we have a caveat. we have no way of distinguishing one apache request from
another. so the standard tricks of 1 ajp worker for one url is not going to work
for ec2. so initially i am going to set up 1 apache - 1 ajp connector for 1k
connections.

we need 4k + delta for apache instances and and 1k+delta /tomcat , so in total we
are talking about 4 jvms of 1200 threads each !!!

but before we dive into jvm arithmetic, we need to know how to setup ajp13 with 1k
connections. here , the trick is to use maxthreads setting for ajp connector in
server.xml file. but then what maxprocessors are doing ?

there were some discussions on tomcat list about maxprocessor attrib. but the long
and short of it is that i just need to set maxthreads and i should be okay. i dont
see a need to use apr (apache portable runtime), because in my case i will be
holding on to the connections. so threads are not wasted.

now, arithmetic

1 jvm - 1200 threads

thread stack size 256k => 8 threads /mb so 256 mb means 1k threads
so we start jvm with minheap 256 mb, maxheap 512 mb
512 mb x 4 = 2 gb for running 4 jvms.
thread stack size can be tuned to 128k also. ($ulimit -s 128)

lets try with one apache + one jvm first!

one apache that can handle 1k simultaneous is covered in apache tuning blog.

chat application is a push type application. we open a socket to server and

browser keeps getting data on that open connection. here, the browser and server
should not cache at any point because you need to push data to users as soon as
they are available.

to enable this behavior we need to set some response headers

you will need an equivalent of these apache server headers
========================================================
header set expires "sat,6 may 1995 12:00:00 gmt"
header set cache-control "no-store, no-cache"
header append cache-control "must-revalidate , no-transform"
header append cache-control "pre-check=0, post-check=0"
header set pragma "no-cache"
=========================================================

when we pluged mod_deflate module in our apache installation , we never realized

that we would hit the caching issues because of compression ;o)

mod_deflate is to compress the response, in our case it can not work till the
response is complete. so to do compression, server will keep caching the data
coming from chat backend and only flush it when the response is complete.

so now we need to do compression based on some conditions like some directories

only etc.

to do list for server tuning

step 1) get a development box that has rh 4.0 installed. we need rh 4.0 because we
need linux 2.6.x kernel series. verify the kernel with $uname -r

step2) we need to bump up the file descriptors limit. we need to raise a ticket
for this. verify the hard limits and then again verify that you can actually bump
up the file descriptors numbers. we can get 8k or maybe 16k descriptors
/process/shell.

3) use this program to verify

==================================================================================
====
[rajeevj01@bdc31035e sandbox]$ cat fd.c
#include <sys/types.h>
#include <fcntl.h>
#include <stdio.h>

void main(void) {
int k = 3;
while (dup(0) != -1)
k++;
printf("opens = %d fdsize = %d\n", k, sizeof(fd_set) * 8);
}

============================================================

4) change the thread stack size with $ulimit -s

change it to 128k or 256k. for the moment lets stick with 256k.
again we can verify the number of threads we can spawn with a program.

5) now fds and threads are okay. we need to move to apache settings. the apache
setting that will let us use 1k simultaneous connections have been covered in a
different blog.
make sure we run a lean apache. check apache 2.0 perf docs also. also make sure
that our apache has been compiled with all the required compiler switches. we need
to arrive at final configurations after doing the required tuning. turn off
debugging options/lots of logging in production systems.

+ tune with various parameters with sendfile on/off

+ we need to turn on keepalive on

finally test your apache for 1k simultaneous connections using the ab tool. you
may use grind/siege etc\. whatever you like, main thing is it should be verified.
6) now we need to move to tomcat. we are using jwal team's ntomcat to run tomcat
after starting the jwal layer. important things to remember are
+ turn off re-loading
+ turn off debugging
+ jvm settings
+ server.xml changes
+ shutdown http connector in production systems
+ tune with and w/o apr.

the ajp connector settings should change. this has been covered in a separate
blog.

after doing the setup , verify the connections.

7) jvm settings tuning

+ min / max heap size
+ -xss switch
other things like incremental gc, starting the jvm in server mode and other tweaks
we can think of.

8) now run the tas tool against our deployment and get the statistics.

================= programs to check max. threads =======================

[rajeevj01@bdc31035e sandbox]$ cat threads.c

/* compile with: gcc -lpthread -o thread-limit thread-limit.c */
/* originally from: http://www.volano.com/linuxnotes.html */

#include <stdio.h>
#include <unistd.h>
#include <pthread.h>

#define max_threads 20000

int i;

void run(void) {
char c;
if (i < 10)
printf("address of c = %u kb\n", (unsigned int) &c / 1024);
sleep(60 * 60);
}

int main(int argc, char *argv[]) {

int rc = 0;
pthread_t thread[max_threads];
printf("creating threads ...\n");
for (i = 0; i < max_threads && rc == 0; i++) {
rc = pthread_create(&(thread[i]), null, (void *) &run, null);
if (rc == 0) {
pthread_detach(thread[i]);
if ((i + 1) % 100 == 0)
printf("%i threads so far ...\n", i + 1);
}
else
printf("failed with return code %i creating thread %i.\n",
rc, i + 1);
 }
exit(0);
}

ajp13 connector options in detail

happy

lets go throught the various attributes of ajp 13 connector

enablelookups : set this to false. we do not want the dns lookups to happen.

protocol this attribute value must be ajp/1.3 to use the ajp handler.

connectiontimeout

the number of milliseconds this connector will wait, after accepting a connection,
for the request uri line to be presented. the default value is infinite (i.e. no
timeout). in our case, we can use the default.

minprocessors and maxprocessors are now not read at all (from 5.5.x series on-
wards )

port : whatever port you want.

sample

<!-- define a ajp/1.3 connector on port 4809 -->

<connector port="4809"
maxthreads="1024"
minsparethreads="25"
maxsparethreads="75"
enablelookups="false"
protocol="ajp/1.3" />

number of threads control how many concurrent requests can be served by this
connector and that is a very important parameter for us.