Forum, Support | Wowza Media Systems - How to do performance tuning

6/6/13 11:45 AM

User Name

Password

Log in

Help

Register

Remember Me?

Articles

Forum

Find a Consultant
Advanced Search

Welcome Quick Start Guide Tutorials Documentation

Home

Performance Tuning

How to do performance tuning

Wowza Media Server 3.6.1 is now available! Click here to get more information. New Wowza GoCoder iOS encoding app is available! Click here to get more information.

Article List Quick Start Guide Tutorials Documentation Performance Tuning Software Updates Wowza for Amazon EC2 Media Players and Devices Live Streaming and Encoders Video On Demand Streaming Closed Captioning Server Administration AddOn: Wowza Transcoder AddOn: Wowza nDVR AddOn: Wowza DRM AddOn: Other Wowza GoCoder for iPhone Server-side Modules and Code Samples List of All Articles

How to do performance tuning

Published on 10-01-2010 11:12 AM Number of Views: 43532 14 Comments

This article provides suggestions for tuning Wowza Media Server on your production hardware. Note: For use with Wowza Media Server 2 and later. 1. It is generally best to run the most recent Sun Java JDK (Java SE Downloads). It is best to run a 64-bit operating system with the 64-bit Java VM. The 64-bit VM enables Java heap sizes greater than 2GB. Note: If you are on a Mac with OS X 10.5.x or greater (Snow Leopard) with 64-bit hardware you can run the Java 6 VM. To change your default Java VM in OSX, open the Java preferences application /Applications/Utilities/Java/Java Preferences and drag the Java SE 6 item to the top of the Java Applications list. 2. It is best to run the -server version of the Java VM. The -server option is enabled on Linux and Mac OS X by default. It is only on Windows where this option is not enabled by default. To enable the server version of the Java VM on Windows, do the following: a. Install the latest version of the Oracle Java JDK and be sure that the JAVA_HOME environment variable is pointing to this Java installation. b. Edit the [install-dir]/bin/setenv.bat file to add the server option to the startup scripts. For example:

Categories Adobe Flash (89) Amazon EC2 (14) Apple iOS (56) Closed Captioning (6) Digital Rights Management (7) IP Camera (16) Live (178) Live Stream Repeater (23) Microsoft Silverlight (37) Mobile (32) RTSP/RTP (46)

Code:
set JAVA_OPTS=%JAVA_OPTS% -server

Note: Affects both Windows Standalone and Windows Service, uncomment line 7. 3. The default Java heap size for Windows is set to 768MB, and for Linux is set to 1200MB, which is great for development, but not large enough for high volume production use. If you are running the 64-bit version of the Java VM and have 4GB or more of RAM in your machine, we suggest a Java heap size of between 3000 MB to 5000 MB. If you have 16 GB or more RAM, we suggest a heap size of 8000 MB. We do not suggest a heap size above 10 GB as it can lead to long GC cycles/pauses. If you are running the 32-bit version of the Java VM or have less than 3GB of RAM in your machine, we suggest a Java heap size of 1500MB. Do not try to set a heap size greater than 1800MB when using the 32-bit VM. On many Linux platforms the
Page 1 of 9

http://www.wowza.com/forums/content.php?46

Forum, Support | Wowza Media Systems - How to do performance tuning

6/6/13 11:45 AM

Security (33) Set-top Box (28) SHOUTcast/Icecast (14) Video Chat (6) Video On Demand (93) Video Recording (14) Wowza DRM (8) Wowza nDVR (14) Wowza Transcoder (31)

server will start but will fail with an OutOfMemory error after a short period of time. Memory settings can be adjusted in the following script files (based on platform): [install-dir]/bin/setenv.sh (Linux, Mac OS X and Solaris, around line 4) Code:
JAVA_OPTS="-Xmx3000M"

[install-dir]/bin/setenv.bat (Windows Standalone and Windows Service, around line 4) Code:

set JAVA_OPTS=-Xmx3000M

4. Garbage collection (GC) tuning in Java is tricky. What works best in one server setup may not work in another. Through trial-and-error and customer feedback, we have several suggested approaches: Garbage Collection Options: We strongly recommend No Additional Settings. The default server tuning (see step #2 above) works well for many streaming situations, so in most cases this tuning works great without additional settings. Only if you are experiencing problems with the Java Heap, such as a build-up of memory usage, should you try implementing one of the following Garbage Collection options: a. Concurrent Collector: The concurrent collector is designed for applications that prefer shorter garbage collection pauses and that can afford to share processor resources with the garbage collector while the application is running. The suggested additional settings are: If heap size is 5000 MB or greater, NewSize=512m If heap size is 3000 MB - 5000 MB, NewSize=256m If heap size is less than 3000 MB, NewSize=128m For example, with 64-bit OS, 8 GB RAM, and heap size of 6000 MB Code:
-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:NewSize=512m

b. Experimental: Garbage-First garbage collector (G1): In the Sun Java 6 (Update 14 or greater) release, there is a new garbage collector that is in the experimental stage. It is designed for low pause time, high-throughput applications such as Wowza Media Server. It is an option to explore. There is more information below. To enable this garbage collector, the additional settings are: Code:

http://www.wowza.com/forums/content.php?46

Page 2 of 9

Forum, Support | Wowza Media Systems - How to do performance tuning

6/6/13 11:45 AM

-XX:+UnlockExperimentalVMOptions -XX:+UseG1GC

To change the garbage collection settings, modify the following scripts: [install-dir]/bin/setenv.sh (Linux, Mac OS X, and Solaris, uncomment line 10). For example: Code:

JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseParN

[install-dir]/bin/setenv.bat (Windows Standalone and Windows Service, uncomment line 10). For example: Code:

set JAVA_OPTS=%JAVA_OPTS% -XX:+UseConcMarkSweepGC -XX:+Use

There is more detailed information here: Java SE 6 HotSpot[tm] Virtual Machine Garbage Collection Tuning Java SE 6 Performance White Paper The Garbage-First Garbage Collector (G1) 5. For low-latency chat applications, it is best to use smaller socket buffer sizes (16000 bytes for read and write). The socket buffer sizes are configured in [install-dir]/conf/VHost.xml: Code:
<ReceiveBufferSize>16000</ReceiveBufferSize> <SendBufferSize>16000</SendBufferSize>

6. On Linux, increase the maximum number of file descriptors. This will address the Too many files open error message. To do this, edit the following two files: /usr/local/WowzaMediaServer/bin/wms.sh from: Code:
#ulimit -n 20000

to: Code:
ulimit -n 20000

/usr/local/WowzaMediaServer/bin/startup.sh from: Code:

http://www.wowza.com/forums/content.php?46

Page 3 of 9

Forum, Support | Wowza Media Systems - How to do performance tuning

6/6/13 11:45 AM

#ulimit -n 20000

to: Code:
ulimit -n 20000

Note: This may fail on some versions of Linux or if you do not have permission to change the descriptor limit. If after making this change Wowza Media Server will not start, consult the documentation for your Linux distribution to see how to increase the descriptor limit. Note: On some versions of Linux, there is also a limit in the kernel that may need to be increased. Consult the documentation for your distribution. You may need to add the following line to your /etc/sysctl.conf file: Code:
fs.file-max=20000

7. On Linux, switch to using the Anticipatory elevator (as) elevator algorithm. It is described here: Choosing an I/O Scheduler for Red Hat 8. On Linux, mount your disk with the noatime option. This operation will differ based on your Linux distribution. Here is a basic description of the setting: Using noatime Attribute 9. To tune your server based on the available CPU resources of your server, use the following guidelines: The [total-core-count] refers to the total number of CPU cores in your server. For example, if you have dual quad core processors (two quad core processors), the [total-core-count] is: Code:
2 (processor) x 4 (cores) = 8

If your server supports hyper-threading, then use the total number of threads. In the above example, if hyper-threading is available, then the total number of threads is: Code:
2 (processor) x 4 (cores) x 2 (threads per core) = 16

With the number of cores and threads per physical processor continually growing, we suggest a maximum number of threads for each value below: In the configuration file [install-dir]/conf/VHost.xml: Code:

http://www.wowza.com/forums/content.php?46

Page 4 of 9

Forum, Support | Wowza Media Systems - How to do performance tuning

6/6/13 11:45 AM

HostPort/ProcessorCount: 2x[total-core-count] (maximum of 24) IdleWorkers/WorkerCount: 2x[total-core-count] (maximum of 24) NetConnections/ProcessorCount: 2x[total-core-count]

(maximum o

RTP/UnicastIncoming/ProcessorCount: [total-core-count] (maximu RTP/UnicastOutgoing/ProcessorCount: 2x[total-core-count] (maxi RTP/MulticastIncoming/ProcessorCount: [total-core-count] RTP/MulticastOutgoing/ProcessorCount: [total-core-count]

(maxi (maxi

HandlerThreadPool/PoolSize: (60x[total-core-count]) (maximum of TransportThreadPool/PoolSize: (40x[total-core-count]) (maximum

This assumes you have at least 1GB of memory per core and that in the case you have 4 or more total cores you are running the 64-bit Java VM using the suggested memory settings above. 10. Running Multiple Virtual Hosts: If you are running more than one virtual host, the resource allocations must be distributed between each VHost. The simplest approach is to divide the settings listed above (which are intended for a single VHost) and distribute the resources across each VHost on your system. The settings do not have to be evenly divided, however, the total should equal what you would allocate if you were configuring for asingle VHost. If one of your VHosts will be idle most of the time, you may allocate more memory than the combined total. Be careful with this setting because excessive allocations are risky. An outOfMemory error will occur in the event both VHosts exceed the combined, available resources. Alternatively, in [install-dir]/conf/VHost.xml, you can set each VHost/HandlerThreadPool/PoolSize to "0" and VHost/TransportThreadPool/PoolSize to "0" which will cause the [install-dir]/conf/Server.xml settings for these properties to be used instead. This will instruct Wowza Media Server to manage the PoolSize amongst VHosts. A mixed approach can also be used with the [installdir]/conf/VHost.xml file by setting the PoolSize to 0 for idle/minimal-use VHosts while using higher values for busy/highperformance VHosts with higher resource requirements. 11. If you are not doing low latency streaming and you have a client side buffer of 3 or more seconds (NetStream.bufferTime), you can reduce the CPU load on the machine and handle more concurrent sessions by editing [install-dir]/conf/VHost.xml and changing the following two values: Code:
IdleWorkers/CheckFrequency: 100 Client/IdleFrequency: 500

12. If you are having problems with multiple incoming multicast streams interfering with each other on Linux you may need to set the Java property java.net.preferIPv4Stack to true. To do this, edit [install-dir]/bin/setenv.sh and uncomment the following line (line 13): Code:

http://www.wowza.com/forums/content.php?46

Page 5 of 9

Forum, Support | Wowza Media Systems - How to do performance tuning

6/6/13 11:45 AM

JAVA_OPTS="$JAVA_OPTS -Djava.net.preferIPv4Stack=true"

13. This really isn't tuning but it seems like a good place... When streaming to the Adobe Flash player, it is important to try and avoid RTMPT (tunneling version of RTMP) as much as possible. RTMPT uses a polling mechanism which is very chatty and CPU intensive. We suggest employing a protocol rollover strategy so that only connections that need RTMPT use RTMPT. Is is described here: http://www.wowzamedia.com/forums/content.php?50 14. On Windows 7, Server 2003 or 2008, it is probably best to turn off TCP auto tuning. There is information about how to do it here: How to disable Windows Vista TCP/IP auto-tuning

Updated: For Wowza Media Server 3.6.0 on 06-05-2013 Click here, if you are having problems or would like to discuss this article. Leave a comment below, if there is some aspect of this article you would like to see changed or improved.

14 Comments
notreg - 01-19-2011, 05:09 AM Why only 3-5GB for heap? How it could be calculated? rrlanham - 01-19-2011, 05:39 AM
Reply Reply

You can allocate more to java heap if you have it, and if you have 64bit OS and 64bit Java. If server is dedicated Wowza server allocate most available ram. Richard alinutzelu - 03-25-2011, 06:37 AM Yes, but "How it could be calculated?" It's any formula for occupied memory? I have 70 live http streams at 250kbps and wowza has reached at 2GB RAM (java process). Thank you Alin rrlanham - 03-25-2011, 08:16 AM Take a look at the tuning guide: http://www.wowzamedia.com/forums/con...ormance-Tuning The suggestions in this guide assume that the server is a dedicated Wowza server. If there are other applications that require a lot of memory, you would have to allow for that. Richard
Reply Reply

http://www.wowza.com/forums/content.php?46

Page 6 of 9

Forum, Support | Wowza Media Systems - How to do performance tuning

6/6/13 11:45 AM

phamkv - 08-30-2011, 04:07 PM Regarding point 9. : anticipatory scheduler has been removed from linux kernel as of kernel Linux 2.6.33 which was released on February 24th, 2010 . charlie - 08-30-2011, 04:20 PM Thanks for the info. We will amend this step. Charlie donASKet - 01-30-2012, 12:15 PM

Reply

Reply

Reply

> If you have 16 GB or more RAM, we suggest a heap size of 8000 MB. We do not suggest a heap size above 10 GB as it can lead to long GC cycles/pauses >This assumes you have at least 1GB of memory per core and that in the case you have 4 or more total cores you are running the 64-bit Java VM using the suggested memory settings above. But i have server with 16 core's. How much memory must i set for JVM Heap? 8GB or 16GB (1GB per core)? Server use for CDN (10-20 stream's with 500-2000 user's per stream) dptekco - 02-06-2012, 04:50 PM
Reply

This is a followup to donASKet's as well, with larger machines and interfaces do the tuning guides recommended limits still apply? In particular: - 10GB heap limit - If heap size is 5000 MB or greater, NewSize=512m, but at 10GB or higher, still 512m? - Do the various VHost.xml maximum values still hold with a 16+ core machine? - With Java 7, do you still recommend the default GC with -server option? With a beefed up machine running huge loads, are any of the optional GC's , including new G1 superior at keeping garbage collection from affecting streams? I know each must test , but I'm curious in regards to best practices already posted and if they still apply with larger machines (16+ cores, 20GB+ ram, 10gbit interface, sustaining over 6gbps throughput) romild0 - 04-23-2012, 06:40 AM
Reply

I would like to know that too. Btw, is there a big difference in java version 1.6 and 1.7? rrlanham - 04-24-2012, 07:10 AM
Reply

If you have multiple VHosts you do have to divide allocations between them. In most cases you can use multiple applications in one VHost and avoid this complication. You will have to look at Oracle's release notes for details on different Java versions, but we are recommending the latest 1.7 release for Wowza Richard steveitl - 11-11-2012, 02:53 PM hello,
http://www.wowza.com/forums/content.php?46 Page 7 of 9

Reply

Forum, Support | Wowza Media Systems - How to do performance tuning

6/6/13 11:45 AM

i not understand, where i need put the code : -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:NewSize=512m in the setenv.bat or WowzaMediaServer-Service.conf ? so, using Win2k8 x64 i need put : [install-dir]/bin/setenv.bat (add at line 8, Windows). set JAVA_OPTS=%JAVA_OPTS% -XX:+UseConcMarkSweepGC XX:+CMSIncrementalMode -XX:+UseParNewGC and : [install-dir]/bin/WowzaMediaServer-Service.conf (Windows). wrapper.java.additional.[n]=-XX:+UseConcMarkSweepGC XX:+CMSIncrementalMode -XX:+UseParNewGC is correct ? thanks steve rrlanham - 11-12-2012, 04:49 PM The GC lines go in /bin/setenv.bat Richard steveitl - 11-13-2012, 03:35 AM Richard, for complete configuration, in setenv.bat, two line : set JAVA_OPTS=%JAVA_OPTS% -XX:+UseConcMarkSweepGC XX:+CMSIncrementalMode -XX:+UseParNewGC and : -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:NewSize=512m in WowzaMediaServer-Service.conf, one line : wrapper.java.additional.[n]=-XX:+UseConcMarkSweepGC XX:+CMSIncrementalMode -XX:+UseParNewGC is correct ? steve Erwin Leon Pentzke - 01-16-2013, 12:51 PM
Reply Reply Reply

Richard and/or Charlie: In order to follow this guide and tune the right way EC2 Wowza servers I need you to help me with this question: Which is the number of threads in the Amazon Web Services EC2 most frequently used instances? (m1.small, m1.large, m1.xlarge, etc) I know that the m1.large has two cores, but there is no information about hyperthreading. Thanks in advance, Erwin

http://www.wowza.com/forums/content.php?46

Page 8 of 9

Forum, Support | Wowza Media Systems - How to do performance tuning

6/6/13 11:45 AM

-- Wowza VB4

Wowza Media Systems Archive Top

All times are GMT -7. The time now is 01:40 AM. 2013 Wowza Media Systems, LLC. All rights reserved

http://www.wowza.com/forums/content.php?46

Page 9 of 9