You are on page 1of 5

Dimensions CM versus ClearCase

Performance Benchmarking
Solution Brief
Serena ® Dimensions ® CM is the industry’s leading end to end, process-centric Application
Lifecycle Management solution for distributed development. It integrates application development
across global sites, stakeholders and platforms, and automates development steps based on
consistent processes to streamline delivery of the applications that drive the business.
Dimensions CM delivers:
• End-to-end traceability
• Streamlined regulatory compliance
• Enhanced team productivity
• Visibility and control

Dimensions CM is built for speed, delivering the superior performance that teams need to
accomplish their development tasks, either in co-located environments or across a wide area
network. Dimensions CM supports the resource intensive development practices that are the
KEY FEATURES norm today, including the use of geographically distributed development teams, offshoring, and
outsourcing.
• Process, change,
configuration, build, Dimensions CM Architecture
and release Dimensions CM is a proven platform for supporting the largest, most complex application
management from a development organizations. Dimensions CM is architected to address the needs of highly
single architecture distributed development teams with the most demanding performance requirements. Dimensions
• Support for advanced CM delivers high performance though:
parallel development • An industry-standard, high-performance RDBMS
and project level • Library caching
merge • Advanced compression
• Tight integrations to
leading IDEs Dimensions CM Performance Characteristics
Serena has established a full-time performance laboratory to help analyze and tune its solutions
to meet the most rigorous standards. In order to provide a real world test environment, Serena
configured the following test environment:

Remote Location (simulation India)

CM Library Cache Server


Clients

WAN (WAN Emulator)


TCP/IP
Latency 350 ms
Network bandwidth 750Kbps

Repository Location (simulation United States)

LAN
Dimensions CM Server
TCP/IP
Latency <1 ms
Clients
ClearCase Server

Clients / CM Library Cache Hardware Server Hardware


OS: Windows XP SP2 OS: Win 2003 Enterprise Server
CPU: Intel 1 x 2.4 GHz Pentium 4 CPU: Intel 4 X 2.7 GHz Xeon
Memory: 2GB, Disk space: 74 GB Memory: 7GB, Disk space: 146 GB
Network Card: 100Mbps Network Card: 1Gbps

Copyright © 2008 Serena Software, Inc. All rights reserved. Serena Dimensions is a registered trademarks of Serena Software. All other product or company names are used
for identification purposes only, and may be trademarks of their respective owners.
Test Configuration and Environment
Serena benchmarked the performance Dimensions CM against IBM Rational ClearCase to
gain a view of their relative performance characteristics. The test environment was chosen
Technical Pre-Requisites: to provide an infrastructure that is commonly in use in contemporary enterprise development
teams today. The hardware configurations used exceeded the minimum recommended for
both tested systems. The server was only moderately provisioned to compensate for
Supported Platforms dedicated use to a single session, rather than the concurrent usage one would expect in a
production environment. Other aspects of the test environment are as follows:
Dimensions CM supports • Dimensions CM and ClearCase servers were mounted on identical quad processor
the following server 2.7 GHz Xeon machines. This machine was configured with 7GB RAM and a 1Gbps
platforms*: Network Card, running Windows 2003 Enterprise Server operating system. License
server for both the products was installed on the same machines as the application
• Windows Server server.
• Sun Solaris • Dimensions CM and ClearCase clients were configured on Intel Pentium 4, 2.7 GHz
• HP-UX Xeon machines with 2GB of RAM, 100 Mbps Network Card, and at least 50GB of
• AIX free hard drive space. These machines ran the Windows XP SP2 Operating
• Red Hat Linux System.
• SUSE Linux • Applicable Dimensions CM and ClearCase software was installed on both the
servers and clients.
Dimensions CM supports • For ClearCase, VOB storage was on the server, and view storage was local to the
the following client client. The benchmark tests were performed with a snapshot view.
platforms*: • Dimensions CM Library Cache server was setup for file transfers across WAN. The
file caching was set to 4096 bytes, and only files with sizes larger than this setting
• Windows 2000 were cached to the Library Cache server.
Professional • TcpAckFrequency was set to 1 on all client machines (TcpAckFrequency is a registry
• Windows XP entry in Microsoft Windows XP and Microsoft Windows Server 2003 that determines
Professional the number of TCP acknowledgments that will be outstanding before the delayed
• Sun Solaris ACK timer is ignored.). More information about this setting is available at Microsoft
• HP-UX knowledgebase, Article ID: 328890.
• AIX • The WAN environment was set up to simulate a connection between an office in the
• Red Hat Linux West coast of United States and India. The network pipe was set at 750 Kbps at
• SUSE Linux both ends as maximum bandwidth for file transfers and the latency was set to 350
ms.
Dimensions CM supports
the following databases: Data Set Characteristics
A common file structure was used for testing both products. Each operated on the same files
• Microsoft SQL Server during testing. Performance tests were made against two data sets - one small and one
larger.
• Oracle
• IBM DB2
Small Data Set
The smaller data set used were ASCII files from the open source Ant build project from the
Apache Software Foundation.

Test Operation for # Files # Size on Disk


small data set Folders
Total Project (open project) 1,634 218 16.3 MB
Average file size: 10 KB
Median file size: 3.4 KB
Get multiple files 401 86 2.6 MB
Average file size: 6.4 KB
Median file size: 0.3 KB
Check in multiple files 30 7 223 KB
Average file size: 7.4 KB
Median file size: 2.7 KB

Copyright © 2008 Serena Software, Inc. All rights reserved. Serena Dimensions is a registered trademarks of Serena Software. All other product or company names are used
for identification purposes only, and may be trademarks of their respective owners.
Larger Data Set

The larger data set was based on the Dimensions CM documentation set, which is
comprised of Adobe Acrobat PDF files and some Microsoft Word documents.
Test operation for # Files # Size on Disk
large data set Folders
Total Project (open project) 34 6 81 MB
Average file size: 2.4 MB
Median file size: 1.3 MB
Compatibility Get multiple files 5 2 18.6 MB
Average file size: 3.7 KB
Median file size: 3.1 KB
Dimensions CM is Check in multiple files 5 2 18.2 MB
compatible with the Average file size: 3.6 MB
following third-party Median file size: 2.7 MB
products*:
A 15% change in the file content of files checked out was implemented as the delta. The
• BMC Remedy ARS multiple file operations for command line client on large data set were conducted on
• Microsoft Visual Microsoft Word files that were password protected using the Microsoft Word security feature.
Studio
This testing was conducted using Dimensions CM v10.1.2 and IBM Rational ClearCase
• Eclipse
v7.0.1.0.
• IBM Rational
Application Developer
• IBM Rational Rose
XDE 2003
Performance Results
• HP/Mercury Quality Tests were conducted to compare performance of both products from both a command line
Center interface and from the Eclipse environment using the Eclipse IDE plug-ins available in both
• HP/Mercury Dimensions CM and ClearCase. In addition, different network latency was considered, to
TestDirector provide a comparison between how the products perform on LAN and WAN environments.
• Telelogic Doors
• Borland JBuilder Comparative benchmarking tests were constructed to reflect real-life steps that developers
• Borland TogetherSoft would take in performing their day-to-day work. In all cases, functionally equivalent sets of
• Powerbuilder commands were used.
* Specific version requirements and
additional integrations exist; please consult
your Serena representative.
Performance Response Times for Command Line Client (in seconds):

Small Data Set Large Data Set


Benchmark 1 Benchmark 2 Benchmark 3 Benchmark 4
LAN WAN* LAN WAN*
Operations in Command
Line CM CC CM CC CM CC CM CC
Open project from source control 26 50 47 9624 12 14 12 2550
Get multiple files 6 16 10 3012 3 5 5 570
Check in multiple files 5 22 12 3762 4 14 233 1758
Total time for operations
above 37 88 69 16398 19 33 250 4878
Figure 1: Test results using the command line interfaces of Dimensions CM 10.1.2 and ClearCase 7.0.1.0
* Benchmark WAN testing uses CM Library Cache

Copyright © 2008 Serena Software, Inc. All rights reserved. Serena Dimensions is a registered trademarks of Serena Software. All other product or company names are used
for identification purposes only, and may be trademarks of their respective owners.
Performance Response Times for Eclipse Plug-In (in seconds):

Small Data Set Large Data Set


Benchmark 5 Benchmark 6 Benchmark 7 Benchmark 8
LAN WAN* LAN WAN*
Operations in Eclipse
plug-in CM CC CM CC CM CC CM CC
Open project from source control 75 61 134 11951 13 20 28 3540
Get multiple files 17 18 27 3625 5 11 8 1470
Check in multiple files 5 67 60 5031 5 20 229 2100
Total time for operations
above 97 146 221 20607 23 51 265 7110
Figure 2: Test results using the Eclipse integration of Dimensions CM 10.1.2 and ClearCase 7.0.1.0
* Benchmark WAN testing uses CM Library Cache

While Serena strives to provide the most accurate performance data possible, customers
should be aware that a myriad of factors can impact the performance of an enterprise-class
application like Dimensions CM. Results may vary depending on those factors.

Summary
In each of the benchmarks, Serena Dimensions CM delivered higher performance than IBM
Rational ClearCase. In high bandwidth, low latency network simulations, Dimensions CM
outperformed ClearCase on average by a factor of 1.8X.

LAN Performance Response times
(Lower is better)
160

140

120
Response Time 

100
(Seconds)

80
Dimensions 
ClearCase
60

40

20

Benchmark 1
Benchmark 3
Benchmark 5
Benchmark 7
Benchmark 

Figure 3: Side by side comparison of Dimensions CM and ClearCase command line client and Eclipse plug-
in on a high bandwidth LAN (Lower is better)

For the lower bandwidth, high latency networks often found where teams employ distributed
or offshore development teams, the performance advantages of Dimensions CM are even
more dramatic. Overall, on low bandwidth, high latency networks Dimensions outperformed
ClearCase by a factor of 60X.

Copyright © 2008 Serena Software, Inc. All rights reserved. Serena Dimensions is a registered trademarks of Serena Software. All other product or company names are used
for identification purposes only, and may be trademarks of their respective owners.
WAN Performance Response times
(Lower is better)

25000

20000

Response Time  15000
(Seconds)

10000
Dimensions 
ClearCase
5000

Benchmark 2
Benchmark 4
Benchmark 6
Benchmark 8

Benchmark 

Figure 4: Side by side comparison of Dimensions CM and ClearCase command line client and Eclipse plug-
in on a low bandwidth WAN (Lower is better)

As the performance benchmarks show, Dimensions CM consistently outperforms ClearCase,


often by a wide margin. The high performance of Dimensions CM empowers enterprises to
get the maximum out of their distributed and offshore development teams. This translates
into an immediate productivity gain as well as long term cost savings.

About Serena Software, Inc.


Serena Software, Inc. provides services to make Enterprises and the business people within
them more productive. More than 15,000 organizations around the world, including 96 of the
Fortune 100, rely on Serena solutions delivered either on premise or on demand, to provide
visibility and efficiency to the application development process. The company provides
software services such as Serena Mariner (Project & Portfolio Management), Serena
Business Mashups, Serena Dimensions (Change & Requirements Management), and will be
providing Agile Lifecycle Management tools in the near future. Serena is headquartered in
Redwood City, California, and has offices throughout the U.S., Europe, and Asia Pacific. For
more information on Serena solutions and services, visit www.serena.com.

Copyright © 2008 Serena Software, Inc. All rights reserved. Serena Dimensions is a registered trademarks of Serena Software. All other product or company names are used
for identification purposes only, and may be trademarks of their respective owners.