Beruflich Dokumente
Kultur Dokumente
June, 2012
Executive Overview
The purpose of the technical document is to detail the performance information for Siebel
8.1.1.7 Reporting BI Publisher integration.
Test Criteria
Goal is for < 10 seconds response time with simple reports (1 page, < 1000 records)
Complex reports with large integration objects (IO), aggregation for small data set
would expect 30-60secs.
Large data set would expect no failures, but realistic generation time.
For extremely large, long running report tests the goal is for no report failures (large
datasets or many small reports and large concurrency).
Test Cases
The test scenarios are designed to test the following common customer scenarios:
Impact on response times for both simple and more complex reports, when scaling
up large numbers of concurrent users running reports for a period of time.
Impact of response times for both simple and more complex reports, when running
with large data sets for a period of time.
Impact on the BI Publisher server and response times when running a report that
contains many aggregations. Also running the same test and increasing the data
size.
2
Oracle White Paper Siebel CRM BI Publisher Integration Performance
Difference between a normal query and selected records query when running a
report.
Impact on response times when generating more complex reports with a larger
Integration Object and data shape.
Reference tests using a standard Account List report with different levels of
concurrency and data set.
The report templates used for the tests are included with this technical document.
Conclusion
Average response times of less than 5 seconds were achieved running simple
reports with 100 records, with many aggregations and also with 100 user
concurrency.
Average response times of less than 10 seconds were achieved running simple
reports with 1000 records, with many aggregations and also with 50 user
concurrency.
Average response time degrades as the high levels of concurrency are increased.
Increasing the number of XMLP Components improves response times in this high
concurrency scenario.
Response time for report generation with large data sets is affected more by the
number of pages than the number of aggregations in the report template.
Time consuming reports with high concurrency will possibly require increasing
WinInet settings. This should be specified on the operating system of the machine
where the XMLP components are running. WinInet was set to 2 during tests
(recommended).
3
Oracle White Paper Siebel CRM BI Publisher Integration Performance
The Average response time has improved considerably from Siebel 8.1.1 - for short
Accounts List reports for 10, 25, and 100 concurrent users times are 5.17, 5.05, 5.79
seconds respectively in Siebel 8.1.1.7.
The time to generate a Selected Records query for a report generation is higher
than a normal query for report generation. Additional memory usage was observed
generating reports in scheduled mode as compared to non-scheduled mode (~450
mb / ~50% more) was observed when testing a large data set report.
Notes
During the testing a SQL performance issue was encountered (bug # 13972140).
With the fix, Database CPU improved by ~60% and Average Response Time
improved by ~19% for 300 user concurrency with 100 SR records scenario. Note that
this bug might only be encountered running reports over large user concurrency. In
general this bug is unlikely to be encountered in a typical deployment.
During the testing a memory leak issue was encountered (bug # 13824469). The fix
improved stability when running large data set reports under concurrency. Note that
this bug might only be encountered running reports with a large data set (50,000
records) and high concurrency. In general this bug is unlikely to be encountered in a
typical deployment.
References
4
Oracle White Paper Siebel CRM BI Publisher Integration Performance
http://www.oracle.com/technetwork/middleware/bi-publisher/overview/oracle-bi-
publisher-best-practices-133345.pdf
Environment Details
5
Oracle White Paper Siebel CRM BI Publisher Integration Performance
6
Oracle White Paper Siebel CRM BI Publisher Integration Performance
records
8 Report with 1, 10, 25, 50, Account List View 1000 Accounts. The related IO
simplified IO 100 > Account List will only draw from Accounts.
(Account list). report Actual Data Shape Used: 1000
Test with Accounts with 10/10/10
standard associations
Account List (Contacts/Activities/Opportunitie
report layout s)
for 1000
records
9 Report with 1 Account List View 10000 Accounts. The related IO
simplified IO > Account List will only draw from Accounts.
(Account list). report
Test with
standard
Account List
report layout
for 10000
records
10 Report with 1 Account List View 50 Accounts. The related IO will
simplified IO > Account List only draw from Accounts.
(Account list) - report Actual Data Shape Used: 50
Normal report Accounts
query. Test
with standard
Account List
report layout
for 50 rows
11 Report with 1 Account List View 50 Accounts. The related IO will
simplified IO > Account List only draw from Accounts. We
(Account list) - report just set the flag to Selected
Selected Records = True
Records report Actual Data Shape Used: 50
query. Test Accounts
with standard
Account List
report layout
for 50 rows
7
Oracle White Paper Siebel CRM BI Publisher Integration Performance
The purpose of this test was to validate the impact on the response time and the BI Publisher
server resources running a report with varied concurrency, small dataset and with a report that
contains many complex aggregations.
Notes
A 1000 SRs data shape was generated with 1 Contact and 1 Account associated to each SR.
Service Request Statistics report is a single page layout with 14 aggregations.
Both 100 and 1000 records report tests produced only a single page (PDF files size is ~4KB).
No slow SQL statements were found for both 100 and 1000 records scenarios.
All statistics were calculated for a 30 mins steady state (45 mins - 1 hour 15 mins).
Key Observations
The Average Response Time for Service Request Statistics Report with 100 records for 10,
50, and 100 users is less than 5 seconds. For 200 and 300 concurrent users average response
times are 20 and 31 seconds respectively.
The Average Response Time for Service Request Statistics Report with 1000 records for 10
and 50 users are 8.47 and 9.67 seconds respectively. The average response time for 100
concurrent users is 21.26 seconds.
The Average Response Time for Service Request Statistics Report with 100 and 1000 records
degrades with an increase in user concurrency. However, increasing the number of XMLP
Components improves the Average Response Time dramatically. This can be seen by
comparing the Average Response Time with Test # 5 and Test # 12.
With 100 SRs and 300 concurrent users the average Application CPU was ~56%. Beyond 300
users the Average Response Time degraded to ~31 seconds.
With 1000 SRs and 100 concurrent users the average Application CPU was ~81%. Beyond
100 users the Average Application CPU reached ~81%.
There were no hangs observed during the test.
The biggest impact on the Average Response Time was due to concurrency. With regards to
report aggregations, the impact on the BI Publisher resources was minimal.
8
Oracle White Paper Siebel CRM BI Publisher Integration Performance
Increasing the number of shared database connections doesnt have much impact on the
response time.
The Average Response Time for Service Request Statistics report with 100 records for 10, 50,
100 and 300 users (with 5 and 6 XMLP OMs) are below 10 seconds while the Average
Response Time for 200 users (with 2 XMLP OMs) is above 10 seconds.
The Average Response Time for Service Request Statistics report with 1000 records for 10
and 50 users are below 10 seconds while the Average Response Time for the 100 users is
above 10 seconds.
The Average Response Time for Service Request Statistics Report with 100 and 1000 records
degrades with increase in user load.
During the testing a SQL performance issue was found in the XMLP report generation, with
the SQL performance fix, Database CPU improved by ~60% for 300 users with 100 SR
records scenario. Average Response Time improved by ~19% (Tests #9 vs #12).
Summary
1 10 100 10 4.51 1.97% 0.46% 2.28% 0.69% 959.88 2 XMLP OM, 20 shared DB connections
2 50 100 10 4.42 8.91% 0.76% 4.63% 2.07% 1298.51 2 XMLP OM, 20 shared DB connections
3 100 100 10 4.84 18.06% 0.96% 8.03% 3.72% 1700.13 2 XMLP OM, 20 shared DB connections
4 200 100 10 20.41 35.67% 1.53% 14.55% 6.44% 2598.53 2 XMLP OM, 20 shared DB connections
5 300 100 10 91.48 37.19% 1.61% 13.98% 6.47% 3269.95 2 XMLP OM, 20 shared DB connections
6 300 100 10 31.78 56.73% 2.21% 19.36% 8.97% 3644.77 3 XMLP OM, 30 shared DB connections
7 300 100 10 9.71 63.39% 2.05% 53.88% 10.73% 2941.42 5 XMLP OM, 50 shared DB connections
8 300 100 10 7.78 61.68% 2.22% 54.50% 10.97% 2864.92 5 XMLP OM, 20 shared DB connections
6 failures
9 300 100 10 7.62 63.84% 2.64% 55.59% 11.51% 3073.95 6 XMLP OM, 30 shared DB connections
1 failure
10 300 100 10 256.54 18.05% 2.02% 43.30% 3.21% 2354.11 2 XMLP OM, 50 shared DB connections
The XMLP report output table was not
cleaned before this run.
11 300 100 10 92.96 38.04% 1.63% 26.71% 6.05% 2532.13 2 XMLP OM, 50 shared DB connections
No failures, consistency run after XMLP
report output table cleanup.
9
Oracle White Paper Siebel CRM BI Publisher Integration Performance
12 300 100 10 6.14 62.23% 2.25% 21.49% 11.35% 3063.02 6 XMLP OM, 30 shared DB connections
With SQL Performance fix:
DB CPU improved by ~60%
Avg RT is improved by 19%
13 10 1000 10 8.47 6.01% 0.39% 4.61% 2.21% 972.29 2 XMLP OM, 20 shared DB connections
No failures
14 50 1000 10 9.67 35.16% 0.79% 15.29% 10.13% 1312.10 2 XMLP OM, 20 shared DB connections
No failures
15 100 1000 10 21.26 81.07% 0.99% 28.95% 19.75% 1734.22 2 XMLP OM, 20 shared DB connections
No failures
Settings
11 Minimal Tracing were enabled to get the statistics for the Siebel Server components
14 Think time used in the script between the transactions/step-groups is 30 seconds (Dynamic Think Time
- 15% to 185%).
Add the following line in the PreInvokeMethod() of the Server script of "XMLP Driver
Service" business service.
Inputs.SetProperty("HTTPSleepTime", "1200000");
Add the following line in the PreInvokeMethod() of the Server script of "EAI HTTP
Transport" business service.
10
Oracle White Paper Siebel CRM BI Publisher Integration Performance
Inputs.SetProperty("HTTPSleepTime", "1200000");
11
Oracle White Paper Siebel CRM BI Publisher Integration Performance
The purpose of this test was to validate the impact on the response time and the BI Publisher
server resources running a report with large data set of 50,000 records and with a report that
contains many complex aggregations. The goal was for the report generation to be stable without
any failures. The test was for both immediate report and scheduled report generation.
Notes
A 50000 Service Request data shape was used.
Service Request Statistics report contains a single page layout with 14 aggregations.
50000 records report contains a single page (PDF files size is ~4 kb).
No Slow SQL statements were found.
All statistics were calculated for a 30 mins steady state (15 mins - 45 mins).
Key Observations
The Average Response Time for the Service Request Statistics report with 50000 records
(non-Scheduled Mode and Scheduled Mode) for 1 user is less than 30 minutes.
There were no crashes or hangs noticed during this run.
Memory leak fix was applied
Summary
Generate
Avg CPU Usage (%)
Report Txn
Siebel -
# # Sched KPI Avg Total
We
user SR uled (Se RT App DB BIP Used Key Configuration
b
s s Mode c) (Sec) Memory
(MB)
1 XMLP OM, 10 shared
180 7.31 0.40 6.12 6.76 DB connections
1 50K No 284.12 674.30
0 % % % % No failures, with
memory leak fix
1 XMLP OM, 10 shared
180 6.77 0.55 5.00 4.30 DB connections
1 50K Yes 288.00 1015.72
0 % % % % No failures, with
memory leak fix
Settings
12
Oracle White Paper Siebel CRM BI Publisher Integration Performance
11 Minimal Tracing were enabled to get the statistics for the Siebel Server components
12 DSMaxFetchArraySize -1
14 Think time used in the script between the transactions/step-groups is 30 seconds (Dynamic Think
Time - 15% to 185%).
13
Oracle White Paper Siebel CRM BI Publisher Integration Performance
The purpose of this test was to validate the impact on the response time running a large report
over varied concurrency with a more complex data shape and increased Integration Object size.
Statistics were obtained at individual component level for analysis.
Notes
A 1000 Accounts data shape was generated, in which 10 Contacts, 10 Opportunities and 10
Activities are associated to each Account.
Account Detail report is a standard Master/Detail report layout. The master section shows
the Account information and detail part contains the list of child records (Contacts, Activities
and Opportunities)
100 Accounts detail report contains 102 pages (PDF files size is ~300 kb).
1000 Accounts detail report contains 1002 pages (PDF files size is ~3 mb).
No slow SQL statements were found for both 100 and 1000 records scenarios.
All statistics were calculated for a 30 mins steady state (15 mins - 45 mins for 1 user, 45 mins
- 1 hour 15 mins for 10, 25 and 50 users scenario).
A memory leak issue was resolved during reliability testing.
Key Observations
The Average Response Time for 1, 10, 25 users is below 60 seconds, the Average Response
Time for 50 users is ~66 seconds.
The Average Response Time degrades with increase in user load.
For 100 accounts with 50 concurrent users the Average Application CPU ~89%.
For 1000 accounts with 10 concurrent users the Average Application CPU ~64%.
There were no hangs during this run. No crash files found during the run.
14
Oracle White Paper Siebel CRM BI Publisher Integration Performance
Summary
Generate
Avg CPU Usage (%)
Report Txn
Siebel -
# # KPI Avg Total
Key
user Account (Se RT App Web DB BIP Used
Configuration
s s c) (Sec) Memory
(MB)
1 XMLP OM, 10
0.35 shared DB
1 100 60 19.10 2.52% % 1.35% 1.42% 706.41 connections
1 XMLP OM, 10
15.57 0.47 shared DB
10 100 60 20.63 % % 5.19% 9.86% 856.52 connections
2 XMLP OM, 20
43.91 0.58 18.54 26.09 shared DB
25 100 60 25.62 % % % % 1489.87 connections
2 XMLP OM, 20
89.48 2.00 19.70 49.21 shared DB
50 100 60 66.75 % % % % 1795.36 connections
1 XMLP OM, 10
165.7 0.35 shared DB
1 1000 60 7 7.45% % 4.05% 5.46% 827.34 connections
2 XMLP OM, 20
296.8 64.83 0.60 35.66 37.31 shared DB
10 1000 60 6 % % % % 1549.73 connections
Timing Breakup:
No. of Query Time Datac Datac Datac Datac Total Repor Total File Total RT
Acco Time taken hunk hunk hunk hunk time t Elaps Save Time from
unts for 1 uploa 2 uploa taken Gener ed Time taken OLT
Recor Data query d1 query d2 for ation Time for run
ds uploa time Time time Time dataq Time on the report
d to uery BIP report
BIP and Serve
serve Uploa r
r d
15
Oracle White Paper Siebel CRM BI Publisher Integration Performance
Notes
A memory leak of ~159KB/Iteration was resolved during initial investigation for eHospitality
OM.
For the first test runs (10 users, 1000 records) the statistics were calculated for 20 hours
steady state.
For the second and third test runs without the memory leak fix, the statistics were calculated
for 2 hours 30 mins steady state.
For the last test run (40 users, 100 records) the statistics were calculated for 18 hours steady
state.
Summary
Generate Report
Avg CPU Usage (%)
Txn
Siebel -
Memory
# # Total
KPI Avg RT Leak
use Accou App Web DB BIP Used Key Configuration
(Sec) (Sec) KB /
rs nts Memory
Iteration
(MB)
2 XMLP OM, 20 shared
61.41 0.66 36.91 38.08 DB connections
10 1000 60 305.48 2100.38 159.18
% % % % Bug# 13824469 for
memory leak
2 XMLP OM, 20 shared
91.21 0.94 34.29 47.07 DB connections
50 100 60 60.81 2223.21 159.76
% % % % Bug# 13824469 for
memory leak
2 XMLP OM, 20 shared
77.58 1.19 30.15 41.17 DB connections
40 100 60 38.92 2051.09 156.00
% % % % Bug# 13824469 for
memory leak
2 XMLP OM, 20 shared
44.80 0.71 60.03 23.73
40 100 60 147.65 2021.19 No leak DB connections
% % % %
With Memory Leak fix
Settings
16
Oracle White Paper Siebel CRM BI Publisher Integration Performance
11 Minimal Tracing were enabled to get the statistics for the Siebel Server components
14 Think time used in the script between the transactions/step-groups is 30 seconds (Dynamic Think
Time - 15% to 185%).
17
Oracle White Paper Siebel CRM BI Publisher Integration Performance
The purpose of this test was to provide a reference test using a standard and simple application
report. The test was run with varied concurrency and data size.
Notes
A 1000 accounts data shape was generated where 10 contacts, 10 opportunities and 10
activities are associated to each Account. The integration object used for the report only
contains the Account entity.
Account list report contains standard OOTB list report layout.
100 accounts list report contains 16 pages (PDF files size is ~47 kb).
1000 accounts list report contains 160 pages (PDF files size is ~447 kb).
No slow SQL statements were found for both 100 and 1000 records scenarios.
All statistics were calculated for 30 mins steady state (45 mins to 1 hour 15 mins).
Key Observations
The Average Response Time for Accounts List Report with 100 records for 10, 25, and 100
users are 5.17, 5.05, 5.79 seconds respectively.
The Average Response Time for Accounts List Report with 1000 records for 10, 25 and 50
users are 15.3, 16.9, 36.3 seconds respectively.
The Average Response Time for Accounts List Report with 1000 records degrades with
increase in user load.
For 100 accounts with 100 concurrent users the Average Application CPU ~21%.
For 1000 accounts with 100 concurrent users the Average Application CPU ~40%.
There were no hangs during this run.
Summary
Generate Report Txn Avg CPU Usage (%)
18
Oracle White Paper Siebel CRM BI Publisher Integration Performance
100 100 60 5.79 21.08% 1.10% 7.44% 13.89% 1697.82 2 XMLP OM, 20
shared DB
connections
Failures after
~1.5 hours due to
memory leak
100 100 60 5.77 22.15% 1.05% 7.48% 13.91% 1694.59 2 XMLP OM, 20
shared DB
connections
Failures after
~1.5 hours due to
memory leak
Settings
# Component Configurations Value
11 Minimal Tracing were enabled to get the statistics for the Siebel Server components
19
Oracle White Paper Siebel CRM BI Publisher Integration Performance
Key Observations
It is important to note that in Siebel 8.1.1 performance testing was with a throughput
scenario, meaning there are no delays between the report requests, with continuous hits to the
server.
With 8.1.1.7 performance testing, a UI scenario was used which involves a dynamic think
time between each request.
The hardware used between 8.1.1 and 8.1.1.7 is different and considerably improved in 8.1.1.7
testing.
The Average Response Time for a shorter Accounts List report was 9 seconds in Siebel 8.1.1.
The Average Response Time for a medium size Accounts List report was 22 seconds in
Siebel 8.1.1.
Summary
Average response times from Siebel 8.1.1 BI Publisher integration with 10 user concurrency:
# of Call XMLP BIP Total (in
Account List report
Pages Center Component Server Seconds)
DB Server (Microsoft SQL Server 2005) 4 x 701 Mhz 3.65 GB Windows Server 2K3
20
Oracle White Paper Siebel CRM BI Publisher Integration Performance
21
Oracle White Paper Siebel CRM BI Publisher Integration Performance
The purpose of this test was to provide a reference test using a standard and simple application
report. The test was run with a large data size (10000 records).
Notes
We have used a 10000 Accounts data shape.
Account List report contains standard list report layout.
10000 accounts list report contains 1591 pages (PDF files size is ~4.5 MB).
No slow SQL statements were found.
Key Observations
Average Response Time for Account List report with 10,000 records was 173.59 seconds, less
than 3 minutes.
Summary
Generate
Avg CPU Usage (%)
Report Txn
# # KPI Siebel - Total
Avg RT Key
user Account (Se App Web DB BIP Used
(Sec) Configuration
s s c) Memory (MB)
1 XMLP OM, 10
360 3.13 0.39 7.59 6.38
1 10K 173.59 699.83 shared DB
0 % % % %
connections
Settings
# Component Configurations Value
22
Oracle White Paper Siebel CRM BI Publisher Integration Performance
11 Minimal Tracing were enabled to get the statistics for the Siebel Server components
23
Oracle White Paper Siebel CRM BI Publisher Integration Performance
The purpose of this test was to see the difference running a standard Accounts List report in two
different modes, standard query and selected records query mode. For selected reports query
mode, a Selected Records version of the same report was executed with individual applet
records selected. For normal query mode, a standard applet query was executed.
Notes
Tests were performed using both a Cold/Warm state for the Object Manager to see any
difference on query caching. Different sets of records were also used for comparison.
An individual breakdown of the report generation timings was done to see where the most
time was taken during report generation.
Key Observations
Time to generate a Selected Records query for a report generation is higher than a normal
query for report generation.
Search Specification in Selected Query contains 50 individual Row IDs in the SearchSpec
text. Search Specification in Normal Query contains single SearchSpec text.
For 50 records using Selected Records Query Spec, using Warm OM, even though SQL
remains the same by using different bind variables, the SQL is taking ~15 sec due to multiple
OR conditions.
Summary
Normal Query Selected Records Query
24
Oracle White Paper Siebel CRM BI Publisher Integration Performance
25
Siebel CRM BI Publisher Integration
Performance
June, 2012
Author: John Bedford
Copyright 2009, Oracle and/or its affiliates. All rights reserved. This document is provided for information purposes only and
the contents hereof are subject to change without notice. This document is not warranted to be error-free, nor subject to any other
Oracle Corporation
warranties or conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability or
World Headquarters
fitness for a particular purpose. We specifically disclaim any liability with respect to this document and no contractual obligations are
500 Oracle Parkway
formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any
Redwood Shores, CA 94065
means, electronic or mechanical, for any purpose, without our prior written permission.
U.S.A.
Worldwide Inquiries: Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective
Fax: +1.650.506.7200
oracle.com 0109