Multi-Tenant Scalability Guidance for

Exchange Server 2013

Customer Advisory Team

Exchange Product Group
Microsoft Corporation

August 2013

Contents
Introduction................................................................................................................ 3
Scalability Guidance Conclusions...............................................................................3
Tenant Scalability.................................................................................................... 3
Modern Public Folders............................................................................................. 4
Limits on Exchange Transport Rules and Offline Address Books..............................4
General Recommendations for High-Scale Deployments........................................5
AD Optimizations..................................................................................................... 5
Exchange Server 2013 Multi-Tenant Scalability Testing Methodology.........................6
Goals....................................................................................................................... 6
Testing Script........................................................................................................... 6
Scale Testing Lab Setup........................................................................................... 7
Distribution of Virtual Server Images on Hyper-V Root Servers............................7
Configuration of Virtual Server Images (RAM, CPU, Disk).....................................8
Test Results at Measurement Points........................................................................9
Results at Scale Target............................................................................................ 9
Validation with Simulated Client Load.....................................................................9
Exchange Logging Sizing Observations.................................................................10
AD Sizing Observations......................................................................................... 10
Summary.................................................................................................................. 10
Legal Notice.............................................................................................................. 11

Introduction
This document describes Microsoft recommendations for scaling a multi-tenant deployment of Exchange
Server 2013 using the methods described in the Multi-Tenancy and Hosting Guidance for Exchange
Server 2013 white paper. Microsoft Exchange Server 2010 Service Pack 2 (SP2) introduced Address
Book Policies (ABP), a feature which allows segmentation of directory information. The ABP feature,
along with other supported configuration, continues to provide the platform for deploying a custom multitenant environment using Exchange Server 2013. For the purposes of this document, a tenant is a hosted
organization which has been configured with a set of administrative policy objects, such as an ABP, along
with a set of users.
In the Multi-Tenancy and Hosting Guidance for Exchange Server 2013 white paper Microsoft describes
the caveats and requirements to implement a multi-tenant solution using Exchange Server 2013. This
document provides additional guidance on the scalability of specific areas of concern with Exchange
2013. This document does not attempt to cover all potential scalability limits in the product, and any highscale deployment should proceed with necessary caution during the planning, testing, deployment, and
scaling phases. The testing that was performed in Microsofts lab is described here to assist you in adding
context to the white papers recommendations.

Scalability Guidance Conclusions

Tenant Scalability
Scale testing performed in Microsofts lab has demonstrated that Exchange 2013 supports up to 50,000
tenants configured in a single Active Directory (AD) forest. Microsoft recommends not exceeding 50,000
tenants in a single forest. Customers who need to deploy capacity for greater than 50,000 tenants should
design a multi-forest architecture with a provisioning system that is capable of supporting multiple
Exchange capacity forests. Additionally, you may need to reduce this per-forest limit significantly if you
plan to provision Exchange transport rules, or the Offline Address Book (OAB) for each of your tenants.
See the section below titled Limits on Exchange Transport Rules and Offline Address Books for additional
information on these limitations.
Each of the tenants were created and given the following objects (additional details of this configuration
are available in the testing section later in this document):

A tenant organizational unit (OU) in AD.

Two universal security groups (USG), one for the tenants admin; one for all of the tenants users.
An accepted SMTP domain.
A global address list (GAL).
An All Rooms address list.
An All Users address list.
An All Contacts address list.
An All Groups address list.
An ABP.
Three mailbox-enabled users: one admin and two users.
Two mail-enabled contacts

Many multi-tenant messaging deployments contain a large number of tenants and a small number of
users per tenant. This testing was designed to evaluate the scalability of an Exchange 2013 solution in
this type of configuration, specifically validating the overall number of tenants that could be successfully
provisioned in a single Exchange organization without focusing on the overall number of users configured
per-tenant or globally within the system. Administrators interested in deployments which contain a large
number of users per tenant can also benefit from this guidance, however they should pay careful attention

to the overall impact of the total number of users on the scalability of the system as that may become a
more limiting factor than the total number of tenants.

Modern Public Folders

A modern public folder hierarchy was created for 10% of the tenants (5,000 tenants total). Testing to this
level indicated we would have been able to go higher as there was no degredation observed in any of the
aspects of this part of the test. Note that this testing focused primarily on the creation and segregation
(access rights) of the folder hierarchy, and did not attempt to test large volumes of data posted within the
public folders. If you choose to offer modern public folders to your tenants, you will need to perform
additional testing to insure your service can scale to the limits and quotas you offer your tenants.
Additional guidance can be found in the Exchange Server 2013 Public Folders TechNet topic.
In this scale lab, eight public folder mailboxes were created, one on each of the mailbox databases that
existed in the scale lab. Throughout the scale testing the creation of new public folders, as well as the
setting of access rights, the replication of the public folder hierarchy; and the posting of new messages
occurred successfully and without any measurable degredation in the time it took to accomplish each of
these tasks.
The public folder hierarchy was created with the following
access rights:
RESELLER: The empty Reseller folder is owned by the
hosters administrator. The default access right was set to
FolderVisible*.
TENANT: Only the sole Tenant administrators were given the
Owner role and that tenants users were given the FolderVisible access right. Default access rights were
set to None to hide this folder, and its subfolders, from all other tenants.
SUB FOLDERS: These folders inherited access rights from the Tenant parent folder. Then additional
rights were assigned to give control of the folder and its contents to the tenants users.
Testing confirmed that the Tenant-level folders were segregated and viewable only by the respective
tenant, that e-mail to the child folder posted successfully, and that edits, deletes and the creation of
additional subfolders behaved as expected based on the roles and access rights assigned to each folder.
*Note: If you offer Public Folders to your tenants, you should be on Cumulative Update 2 (CU2) for
Exchange Server 2013. There is an issue on earlier versions where you are unable to set explicit access
rights when creating a new public folder. The new folder will replicate immediately with only inherited
rights, and any explicit rights you set will miss the first replication and will take effect only after the next
replication, which may be hours. This was fixed in CU2 by allowing explicit permissions to be set at the
same time you create a new public folder.

Limits on Exchange Transport Rules and Offline Address Books

Based on testing performed for the creation of this document, as well as sizing and scalability of internal
hosting deployments, Microsoft recommends not exceeding 5,000 transport rules or 25,000 OABs per
forest.
Transport Rule behavior: Functional issues were observed in testing where the efficiency of the
transport service degraded as additional rules were created above 5,000. High CPU consumption
by the EdgeTransport.exe process was observed; as well as low sustained rates of message
processing; and the slow creation of new transport rules. Given this limitation, if customer
requirements involve one or more transport rules per tenant, then the number of transport rules
would limit the number of tenants that could be hosted in a given Exchange organization (or AD
forest).
Offline Address Book (OAB) behavior: A functional issue was observed above 25,000 OABs
that would cause Outlook client downloads to fail. Given this limitation, if customer requirements

involve one or more OABs per tenant, then the number of OABs will limit the number of tenants
that could be hosted in a given Exchange organization (or AD forest).

General Recommendations for High-Scale Deployments

Microsoft provides generic performance and scalability guidance for Exchange 2013 in the Scalability
section of the Multi-Tenancy and Hosting Guidance for Exchange Server 2013. The sizing process for
high-scale hosted deployments requires careful attention to domain controller scalability because hosted
deployments tend to be associated with extremely large directories. The test results quoted later in this
document provide a sense of AD database growth as the number of test tenants increased.
Sizing should be performed with an understanding of user concurrency. Rather than sizing for the number
of provisioned tenants and users, the deployment should be sized for the maximum concurrent number of
users expected on the system. Historical user concurrency should be measured from an existing hosted
Exchange deployment, or if transitioning from another messaging platform, equivalent measurements for
the supported messaging protocols can be used as a starting point. Without a clear understanding of user
concurrency to base sizing calculations on, the safest option is to size for 100 percent concurrency per
the Microsoft recommendation for a typical enterprise deployment.
The deployment should be sized in such a way that additional capacity of a given physical resource can
be quickly provisioned to handle unanticipated increases in user demand. Microsoft recommends
deploying a significant amount of extra capacity during the initial deployment as a buffer to handle these
unanticipated increases, given the dynamic nature of your customer base. As with any high-scale
deployment, Microsoft recommends lab testing prior to production deployment to ensure that the specified
hardware is capable of meeting the workload requirements. Given the extreme scale of hosted
deployments, it may be difficult or impossible to perform lab validation of an entire forest simultaneously,
so Microsoft recommends that the design include a building block or scale unit approach in which a
subset of the hardware can be validated as a unit and multiple units can be put together to build the
forest.

AD Optimizations
Large-scale hosted Exchange deployments may see a significant volume of authentication requests on
the Client Access (CAS) servers, especially during peak periods when many users are trying to logon
concurrently. The authentication configuration should be evaluated to help ensure that the authentication
process works properly in a hosting deployment:
1. When client protocols use NTLM or basic authentication, the Netlogon service on the server that
the credentials were presented to will be responsible for validating the credentials via an AD
domain controller. A limited number of threads are available for concurrently processing these
authentication request. The number of threads is adjustable via the MaxConcurrentApi registry
key, as described in the Microsoft Knowledge Base article, 975363 -- You are intermittently
prompted for credentials or experience time-outs when you connect to Authenticated Services.
The default number of threads available is dependent on the server operating system used for
your CAS and domain controller servers. For example, Windows Server 2008 R2 and prior
versions default to two threads, and can be increased to a maximum ten threads. Windows
Server 2012 uses a default of ten threads, and this default was used in all scale testing covered
in this document. Windows Server 2008 R2 with SP1 and Windows Server 2012 can be
increased to a maximum of 150 threads. The impact of tuning the MaxConcurrentApi key may be
monitored via the Netlogon performance counters described in the Knowledge Base article,
975363.
2. This applies to Windows Server 2008 only: Kerberos authentication may also be improved via
application of the hotfix described in Knowledge Base article 2545833 -- Slow performance
occurs when many user authentication requests are handled in Windows Server 2008 R2.

Exchange Server 2013 Multi-Tenant Scalability Testing

Methodology
Goals
Functionality tests were carried out at defined points along the way as the number of tenant organizations
grew. The principal goal of this scale testing was to incrementally provision up to 50,000 tenant
organizations, each with a set of users, a unique ABP assigned to its users, one accepted domain, one
transport rule, one OAB, etc. Then we determined if there were any performance bottlenecks, scalability
limitations, lessons-learned, or other impacts to client functionality at that scale. Having said this, it is
important to note that tenants who are configured with multiple accepted domains* per tenant, or need to
assign transport rules and/or OABs to every tenant; this will impact the overall sizing of the Exchange
deployment, as discussed earlier in this document.
*Note: In this scale testing only one accepted domain was created per tenant, for a total of 50,000
accepted domains. However, 150,000 accepted domains is the maximum recommended by Microsoft, as
discussed in the Multi-Tenancy and Hosting Guidance for Exchange Server 2013 white paper.

Testing Script
The test team deployed a virtualized hosted lab environment of four Exchange Server 2013 Enterprise
servers. Tenant organizations were provisioned via a custom Exchange Management Shell script. Each
tenant organization included the following:

A tenant OU in AD.

Two global security groups.

An accepted SMTP domain.

A GAL, filtered by tenant [based on CustomAttribute1].

An All Rooms address list, filtered both by tenant (as above) and a RecipientDisplayType of
ConferenceRoomMailbox.

An All Users address list, filtered both by tenant (as above) and ObjectClass=user.

An All Contacts address list, filtered both by tenant (as above) and ObjectClass=contact.

An All Groups address list, filtered both by tenant (as above) and ObjectClass=group.

An OAB for the first 25,000 tenants, based on the tenant organizations GAL.

A single shared OAB that contained a single contact (hosters Support Desk, for example) which
was assigned to all tenants that did not have their own unique OAB.

An ABP, with the tenants GAL, address lists, and OAB (if it existed) linked to it.

Three mailbox-enabled users, with their CustomAttribute1 set to the tenant name.

Two mail-enabled contacts, with their CustomAttribute1 set to the tenant name.

Ten percent of tenants (5,000 total) were given a Public Folder hierarchy, with access rights such
that these folders were viewable by the tenants users only.

Mailboxes were evenly distributed across all databases; of which there were two databases per server for
a total of eight databases on four multi-role servers. All databases were members of a two-node database
availability group (DAG).
After all the above had been provisioned, the tenants OAB was updated using the UpdateOfflineAddressBook PowerShell cmdlet, and a new transport rule was created based on a
SubjectContainsWords filter. For reasons described earlier in this document and shown below in Table
2, OAB creation was stopped at 30,000 and OABs were then reduced to 25,000 total for the remainder of
testing. Also transport rule creation was stopped at 10,000 rules and were then reduced to 5,000 total for
the remainder of testing.
Milestone breakpoints were defined at the following tenant organization counts: 1,000; 5,000; 10,000;
20,000; 25,000; 30,000; 40,000; and 50,000. At each point, the following information was gathered:

Average end-to-end time to fully provision a tenant organization

Average time for each provisioning cmdlet to run.

At each break point, the following functionality tests were run:

Outlook Web App Test: Login to a new tenant users mailbox (for example, user01@tenant1.com)
using Outlook Web App
o Open the Address Book feature in Outlook Web App.
Verify that only those address books assigned by the tenants ABP are visible.
Verify each of those address lists are populated as expected.
o Send an e-mail to another user in the same tenant (for example, user02@tenant1.com).

Outlook Test: Create an Outlook 2010 user profile for another user in the same tenant (for
example, user02@tenant1.com)
o Verify that profile creation succeeds.
o Open the mailbox in Outlook 2010.
o Open the Address Book feature in Outlook.
Verify that only those address books assigned by the tenants ABP are visible.
Verify each of those address lists are populated as expected.
o Verify that the Outlook client can download the tenant OAB.
o Switch to offline mode and verify that the OAB is populated correctly.
o Verify that the e-mail sent from Outlook Web App has arrived.
o Reply to the e-mail from the previous test and verify that the replies appear in both the
Outlook Web App and Outlook clients.

Scale Testing Lab Setup

Distribution of Virtual Server Images on Hyper-V Root Servers
This section provides an overview of the number and types of servers in the scale testing lab, and shows
how they were distributed to Windows Server 2012 Microsoft Hyper-V host servers.

Distribution of Virtual Server Images on Windows 2012 Hyper-V Host Servers

Configuration of Virtual Server Images (RAM, CPU, Disk)

Table 1 shows the hardware configuration of both the Hyper-V host (physical) servers, as well as the RAM
and CPU allotments to the Hyper-V virtual server guests. Each of the four Exchange 2013 servers (MBXCASx) were deployed as a CAS and Mailbox combined multi-role server. Dont use this table as a
reference architecture, it is simply a definition of a hypothetical test topology. The disk, CPU and memory
sizing represented here are not designed to match the needs of any specific production workload and
indeed a real world environment with thousands of tenants and a corresponding number of users would
quite likely require an increase in hardware resources, above those that are listed below.
Table 1 Hardware configuration of Hyper-V root (physical) servers with guest image allotments

Hyper-V physical host

Hyper-V Host1: 8 Cores
2.27GHz Intel Xeon, 32GB
RAM, 13-drive RAID 6 Array
Hyper-V Host2: 8 Cores
2.4GHz Intel Xeon, 32GB
RAM, 13-drive RAID 6 Array
Hyper-V Host3: 8 Cores
2.27GHz Intel Xeon, 48GB
RAM, 13-drive RAID 6 Array
Hyper-V Host4: 8 Cores
2.4GHz Intel Xeon, 32GB
RAM, 13-drive RAID 6 Array
Hyper-V Host5: 8 Cores
2.27GHz Intel Xeon, 32GB

Guest virtual server name

AD1

Guest cores
8

Guest RAM
16 GB

AD2

16 GB

AD3
Win 8 Client with Outlook 2013

6
2

16 GB
4 GB

MBX-CAS1

16 GB

MBX-CAS2

16 GB

RAM, 13-drive RAID 6 Array

Hyper-V Host6: 8 Cores
2.27GHz Intel Xeon, 32GB
RAM, 13-drive RAID 6 Array
Hyper-V Host7: 8 Cores
2.4GHz Intel Xeon, 32GB
RAM, 13-drive RAID 6 Array

MBX-CAS3

16 GB

MBX-CAS4

16 GB

Test Results at Measurement Points

This section presents the performance benchmarks at each breakpoint of the scale testing.
The performance benchmarks listed in Table 2 were captured during maximum provisioning load; in other
words, tenant organizations (including mailbox-enabled users and mail-enabled contacts) were being
created as quickly as possible.
Table 2 Performance benchmarks at each phase of the scale testing

Total tenants
provisioned
1,000
5,000
10,000
20,000
25,000
30,000
40,000
50,000

Total transport
rules created
1,000
5,000
10,000*
5,000
5,000
5,000
5,000
5,000

Total OABs
created
1,000
5,000
10,000
20,000
25,000
30,000*
25,000
25,000

Average time to
provision tenant
9 seconds
30 seconds
74 seconds*
64 seconds
74 seconds
88 seconds
109 seconds
135 seconds

Outlook Web
App Test
Success
Success
Success
Success
Success
Success
Success
Success

Outlook
Test
Success
Success
Success
Success
Success
Failed**
Success
Success

* After the test environment had scaled to 10,000 tenants, including 10,000 transport rules, the transport
service continued to service requests, however the transport services ability to process at scale was
diminished. It was determined that a limit of 5,000 transport rules should not be exceeded in a single
Exchange Organization (or AD Forest). The script was updated to not create additional transport rules
and rules already created above 5,000 were removed.
**After the environment had scaled past 25,000 tenants and OABs, OAB creation began to fail and
Outlook could no longer reliably download the OAB. It was determined that a limit of 25,000 OABs should
be the limit in a single Exchange organization (or AD Forest). All other Outlook tests were successful. The
script was updated to not create additional OABs, and OABs already created above 25,000 were
removed.

Results at Scale Target

After reaching 50,000 tenant organizations and 150,000 total mailboxes, the environment was stable and
demonstrated acceptable performance. The Outlook Web App and Outlook tests succeeded with no
issues. No other scalability or performance issues were encountered.

Validation with Simulated Client Load

After the target scale of 50,000 tenant organizations had been reached, load testing was performed
against a single virtualized Client Access server to ensure all features remained functional with client load
at target scale.
Load testing consisted of Outlook Web App load, which was generated using the Microsoft Exchange
Load Generator (LoadGen) tool. The first test pass of the Outlook Web App load consisted of 500
concurrent Outlook Web App sessions for 8 hours, running the Outlook Web App 2013 Enterprise script

included with the LoadGen tool. The second test pass of Outlook Web App load consisted of 2,000
concurrent Outlook Web App sessions for 8 hours, again running the Outlook Web App 2013 Enterprise
script.
Performance was evaluated after the load testing was complete, and no abnormal issues were found in
the system.
The load simulation wasnt designed to simulate realistic client load at scale. It was simply designed to
ensure that the system remained functional with a lite amount of client load after the system had been
provisioned to the target level of tenant scale.

Exchange Logging Sizing Observations

New to Exchange 2013 is the default-enabled logging of various Exchange components, including the
capture of seven days of performance logs, as well as logs that capture the installation and configuration
of Exchange. These log files are kept in the %ExchangeInstallPath%\Logging folder. During scale testing
we captured the growth of this folder, the results of which are shown below in Table 3.

AD Sizing Observations
As the number of tenant organizations grows, the number of objects stored in AD grows. In turn, this
increase results in an increase in disk space consumed by the AD database file (NTDS.DIT), as shown in
Table 3. To assist with capacity planning, the NTDS.DIT size was captured during this scalability testing
process. The actual NTDS.DIT size of a given Exchange deployment may vary greatly depending on total
number of objects and utilization of various attributes. For example, storage of certificates, deployment of
third party tools and/or additional Microsoft products which use the AD database as a configuration store.
Table 3 Exchange Logging and AD NTDS.DIT sizing observations

Number of Tenant
Organizations
1,000 tenants
5,000 tenants
10,000 tenants
20,000 tenants
25,000 tenants
30,000 tenants
40,000 tenants
50,000 tenants

Size of NTDS.DIT
184 MB
694 MB
1.33 GB
2.60 GB
3.20 GB
3.83 GB
5.64 GB
6.32 GB

Size of Exchange
Logging Folder
2.1 GB
3.38 GB
6.19 GB
12.57 GB
13.23 GB
13.87 GB
15.14 GB
16.40 GB

The total number of objects stored in the AD database may be a limiting factor on the scalability of a given
multi-tenant Exchange forest because the size of the database may impact the time required to on-board
new domain controllers, backup domain controller databases, and may also impact the ability of AD to
successfully replicate changes throughout the AD topology. For more information about AD scalability
limits, see Active Directory Maximum Limits - Scalability. Exchange forests which will be designed to grow
to extremely large user counts will require additional pre-deployment testing to ensure that the design is
achievable given anticipated rate of AD object churn, requirements for time to on-board new domain
controllers, requirements for time to recover from a domain controller hardware failure, and other specific
deployment requirements.

Summary
Building a large-scale multi-tenant Exchange 2013 deployment is a very complex undertaking. Great care
should be taken during the design, pre-production testing, and deployment phases to ensure that all
scalability aspects of the system have been considered and are being closely monitored. No large scale
deployments of this type are the sameevery deployment has customizations that impact how the

10

system will function as the number of tenants and users increases. Therefore, testing, monitoring,
capacity planning, and risk mitigation are all critical aspects of a successful multi-tenant Exchange
deployment.
This document provides some high-level scalability guidance and deployment recommendations. We
welcome your feedback and comments on this document and will incorporate them into future versions if
required. If you have feedback, please send it to ExHostingFeedback@microsoft.com.

Legal Notice
This document is provided as-is. Information and views expressed in this document, including URL and
other Internet Web site references, may change without notice. You bear the risk of using it.
Some examples depicted herein are provided for illustration only and are fictitious. No real association or
connection is intended or should be inferred.
This document does not provide you with any legal rights to any intellectual property in any Microsoft
product. You may copy and use this document for your internal, reference purposes.
2013 Microsoft Corporation. All rights reserved.
Microsoft, MS-DOS, Windows, Windows Mobile, Windows Server, Active Directory, ActiveSync, Forefront,
Exchange, Outlook, and SharePoint are trademarks of the Microsoft group of companies.
All other trademarks are property of their respective owners.

11