Microsoft IT Tests LOB Compatibility with

Office 365 ProPlus

Published: June 2013
The following content may no longer reflect Microsofts current position or infrastructure. This content should be viewed as reference
documentation only, to inform IT business decisions within your own company or organization.

Microsoft IT gains additional efficiencies during the new Office deployment by

incorporating changes into their mature application compatibility testing process.

Intended Audience

Technical Decision Makers

Business Decision Makers
Products & Technologies
Office 365 ProPlus

Introduction
Application compatibility testing is critical for the successful deployment of new software in any enterprise environment. Microsoft
Information Technology (Microsoft IT) faces the same challenge maintaining an application portfolio that consists of approximately
1,500 applications in its internal line-of-business (LOB) application portfolio. Prior to each new client product release (Windows,
Microsoft Office, and Microsoft Internet Explorer), including the early validation of pre-release software (also known as dogfood
testing), Microsoft IT conducts compatibility testing on the most critical LOB applications and websites used within Microsoft. Testing
results provide feedback to Microsoft product groups to help improve Microsoft products before their release and allow LOB
application owners to update their code and inform users of any issues or workarounds to help ensure a good user experience.
With each testing program that Microsoft IT conducts, they review the processes of past programs and introduce new efficiencies
wherever possible, helping to reduce their IT costs. This article describes the processes that Microsoft IT used for their application
compatibility testing program during the new Office deployment.

Planning
The first step in testing for application compatibility is understanding the applications present in an environment and identifying the
subsets that are business-critical. This information will enable you to build a test plan and test matrix of the type and level of testing
that should be performed on each key application.

Application Portfolio Management

Microsoft IT uses an Application Portfolio Management (APM) tool to track a wealth of historical application data. This enables them
to identify which applications are most likely to provide valuable feedback to the product group.
From an application portfolio perspective, Microsoft IT tracks LOB dependencies on Office, Windows, Internet Explorer, and the
Microsoft .NET Framework to help target groups of applications that are dependent on specific client technologies or products. They
also use two types of application designations, primary and secondary.

Primary. Critical applications required for business continuance, such as payroll and expense reporting, procurement, hiring
and resource management tools, sales, and so on. Testing of primary applications is required during all test cycles.

Secondary. Less-critical key applications that the business depends on and identified through voluntary participation from
internal groups. Testing of secondary applications is voluntary and may occur at any point during the test cycle.

Tracking this information helped Microsoft IT focus their testing on a much more manageable subset of the application
portfolio.

Testing Plan
Your testing plan needs to account for not only the LOB applications with dependencies on Office, but also any changes to
the operating system and browser that may have an impact on your end users.
Number and Timing of Test Passes
Internal testing for both Windows 8 and the new Office started within weeks of each other. Since Microsoft IT identifies
which LOB applications have an Office and a Windows dependency in their APM tool, they were able to gain efficiency and
reduce the impact on their internal testing resources by combining testing of both products later in the release cycle. Early in
the testing process, Microsoft IT does not combine products in order to more easily identify the source of any issues found.
For the new Office release, Microsoft IT focused iterative (repeated) testing on approximately 59 primary applications during
each compatibility test cycle. In addition, Microsoft IT performed voluntary testing on secondary applications on an ad-hoc
basis, increasing coverage to 148 Office-dependent LOB applications tested on Windows 7. Later, integration with Windows
8 testing increased the number of unique LOB applications tested with the new Office to 266. Helpdesk calls were monitored
as a means of validating that the right set of applications were being tested.
Testing Configuration Matrix
As part of the application compatibility testing for the new Office deployment, Microsoft IT needed to develop a test matrix
that enabled them to test various combinations of OS, browser version, version of Office, and deployment means to ensure
the best end user experience. This matrix resulted in testing LOB applications on both the Windows 7 and Windows 8
platforms with Internet Explorer 10, deploying Microsoft Office 2010 and Microsoft Office 365 ProPlus in stand-alone and
side-by-side scenarios, and evaluating using MSI versus Click-to-Run as a delivery mechanism (in the past, Microsoft IT has
always used the MSI version of Office).
Microsoft IT developed a plan that used testing best practices to limit the number of configuration changes introduced into the
environment. Since Office 2010 and Windows 7 were both known compatible environments, their test plan started with
testing Office 2010 on Windows 8 and testing Office 365 ProPlus on Windows 7. These two scenarios mirrored the most
common configurations seen in the Microsoft user's environment, where the largest set of users were running Windows 7
and a pre-release version of Office 365 ProPlus. Then combining the testing of Windows 8 and Office 365 ProPlus as both
deployments gained critical mass and would soon be the dominant configuration in the environment.
Microsoft IT tested the following configurations during the new Office testing program:
Windows

Office

Internet Explorer

Windows 7 SP1

Office Professional 2013

Internet Explorer 9

Windows 7 SP1

Office 2010 + Office 365

ProPlus side-by-side

Internet Explorer 9

Windows 8

Office 2010 + Office 365

ProPlus side-by-side

Internet Explorer 10 Classic

Browser

Windows 8

Office 365 ProPlus

Internet Explorer 10 Classic

and Modern Browsers

Testing Considerations
There are a large number of compatible configurations available. Customers can adopt Office Professional Plus 2013 (MSI)
or Office 365 ProPlus (Click-to-Run), running on Windows 7 or Windows 8. This allows customers a wide variety of options
for how they want to run their IT.

Microsoft IT Tests LOB Compatibility with Office 365 ProPlus

Page 2

Internally at Microsoft, Microsoft IT validates everything that is being released and has tested the various compatible
configurations. Customers need to take an approach that matches the configuration they will deploy.
The following provides some guidance on items to consider when moving to the new Office.

Transitioning to Windows 7 or Windows 8. If you are running Windows XP and are contemplating moving to
Windows 7 with the new Office, Microsoft IT has tested both Windows 7 and Windows 8 with the new Office, helping
to ensure that the new Office would work for customers transitioning to either platform.

Classic versus Modern browser. While testing browser compatibility was not strictly a requirement for the new
Office deployment, approximately 94 percent of Microsoft IT's application portfolio is web-based (the remainder is
installed desktop/smart client applications developed with .NET), so Microsoft IT focused compatibility testing on
browser updates to learn how this might affect any key internal web-based LOB application. As Windows 8 offers
two different browser experiences, Microsoft IT recommends testing LOB applications for compatibility against both
the Classic and Modern browsers. If incompatibilities occur during testing with the Modern browser, customers may
be able to use their business critical applications with Classic browser while they upgrade their applications to be
more WC3 standards compliant or until the application is retired.

Note: For more information about Windows 8 and Internet Explorer 10 considerations, see Managing Windows 7 and
Windows 8 Side-by-Side at http://technet.microsoft.com/en-us/library/dn151702.aspx and Compatibility in Office 2013 at
http://technet.microsoft.com/en-us/library/ff394407.aspx.

Testing
Testing should be automated where possible. A virtual machine (VM) testing environment provides for an efficient, safe, and
non-destructive way to test configurations before a wide-scale rollout without impacting the day-to-day hardware. In addition,
how you report the status of applications tested should be consistent and easily understood by all teams reviewing reports.

Open Testing Model

While application compatibility testing has been part of previous deployments, Microsoft IT took some new approaches to
increase efficiency in this deployment. To minimize cost and effort associated with the testing process, Microsoft IT leveraged
desktop virtualization and virtualization management features available in Windows Server 2008 R2 and Microsoft System
Center Virtual Machine Manager 2008 R2. Using dynamically provisioned virtual machines (VMs), Microsoft IT tested LOB
client applications on pre-configured virtual desktops. This provided a higher level of control and efficiency by providing
identical configurations for all testers, eliminating the need and time required to build out physical machines. Issues found
were attributable only to the new software, because the only change to the testing environment was the version of Office. All
of the other software components installed were known to be compatible from previous testing programs.
Another big change was that Microsoft IT moved to a new open testing model. In the past, testers were required to submit
VM requests via email, and then had to wait for notification when their pre-configured VMs were available, delaying the
testing process. With the new open testing model, based on Microsoft System Center Virtual Machine Manager technology,
testers can provision one or more VMs using templates through a self-service portal. Testers can interact with the VM as
soon as the creation process is complete. This change vastly reduced lost testing time, eliminating the need for testers in
remote locations to wait for their requests to be completed by Puget Sound-based support personnel.
Looking ahead, Microsoft IT plans to increase their efficiency further in two waysfirst, by changing the way that VMs are
hosted using System Center Virtual Machine Manager 2012. And second, they plan to increase efficiency by moving from a
distributed voluntary model, with testers in various Microsoft IT groups performing LOB application testing when requested to
a centralized dedicated group whose sole purpose is to repeatedly test Microsoft ITs primary applications year-round on new
client configurations. Moving to a dedicated group will significantly reduce the number of testers participating while still
providing the same valuable feedback, and result in continuous feedback to Microsoft product groups rather than only
periodic feedback during release cycles.

Microsoft IT Tests LOB Compatibility with Office 365 ProPlus

Page 3

Status Reporting
A new scorecard was created leveraging Microsoft SQL Server Reporting Services that allowed key test managers to
monitor the progress of their test teams in real time. Test managers were able to refresh the report at any time to show the
latest testing data entered in Microsoft ITs LOB portfolio management tool. The new scorecard included a report divided by
organization and group. Each group was monitored with a percentage score tracking completion of their testing of the
applications most critical to daily business. Prior to the new reporting solution, participation and current status were tracked in
separate reports that were manually updated with the results of SQL queries made against the LOB application portfolio
database. The data displayed was static until the next time the report was manually updated.
Participating in pre-release testing of Microsoft client products is an integral part of Microsofts culture. With the inclusion of
non-Microsoft IT groups, the LOB client application compatibility stakeholder community swelled from about 500 members to
over 850. Of those individuals, approximately 150 were directly involved in the testing of LOB applications. An Active
Directory-based security group was created to manage membership and to provide a means to easily communicating
status updates and changes to all stakeholders. It also provided access to security-restricted resources, such as the Product
Groups bug database, and Microsoft ITs LOB Application Compatibility Microsoft SharePoint site, where testing program
reports, bug templates, and the LOB testing schedule were published.

Deployment
Pilot programs are familiar to anyone who has worked on a deployment project. In a pilot program, an application or product
is deployed to a small control group of users, typically skilled users in multiple affected groups. The small number prevents
costly down time to a business group if issues are encountered, yet the number must be large enough to establish
statistically meaningful field test data.

Deployment Milestones and Targets

Microsoft IT takes a ramped approach to deployment. Microsoft IT tests pre-release and dogfood builds of products released
to internal Microsoft users prior to the final Release to Market (RTM) build.
This ramped process was no different with the new Office deployment.

Initial release was to a controlled group of full-time employees, approximately 31,000 users, with the builds hosted
on a restricted access share to control distribution.

Beta 2 (also known as Customer Preview) release was open to the entire company with a deployment goal of
55,000.

RTM release ramped up to full deployment with the goal of 95 percent of all employees using the new Office.

Beyond the testing that occurs in the IT labs, the user community provides a larger, shallow level testing of LOB applications
as they are using the application in day-to-day business and running into interactions, scenarios, and/or other configurations
that Microsoft IT cannot cover in their clean install approach. VMs represent a pristine environment, whereas the user's
desktop may have additional items installed that can impact a LOB application.
Note: For more information on deploying Office 365 ProPlus, see Microsoft IT Deploys and Manages Office 365 ProPlus at
http://technet.microsoft.com/library/dn283378.aspx.

Compatibility Guidance
Microsoft IT strives to achieve a high level of compatibility before deploying a new product, such as Office, but does not
require 100 percent compatibility before internal deployment. Microsoft IT moves forward without major impact in productivity
before product development work is complete. Where there are remaining issues with the new product, Microsoft IT focuses
on finding workarounds to mitigate the impact to early adopters.
Microsoft IT publishes results of application compatibility testing through the Microsoft internal website, IT Web. For the new
Office deployment, IT Web provided users with a single, trusted source for information to learn about the state of LOB
applications, any known issues or workarounds to allow the user to make an informed decision whether to move forward with
adopting the new Office. Microsoft IT created a distribution group that was used to send notification when application
Microsoft IT Tests LOB Compatibility with Office 365 ProPlus

Page 4

compatibility results changed. Users could join the group with a single click from the applications compatibility results page
on IT Web. This allowed users to move to the new Office as soon as any issues with their key applications were resolved.
Providing a centralized mechanism to make users aware that there are application issues is one of the enablers that allows
for a rolling deployment.

Conclusion
Application compatibility testing is crucial for the successful deployment of new software in an enterprise environment.
Microsoft IT has developed an efficient approach to LOB application compatibility testing and applied that approach when
deploying the new Office. Microsoft IT recommends that you:

Understand your environment. Survey of your systems and identify both LOB application dependencies with the
new Office but also on any OS and browser dependencies. Store historical data to help you focus your testing and
avoid having to test your entire application portfolio. Test a subset of your portfolio, testing either the most complex
applications, applications that historically have been known to have issues, or the applications that are most critical
for the business. By picking the right subset of applications that leverage Office products and technologies the same
way, you can become efficient and reduce the number of applications that you need to test.

Create a test plan. A test plan and test matrix should be sufficient to enable you to test the various configurations in
your environment and to represent the typical LOB application to minimize the need to test every LOB application.

Use testing best practices. Introduce a single change into a controlled test environment to make it easier to
identify the source of compatibility issues.

Gain efficiencies through automation. Using a test environment based around VM technology will enable you to
perform a large number of tests in a short period without impacting your day-to-day hardware.

Use a rolling deployment. Use a pilot approach, deploying the new Office to a small set of users, and leverage
their feedback on interaction with their application portfolio to identify issues. Make sure that you provide users with
guidance on what issues exist and any workarounds so they can continue to be productive and you can continue to
resolve compatibility issues.

Benefit from the side-by-side installation opportunity presented by Click-to-Run. The new Office Click-to-Run
deployment model allows subscription customers to install the new Office alongside an existing Office release,
which can help unblock any LOB applications that are not yet compatible with the new Office.

For More Information

For more information about Microsoft products or services, call the Microsoft Sales Information Center at (800) 426-9400. In
Canada, call the Microsoft Canada Order Centre at (800) 933-4750. Outside the 50 United States and Canada, please
contact your local Microsoft subsidiary. To access information via the World Wide Web, go to:
http://www.microsoft.com
http://www.microsoft.com/technet/itshowcase
To learn more about the new Office deployment at Microsoft, review the two companion articles:

Microsoft IT Deploys and Manages Office 365 ProPlus http://technet.microsoft.com/library/dn283378.aspx

Microsoft IT Helps Users Embrace Office 365 ProPlus http://technet.microsoft.com/library/dn283375.aspx

2013 Microsoft Corporation. All rights reserved.

This document is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY. Microsoft,
Active Directory, Internet Explorer, Office 365, SharePoint, SQL Server, Windows, and Windows Server are either registered trademarks or
trademarks of Microsoft Corporation in the United States and/or other countries. The names of actual companies and products mentioned herein may
be the trademarks of their respective owners.

Microsoft IT Tests LOB Compatibility with Office 365 ProPlus

Page 5