AI - Introducing ModelOps To Operationalize AI

Licensed for individual use only
Introducing ModelOps To Operationalize AI

The Core Capability That Enterprises Need To Deploy, Monitor, And Govern Machine Learning
Models
by Kjell Carlsson, PhD and Mike Gualtieri

August 13, 2020
Why Read This Report Key Takeaways

Model operations (ModelOps) is a must-have Sleeping With One Eye Open Is Unsustainable
capability to operationalize AI at scale. Also A top complaint of data science, application
known as machine learning operations (MLOps), development and delivery (AD&D) teams, and,
ModelOps comprises tools, technologies, and increasingly, line-of-business leaders is the
practices to enable organizations to deploy, challenge in deploying, monitoring, and governing
monitor, and govern machine learning models machine learning models in production.
and other analytical models in production Manual handoffs, frantic monitoring, and loose
applications. Technology and data science governance prevent organizations from deploying
pros should read this report to understand why more AI use cases.
ModelOps is important, the capabilities they
ModelOps Is About More Than Moving Bits
need, and how to develop or acquire them.
Deploying models doesn’t end with provisioning
infrastructure and copying code. Machine
learning models are unique in that they must be
constantly monitored while in production and
regularly retrained, requiring the collaboration
of a host of stakeholders from data scientists to
ops pros.
Vendors Are Flexing Their ModelOps Muscles

This is a breakout year for ModelOps solutions
from a diverse set of vendors. Independent
ModelOps vendors, predictive analytics and
machine learning (PAML) tool vendors, and
cloud and infrastructure vendors all offer
ModelOps solutions.
This PDF is only licensed for individual use when downloaded from forrester.com or reprints.forrester.com. All other distribution prohibited.
forrester.com
For Application Development & Delivery Professionals

The Core Capability That Enterprises Need To Deploy, Monitor, And Govern Machine Learning
Models
by Kjell Carlsson, PhD and Mike Gualtieri

with Srividya Sridharan and Jeremy Vale
August 13, 2020
Table Of Contents Related Research Documents

2 Your AI Transformation Is Doomed Without The Forrester Wave™: Multimodal Predictive
ModelOps Analytics And Machine Learning Solutions, Q3
2018
3 Start Flexing Three Sets Of Must-Have
ModelOps Muscles The Forrester Wave™: Notebook-Based
Predictive Analytics And Machine Learning
No. 1: Deploy And Serve AI Models
Solutions, Q3 2018
No. 2: Monitor Model Metrics
Now Tech: Predictive Analytics And Machine
No. 3: Govern The AI Lifecycle Learning, Q2 2020
10 Build Or Buy Your ModelOps Muscles Shatter The Seven Myths Of Machine Learning
Recommendations
11 Do ModelOps Or Learn The Hard Way
Share reports with colleagues.

Enhance your membership with
Research Share.
Forrester Research, Inc., 60 Acorn Park Drive, Cambridge, MA 02140 USA

+1 617-613-6000 | Fax: +1 617-613-5000 | forrester.com
© 2020 Forrester Research, Inc. Opinions reflect judgment at the time and are subject to change. Forrester®,
Technographics®, Forrester Wave, TechRadar, and Total Economic Impact are trademarks of Forrester Research,
Inc. All other trademarks are the property of their respective companies. Unauthorized copying or distributing
is a violation of copyright law. Citations@forrester.com or +1 866-367-7378
For Application Development & Delivery Professionals August 13, 2020
The Core Capability That Enterprises Need To Deploy, Monitor, And Govern Machine
Learning Models
Your AI Transformation Is Doomed Without ModelOps

Enterprises need to deploy AI solutions to become industry leaders, but they struggle to deploy, monitor,
and govern even a single AI model in production applications. As one team put it, “You build and train
your model and then you run into the nightmare of putting it into production.”1 This process takes too
long and is fraught with technical and business challenges, just with one model. What about a dozen
use cases and models? A hundred? A thousand? To handle that, enterprises need ModelOps — a rapid,
repeatable, and scalable process that AI teams can use to implement more high-ROI AI use cases and
manage them continuously across the AI model lifecycle (see Figure 1).2
Forrester defines ModelOps, also known as MLOps, as:3
Tools, technology, and practices that enable cross-functional AI teams to efficiently deploy,
monitor, retrain, and govern AI models in production systems.
FIGURE 1 ModelOps Is The Crank That Turns The AI Lifecycle Wheel
Identify data that is relevant

to the business goal.
Measure the effectiveness Integrate and enrich the

of the model in the real data into an analytical
world. Understand data data set.
Monitor Prepare data
Model
lifecycle
governance
Deploy Model
Use the model in Evaluate Run statistical and

applications. machine learning
algorithms to find the
model.
Test the model to make

sure it will work.
© 2020 Forrester Research, Inc. Unauthorized copying or distributing is a violation of copyright law. 2
Citations@forrester.com or +1 866-367-7378
Learning Models
Start Flexing Three Sets Of Must-Have ModelOps Muscles

The AI lifecycle must be continuous if it is to be successful. Data engineers acquire and prepare
data. Data science and machine learning (ML) teams analyze that data to create models. Developers
integrate those models into applications. Infrastructure operations pros provision and manage
the infrastructure needed to run applications and models. The process cycles on as teams retrain
models from new data to redeploy them. For this process to work, companies require three sets of
ModelOps capabilities.
No. 1: Deploy And Serve AI Models
Developing an AI model is just the first part; now you need to get it into production. This means
connecting it to production data, integrating it into applications, and provisioning infrastructure.
Without ModelOps this can take weeks, months, or never happen at all. For example, it took a major
US bank two weeks to develop a fraud model, two weeks to document it, and eight months to
integrate into its systems — and that was considered a success story. With this type of time lag and
redevelopment, models can perform very differently than during development, which reduces their
performance and leads to additional rework. ModelOps deployment capabilities help enterprises:
›› Orchestrate deployment and serving. ModelOps helps teams orchestrate the deployment of the
model. This includes provisioning of infrastructure, staging the model, managing dependencies,
orchestrating the multiple steps that occur when a model is called, and serving the model in a
robust, scalable, and high-availability fashion. For example, a major pharmaceutical company
leveraged ModelOps pipelines to train a model that predicts pulmonary diseases in patients,
evaluates it, deploys it on a public cloud, and periodically monitors and retrains it.4
›› Deploy models from anywhere . . . Enterprises usually use multiple predictive analytics and
machine learning (PAML) tools based on the skills of their data science teams and to take
advantage of the latest innovations in AI. Hence, they need to support the deployment of models
developed using multiple tools. This includes open source ML programming languages and
frameworks like Python and R as well as commercial ML tools so enterprises can take advantage
of all best-of-breed solutions.
›› . . . to anywhere. Organizations must develop the ability to deploy models across an ever-growing
range of on-premises, private cloud, public cloud, hybrid cloud environments, and specialized
AI hardware. Not only does this help them meet the data and application requirements of their
AI use cases, but it also helps them take advantage of the cost, scalability, latency, and security
tradeoffs of each environment. For example, Datactics deploys models across a host of different
cloud environments, depending on what its customers use, and with a combination of open source
technologies, including Data Version Control for versioning, Airflow for orchestration, and MLflow to
manage the model lifecycle.
Learning Models
No. 2: Monitor Model Metrics
Models don’t get better on their own; they usually get worse as the world changes and the data drifts
further and further from the data the model was trained on (see Figure 2). ModelOps helps you monitor
important production model metrics and alerts you when your models may need to be retrained. It also
accelerates your ability to retrain your models by automatically capturing and selecting the new training
data. An online service provider found that without ModelOps, it took 21 days to detect that a model
was underperforming and an additional four to five weeks to assemble the necessary data to retrain it.5
Enterprises must use ModelOps to monitor their models for:
›› Data drift. The world is always changing, and the data feeding your models changes with
it, whether that is due to user behavior or your changing internal systems. By monitoring the
distribution of data feeding a model and detecting when it changes, you get early warning that
your model may not be getting the right data or may no longer be up to date (see Figure 3). An
eCommerce firm shortened the time it took to identify performance issues in its fraud detection
models from one to three months to under a day by monitoring its models for data drift and
assessing the impact, instead of waiting to see a spike in fraudulent transactions.6
›› Prediction distribution drift. Maybe your model can handle the fluctuations in the input data (e.g.,
because that data isn’t very important for the model), or maybe it can’t. By tracking your models’
predictions over time, ModelOps helps you detect when your models are making substantially
different predictions compared to normal, which can be a sign that your model needs to be
retrained or your pipeline isn’t working correctly. For example, a sudden spike in the number of
mortgage applications that your model thinks are low risk could indicate that your efforts to market
to low-risk customers are paying off, but it’s more likely a sign that your model needs retraining. By
detecting the spike, ModelOps lets you know when you should investigate.
›› Concept or business KPI drift. The ultimate proof of whether your model needs to be retrained is
when your model starts making incorrect predictions that affect your important business outcomes.
ModelOps helps you close the loop, enabling you to compare what actually happened to the
model’s prediction and its impact on business KPIs. Better still, it helps you identify the drivers
causing that incorrect prediction (see Figure 4).
›› Explainability and fairness. Enterprises can use ModelOps to provide explanations for model
predictions, using techniques like SHAP or surrogate models, or to monitor models for fairness.7
However, not all enterprise use cases will need prediction-level explanations or monitoring for
ethical considerations, and other tools can also meet these needs (see Figure 5).
Learning Models
FIGURE 2 Enterprises Must Monitor, Retrain, And Redevelop Machine Learning Models Over Time
Business
value
Maximum
business value
Time deployed
Retrain or
redevelop
Learning Models
FIGURE 3 Example Of Monitoring Models For Data Drift
Selectable data drift

test type
Visualizations of Compare threshold

changes in distribution versus actual drift.
Source: Domino Data Lab
Learning Models
FIGURE 4 Example Of Monitoring Model Accuracy And Metrics Impacting Accuracy
System triggers alerts Period of abnormally

in real time to detect low accuracy relative
anomalies. to baseline
Relevant metrics
correlated with
accuracy drop
Source: superwise.ai
Learning Models
FIGURE 5 Example Of Monitoring A Model For Age Discrimination
Groups to monitor for

fairness and other
types of bias
Source: IBM Watson OpenScale
No. 3: Govern The AI Lifecycle
AI is ultimately a type of software. As such, to be governed it must have versioning, lifecycle lineage,
and security capabilities — so that any team member can access what others have done, replicate
it, validate it, and build on it. Tools accessible by and designed for broader AI teams of business,
AD&D, IT ops, and data science enable collaboration. Most importantly, enterprises need ModelOps
governance capabilities that:
›› Track metadata, lineage, provenance, data, and other dependencies of models. Step 1 in
governing the AI lifecycle and driving collaboration among teams is tracking and documenting
what was done using what. These ModelOps capabilities help you reproduce predictions, fix faults,
accelerate your ability to both retrain models and reuse pipelines, and provide an audit trail for
regulators (see Figure 6).
›› Orchestrate workflows. Data and ML model pipelines are often complicated, involving numerous
data sources, transformations, and multiple models. The workflows to monitor, notify, and update
these pipelines as well as approve changes are similarly complex to manage. Enterprises need the
capabilities to define, manage, and update these workflows to coordinate the disparate teams. For
Learning Models
example, some ModelOps solutions can implement workflows that detect data drift, notify a data
engineer to investigate, determine whether a data scientist needs to retrain that model, and solicit
approval from a business user to deploy the updated model (see Figure 7).
›› Provide security and compliance. Organizations need to ensure that everyone is playing the right
role and working together effectively. That means having capabilities to determine access and
permissions for data, infrastructure, tools, model artifacts, and project repositories as well as to
both define and enforce policies for usage, sharing, data protection, and so on. For example, the
United Nations used a ModelOps platform to build a ML model repository and provide access to its
data scientists and member states globally.8
FIGURE 6 Example Of Tracking Model And Data Lineage
Source: Cloudera
Learning Models
FIGURE 7 Example Of A Model Update Workflow
Source: ModelOp
Build Or Buy Your ModelOps Muscles

How do you start building these vital ModelOps capabilities? ModelOps is increasingly recognized as
the key to operationalizing AI at scale, and there has been an explosion of development in the last two
years with existing vendors upgrading their capabilities and numerous new entrants. To develop your
ModelOps capabilities, consider:
›› Existing predictive analytics and machine learning solutions. Odds are the PAML solutions that
your data science teams use today to develop AI models have basic ModelOps capabilities that
you can leverage immediately for the price of an additional license. Further, many PAML solutions
are rapidly expanding their ModelOps capabilities to encompass more must-have capabilities.
While most focus on deploying and managing models developed with their solution, they
increasingly support a wider array of models developed with other tools, usually the most popular
open source tools. They may not provide a full solution, in terms of supporting all your AI models or
providing all the capabilities you need — but you’ll be able to get ModelOps capabilities rapidly.9
Learning Models
›› Specialized third-party ModelOps solutions. A new market has sprung up composed of

startups offering specialized solutions for ModelOps. Vendors like Algorithmia, ModelOp, Modzy,
Quickpath, and superwise.ai are agnostic as to where a model comes from and focus solely on the
deployment, monitoring, and governance of models. They typically have unique strengths in either
deploying models across many environments, monitoring operational KPIs, managing workflows,
or providing security and regulatory compliance.
›› Building it yourself. Enterprises with the most mature AI use cases have had to build their own
solutions, and most continue to do so today. Using an ecosystem of open source components —
often Jupyter Notebooks, Kubernetes, Data Version Control (DVC), and a few open source machine
learning frameworks — you can build your own ModelOps solutions too. For enterprises that
have already invested in homegrown solutions or that have specialized needs, this may be your
best option . . . for now. Most enterprises, however, will want to forgo the time, effort, and cost of
building and maintaining their own ModelOps platforms.
Recommendations
Do ModelOps Or Learn The Hard Way

Enterprise AI is here to stay because every industry has promulgated high-ROI use cases. It’s common
for even a single use case to result in millions of dollars of business value. The more use cases, the
more value. Enterprises mustn’t “hit the wall” of AI by being either nonchalant or blissfully ignorant
toward operationalizing ML models. You must:
›› Do it now. Given the rapid developments in ModelOps offerings, many enterprises will be tempted
to wait until the offerings have matured and the market has stabilized. That is not an option.
To unlock the value of their AI initiatives today, all organizations need to rapidly improve their
ModelOps capabilities using the options that are available to them now.
›› Learn lessons from your DevOps team. Most enterprises have a DevOps functions that has
learned, perhaps the hard way, the machinations necessary to deploy software to production
environments. ModelOps is not DevOps, but there is much overlap with the working collaborations
to make ModelOps a success.
›› Put someone in charge. ModelOps is underdeveloped in most organizations because it has fallen
through the cracks, with no team having the knowledge, responsibility, or authority to develop it
into a cross-functional capability. Unless you empower someone and give them the resources and
mandate, expect slow progress. Look to leaders in DevOps, operations, and/or integrated data
science teams that have a proven track record of putting assets into production and have them
build a shared ModelOps capability for the organization.
Learning Models
›› Formalize and routinize your existing ModelOps processes. They might not be ideal, but if
you’ve put models into production, then you have ModelOps processes that you can incrementally
improve. Document them, look for ways to improve them quickly, and make them more widely
available so your organization can start taking advantage of AI use cases faster.
›› Evaluate the offerings regularly. If you did an assessment of ModelOps offerings six months ago,
that assessment is out of date, as nearly every solution on the market has developed significantly
since then. Assess the market’s current offerings, leverage the ones that suit your needs now, and
look again in six months to a year.
›› Have a plan if you have to build. Some enterprises will find they need to build their own
ModelOps solutions in-house; do this as a last resort after you have assessed the costs and
benefits of doing so relative to the solutions on the market. And when you do build your own
ModelOps, plan for how you’ll develop this solution to meet the needs of the broader enterprise.
Beware the temptation to let individual teams explore building their own platforms. It may look
cheap and quick in the short run, but it usually magnifies the pain later on (in terms of difficulty to
maintain and scale).
Learning Models
Engage With An Analyst

Gain greater confidence in your decisions by working with Forrester thought leaders to apply
our research to your specific business and technology initiatives.
Analyst Inquiry Analyst Advisory Webinar
To help you put research Translate research into Join our online sessions
into practice, connect action by working with on the latest research
with an analyst to discuss an analyst on a specific affecting your business.
your questions in a engagement in the form Each call includes analyst
30-minute phone session of custom strategy Q&A and slides and is
— or opt for a response sessions, workshops, available on-demand.
via email. or speeches.
Learn more.
Learn more. Learn more.
Forrester’s research apps for iOS and Android.

Stay ahead of your competition no matter where you are.
Endnotes
Source: Forrester client inquiry.
1
Source: “ModelOps: AI At Scale Isn’t Possible Without It,” Forrester (https://www.forrester.com/

2
fn/5NVraa31eBopO5IqYqUvx7).
Technically, machine learning ops (or MLOps) refers only to the machine learning model lifecycle, while ModelOps also
3
includes other types of models, such as decisioning models and optimization models. However, the two terms are
frequently used interchangeably.
Source: Customer example shared by Brillio.

4
Source: Phone interview with Ofer Razon, cofounder and CEO of superwise.ai, June 2, 2020.
5
Source: Phone interview with Ofer Razon, cofounder and CEO of superwise.ai, June 2, 2020.
6
SHAP: SHapley Additive exPlanations. Source: Scott M. Lundberg and Su-In Lee, “A Unified Approach to Interpreting
7
Model Predictions,” arXiv.org, November 25, 2017(https://arxiv.org/pdf/1705.07874.pdf).
Learning Models
Source: “The United Nations Global Working Group brings machine learning to the world using
8
Algorithmia,” Algorithmia, 2018 (https://info.algorithmia.com/hubfs/2018/Case-Studies/Algorithmia-

United-Nations-Global-Working-Group-Case-Study.pdf?__hstc=69686351.48d92db22b113c87599fd5724
1b40600.1559587119844.1562700368303.1562704479583.51&__hssc=&hsCtaTracking=90e24364-80ee-4b88-9842-
4fa0022e62cb%7Ccd26c6d3-4fe4-4840-a6e6-ed157f9b0773&hsLang=en-us).
For evaluations of the ModelOps capabilities of the most important PAML vendors, see the Forrester report “The
9
Forrester Wave™: Notebook-Based Predictive Analytics And Machine Learning Solutions, Q3 2018” and see the
Forrester report “The Forrester Wave™: Multimodal Predictive Analytics And Machine Learning Solutions, Q3 2018.”
forrester.com
We work with business and technology leaders to drive customer-

obsessed vision, strategy, and execution that accelerate growth.
Products and Services
›› Research and tools
›› Analyst engagement
›› Data and analytics
›› Peer collaboration
›› Consulting
›› Events
›› Certification programs
Forrester’s research and insights are tailored to your

role and critical business initiatives.
Roles We Serve
Marketing & Strategy Technology Management Technology Industry
Professionals Professionals Professionals
CMO CIO Analyst Relations
B2B Marketing ›› Application Development
B2C Marketing & Delivery
Customer Experience Enterprise Architecture
Customer Insights Infrastructure & Operations
eBusiness & Channel Security & Risk
Strategy Sourcing & Vendor
Management
Client support
For information on hard-copy or electronic reprints, please contact Client Support at
+1 866-367-7378, +1 617-613-5730, or clientsupport@forrester.com. We offer quantity
discounts and special pricing for academic and nonprofit institutions.
160698

AI - Introducing ModelOps To Operationalize AI

Hochgeladen von

Dokumentinformationen

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

AI - Introducing ModelOps To Operationalize AI

Hochgeladen von

Copyright:

Verfügbare Formate

Licensed for individual use only

Introducing ModelOps To Operationalize AI

by Kjell Carlsson, PhD and Mike Gualtieri

Why Read This Report Key Takeaways

Vendors Are Flexing Their ModelOps Muscles

Introducing ModelOps To Operationalize AI

by Kjell Carlsson, PhD and Mike Gualtieri

Table Of Contents Related Research Documents

11 Do ModelOps Or Learn The Hard Way

Share reports with colleagues.

Forrester Research, Inc., 60 Acorn Park Drive, Cambridge, MA 02140 USA

Your AI Transformation Is Doomed Without ModelOps

Forrester defines ModelOps, also known as MLOps, as:3

FIGURE 1 ModelOps Is The Crank That Turns The AI Lifecycle Wheel

Identify data that is relevant

Measure the effectiveness Integrate and enrich the

Monitor Prepare data

Use the model in Evaluate Run statistical and

Test the model to make

Start Flexing Three Sets Of Must-Have ModelOps Muscles

No. 1: Deploy And Serve AI Models

No. 2: Monitor Model Metrics

FIGURE 3 Example Of Monitoring Models For Data Drift

Selectable data drift

Visualizations of Compare threshold

Source: Domino Data Lab

FIGURE 4 Example Of Monitoring Model Accuracy And Metrics Impacting Accuracy

System triggers alerts Period of abnormally

FIGURE 5 Example Of Monitoring A Model For Age Discrimination

Groups to monitor for

Source: IBM Watson OpenScale

No. 3: Govern The AI Lifecycle

FIGURE 6 Example Of Tracking Model And Data Lineage

FIGURE 7 Example Of A Model Update Workflow

Build Or Buy Your ModelOps Muscles

›› Specialized third-party ModelOps solutions. A new market has sprung up composed of

Do ModelOps Or Learn The Hard Way

Engage With An Analyst

Analyst Inquiry Analyst Advisory Webinar

Forrester’s research apps for iOS and Android.

Source: “ModelOps: AI At Scale Isn’t Possible Without It,” Forrester (https://www.forrester.com/

Source: Customer example shared by Brillio.

Model Predictions,” arXiv.org, November 25, 2017(https://arxiv.org/pdf/1705.07874.pdf).

Algorithmia,” Algorithmia, 2018 (https://info.algorithmia.com/hubfs/2018/Case-Studies/Algorithmia-

We work with business and technology leaders to drive customer-

Forrester’s research and insights are tailored to your

Das könnte Ihnen auch gefallen