CHECKLIST REPORT

2018

Data Management for

Data Lakes in the Cloud
By Philip Russom

Sponsored by:
 FEBRUARY 2018

T DW I CHECK L IS T RE P OR T TABLE OF CONTENTS

Data Management for

Data Lakes in the Cloud 2 FOREWORD

3 NUMBER ONE
Catalog Data to Keep the Lake from Becoming a Swamp
By Philip Russom
4 NUMBER TWO
Address the Data Lake’s Aggressive Data Ingestion
Methods with DM Practice Adjustments and Data
Integration Tools

5 NUMBER THREE
Design a Cloud-Based or Hybrid Architecture for Your
Data Lake

6 NUMBER FOUR
Consider iPaaS as Tooling for Cloud DM

6 NUMBER FIVE
Make Your Cloud-Based Data Lake a Nexus for Sharing
Modern and Traditional Data, with a Focus on External
Sources

8 NUMBER SIX
Use the Data Lake as a Self-Service Data Exploration
Platform

9 ABOUT OUR SPONSOR

9 ABOUT THE AUTHOR

9 ABOUT TDWI RESEARCH

9 ABOUT TDWI CHECKLIST REPORTS

555 S. Renton Village Place, Ste. 700

Renton, WA 98057-3295

T 425.277.9126
F 425.687.2842
E info@tdwi.org © 2018 by TDWI, a division of 1105 Media, Inc. All rights reserved. Reproductions in whole or in
part are prohibited except by written permission. Email requests or feedback to info@tdwi.org.

tdwi.org Product and company names mentioned herein may be trademarks and/or registered trademarks
of their respective companies. Inclusion of a vendor, product, or service in TDWI research does not
constitute an endorsement by TDWI or its management. Sponsorship of a publication should not be
construed as an endorsement of the sponsor organization or validation of its claims.
TDWI CHECKLIST REPORT: DATA M A N AGEMENT FOR DATA L A K ES IN THE CLOUD
FOREWORD
This report discusses the leading data management (DM) best
practices you need for data lakes to be successful when deployed in
the cloud. It will drill into those best practices—which are a mix of
business and technology matters—and will look at numerous other
success factors for lakes and clouds, with a focus on DM
requirements for platforms, architectures, analytics, and integration.
To set the scene, let’s start with an inspirational mission statement
from a user who has designed and managed a cloud-based data
lake successfully. Then let’s discuss the mission.
Our opportunity is to disrupt
our existing business processes
by leapfrogging standard data
architectures and adopting new and
leading data architectures to explore
business innovation and support our
digital journey.
Business and Technology Drivers for Cloud-Based
Data Lakes
Data lake disruption. Many organizations are facing a flood
of new data types and sources coming from big data, customer
channels, social media, the Internet of Things (IoT), and numerous
external sources (such as partners and third-party data providers).
They know they need to disrupt “business as usual” because older
DM best practices—and the ways a business gets value from
data—don’t necessarily manage new data assets appropriately or
generate business value. Organizations need disruptive database
designs (such as data lakes) and modern computing platforms (such
as clouds) that are optimized for the aggressive ingestion and agile
use of new data assets.
Leapfrogging standard data architectures. A lake or cloud
can breathe new life into established enterprise data architectures
(data warehouses, marketing channel data, digital supply chains)
or create new and different ones (analytics labs and sandboxes,
ecosystems of cloud-based operational applications). This way,
traditional approaches continue to deliver value while modern
approaches pioneer new practices and value and enable the future.
Data-driven business innovation. Nowadays, digital
enterprises serve and manage customers through data; imagine
and design new products through data; discover and deploy new
2  TDWI RESE A RCH
business processes through data; and compete and lead through
data. A cloud-based data lake can integrate old and new data at
massive scale in new and creative ways to enable the data discovery
and analytics that lead to data-driven business innovation.
Analytics is the data lake’s overarching use case.
Sometimes analytics is the sole use case, as when a lake is the core
of an analytics program or an extension of a data warehouse. Other
times, analytics is a significant component within an operational
solution, as with marketing or supply chain data lakes. Some data
lakes are built purely for self-service data exploration and discovery,
which often lead to visualization or some other form of analytics.
Hence, with a data lake, the path to business value usually leads
through analytics.
Data Management Challenges for Cloud-Based
Data Lakes
The cloud-based data lake has clear and compelling benefits.
However, it also faces many challenges in the realm of DM:
• A data lake needs solid semantics—metadata, glossaries, and
cataloging—to keep from becoming a swamp, despite the fact
that many big, new, external, and cloud data sources have no
metadata or equivalent semantics.
• A data lake must ingest highly diverse data, at scale, at multiple
latencies, for immediate use, while interoperating with a long list
of systems, in the cloud and on premises.
• Data lakes tend to have complex multiplatform data
architectures (MDAs) and they exchange data with other MDAs.
DM must stitch together these complex hybrid environments.
• DM tools used in hybrid environments should support traditional
best practices (integration, quality, master data management)
as well as new ones (microservices, orchestration) via an
integrated tool platform. One new tool-based solution for this
problem is iPaaS.
• For the richest analytic correlations, a data lake should
integrate traditional and modern data—both structured and
unstructured—from many sources at multiple latencies. The
explosion of data from external sources is both a problem and an
opportunity.
• Data lakes mostly manage extracted or streamed source data
in its original raw form so it can be repurposed repeatedly for
multiple use cases. Yet you must also present this data in forms
that are conducive to data exploration and other self-service
practices for a wide range of user types.
This report will now discuss these challenges and offer practical
solutions.
tdwi.org
TDWI CHECKLIST REPORT: DATA M A N AGEMENT FOR DATA L A K ES IN THE CLOUD
NUMBER ONE
CATALOG DATA TO KEEP THE LAKE FROM BECOMING
A SWAMP
An all-too-common misconception about data lakes is that they
are dumping grounds for any random data that anyone wants
to put there. The problem with data dumping is that data is not
appropriately named and documented via semantics as it comes
into the data lake. Semantics includes various forms of metadata,
cataloging, glossaries, tags, and file headers. The problem is
exacerbated in cloud environments, which regularly deal with data
from the Web, streams, and other external sources that notoriously
lack metadata or any other form of semantics. Additionally,
developers cannot attach to external sources to extract metadata
or its equivalent.
Experienced data professionals know that data dumping is a “worst
practice” that can turn a data lake into the so-called “data swamp.”
Due to the lack of proper semantics, a data swamp is difficult to
use, trust, govern, browse, query, or analyze. Hence, getting full
business use and value from a cloud-based data lake requires
significant semantics, as follows:
Next-generation metadata management. Because most tool
types and user query practices require metadata, tools should be
in place to develop, capture, and automatically deduce technical
metadata. These scenarios involve traditional development as well
as new situations.
For example, as users explore a lake’s data (a common business
requirement for a lake), the tool should help them develop metadata
as they go. For metadata-free source data, a tool should parse the
incoming file, data set, or message to deduce its schema and turn
that into reusable technical metadata. This valuable automation
may be enabled by a rules engine or by an artificial intelligence and
machine learning (AI/ML) algorithm, and it may be curated by a user
or applied autonomously without human intervention.
Business metadata and business glossaries. The number of
business users and other nontechnical users continues to increase.
They demand easy-to-use tools and data semantics that will enable
them to access and use a lake’s data autonomously. At a minimum,
they need business metadata, which translates technical metadata
into business-friendly descriptions of data that they can understand.
On a more sophisticated level, this class of user is progressively
demanding a business glossary with which they can work
collaboratively with others to create, define, and apply terms
describing common business entities, such as customer, profitability,
and production yield.
3  TDWI RESE A RCH
Data cataloging. In a lot of ways, data cataloging goes beyond
semantics—a straightforward description of data—to intelligence,
which is rich information about data and relations among data
elements. The intelligence can be interpreted to enrich browsing,
searches, queries, reports, and analytics, which in turn greatly
raises data’s value to users and the business. Modern intelligent
practices in data cataloging have much to offer data lake and cloud
data users:
1. Holistic view. A mature catalog can literally visualize an
entire data lake, or subsets of it, to illustrate the data
available and relations among data elements. Ideally, a data
catalog integrates with metadata repositories and business
glossaries for consistent terminology and definitions across the
holistic view.
2. Single entity, cataloged multiple ways. Users may
categorize data by data domain, source, lineage, personally
identifiable information, compliance sensitivity, and so on. This
enables them to scan data by numerous criteria at the semantic
level while reducing source data processing.
3. Multiple data access methods via the catalog.
Every user and use case is different. To satisfy diverse needs,
a catalog should provide access via browsing, searching,
and querying.
4. Crowd-source the development of data intelligence.
Users don’t just access a data catalog; they help to develop it.
Users can enter data elements into the catalog; review and rank
elements by quality, trust, and usability; and curate entries in
the catalog for accuracy or proper placement in a taxonomy
or hierarchy.1
General semantics requirements. Whether metadata, glossary,
or catalog, semantics should be centralized for a single view that is
accessible by broadly deployed users and technologies. Recognize
that data lake users will demand self-service data practices (data
exploration, preparation, visualization) and these are not possible
without friendly and broad semantics. For all forms of semantics
(and many other DM tasks), you should demand tools that support
productive automation through a rules or an AI/ML engine.
1
For a detailed discussion of modern data cataloging, read the TDWI
Checklist Report The Data Catalog’s Role in the Digital Enterprise, online
at tdwi.org/checklists.
tdwi.org
TDWI CHECKLIST REPORT: DATA M A N AGEMENT FOR DATA L A K ES IN THE CLOUD
NUMBER TWO
ADDRESS THE DATA LAKE’S AGGRESSIVE DATA INGESTION
METHODS WITH DM PRACTICE ADJUSTMENTS AND DATA
INTEGRATION TOOLS
Lake ingestion methods differ from other database-
driven practices.
A hallmark of the data lake is its unique approach to data ingestion.
On the one hand, a data lake of any maturity will continuously ingest
large volumes of data from hundreds of sources. On the other hand,
a lake stores most data in its raw source form so that the original
details are preserved for future analytics, reporting, and operations
that may need them.
To satisfy these two DM requirements, the data integration and
other processes feeding the lake store incoming data as quickly as
possible, with little or no alteration or improvement. This allows the
ingestion process to perform and scale by delaying data processing
workloads until read time. If a lake’s data needs processing to be
repurposed for various uses cases, that processing occurs later,
typically well after ingestion and increasingly on the fly as data is
explored or a report is refreshed.
The data lake’s “process on read” strategy differs from the data
warehouse’s “process before ingestion” strategy, which is fine
because the two serve complementary use cases with different
requirements. A data lake is a raw detail store supporting
mostly discovery-oriented analytics and which by nature accepts
whatever data models, quality states, and semantics it gets.
A data warehouse is a store of mostly calculated and carefully
structured values for recurring reports which have rather stringent
requirements and standards for modeling, quality, accuracy, and
audit. Hence, data warehouse professionals have the skills they need
for data lakes but will need to adjust them slightly to align with the
unique requirements of the data lake.
Most challenges of the lake’s data ingestion
methods are met by data integration tools.
Speed and scale. Most data lakes are ingesting data around the
clock, often at Internet volumes. Achieving speed and scale depends
on a data integration platform that can parallelize all jobs for
maximum throughput.
Containers from files to messages. A cloud-based data lake
will ingest data from traditional on-premises sources and modern
Web-based ones, which amounts to many container types (including
XML and JSON files), SQL result sets, bulk and block data sets, and
streaming events and messages. The data integration tool must
ingest all these quickly and at scale.
4  TDWI RESE A RCH
Unpredictable ingestion and processing loads. One reason
data lakes are being deployed more on clouds is to take advantage
of cloud’s automatic allocation of resources as workloads in the lake
ramp up and subside. Cloud-based integration processing can do
the same.
Rich library of APIs and open interfaces. A data lake of any
maturity will ingest data from a long list of tools and platforms,
which means its data integration has to support many demanding
interfaces. Besides the usual enterprise applications, this also
involves SaaS apps such as Marketo, Salesforce, and NetSuite, as
well as cloud-based PaaS providers such as Amazon, Google, and
Microsoft. Given the mass and complexity of interfaces, a modern
data integration tool should support API management.
Automation for the rapid addition of data sources. Data
lakes and cloud DM regularly take on data from new sources. For
example, in industries involved in IoT (such as logistics, utilities,
and supply chains), new machines, devices, vehicles, and sensors
can come online every week, generating valuable data for both
operational and analytics use. To get to such business value quickly,
the data integration tool must have easy, agile, and repeatable
functionality for source onboarding. Ideally, onboarding should be
automated via rules or machine learning.
Judicious data quality. As discussed earlier, data lakes tend
to store data in its original form, without improvement until later.
Even so, over time, data professionals managing the lake will learn
(as they would with any data set) where data standardization is
needed to facilitate self-service, data exploration, reporting, or query
performance. Such improvements should be applied to documented
copies of data so details of the original source are not lost. When
lake data is cataloged properly, information captured in the catalog
can be the equivalent of standardization, but performed at the
semantic level without altering actual data.
Data landing and staging on steroids. In many ways, a data
lake is like a massive data landing area, with extreme levels of
ingestion but low levels of staging. For this reason, many data lakes
serve the primary purpose of discovery analytics while also serving
as data landing and staging for other systems, such as a data
warehouse or channel marketing data ecosystem.
tdwi.org
TDWI CHECKLIST REPORT: DATA M A N AGEMENT FOR DATA L A K ES IN THE CLOUD
NUMBER THREE
DESIGN A CLOUD-BASED OR HYBRID ARCHITECTURE FOR
YOUR DATA LAKE
One of the strongest—and most challenging—trends in data
management today is toward environments consisting of numerous
data platforms where data is physically distributed across multiple
database servers, file systems, and storage. This is certainly the
case with most data lakes, especially those involving clouds. When
diverse data platforms and data sets are integrated this way, the
result is called a multiplatform data architecture (MDA). Synonyms
include distributed data architecture and hybrid data ecosystem.2
The trend toward MDAs affects data lakes in
multiple ways.
A data lake is a data architecture. Note that a data lake is
a method for managing data, not a platform per se. The method
assumes you are willing and able to create a large-scale architecture
(sometimes called a design pattern) for your data lake, similar to
how you’ve modeled and architected databases, data warehouses,
data hubs, and data stores. A lake’s architecture may involve one
or several data platforms and—like most architectures—it has
multiple components and layers. Furthermore, the lake’s architecture
may overlap with other MDAs, as when a lake extends a data
warehouse environment.
Most data lakes are hybrid MDAs. Research by TDWI has
revealed that more than half of data lakes are hybrids, spanning
a mix of Hadoop clusters, file systems (e.g., Amazon S3 and Azure
DLS), and diverse brands of relational database management
systems (RDBMSs). Many lakes involve data platforms located both
on premises and in the cloud. Hence, you should probably assume a
hybrid MDA when designing your data lake.3
The cloud is an emerging data lake platform. As cloud
usage for DM has matured, user organizations have started
leveraging a variety of data systems that most cloud providers
support. For example, at TDWI we see users turning to Hadoop,
Amazon S3, and Azure DLS for cloud-based data lakes, as well
as Redshift, Snowflake, and SQL Server on Azure for cloud-based
data warehousing. Hence, most MDAs already include cloud-based
systems and they will no doubt involve more over time, especially for
data lakes, warehouses, and analytics.
2
For a detailed definition of MDAs, replay the archived TDWI Webinar
Defining the Multiplatform Data Architecture and What It Means to You,
online at tdwi.org/webinars.
3
For a detailed rundown of lake platforms, see the TDWI Best Practices
Report Data Lakes: Purposes, Practices, Patterns, and Platforms, online at
tdwi.org/bpreports.
5  TDWI RESE A RCH
The point of the MDA is to provide options for data’s
diverse structures and uses.
For example, as businesses deploy more sensors, customer channels,
applications, and social media, the breadth of source data’s schema,
latencies, and containers is driven up, which in turn requires
more types of data platforms to capture and process the diverse
data appropriately. Similarly, relational and other structured data
types are being joined by a widening range of multistructured and
unstructured data types.
As another example, businesses are diversifying the use of analytics,
reports, and data-driven business monitoring, all of which have
unique requirements for data capture, storage, and processing.
These in turn drive up the diversity of data platforms and related
tools. TDWI sees users succeeding with MDAs in data warehousing,
analytics, multichannel marketing, digital supply chain, IoT, and
other data-driven enterprise programs.
MDA complexity is the problem.
Extreme complexity results from the number of systems involved,
multiplied by the extreme diversity of platform types that may
integrate multiple brands of database management systems (both
old and new), NoSQL platforms (especially Hadoop), and tools for
data integration, analytics, and stream processing. These may be
deployed on premises, in the cloud, or in hybrid combinations.
The complexity of MDAs is amplified by the growing popularity of
popular cloud platforms in data-driven solutions, namely Amazon
Web Services (AWS), Google Cloud, and Microsoft Azure. Even more
dramatic is the surging portfolio of cloud-based SaaS applications
users are subscribing to, including Salesforce, Marketo, NetSuite,
Paylocity, and Workday. We say “the cloud” as if there is only one,
but many organizations are managing data across many cloud-based
data platforms and SaaS apps.
There are solutions to the MDA complexity problem.
DM tools. Although data is strewn about physically, there are
usually cross-platform architectural layers that stitch together the
MDA and its data. For example, architectural layers for DM can unify
an MDA, especially the layers for metadata and other semantics.
Virtual DM techniques such as federation and data virtualization can
create views that make data look simpler and more unified than it
actually is. Data travels—a lot—among the platforms of an MDA,
such that a well-constructed network of data integration data flows
constitutes yet another unifying architectural layer.
Another emerging best practice for orchestrating data integration
data flows is the integration hub, which employs a publish-and-
subscribe method of sourcing and distributing data. An integration
hub makes it easier to introduce and maintain data and avoids the
integration morass that too many data architectures deteriorate into.
Note all these solutions to the MDA complexity problem require a
well-integrated suite of diverse DM tools.
tdwi.org
TDWI CHECKLIST REPORT: DATA M A N AGEMENT FOR DATA L A K ES IN THE CLOUD
Architecture and governance teams. Tools aside, data
architects are indispensable to the success of an MDA, including
those responsible for data lakes and clouds. An enterprise data
architect (or data warehouse architect, etc.) typically heads a team
of people who influence the selection of data platforms and DM tools
relative to MDAs. The team also fosters enterprise data standards
that facilitate the integration of data across platforms, whether
on premises or in the cloud. A data governance or stewardship
program may provide similar assistance with data standards and
architectural preferences.
NUMBER FOUR
CONSIDER IPAAS AS TOOLING FOR CLOUD DM
Integration Platform-as-a-Service (iPaaS)
is a response to new business and technical
requirements.
Many user organizations have compelling reasons or executive
mandates to move to the cloud. They seek to follow cloud-first
policies, control IT costs, integrate disparate applications, deliver
data-driven solutions faster, and provide integration infrastructure
for complex multiplatform data environments that include clouds.
Organizations facing any combination of these evolving business
and technology requirements are realizing that their traditional
on-premises integration solutions are not a good fit for fast-paced
cloud operations or complex hybrid environments. Furthermore, users
migrating data and applications to the cloud need richer cloud-
based integration toolsets both to facilitate the migration and to
support daily cloud-native integration flows. As a result, Integration
Platform-as-a-Service (iPaaS) has emerged to address today’s
requirements. IPaaS is a suite of cloud microservices enabling the
development, execution, and governance of integration flows and
data pipelines. It also enables modern, cloud-based DM, which can
ably apply to cloud-based data lakes.
DM requirements for iPaaS
In a nutshell, here are the essential requirements for iPaaS, plus
some of its uses and benefits:
Powered by the cloud. IPaaS is cloud based to take advantage
of cloud elasticity, scalability, flexibility, and low cost. This also
allows it to interface directly with Internet-based applications
(Salesforce, Marketo, NetSuite) and data sources (Web apps, B2B
partners, IoT).
Fast track to integration solutions. IPaaS can accelerate
application and data integration. For example, when a public cloud
has multiple tools already set up and optimized, it prevents users
6  TDWI RESE A RCH
from burning up valuable time and personnel on system integration.
This way, cloud-based systems present minimal time until business
use. In many cases, users can set up interfaces, load data, migrate
users, and put the solution into production in a few days.
All forms of integration. Vendor products for iPaaS vary, but
the comprehensive ones support the many functions of both data
integration and application integration. This is because, in addition
to data integration, many users need a data-driven toolset for
reliably migrating, consolidating, and delivering application data,
plus managing data from SaaS apps.
Multitool suite. IPaaS suites tend to include many integration
tool types through a single development and management console,
including those for data integration, quality, and master data
management (MDM); plus application integration, orchestration,
and process management. The unified toolset boosts developer
productivity, the creation of consistent standards, and broad
curation. Furthermore, it fosters the modern design of flows and
pipelines that incorporate multiple forms of integration technology.
Microservices. IPaaS functionality is available via cloud-based
microservices. Almost any data- or application-integration function
you can think of is now a service.
API driven. To fulfill its aggressive integration goals, an
iPaaS suite must support all modern and traditional application
programming interfaces (APIs), in addition to including special
functionality for managing API portfolios and performance.
Built for hybrid environments. Despite the focus on the
cloud, iPaaS also provides integration microservices that can be
used by on-premises applications and tools. In fact, organizations
with iPaaS typically use it as a nexus that provides rich integration
and interoperability for the many platforms of a hybrid data and
application environment. Given that many data lakes are cloud
based and hybrid, iPaaS can be an appropriate DM solution
for them.4
NUMBER FIVE
MAKE YOUR CLOUD-BASED DATA LAKE A NEXUS FOR
SHARING MODERN AND TRADITIONAL DATA, WITH A
FOCUS ON EXTERNAL SOURCES
One of the fastest growing trends in information technology today
concerns the burgeoning number of parties and applications that
are external to an enterprise. Most of these generate data and may
demand data in return, and that data is critical to business success.
Therefore, data management professionals are under pressure to
4
For more information, read the TDWI Checklist Report Data Management
Best Practices for Cloud and Hybrid Architectures, online at tdwi.org/checklists.
Also, replay the archived TDWI webinar Achieving Integration Agility, Scale, and
Simplicity via Cloud-Based Integration Platform-as-a-Service, available online at
tdwi.org/webinars.
tdwi.org
TDWI CHECKLIST REPORT: DATA M A N AGEMENT FOR DATA L A K ES IN THE CLOUD
capture and manage data coming from sources and technologies
that are new to them. They must also support new business use
cases by processing data in ways that are likewise new. Here are
several of the high-profile and high-value use cases for external
data sources that are well-served by cloud-based data lakes.
General analytics. Analytics is the overarching use case for
almost all data lakes. Analytics is also mostly about correlating
disparate data points, as seen in advanced techniques such as
data mining, clustering, and graph databases. A data lake can
bring together massive quantities of data from many sources,
structures, and latencies for the richest cross-source correlations
possible. These rich correlations in turn lead to richer analytics,
more comprehensive reporting, more complete views of customers,
more discoveries of fraud, and more revelations about business
characteristics. When the sources required for such an eclectic data
mix are distributed across enterprise locations and Internet-based
applications and parties, deploying a data lake in the cloud can
be an advantage for creating a shared, multitenant intersection of
extremely diverse data and its analytics.
Multichannel marketing data. TDWI sees this as the next-
largest use case for data lakes after general analytics. The so-called
marketing data lake is gaining adoption because it provides a single
repository for the many data-driven functions of modern marketing,
including data from numerous channels and customer touch points,
360-degree customer views, third-party data about consumers,
campaign design and execution, and customer analytics (profiling,
segmentation, profitability, etc.). Some customer channels and
touch points are inherently Internet-based, especially those involving
website visitor behavior and e-commerce. External data providers
are likewise on the Internet, ranging from traditional consumer
demographics to modern social media.
Furthermore, marketing departments tend to prefer SaaS-based
applications for sales force automation and marketing campaign
management, both of which involve larger regular data loads and
extracts across the Internet than even most other SaaS applications.
Additionally, marketing data solutions are not complete without
functions for MDM and data quality. Finally, these data-driven,
digital approaches to marketing are universally practiced by large
firms, where marketers are geographically distributed but need to
share large volumes of data easily. All these data types, sources,
and scenarios point to the cloud-based data lake, augmented with
modern DM and analytics tooling, possibly with an iPaaS platform.
Business-to-business (B2B) partner data. Procurement
and supply chain operations are mission-critical in manufacturing
and retail industries. Yet these industries have only recently begun
serious modernization efforts as they move from faxes and phone
calls to managed file transfers and automated transactions. Data
lakes are easily optimized for file-based data and so are a good
fit for landing, processing, archiving, and analyzing partner data
exchanged via EDI, XML, and JSON files. Because these files arrive
7  TDWI RESE A RCH
from and depart through the Internet, a cloud-based data lake can
be proximity appropriate.
The Internet of Things. TDWI has a number of members who
work in logistics firms, especially those for truck and rail freight.
These firms have invested significantly in telematics and other
sensor systems for their trucks, trailers, locomotives, and railcars.
They’ve also deployed data gathering stations in their truck
depots and rail yards. Some sensors stream data continuously
(as with GPS) while others are intermittent (as when an RFID chip
comes into proximity of an RFID reader). Every sensor and read
context is different, meaning that data falls into many schemas,
some standard and some proprietary. Sensor data may be
transmitted over the Internet (often via WiFi), telephony, satellites,
or enterprise networks.
When a data lake’s DM solution supports many ingestion methods,
it can handle incoming IoT data regardless of its latency, schema,
or transmission infrastructure. The lake and its DM solution can
also have data ready for immediate use, which is important because
most IoT data has urgent operational functions (tracking shipments,
spotting vehicle malfunctions, monitoring operator behavior, and
operational reporting) as well as latent analytics uses (optimizing
routes, vehicle maintenance, and customer service).
Deploying the data lake in the cloud makes sense when IoT data is
shared with people outside the enterprise, usually through browser-
based applications. For example, manufacturers of high-end farm
machinery collect data from combines so the combine owner can
analyze the vehicle’s performance and his use of the machine. Some
utility firms share data about electricity consumption so property
owners can understand and optimize their consumption.
NUMBER SIX
USE THE DATA LAKE AS A SELF-SERVICE DATA EXPLORATION
PLATFORM
As we just saw, the onslaught of new data from new sources is
both a DM problem and a business opportunity. The cloud-based
data lake is a likely solution for capturing new data. How does an
organization get beyond data collection to business value?
Data exploration leads to business value and
compliance.
Data exploration (sometimes called data discovery) is a recently
emerged best practice that is critical to wringing business value
from data that is new to an organization. Diverse users are now
exploring data regularly, from technical people such as data analysts
and scientists, to business domain experts such as marketers,
accountants, and procurement specialists.
tdwi.org
TDWI CHECKLIST REPORT: DATA M A N AGEMENT FOR DATA L A K ES IN THE CLOUD

With new data, there is the prospect of new insight. A desirable your peril because without them, most users will not use a data lake
outcome of data exploration is the discovery of facts about the and it will be considered a failure.
business or one of its processes that were previously unknown.
Note that self-service data exploration often leads directly into
However, users also explore to simply understand the new data.
other self-service practices and tool types. For example, once a user
Once they see which business domains, entities, and activities
has explored individual records of customers who churned recently,
are represented in it, they can decide how this information can
he or she might immediately move to a self-service data prep tool
be applied to solutions in DM, operations, and analytics. This
(which ideally is in the same toolset as the exploration tool) to pull
understanding also helps them decide which data governance,
all similar records into a data set, then augment the data set with
privacy, and security policies to apply to the new data.
demographics of these and similar customers. The user then applies
Data profiling leads to DM solutions that deliver self-service data visualization to the data set to get a different or
quality data. deeper understand of the current form of churn. Once the data set
and its visualization are polished, the user may publish them via
The technical adjunct to exploration is profiling. Technical data self-service so other users can subscribe to them and understand
management professionals (or special business users such as the latest form of customer churn.
data stewards) profile new data to understand its technical state
in terms of structure (format, model, or schema); metadata and A data lake and its tooling can enable these self-service tasks.
other semantics (whether embedded, extracted, or deduced); The lake provides the data and the semantics appropriate to
condition (quality, standardization, completeness); and details of exploration. The lake and its DM tools enable the queries that are
delivery or storage (files, streams, preferred interfaces, and so on). at the heart of most exploration and data prep. The lake can
This information—called a data profile or data intelligence—is store self-service data sets, preferably in special areas such as
indispensable for getting started with new data as well as for sandboxes, so that various users and tools can subscribe to sandbox
improving, storing, and presenting it for fullest business use. content. To avoid swamp practices such as abandoned data sets,
the lake’s tools should automatically delete sandboxes that lie
A data lake can enable broad data exploration unused for long periods of time.
and profiling.
Data exploration and data profiling usually require detailed source
data, and large quantities of it—features at which all data lakes
excel. The multiple ingestion methods supported by most lakes also
mean they excel at landing and staging all kinds of data; hence,
a data lake is ideal for the first capture of new data and the first
study of it.
New data aside, the average data lake contains highly diverse
data, both modern and traditional, from both internal and external
sources. The lake’s eclectic mix of data makes it ideal for data
exploration sessions in which the user develops analytic correlations
among disparate data points in hopes of discovering connections.
For example, correlations can reveal a person who is present at
multiple accidents (suggesting fraud), seasonal conditions that
lead to damages (enlightening actuaries), or bottleneck steps in a
process (suggesting efficiency improvements).

Data exploration, data prep, and other data lake-

based self-service practices are coalescing into
a unified analytics approach demanded by both
business and technology users.
As mentioned, self-service business practices are on the rise among
business domain experts and they expect it with their data lakes.
These users need a combined data lake-and-DM solution that
provides well-developed business-friendly semantics. They also need
end-user tools with strong ease-of-use. Ignore these requirements at

8  TDWI RESE A RCH tdwi.org

TDWI CHECKLIST REPORT: DATA M A N AGEMENT FOR DATA L A K ES IN THE CLOUD
ABOUT OUR SPONSOR
informatica.com
Informatica is the only enterprise cloud data management leader
that accelerates data-driven digital transformation. Informatica
enables companies to unleash the power of data to fuel innovation,
become more agile, and realize new growth opportunities, resulting
in intelligent market disruptions. With over 7,000 customers
worldwide, Informatica is the trusted leader in enterprise cloud
data management.
Businesses must transform to stay relevant and data holds the
answers. We’re prepared to help you intelligently lead—in any
sector, category, or niche. Because our focus is 100% on everything
data, we offer the versatility needed to succeed. We invite you to
explore all that Informatica has to offer—and unleash the power of
data to drive your next intelligent disruption.
With full access to data, IT roles shift dramatically to become
more strategic, more essential, to become partners in leading
the business. With Informatica, you can use data to develop new
business models and capture growth opportunities. We enable you to
unleash the power of data to intelligently disrupt your industry.
The Informatica Intelligent Data Platform is the industry’s
most complete and modular solution, built on a microservices
architecture, to help companies unleash the power and value
of all data across the hybrid enterprise. The AI-driven platform
spans on-premises, cloud and big data anywhere—ensuring
data is trusted, secure, governed, accessible, timely, relevant and
actionable. This enables the world’s most progressive companies to
deliver data-driven digital transformation outcomes.
9  TDWI RESE A RCH
ABOUT THE AUTHOR
Philip Russom, Ph.D., is senior director of
TDWI Research for data management and is
a well-known figure in data warehousing,
integration, and quality, having published
over 550 research reports, magazine articles,
opinion columns, and speeches over a
20-year period. Before joining TDWI in 2005, Russom was an
industry analyst covering data management at Forrester Research
and Giga Information Group. He also ran his own business as an
independent industry analyst and consultant, was a contributing
editor with leading IT magazines, and was a product manager at
database vendors. His Ph.D. is from Yale. You can reach him at
prussom@tdwi.org, @prussom on Twitter, and on LinkedIn at
linkedin.com/in/philiprussom.
ABOUT TDWI RESEARCH
TDWI Research provides research and advice for BI professionals
worldwide. TDWI Research focuses exclusively on analytics and
data management issues and teams up with industry practitioners
to deliver both broad and deep understanding of the business and
technical issues surrounding the deployment of business intelligence
and data management solutions. TDWI Research offers reports,
commentary, and inquiry services via a worldwide membership
program and provides custom research, benchmarking, and strategic
planning services to user and vendor organizations.
ABOUT TDWI CHECKLIST REPORTS
TDWI Checklist Reports provide an overview of success factors for
a specific project in business intelligence, data warehousing, or
a related data management discipline. Companies may use this
overview to get organized before beginning a project or to identify
goals and areas of improvement for current projects.
tdwi.org