Hortonworks /

Informatica /
Teradata
PAGE 24
DESIGNING
THE DATA LAKE
FOR FASTER TIME
TO VALUE

DATA
Progress
DataDirect
PAGE 27
LOADING YOUR
SAAS DATA INTO
A BIG DATA LAKE

LAKES
BEYOND THE HYPE AND
READY FOR THE ENTERPRISE

Best Practices Series

 CALMING THE
TURBULENT WATERS OF
DATA LAKES
Best Practices Series
Data lakes have become a mainstream strategy for many
enterprises over the past couple of years, with promises of
greater flexibility in the way data is handled and made available
to decision makers. A recent survey by Unisphere Research, a
division of Information Today, Inc., found that 20% of data
managers and professionals are currently deploying data lakes,
and 45% are learning and researching about them. A majority,
56%, have a positive impression of the concept in that it may
serve some value to their businesses. At least 38% indicate their
companies are committed to data lake strategies (“Data Lake
Adoption and Maturity Survey Findings Report,” Unisphere
Research, October 2015).
In most cases, data lakes are defined as data environments
that capture and store raw data. A data lake comprises data in its
original format, to be transferred and transformed at a later date
as applications and end users demand. The thinking behind the
concept is that the analytics or questions to be applied against the
data may have not yet been identified, and by holding the data in a
relatively accessible environment, it is open for future innovation.
22 BI G D ATA QU A RTERLY | SPRIN G 2016
However, as with any major enterprise data initiative, the con-
cept has to be sold to the enterprise. Data lakes absorb data from
a variety of sources and store it all in one place, with all the nec-
essary requirements for integration and security. Data lakes are a
response to the eternal problem of data silos, attempting to bypass
these various, fragmented environments to finally maintain data
all in one place. The data lake also reduces the requirement for
immediately processing or integrating the wide variety of data
formats that comprise big data.
Here are some best practices for successful implementation
of a data lake:
THINK LONG TERM
The advantage that data lakes offer is the ability to keep data at
the ready for applications or queries that have yet to be designed.
The bottom line is no one knows what data will be valuable in 5
years, or how it might be used. There may be entirely new business
lines built around data that is currently being cast aside in today’s
environments. As a result, data needs to be maintained and stored

for purposes and applications that have yet to be determined.
Some data will be needed immediately, while other data will need
to be stored. To see this through, enterprise input is essential to
identify those information areas that decision makers see as hav-
ing future potential.
VIEW DATA LAKES AS A BUSINESS GROWTH
OPPORTUNITY VERSUS A COST-SAVING MEASURE
As with any major IT initiative, cost-savings drives many data
lakes initiatives. However, the value will ultimately be realized
in the potential avenues it offers for business growth. The next
frontier for data lakes is providing organizations with greatly
enhanced analytical opportunities. The analytics made possible
by large stores of information in data lakes facilitates customer
relationship management, predictive analytics, preventive main-
tenance, fraud detection, and a range of other applications.
IMPLEMENT SOUND DATA GOVERNANCE
Data governance is just as important to data lakes as it has
been to data warehousing and other big data projects in recent
years. The data lake is increasingly recognized as both a viable and
compelling component within a data strategy, with companies
large and small continuing to move toward adoption. Governance
is the key challenge to data lakes, cited by 71% of respondents in
the Unisphere Research survey.
When data lakes first appeared on the scene, they were catch-
alls for structured and unstructured data, even though the data
might not have been useful at the moment. As applications
and functions are developed, custom adjustments or recoding
is performed to meet the requirements of the situation. How-
ever, organizations may not have the data talent to address these
requirements on an ever-increasing scale. Data lakes are, from
the start, an enterprise project that requires enterprise input and
ownership.
PUT SEMANTICS IN PLACE
The best way to avoid having data lakes decay into haphazard
“data swamps” is to have a strong, semantic structure to provide
consistency and enable the easy discovery and access of essen-
tial data. It’s also important to map relationships between data,
which provides the foundation for trusted data that is used within
analytic applications. Semantic models also need to be clear to
human operators, as well as map to the needs of the business.
DATA LAKES SHOULD AUGMENT—NOT REPLACE
—CURRENT TECHNOLOGY
There remains a great deal of confusion in the market, with
many managers and professionals assuming that the addition of
a data lake reduces the need for an enterprise data warehouse.
Nothing could be further from the truth. While there is uncer-
tainty—and perhaps some purposeful ambiguity—in the market
sponsored content
about what data lakes are and what role they serve, there are clear
capabilities for data lakes that vary from data warehouses. For
example, while data lakes are intended for the storage of raw
data, making it easy to retrieve, by contrast, data warehouses
are intended to serve as sources of extracted and vetted data for
specific areas of the business. Importantly, however, data lakes
will typically need to sit in front of data warehouses or other
Hadoop deployments, with data that may be processed within
those environments at a later date.
The data lake concept: The analytics or
questions to be applied against the data
may have not yet been identified, and by
holding the data in a relatively accessible
environment, it is open for future innovation.
ADDRESS DATA SECURITY
AND COMPLIANCE ISSUES
Data lakes help reduce the need to move data through the
organization, which is a costly and complex undertaking, not to
mention the security implications that go with it. At least 67%
of respondents to the the Unisphere Research survey indicated
they are concerned about data security in these settings. There is
considerable concern that data lakes will serve as repositories for
all types of unvetted and insecure data or even files and docu-
ments that may be in violation of compliance mandates—again,
the “data swamp” fears. Data lakes need to have the same secu-
rity, governance, and accountability as any other data environ-
ment within the enterprise. Authentication, authorization, and
encryption are key to managing a secure and compliant data lake
environment.
AUTOMATE AS MUCH AS POSSIBLE
Data lakes will have a lot of moving parts, handling ingestion,
integration, processing, and storage, among other important
functions. This all needs to be built and embedded into the enter-
prise data framework to the point where it’s virtually invisible
to end users. The more these various functions are automated,
the more data managers and business users can focus on their
higher-level core competencies. The various functions of data
lakes—from ingestion to storage—can all be automated. Automa-
tion also makes real-time data movement and analytics possible.
 —Joe McKendrick
DBTA. COM/ BI GDATAQUARTERLY 23
 sponsored content
Designing the
Data Lake for Faster
Time to Value
THE DATA LAKE OPPORTUNITY
Data has become the lifeblood
of nearly every industry-leading
company. But the ability to turn this
data into valuable business insights
with the right data delivered at the right
time is what separates industry leaders
from laggards. The concept of the data
lake pattern has developed as a means
to economically harness and derive
value from exploding data volume and
variety. New data sources such as web,
mobile, and connected devices along
with new forms of analytics such text,
graph, and pathing have necessitated a
new data lake design pattern to augment
traditional design patterns such as the
data warehouse.
Enterprise data warehouses have
always struggled to balance time
to delivery against auditability,
stability, ease of use, data quality, and
performance. The new challenge in a
world with growing volume, variety, and
velocity of data is that business analysts
must manually find and reconcile
fragmented, replicated, incomplete, and
inconsistent data across the organization.
As a result, business analysts face delays
in accessing needed data and sharing it
24 BI G D ATA QU A RTERLY | SPRIN G 2016
with one another in a timely manner.
Moreover, with the exponential increase
in data volume and data proliferation
across systems, business analysts run
the risk of delivering inaccurate reports
and predictions because of data that is
insufficient, incomplete, inconsistent,
inaccurate, or insecure.
FIT-FOR-PURPOSE APPROACH
Prior to the advent of the trend
toward big data, there was a singular
and traditional approach to data
management whereby data was fully
normalized and persisted in a relational
data warehouse before any value
could be derived. The theory behind
this tightly coupled schema-on-write
philosophy was that it was essential to
absorb the costs of data management
up-front so that all available data was
of high fidelity. The requirement for
high fidelity data for high performance
analytics continues to grow, but today
the relational data warehouse approach
is no longer sufficient to manage all
enterprise data.
Meanwhile, infrequently used data
or so-called “dark data” has historically
just been deleted as part of information
lifecycle policies usually because it is
unstructured and in large quantities.
Since no one explores it, there is no
demand for that data from the user
community. Based upon the assumption
of infrequent use, the data is not
examined for the possibility of gold
nuggets of value. Absent this discovery
or exploration, the data naturally goes
unused. This creates a chicken-and-egg
puzzle whereby historical access patterns
bias future availability.
This brings up the central challenge
of how to deal with data of unknown or
questionable value from a burgeoning
number of data sources and types.
Moving it into the data warehouse or
database management system isn’t ideal
for two reasons.
One is that the schema has to be pre-
defined, specifically as a “compromise,”
before data can be loaded into either of
these environments. This isn’t a trivial
effort and it often limits organizations
to only using data they know to have
value beforehand. Also, the act of
creating the schema is likely to constrain
data to a few anticipated use cases.
Much of the predictive and diagnostic

value contained in the raw data may
then be lost.
And, two, the storage cost model for
the data warehouse or DBMS may not
lend itself to wholesale data ingestion.
This leaves organizations searching for
a place where they can explore the data
for alternative interpretations to the
compromise, then discover the value of
those alternatives.
Newer big data technologies and
approaches have enabled a new fit-
for-purpose approach for delivering
the right data to the right people at
the right time. These newer, loosely
coupled schema-on-read philosophies
are enabling organizations to leverage
a “data lake pattern” that collects all
of their raw data into persistence
systems prior to any modeling or
transformation.
DATA SECURITY & GOVERNANCE
Still, data lake patterns are not
without their challenges. With the
growing interest in self-service
analytics tools, organizations are
seeing data proliferate through teams
with increasing risk for security and
governance failures. In the face of such
risks, some organizations are locking
down access to self-service tools and
Hadoop clusters, thus defeating the
purpose of leveraging data lake patterns.
However, organizations need not make a
false tradeoff between data anarchy and
data tyranny.
Through collaborative and machine-
learning guided curation processes,
datasets can be refined from raw form
into datasets with increasing levels of
quality. This enables a bare minimum
of data rationalization to occur post-
ingestion for data consumers who don’t
require the highest fidelity of data.
Meanwhile, new curation processes
discovered by data consumers can also
be operationalized into automated
and repeatable ETL-like processes. By
enabling secured and governed access
to data by data consumers themselves,
organizations can achieve a balanced
and well-governed data democracy.
“The purpose of a data lake is to
present an unrefined view of data to only
the most highly skilled analysts,” notes
Gartner (2015), “to help them explore
their data refinement and analysis
techniques independent of any of the
system of record compromises that may
exist in a traditional analytic data store
(such as a data warehouse).”
When data finally exits the
curation process inside the data lake
environment, it can be loaded into a
downstream integrated data warehouse
(IDW). The IDW is now optimized for
storing the highest fidelity data and
performing high performance analytical
queries. A fully leveraged IDW will be
deployed with access to its structured
data products from across multiple
lines of business within the enterprise.
This promotes reuse of high fidelity
data through a service-oriented model
while datasets curation happens quickly
and confidently in the data lake for
maximum business value.
DEFINING THE DATA LAKE
Confusion regarding the definition
of a data lake abounds in the absence
of a large body of well understood best
practices. Drawing upon many sources
as well as on-site experience with leading
sponsored content
data-driven customers, we define the
data lake as a collection of long-term
data containers that capture, refine, and
explore any form of raw data at scale,
enabled by low-cost technologies, from
which multiple downstream facilities
may draw upon.
It is important to note that a design
pattern is an architecture and set of
corresponding requirements that
have evolved to the point where there
is agreement and best practices for
implementations. How you implement
it varies from workload to workload,
organization to organization. While
technologies are critical to the outcome,
a successful data lake needs a plan. A
data lake design pattern is that plan.
The data lake definition does
not prescribe a technology, only
requirements. While data lakes are
typically discussed synonymously
with Hadoop – which is an excellent
choice for many data lake workloads
– a data lake can be built on multiple
technologies such as Hadoop, NoSQL,
S3, RDBMS, or combinations thereof.
One way to look at it is that Hadoop
is a technology, while the data lake is a
design pattern fit for certain workloads.
Hadoop does not perform all of those
workloads. You have to go get ETL,
data wrangling, and data governance
tools, as well as many others in order to
make the data lake the best it can be. In
contrast, the data lake is not a real-time
complex event processing tool. In fact,
it’s not a tool at all. Within the data
lake definition and design, we do not
find things such as Hbase or real-time
applications development. Nor do we
expect data lakes to be data products
DBTA. COM/ BI GDATAQUARTERLY 25
 sponsored content
such as recommendation engines.
What this means is that Hadoop can be
applied to other design patterns, just like
an RDBMS can be applied to workloads
other than data warehousing.
BRINGING THE RIGHT DESIGN
AND TOOLS TOGETHER
The flood of new data from
new sources has necessitated a new
methodology and set of economics
for capturing and analyzing raw data
at scale—known as the data lake.
The data lake enables organizations
to realize new insights from data of
unknown and under-appreciated
value, perform new kinds of analytics
that were not previously possible,
retain a longer corporate memory, and
optimize existing processes such as
data integration. Organizations that are
leading the way in delivering value from
the data lake realize the importance of
starting with a solid design based on
required capabilities, and satisfying those
capabilities with enterprise-grade big
data technologies.
As close partners, Informatica,
Hortonworks, and Teradata have worked
to integrate technologies that enable the
data lake design pattern. This approach
to data lakes leverages low-cost data
platforms to provide a collaborative and
flexible data curation environment to
capture, integrate, refine, and analyze
data at scale.
Informatica provides the Big Data
Management layer which enables you to
find, prepare, govern, and protect the big
26 BI G D ATA QU A RTERLY | SPRIN G 2016
data that your business needs to quickly
and repeatedly get business value. It
accelerates developer productivity and
analyst productivity, while enabling IT
agility in a fully secure and governed
environment. Data security and data
governance are not merely enforced
through manual intervention. Instead,
a universal metadata catalog is used to
enable more data consumers to quickly
and repeatedly get more business value,
from more data, without more risk.
You get simplified data ingestion and
self-service data preparation capabilities
alongside data lineage and risk-centric
data security.
Hortonworks provides Open
Enterprise Hadoop, the open source
framework for distributed storage
and processing of large sets of data on
commodity hardware. Hadoop enables
businesses to quickly gain insight from
massive amounts of structured and
unstructured data. With YARN as the
architectural center, Open Enterprise
Hadoop enables enterprises to run
diverse applications on the common
data lake.
Teradata provides a variety of
products and services to help enterprises
accelerate their data lake journey. Think
Big—A Teradata Company—provides
services for Data Lake Architecture,
Data Lake Foundation, and Data
Lake Analytics. Performance hurdles,
prolonged implementation periods,
and reliability issues—are solved by
the Teradata Appliance for Hadoop
when compared to solutions that are
not preconfigured. Teradata does the
hardware and software integration plus
plenty of testing so you don’t have to do
it. The Teradata Appliance for Hadoop
is delivered ready-to-run and optimized
for enterprise-class big data storage and
discovery. Teradata also provides self-
service tools and accelerators for reliable
data ingest and powerful multi-genre
analytics for the data lake.
DESIGNING FOR FASTER
BUSINESS VALUE
The data lake design pattern focuses
your organization’s resources on the
difficult job of leveraging fit-for-purpose
data to derive compelling business
insights. But, in order to move the
data lake from concept to real-world
implementation, you need solutions
that work together effectively and
span the wide array of data-related
considerations. The path to the data
lake can be challenging, but it doesn’t
have to be if you follow a trusted route
that experts have already mapped out.
With an efficient, high-performance,
and highly-scalable solution from
Informatica, Hortonworks, and
Teradata, the data lake design pattern
can accelerate your organization’s time
to business value.
FOR MORE INFORMATION
ABOUT THE UNIFIED DATA LAKE
ARCHITECTURE, VISIT
http://informatica.com/bigdata,
http://www.teradata.com/products-
and-services/data-lake-products, and
www.hortonworks.com.

Loading Your SaaS Data
Into a Big Data Lake
Enterprises today are using analytics
—descriptive, predictive and prescriptive
—to quickly react to market changes
and predict customer needs. As a result,
data has become a critical resource for
every business. Research shows that by
2018, the world will produce around
50,000 GB of data every second, 90% of
which will be unstructured. However,
traditional data warehouses are
optimized for storing structured data. So
every business needs to ask, how can we
get the most value out of that 90%?
ADOPTING AN EFFECTIVE DATA
MONETIZATION STRATEGY
Big data lakes provide a cost-effective
alternative to store unstructured data.
A schema is created after a data scientist
selects the data he wants to use—
commonly known as schema-on-read.
Enterprises can reap big benefits by this
approach since it enables IT to retain all
the collected data and processing only
useful content on demand. This enables
big data analytical tools to perform
broader analyses of data because there is
real-time access to all the raw data being
produced. Thanks to this innovation,
organizations are now able to derive
more profitability from the data—
known as data monetization.
EASY ACCESS TO YOUR
SAAS APP DATA TRANSLATES
TO BETTER BI
Data lakes thrive on the 4 Vs of
data—volume, velocity, variety and
veracity. So, having easy access to data
is one of the basic prerequisites for a
data lake. While consumer data can
be integrated across mobile, social,
web and IoT devices, it is very difficult
to pull together enterprise customer
information from across various SaaS
applications. Most B2B companies find
it particularly difficult to gather enough
customer data that is consistent, timely
and accessible. However, new SaaS-based
solutions have empowered businesses
to get access to high-quality customer
information.
Currently, there are three fast-
growing types of cloud applications:
1. Marketing Automation: Marketo,
Hubspot and Oracle Eloqua help
identify potential customers through
lead-generation, scoring, nurturing
and conversion activities.
2. Sales Automation: SalesForce,
SugarCRM and Microsoft CRM
enable businesses to convert leads
into customers by customizing the
sales cycle based on lead score.
3. Customer Service Automation:
Oracle Service Cloud and ServiceMax
help provide excellent customer
support. Retaining a customer is
much easier than winning a new
customer and is critical for long-term
success.
STANDARDS-BASED
CONNECTIVITY IS THE KEY
Critical customer information such as
channel activities, customer web journey
and post-sales support is now hosted on
the cloud. These applications can also
provide access to historical trends, reports
and predictions. Many organizations use
Apache Sqoop to transfer all this data
between relational database systems and
data lakes. Although Apache Sqoop can
connect to any data source supporting
sponsored content
JDBC interface, it is not easy to develop
standards-based connectivity for ach
SaaS application because each source
has a proprietary REST or SOAP API
and sometimes both. To add to the
complexity, each of these APIs return
the results in a different format—JSON,
XML, or CSV.
DATA CONNECTIVITY
MADE SIMPLE
Progress® DataDirect®, the leader in
data connectivity solutions, provides a
sophisticated solution to this complex
ingestion problem. They provide highly
scalable and reliable JDBC drivers for
all the major SaaS data sources. With no
web services APIs to deal with, it’s simple
to quickly deploy and easily manage
drivers. And if the data lake is behind a
firewall, Progress® DataDirect Cloud®
provides a powerful data connectivity
service that provides universal
connectivity to all your data sources—
wherever they are, with no complex
firewall changes required. The simple
configuration in DataDirect connectors
frees you from managing API changes
and can improve CPU efficiency up to
400% and save over 550% in memory.
Sumit Sarkar, the Chief Data
Evangelist at Progress Software, says
that “We’re seeing a trend in accessing
customer data from SaaS applications
to build data lakes. This brings the most
valuable data, customer data, into the
big data ecosystems where organizations
are making significant investments to
derive insight.”
PROGRESS DATADIRECT
www.datadirect.com
DBTA. COM/ BI GDATAQUARTERLY 27