This article has been accepted for publication in a future issue of this journal, but has not been
fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TFUZZ.2018.2812157, IEEE
Transactions on Fuzzy Systems
Soft and Declarative Fishing of Information
in Big Data Lake
Bożena Małysiak-Mrozek, Member, IEEE CI, Marek Stabla, and Dariusz Mrozek, Senior Member, IEEE SMC,
EMBS
Abstract—In recent years, many fields that experience a sudden
proliferation of data, which increases the volume of data that
must be processed and the variety of formats the data is stored
in have been identified. This causes pressure on existing compute
infrastructures and data analysis methods, as more and more
data is considered as a useful source of information for making
critical decisions in particular fields. Among these fields exist
several areas related to human life, e.g., various branches of
medicine, where the uncertainty of data complicates the data
analysis, and where the inclusion of fuzzy expert knowledge in
data processing brings many advantages.
In this paper, we show how fuzzy techniques can be incor-
porated in Big Data analytics carried out with the declarative
U-SQL language over a Big Data Lake located on the Cloud. We
define the concept of Big Data Lake together with the Extract,
Process, and Store (EPS) process performed while schematizing
and processing data from the Data Lake, and while storing results
of the processing. Our solution, developed as a Fuzzy Search
Library for Data Lake, introduces the possibility of (1) massively-
parallel, declarative querying of Big Data Lake with simple and
complex fuzzy search criteria, (2) using fuzzy linguistic terms in
various data transformations, and (3) fuzzy grouping. Presented
ideas are exemplified by a distributed analysis of large volumes
of biomedical data on Microsoft Azure cloud.
Results of performed tests confirm that the presented solution
is highly scalable on the Cloud and is a successful step toward soft
and declarative processing of data on a large scale. The solution
presented in this paper directly addresses three characteristics
of Big Data, i.e., volume, variety, and velocity, and indirectly
addresses, veracity and value.
Index Terms—Big Data, fuzzy logic, querying, Cloud comput-
ing, biomedical data analysis, declarative languages.
I. I NTRODUCTION
T HE pace of changes in human life, especially in big cities,
is constantly increasing. Such fast pace of life does not
remain indifferent to human health, and has a negative effect
on the state of health. Intense and often stressful work that
frequently exceeds the allowed number of hours, poor diet,
smoking, lack of adequate sleep, sufficient portion of physical
exercises, and time to rest and relax, cause a sharp increase in
the incidence of cardiovascular diseases, including high blood
pressure, high cholesterol, cardiac arrhythmias, and finally
heart attacks or strokes. Currently, many cardiology centers
have been performing research on the relationship between the
quality of lifestyle, proper diet, smoking, and the occurrence of
cardiovascular diseases. The collected data describing patients,
B. Małysiak-Mrozek and D. Mrozek are with the Institute of Infor-
matics, Silesian University of Technology, Gliwice, 44-100 Poland e-mail:
bozena.malysiak@polsl.pl
M. Stabla was with the Institute of Informatics, Silesian University of
Technology, Gliwice, 44-100 Poland.
Manuscript received April 19, 2017; revised February 26, 2018.
1063-6706 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
1
their daily habits, including diet, physical activity, or smoking
cigarettes, and finally identified diseases, are stored in special
repositories. These repositories grow in size due to the growing
number of patients, cases, and collected features that can be
explored in order to draw interesting conclusions. The amount
of data that may constitute a value for making important
decisions is constantly growing due to modern techniques of
data harvesting, newly identified data sources, and growing
capacities of storage systems that can accommodate the growth
of incoming large data volumes. The era of Big data that
we entered several years ago has changed our imagination
about the type and the volume of data that can be processed,
as well as the value of data. This is now visible in many
fields which are experiencing an explosion of data that are
considered relevant, including social networks [1], [2], [3], [4],
[5], [6], [7], [8], multimedia processing [9], internet of things
(IoT) [10], intelligent transport [11], [12], [13], [14], medicine
and bioinformatics [15], finance [16], and many others [17],
[18], that face the problem of big data. The big data problem
(or opportunity) usually arises when data sets are so large
that the conventional database management and data analysis
tools are insufficient to process them [19]. However, the large
amount of data that must be processed (large volume) is not
the only characteristic of big data. Apart from volume, big data
also have other characteristics: velocity, variety, veracity, and
value, which are together known as 5V model. The big data
solutions, including the one presented in this paper, usually
address more than one of the Vs.
Biomedical data are an example of the type of data that
have extensively proliferated in recent years. This prolifera-
tion has resulted from the ongoing technological progress in
monitoring patients and collecting appropriate information on
their state of health directly by doctors or, remotely, through
telemedicine systems. Biomedical data also are the type of
data where the use of fuzzy techniques for data processing is
justified and that brings many advantages. This is motivated by
two factors. (1) Fuzzy logic allows efficient representation of
expert knowledge, especially in the case of imprecise data,
and incorporate it into the data analysis process [20]. (2)
Biomedical data tend to miss a lot of attributing values in
a diagnosis by a doctor in clinics and health fields, which
increases the uncertainty of information [21]. Fuzzy sets [22]
can be used to represent the information with uncertainty
and inaccuracy, or to formulate imprecise search criteria in
the information retrieval process. For example, similar values
of health markers and results of laboratory tests, like blood
pressure, cholesterol, glycemic index, BMI, together with
other data, like age and gender, may lead to preparation of
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TFUZZ.2018.2812157, IEEE
Transactions on Fuzzy Systems
similar treatment scenarios for patients with similar symptoms.
On the other hand, incorporating routines for fuzzy processing
in the data analysis pipeline allows generalizing the data,
group it and aggregate, and thus, change the granularity of
information that we have to deal with. As a consequence,
this provides a way to quickly reduce the volume of data
from big to small, which is highly required in the era of Big
Data. Biomedical data analysis, performed especially for big
biomedical data sets, call for mechanisms that would allow
approximate information retrieval. This constitutes the first aim
of our works presented in this paper.
Apart from large data sets that are generated as a result of
technology-driven data proliferation, biomedical data is usu-
ally delivered in a variety of formats and may have complex
structure, e.g., this can be numerical data of biochemical mark-
ers from laboratory testing, image data for X-ray pictures or
computed tomography, time series from EKG or EEG, or DNA
microarray or next-generation sequencing data from molecular
profiling. This makes the integration and structuralization of
the data difficult or expensive, and leads to solutions that allow
storing and processing the data in its native form. Big data lake
is a central location in which users can store all their data in
its native form, regardless of its source or format. Big data
lake can be used as an environment for the development of
in-depth analytics oriented toward fast decision making on the
basis of raw data. Data analysis can be performed dynamically
and on an ad hoc manner. Data shall not be prepared and
structuralized far before the begining of the analysis. Usually,
only a fraction of the data gets structuralized dynamically
during the analysis and for the purpose of the analysis or
possible secondary analyses performed later. This schema-on-
read approach allows skipping expensive schematization of
data, applying a schema as data are pulled out of a stored
location, focusing on analyses driven by current needs, and
keeping all the data even if just a part of the data constitutes
a value for the analysis performed at the moment. As a
consequence, data analysts are able to ask more powerful
queries that translate data into more actionable insights. Such
an approach speeds up the phase of data preparation before the
data are analyzed and visualized, but it usually comes as a cost
of losing the possibility of declarative retrieval of information,
which is typically performed in relational databases through
SQL-based querying. Moreover, it usually requires dedicated
programmatic procedures, utilization of specific computational
frameworks, like Apache Hadoop [23] or Apache Spark [24],
for efficient processing of terabytes of data, and extensive
computing resources that can be provisioned on demand from,
e.g., the Cloud [25]. Declarative data manipulation in Big Data
Lakes constitutes the second motivation for our works.
A. Related Works
World literature provides various solutions in terms of
approximate processing of Big Data and declarative data ma-
nipulation. The first group of works is focused on declarative
processing and transformation of data stored in relational
databases and NoSQL databases by querying the databases
with query languages, like SQL, containing fuzzy extensions.
1063-6706 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
2
The declarative character of the SQL language encouraged
scientists working with uncertainty and soft knowledge of
experts to extend the relational database management systems
(RDBMSs) toward fuzzy data processing by implementing
appropriate procedures and functions in the programming
language that is native for the particular database management
system (DBMS). Examples of such implementations are SQLf
[26], FQUERY [27], Soft-SQL [28], fuzzy Generalised Log-
ical Condition [29], FuzzyQ [30], and fuzzy SQL extensions
for relational databases [31], [32], [33], possibilistic databases
[34], and data warehouses [35], [36]. Mentioned extensions to
the SQL query language are noteworthy, as they deliver various
fuzzy techniques for data exploration, like fuzzy filtering,
fuzzy inference, generalization with fuzzy linguistic variables,
fuzzy grouping, which are also implemented in the solution
presented in the paper. However, they do not address the
problem of Big Data and any of the V characteristics, since
they are mainly devoted to relational databases.
Recent trends in developing and using NoSQL databases
naturally led to the transition of fuzzy techniques used in
relational databases to the NoSQL model. Although NoSQL
database systems are more specialized and domain-oriented,
they are also more scalable to cope with large volumes of data.
Works presented by Castelltort and Laurent [37], [38], [39]
show fuzzy extensions, including fuzzy filtering and linguistic
summaries, to the declarative Cypher query language used in
querying Neo4j graph databases. Similar attempts can be found
in [40], where authors proposed a fuzzy NoSQL model to deal
with large fuzzy databases hosted on Neo4j graph database.
In [41], Kacem and Touzi show how they extended document-
oriented MongoDB database and Mongo Query Language
toward flexible querying with linguistic labels. Those solutions
focus mainly on the volume characteristic of the Big Data
basing on the assumption that NoSQL databases are highly
scalable. However, no results of performance tests were pro-
vided in either of these works and not all of the solutions
support declarative querying.
The second group of works describes approaches devoted
to the processing and analysis of Big Data with the use of
fuzzy techniques. Most of the works concentrate on clustering
and classification of Big Data with the use of computational
frameworks, like Apache Hadoop and Apache Spark. In works
[42], [43] authors have showed the classification of big data
using Fuzzy K-Nearest Neighbor classifier speeding up the
classification process on Apache Hadoop. MapReduce-based
procedures for the classification task were also developed for
fuzzy rule based associative classifier presented in [44], Fuzzy
Rule Based Classification System (ChiFRBCS-BigData) pro-
posed in [45], and its extension [46]. In [47], Segatori et al.
show efficient and scalable MapReduce-based implementation
of Fuzzy Decision Trees used for fuzzy classification per-
formed on Apache Spark. In [48], authors propose a parallel
implementation of the fuzzy minimals clustering algorithm
(PFM), reporting a linear increase in the performance. Re-
duction of the volume of the processed data is proposed in
[49], where Li et al. apply two techniques, i.e., discretization
of conditional attributes and fuzzification of class labels, to
transform the original data set into a smaller one. There is
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TFUZZ.2018.2812157, IEEE
Transactions on Fuzzy Systems
also a group of papers, like [50], [51], [52], [53], [54], [55],
[56], [57], [58], [59], which adopt fuzzy clustering methods,
including various versions of Fuzzy C-Means algorithm, to
Big Data collections, very large databases, and data streams.
Finally, several works, including [60], [61], [62], [63], show
that Fuzzy Cognitive Maps and Linguistic Fuzzy Cognitive
Maps are able to handle large scale data in pattern classifica-
tion applications. Elements of fuzzy sets theory are also used
in (big) data preprocessing [64], information searching [65],
managing data privacy [66] and data access [67]. Querying Big
Data has been raised in [68], where Novikov et al. discuss an
algebraic layer for complex query processing by using adaptive
abstract operations based on the concept of fuzzy set, which
are needed to support uniform handling of different kinds of
similarity processing. However, no implementation was made.
It is also worth noting a few works that are devoted to
the application of fuzzy sets theory in medical data analysis,
since various branches of medicine deliver large volumes of
data [15]. In [69], Sundharakumar et al. propose a cloud-based
system containing fuzzy inference module for monitoring the
health of patients on the basis of the analysis of streams of
body sensor data with the use of Storm (the real-time compu-
tation system). The system is hosted on a private cloud, which
thereby ensures security and scalability, and the approach
addresses mainly the velocity characteristics of Big Data. How-
ever, no performance results were presented. An interesting
approach for the reduction of volume of data is proposed in the
paper [70], where Azar and Hassanien define and show a new
neural-fuzzy classifier for dimensionality reduction in medical
big data. Authors suggest that the proposed method simplifies
the classification tasks by reducing the dimensionality of
large data sets and by speeding up the learning process. In
[71], Behadada et al. present a novel method to define semi-
automatically fuzzy partition rules to provide a powerful and
accurate insight into cardiac arrhythmia. The approach is based
on text mining techniques applied to big data sets of freely
available scientific papers provided by PubMed [72].
Broader classification of fuzzy set techniques used in Big
Data processing is presented by Wang et al. in [73]. The wide
spectrum of published works proves that the application of
fuzzy techniques for data processing brings many advantages.
Those of the presented solutions that are dedicated to Big
Data usually assume reduction of the volume of processed
data by using various fuzzy techniques while generalizing,
grouping, and aggregating the data and classifying or assigning
to clusters. However, they do not allow performing these
operations in a declarative way. On the other hand, those
solutions that enable declarative and approximate information
retrieval are not dedicated for Big Data (SQL-based solutions),
or need a dedicated storage model (NoSQL attempts).
B. Scope of this Work
In this paper, we present a novel, scalable, and universal
solution for processing the data stored in Big Data Lake with
the use of fuzzy techniques and declarative U-SQL query
language. Presented solution allows exploring data coming
from various domains by querying large data sets with the
1063-6706 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
3
use of declarative U-SQL language. We extended the U-SQL
language for parallel querying by incorporating a variety of
techniques that enable fuzzy data processing, including fuzzy
searching, fuzzy transformation, linguistic terms assignment,
and fuzzy grouping. These operations can now be performed
on a large scale to transform the data, and thereby, allow
searching and grouping similar data on the basis of experts
knowledge, reduce the volume of the data, and prepare the
data for further analysis and visualization. Data processing is
highly parallelized in the Extract, Process, and Store (EPS)
process, which can be scaled out in order to adjust to the
volume of data stored in the Data Lake. Storing and processing
of the data is performed on Microsoft Azure public cloud,
which ensures large scalability of the presented approach.
Performance and scalability of the proposed solution are tested
with the use of several data sets containing up to one billion
rows of medical data related to cardiac disease. Presented
work addresses directly at least 3Vs that characterize Big Data,
i.e., volume, velocity, and variety, and indirectly, veracity and
value. Moreover, the paper extends the spectrum of existing
works by: (1) Showing that declarative, fuzzy querying can
be performed efficiently and effectively in big data sets when
implemented in highly scalable environments, which is proved
by performed experiments (Sect. IV). (2) Enabling fuzzy tech-
niques that allow processing and analyzing raw and schema-
free data coming from various data sources (Sect. III). (3)
Providing formal definition of the Big Data Lake concept and
EPS process (Sect. II-A). (4) Providing formal definitions to
the fuzzy operations performed on big data (Sect. II-B).
II. M ETHODS
Big Data Lake changes the way how data are stored and
managed within the IT infrastructure of an institution, and
enables responding to the changing requirements for the data
analysis in various domains. While this shift often enables an
immediate access to the data, without waiting to be cleaned,
modeled, structuralized, and loaded, it also requires to change
the way how data are prepared for the analysis.
A. Data Lake and EPS Process
Data lake allows quickly consolidating various types of data
in one place. The data may come from various data sources,
can be logically related, and can be stored in the raw form —
structured, semistructured, or unstructured.
Definition 1: Formally, we define Data Lake as a pair:
DL = {V, M }, (1)
where V is a set of values in the Data Lake, and M is a set
of metadata describing values in the Data Lake DL.
If:
∀v∈V ∃md ∈M fd (v) = md , (2)
where md represents metadata that describe the name of the
attribute for the value v, and fd is a function assigning the
name of the attribute to the value, then we call the data lake
DL as fully described in terms of attribute names.
If:
∀v∈V ∃mt ∈M ft (v) = mt , (3)
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TFUZZ.2018.2812157, IEEE
File 1 File 2 File 3
Query 1 Query 2 Query 3
(U-SQL) (U-SQL) (U-SQL)
R1 R2
Data Lake
Fig. 1. Extract, Process, Store (EPS) over a Data Lake. Data extracted to
rowsets R1 and R2 are processed and queried, and results are stored in output
files.
where mt represents metadata that describe the type of the
value v, and ft is a function assigning the type to the value,
then we call the data lake DL as fully described in terms of
data types.
If data lake DL is fully described in terms of attribute names
and in terms of data types, then we call the data lake DL as
completely described. If the data lake is completely described
it can usually be processed automatically without additional
knowledge of experts. Otherwise, the knowledge is needed in
various phases of processing data stored in the data lake.
Definition 2: Extract, Process, Store (EPS) refers to a
process of using data stored in a data lake by extracting appro-
priate information, structuralizing it, processing and querying,
and storing the information in a specific form (Fig. 1). Partic-
ular steps of the EPS process will be defined in the following
parts of the section.
Definition 3: Extract: Data extraction is the first phase
of the EPS process, in which data are read from the Data
Lake and represented as one or several rowsets. A single
rowset is a set of tuples and it is equivalent to the relation in
relational databases. Schema of the relation/rowset is defined
dynamically, on read, i.e., it is neither predefined, nor known
priorly (schema on read). Although, we do not define any
additional constraints between any two rowsets, like foreign
keys in relational databases, logical relationships may exist
between rowsets created in the Extract phase.
A single extraction Ei is an operation that produces a rowset
Ri (Ai1 , Ai2 , ..., Ain ) with n attributes, based on values and
metadata coming from data lake DL, and optional transfor-
mation rules provided by an expert during the extraction step:
Ei (Vi , Mi , TEi ) −→ Ri (Ai1 , Ai2 , ..., Ain ),
where: Vi ⊂ V represents the subset of values extracted from
1063-6706 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Transactions on Fuzzy Systems
4
the data lake DL in the extraction Ei ; Mi ⊂ M represents
Store subset of metadata describing values extracted in the extraction
Ei ; TEi is a set of transformation rules.
Transformations tE ∈ TEi can be applied to the extracted
values or to the metadata, e.g., they can be used to supplement
Process a missing name of an attribute or to define the type for data
extracted from the partially described data lake, or to derive
some values on the basis of other values in the data lake.
They are used to properly read the complex unstructured data,
to clean and standardize the data, and finally to format the
Extract
data as a rowset (e.g., see Sect. IV-B).
Definition 4: Process: Data processing covers all operations
applied to the produced rowsets that allow converting the data
from one form to another in order to meet requirements of the
final analysis or the visualization performed by a customer
(consumer of the results).
The set of operations may include:
• Selecting only certain attributes of the produced rowsets
to be presented to the final user;
• Joining data from multiple rowsets and producing new
rowsets;
• Filtering rowsets based on simple and complex search
conditions;
• Deriving a new calculated attribute based on other at-
tributes of processed rowsets;
• Sorting or ordering data based on selected attributes;
• Grouping and aggregating data;
• Labeling data;
• Looking up for relevant data in other rowsets;
• Validating data and cleaning with custom procedures and
functions.
Examples of various operations performed in the Process phase
are shown in query scenarios SQ1–SQ4 in Sect. IV-C, and
Appendix B (SQ5) and Appendix C in the Supplement to the
paper.
Definition 5: Store: Data storing is the last phase of the
EPS process, in which data are read from the selected rowsets
and sent to a place of destination. The place of destination
can be a relational database, a simple delimited flat file, or a
formatted file with a more complex structure.
If the destination place is a relational database, a single
output Oi is an operation that, on the basis of the rowset
Ri (Ai1 , Ai2 , ..., Ain ) with n attributes, produces a relation
RiDB with the same schema in the relational database:
Oi (Ri ) −→ RiDB (Ai1 , Ai2 , ..., Ain ). (5)
If the destination place is a file, a single output Oi is an
operation that, on the basis of the rowset Ri (Ai1 , Ai2 , ..., Ain )
with n attributes and optional transformation rules provided by
an expert during the Store step, produces a data output file Fi
with the data from the rowset:
Oi (Ri , TOi ) −→ Fi , (6)
where TOi is the set of transformation rules applied to the
stored data. It may comprise operations, such as proper,
(4) custom formatting of the output file, or compression of the
output file. Example of the Store phase is shown in query
scenario SQ5 located in the Supplement to the paper.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TFUZZ.2018.2812157, IEEE
Transactions on Fuzzy Systems
5

µCh(x) Fuzzy selection can be performed on the basis of compound

Cholesterol Cholesterol
=197 high =248 fuzzy search conditions (or mixed with crisp search condi-
1
tions), e.g.:
0.7 λ
σ̃A i ≈vi θ...θ Aj ≈vj
(R)
λ (10)
= {t : t ∈ R, (t(Ai )≈vi θ...θ t(Aj )≈vj ) ≥ λ},
0.2 Cholesterol
where Ai , Aj are attributes of rowset R, i, j = 1...n; i 6= j,
190 200 240 250 vi , vj are fuzzy sets; λ is a minimum membership degree
for which the whole complex fuzzy condition is fulfilled;
Fig. 2. Definition of high cholesterol fuzzy set and interpretation of fuzzy
selection with the fuzzy search condition cholesterol is high showing two and θ can be any of the fuzzy operators of conjunction and
values of the Cholesterol attribute: 197 that satisfies the condition, and 248 disjunction (we implemented Zadeh T-norm and S-norm [22],
that does not satisfy the condition for λ = 0.5. [74]). Therefore,
λ
σ̃A i ≈vi θ...θ Aj ≈vj
(R) = {t : t ∈ R, (µvi (t(Ai ))
B. Fuzzy Extensions to the Process Phase (11)
θ...θ µvj (t(Aj ))) ≥ λ},
For fuzzy exploration of data stored in a data lake we have
where µvi , µvj are membership functions of fuzzy sets vi , vj .
extended a collection of standard operations performed on the
generated rowsets. We will use the following sample rowset Definition 7: Extended projection with fuzzy transforma-
to exemplify the presented ideas throughout this section. tion: Given a rowset R with n attributes:
Patients = Id Age Cholesterol R = {A1 A2 A3 ...An }, (12)
-- --- -----------
1 45 197
2 67 248
a projection with fuzzy transformation π̃ is a unary operation,
3 25 165 in which attributes of the rowset R are restricted to the set
4 38 192
5 72 210
{A1 , A2 , ..., T̃ (Ai ), ..., Ak }, where {A1 , A2 , ..., Ai , ..., Ak } ⊆
{A1 , A2 , A3 , ..., An }, k ≤ n, and one (or more) of the at-
Definition 6: Fuzzy selection: Given a rowset R with n tributes is subjected to transformation T̃ :
attributes:
R = {A1 A2 A3 ...An }, (7) π̃A1 A2 ...T̃ (Ai )...Ak (R)
(13)
= {t[A1 A2 ...T̃ (Ai )...Ak ] : t ∈ R, k ≤ n},
a fuzzy selection σ̃ is a unary operation that denotes a subset
of a rowset R on the basis of fuzzy search condition: where t[A1 A2 ...T̃ (Ai )...Ak ] is the restriction of the tuple t to
λ the set {A1 , A2 , ..., T̃ (Ai ), ..., Ak } so that:
σ̃ λ (R) = {t : t ∈ R, t(Ai ) ≈ v}, (8)
Ai ≈v
t[A1 A2 ...T̃ (Ai )...Ak ] =
λ
where Ai ≈ v is a fuzzy search condition, Ai is one of the {(A, x) : x = t(A),
attributes of the rowset R for i = 1..n, n is the number of (14)
A ∈ {A1 , A2 , ..., A0i , ..., Ak },
attributes of the rowset R, v is a fuzzy set represented by a
A0i = T̃ (Ai )}.
membership function (e.g., young person, age near 30, normal
blood pressure), ≈ is a comparison operator used to compare The transformation T̃ can be one of the two specific types
the crisp value of attribute Ai for each tuple t from the rowset of transformations T̃ ∈ {T̃µ , T̃L }:
R with the fuzzy set v, λ is a minimum membership degree
• T̃µ (Ai ) calculates the membership degree of the value
(threshold) for which the search condition is satisfied (Fig. 2).
t(Ai ) of attribute Ai for each tuple of the rowset R to
The selection σ̃ λ (R) denotes all tuples in R for which
Ai ≈v the fuzzy set v:
≈ holds between the attribute Ai and the fuzzy set v with the
membership degree greater or equal to λ. Therefore, ∀t∈R T̃µ (t(Ai )) = µv (t(Ai )). (15)

σ̃ λ (R) = {t : t ∈ R, µv (t(Ai )) ≥ λ}, (9) • T̃L (Ai ) assigns and returns a linguistic value l of the
Ai ≈v
defined linguistic variable L = {l1 , l2 , ..., lm }, m ∈ N+
where µv is a membership function of a fuzzy set v. for the value of the attribute Ai from each tuple of the
The fuzzy set v can be defined by various types of member- rowset R:
ship functions, including triangular, trapezoidal, and Gaussian.
∀t∈R ∃l∈L T̃L (t(Ai )) = l
For example, the fuzzy selection with the search condition
Cholesterol is high for the fuzzy set high cholesterol defined ∧ µl (t(Ai )) = (16)
as presented in Fig. 2, and λ = 0.5 produces the following max{µl1 (t(Ai )), µl2 (t(Ai )), ..., µlm (t(Ai ))}.
rowset (σ̃ 0.5 (P atients)).
Cholesterol ≈ high For example, the extended projection with fuzzy trans-
Id Age Cholesterol formation (both types) of the Cholesterol attribute on the
-- --- -----------
1 45 197
basis of the Cholesterol linguistic variable (ChV), defined
5 72 210 as presented in Fig. 3, produces the following rowset.

1063-6706 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TFUZZ.2018.2812157, IEEE
Transactions on Fuzzy Systems
6

ChV linguistic variable

µCh(x) Azure Data Lake
normal high very high
1
Analytics
Fuzzy Search Domain Specific
Library Libraries
HDInsight
192 U-SQL

210
197

248
165

Cholesterol

190 200 240 250

Fig. 3. Assignment of sample values of the Cholesterol attribute to linguistic
values (normal, high, and very high cholesterol ) of a sample Cholesterol
linguistic variable (ChV).
π̃Id,Age,Cholesterol,T̃ChV (Cholesterol),T̃µ (Cholesterol)
ChV
(P atients) =
Id Age Cholesterol ChV u Fig. 4. General architecture of Azure Data Lake service with fuzzy extensions
-- --- ----------- --------- ---
1 45 197 high 0.7
2 67 248 very high 0.8
3 25 165 normal 1.0
4 38 192 normal 0.8
5 72 210 high 1.0
Definition 8: Fuzzy grouping: Fuzzy grouping performs
classification of crisp values of a selected attribute Ak of the
rowset R to particular groups represented by linguistic values
of a predefined linguistic variable L = {l1 , l2 , ..., lm }, m ∈
N+ . Each value from the rowset is then transformed by the
fuzzy transformation T̃L (Ak ) and represented by a fuzzy set,
described by a membership function. The grouping algorithm
compares each value from the rowset with particular values
of the linguistic variable and produces values of membership
degree. A value (tuple) from the rowset is then assigned to a
group, represented by a fuzzy set, for which the value of the
membership degree is the highest among membership degrees
calculated for all linguistic values of the variable. Therefore,
each of the rowset value is assigned to a linguistic value, which
it fits the most.
Grouping algorithm: Given is a linguistic variable with a
set of linguistic values L = {l1 , l2 , ..., lm }, m ∈ N+ . Each
linguistic value li , i = 1, ..., m is described by an arbitrarily
defined membership function µi (x). For each value t(Ak ) :
t ∈ R, k < n, the algorithm takes the following steps:
1) ∀li ,i=1,...,m calculates a membership degree µi (t(Ak )),
2) Determines the maximum membership degree for the
value t(Ak ) on the basis of membership degrees calcu-
lated in step 1:
µmax (t(Ak )) = max |µi (t(Ak ))|.
i=1,...,m
3) Assigns the value t(Ak ) to the group with maximum
membership degree.
For example, fuzzy grouping by the Cholesterol attribute
from the sample Patients rowset on the basis of the Cholesterol
linguistic variable (ChV), defined as presented in Fig. 3,
would produce three groups: normal cholesterol containing
two tuples, high cholesterol with two tuples, and very high
cholesterol with one tuple of the sample rowset (as shown
in the example presented in Definition 7). Having the data
YARN
WebHDFS
Store
(unstructured, semi-structured, structured)
(Fuzzy Search Library, marked in yellow) and Domain Specific libraries
(marked in green).
grouped, users can aggregate the data with the use of standard
or custom aggregation functions (e.g., see SQ4 in Sect. IV-C4).
III. I MPLEMENTATION
Particular mechanisms presented in Sect. II were imple-
mented in Azure Data Lake environment, a hyperscale repos-
itory for big data analytic workloads in the Microsoft Azure
cloud.
A. Azure Data Lake
Azure Data Lake (ADL) is a scalable, cloud environment for
storage and analytics. It allows for interactive batch analysis
of various types of data, including structured, semi-structured,
and unstructured data, in real time [75]. General architecture
of the Azure Data Lake is presented in Fig. 4. It is comprised
of two main parts:
1) Data Lake Store (DLS) provides petabyte scale, unlim-
ited storage for data lake DL. It distributes large data
sets located in files across many storage servers, which
allows performing read operations in parallel and to
improve read performance.
2) Data Lake Analytics (DLA) allows for an efficient and
scalable analysis of data stored in Big Data Lake by
parallelizing the analysis on a distributed infrastructure
in the Azure cloud. It provides HDInsight services,
(17) including Hadoop and Spark, to analyze Big Data, and
U-SQL distributed execution environment for declarative
processing and analysis of data.
The U-SQL, which is used in our solution, is a big data
query language that combines declarative capabilities of the
SQL query language and expressive power of C# code. Data
processing and transformation (the Process phase of EPS) is
performed with the use of U-SQL query expressions, includ-
ing the SELECT expression, and the PROCESS, REDUCE,
COMBINE expressions that apply the custom or user-defined
operators (UDOs). These query expressions produce rowsets

1063-6706 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TFUZZ.2018.2812157, IEEE
Fuzzy Search Library
Membership
Linguistics
Functions
Udfs
Fig. 5. Main modules of the Fuzzy Search Library library that allows fuzzy
data processing in Data Lake.
that can be assigned to rowset variables. Rowset variables are
populated in the Extract phase implemented by the EXTRACT
U-SQL phrase with the use of appropriate data extractor (can
be built-in, e.g., CSV or TSV, or custom, dedicated). Data
output (the Store phase of the EPS) is implemented in the U-
SQL OUTPUT phrase that uses appropriate data outputter,
which, as in the Extract phase, can be built-in or custom.
Sample U-SQL scripts are presented in Sect. IV and in the
Supplement to the paper. We have enabled fuzzy processing
of the big data stored in Data Lake by developing fuzzy
extensions to the U-SQL language available in DLA, and by
providing new set of operations in the Process phase of the
EPS. These extensions are marked in yellow and green in the
general architecture of Azure Data Lake presented in Fig. 4.
B. Modules and Methods for Fuzzy Data Processing
Fuzzy transformations performed in the Process phase of
the EPS process can be implemented in two methods: (1)
as custom rowset processors in C# invoked in the U-SQL
PROCESS expression, optionally accompanied by custom RE-
DUCER and COMBINER expressions, or (2) as user-defined
functions (UDFs) nested in U-SQL SELECT expressions. The
second approach usually allows building more optimal U-SQL
execution plans and was adopted in the presented solution.
Common methods for fuzzy data processing, including fuzzy
selection, extended projection with fuzzy transformation, fuzzy
grouping, defining linguistic variables and assigning data to
linguistic variables, are assembled in the Fuzzy Search Library
(FuzzySearchLib, Fig. 5) that we have developed for the Pro-
cess phase. These methods are exposed to users as user-defined
functions. Additional processing methods, custom extractors,
and outputters specific for the domain of the processed data
are implemented as Domain Specific Libraries.
The Fuzzy Search Library contains four main component
modules that provide methods for fuzzy data processing:
• MembershipFunctions that allows creating trapezoidal,
triangular, and Gaussian membership functions to rep-
resent particular fuzzy sets or fuzzy values. The module
also provides a dedicated Result data type that is usually
used to represent a similarity degree when comparing
a crisp value of a processed attribute of a rowset to
a fuzzy set represented by and appropriate membership
1063-6706 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Transactions on Fuzzy Systems
7
function. This comparison involves calculation of a value
of the membership function for the crisp value of the
attribute. In such cases, the Result data type is used as an
intermediate data type and it was implemented in order to
overload the & and | operators to work according to the
Zadeh t-norm and s-norm. Both operators are provided
by this module for fuzzy selections containing complex
search conditions joined according to Zadeh t-norm and
Udl s-norm. They are nonoverloadable for the standard double
data type used to represent membership degrees, and for
this reason, we had to use the intermediate Result data
type to overload these logical operators.
• Linguistics allows creating linguistic variables and lin-
guistic values. For any linguistic variable, this module
also enables two important functions:
– Values that returns linguistic values of the linguistic
variable,
– Group that allows for linguistic grouping.
• Udfs provides functions that are used while creating U-
SQL queries, e.g., performing fuzzy selection, including:
– Is, which assesses how much a given crisp value
matches a specified linguistic value or a defined
fuzzy set (overloaded function),
– Around, which assesses how much a given crisp
value matches to a fuzzy value represented by a
Gaussian membership function,
– Value, which converts a value of a membership
function (the Result object) to a numerical value,
• Udl (User dynamic linguistics) provides functions to
operate on linguistic variables created dynamically in the
U-SQL code, e.g.:
– Get, which creates and returns a linguistic variable
and its values on the basis of a formatted character
string passed as an argument,
– GetLV (Get Linguistic Value), which returns a speci-
fied linguistic value for a specified linguistic variable.
C. Defining Fuzzy Linguistic Variables
Fuzzy linguistic variables are objects that allow adding
semantics to the processed data by assigning linguistic labels
to elements of a universe and by providing a confidence of the
assignment. Linguistic variables can be defined in two ways:
• they can be implemented as classes in C# programming
language and delivered by a domain-specific library ex-
tending U-SQL,
• they can be implemented dynamically in the U-SQL code.
Linguistic variables, together with linguistic values (terms),
delivered by specific libraries are usually dedicated for par-
ticular domains of processed data. They can be predefined
on the basis of expert knowledge. Programmatically, they are
represented as classes in the C# programming language and
are created through implementation of interfaces provided by
the Fuzzy Search Library (in the Linguistics module). The
capability of defining linguistic variables in this way is very
beneficial — other users (developers) may independently de-
liver external libraries for U-SQL-based fuzzy data processing
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TFUZZ.2018.2812157, IEEE
Transactions on Fuzzy Systems
8

with linguistic variables defined based on the knowledge of U-SQL script extracts data by using the EXTRACT phrase
experts specific for particular data domains. This allows to (line 10) from the big data file (lines 8 and 13) stored in
independently create a collection of libraries for processing the CSV format. The CSV built-in extractor is used for this
various types of data. Sample C# code for defining linguistic purpose (line 14). The extraction from the Big Data Lake
variables is presented in Appendix D in the Supplement. produces the rowset @data (line 9), which is used in the
We also have made possible to define linguistic variables Process phase of the EPS. Schema of the rowset is defined
directly in the U-SQL code. This allows for dynamic defining on read (lines 10–12) by specifying a column name and C#
of variables that are not known before the implementation of type name per each extracted attribute of the Data Lake. Data
the Process phase. In the U-SQL code they are represented extraction is preceded by loading the Fuzzy Search Lib and
as string variables and formatted according to the grammar Heart Disease Data libraries into the script context (lines 1–
presented in Appendix E (Supplement). Dynamically defined 2). The former exposes all fuzzy extensions to U-SQL (fuzzy
linguistic variables are managed by methods available through modules described in Sect. III-B). The latter delivers domain-
the Udl module of the Fuzzy Search Library. The capability of specific fuzzy linguistic variables used in data processing.
defining linguistic variables in this way has two advantages: Short names for modules exposed by both libraries are mapped
• Dynamically defined linguistic variables can be created in lines 3–5 to simplify the syntax of U-SQL queries used in
ad hoc in the U-SQL scripts. the Process phase.
• They do not require to be compiled to external libraries. 1 REFERENCE ASSEMBLY H e a r t D i s e a s e . H e a r t D i s e a s e D a t a ;
2 REFERENCE ASSEMBLY H e a r t D i s e a s e . F u z z y S e a r c h L i b ;
Sample definition of a linguistic variable created in this way 3 USING Udfs = F u z z y S e a r c h L i b . Udfs ;
is presented in Listing 1. 4 USING V a r i a b l e s = H e a r t D i s e a s e D a t a . V a r i a b l e s ;
5 USING Udl = F u z z y S e a r c h L i b . Udl ;
1 DECLARE @ c h o l e s t e r o l V a r s t r i n g = ” C h o l e s t e r o l : Normal 6
, T , 0 , 0 , 1 9 0 , 2 0 0 / High , T , 1 9 0 , 2 0 0 , 2 4 0 , 2 5 0 / VeryHigh , T 7 / / Extract data
,240 ,250 ,1000 ,1000 ” ; 8 DECLARE @ d a t a P a t h s t r i n g = ” / i n / H e a r t D i s B i g . c s v ” ;
Listing 1. Sample U-SQL code for ad hoc defining of the linguistic variable 9 @data =
Cholesterol with its linguistic values. 10 EXTRACT I d s t r i n g , Age i n t , Sex i n t , . . . ,
11 Cholesterol int , CigarettesPerDay int , . . . ,
12 RestHr i n t , . . . , Disease bool
IV. E XPERIMENTAL R ESULTS 13 FROM @ d a t a P a t h
14 USING E x t r a c t o r s . Csv ( ) ;
Our extensions for fuzzy data processing in Big Data Listing 2. A part of a U-SQL script extracting data and producing the @data
Lake have been tested extensively, in order to verify its rowset.
functionality, effectiveness, and performance. In this section, A part of the rowset produced by the presented U-SQL
we show sample U-SQL queries that demonstrate capabilities script has the following form (column headers were formatted
of the proposed extensions and the implementation of the manually).
EPS process. We also present results of performance tests that Id Age Sex Chol-l Cig-PerDay RestHR Disease
----------- --- --- ------ ---------- ------ -------
reveal scalability of our solution for Big Data. 474a1bce-.. 57 M .. 192 75 .. 86 .. True
d3258689-.. 53 M .. 203 20 .. 86 .. False
3ee478ee-.. 48 M .. 229 0 .. 75 .. False
A. Data Set and Processing Environment e5724521-.. 54 M .. 239 20 .. 86 .. True
33be448a-... 49 M .. 188 30 .. 78 .. False
All tests of the presented solution were carried out with 93e06584-.. 64 M .. 211 30 .. 58 .. True
8e4d191d-.. 66 F .. 226 0 .. 49 .. True
the use of biomedical Heart Disease data set [76] from ...
UCI Machine Learning Repository. This data set contains
75 attributes, but we present our experiments by using a C. Examples of U-SQL Queries Extended with Fuzzy Process-
subset of them (mainly based on the Cleveland database). ing Capability
In order to test the performance, records were proliferated In this section, we show the use of various fuzzy techniques
up to 100 million, 500 million, and 1 billion for various in data transformations performed in the Process phase. Pre-
experiments. To examine flexibility of fuzzy querying, during sented sample query scenarios (SQ1–SQ4) allow understand-
the data proliferation, existing values of particular attributes ing how the Fuzzy Search Library can be used to process data
were multiplied by a randomly generated factor from the range in Data Lake.
[0;1] in order to differentiate the data and obtain similar, i.e., 1) Fuzzy selection with the use of fuzzy values (SQ1): This
not identical, cases. Data proliferation and performance tests simple example shows filtering with the use of fuzzy condition
were conducted in Azure Data Lake environment on Microsoft built on the basis of a fuzzy value. In this example we perform
Azure public cloud. Proliferated data were stored as CSV files the following operation: Retrieve patients, whose age is around
in the Data Lake Store on the Clcud. U-SQL scripts were 50 years old. The U-SQL query implementing this operation
executed as distributed jobs in DLA, thus, performing parallel is shown in Listing 3.
data processing for all phases of the EPS. 1 @age =
2 SELECT Id , Age ,
3 Udfs . Around ( Age , 5 0 , 4 5 ) . V a l u e AS MemberDg
B. Data Extraction 4 FROM @data
Data extraction (the Extract phase) was performed with 5 WHERE Udfs . Around ( Age , 5 0 , 4 5 ) . V a l u e >= 0 . 5 ;
Listing 3. Sample U-SQL query retrieving patients, whose age is around 50
the use of the code presented in Listing 2. This part of the years old.

1063-6706 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TFUZZ.2018.2812157, IEEE
Transactions on Fuzzy Systems
9

This U-SQL query processes and transforms the rowset ...

@data to the rowset @age containing only two attributes (Id
3) Extended projection with fuzzy transformation and fuzzy
and Age). Selection in the WHERE clause (line 5) is based
selection (SQ3): This example shows extended projection with
on the fuzzy condition that uses the Around function from
T̃µ fuzzy transformation, filtering on the basis of complex
the Udfs module of the Fuzzy Search Library to represent the
search condition, and the use of overloaded disjunction oper-
fuzzy set age around 50 with the use of the Gaussian mem-
ator | (working according to Zadeh s-norm). In this example,
bership function. The third argument of the Around function
we perform the following operation: Retrieve data for patients,
(45) is the left crossover point, i.e., the element in universe,
whose resting heart rate is good or competitive (athlete’s).
left to the midpoint (50), whose membership value is equal
The U-SQL query implementing this operation is shown in
to 0.5. The right crossover point is calculated automatically,
Listing 5.
e.g., for the given example the right crossover point is 55. The
query retrieves only those rows that satisfy the fuzzy search 1 @hr =
condition with a minimal membership degree greater or equal 2 SELECT Id , Age , RestHr ,
3 ( Udfs . I s ( RestHr , V a r i a b l e s . R e s t H r C o n d i t i o n .
to 0.5 (λ = 0.5). Results produced by the query have the V a l u e s . Good ) |
following form. 4 Udfs . I s ( RestHr , V a r i a b l e s . R e s t H r C o n d i t i o n .
V a l u e s . C o m p e t i t i v e ) ) . V a l u e AS MemberDg
Id Age MemberDg 5 FROM @data
------------ --- --------
d3258689-... 53 0.779
6 WHERE Udfs . V a l u e ( Udfs . I s ( RestHr ,
3ee478ee-... 48 0.895 7 V a r i a b l e s . R e s t H r C o n d i t i o n . V a l u e s . Good ) |
e5724521-... 54 0.641 8 Udfs . I s ( RestHr , V a r i a b l e s . R e s t H r C o n d i t i o n .
8fba1e44-... 50 1.000 V a l u e s . C o m p e t i t i v e ) ) >= 0 . 2 ;
1e0340c4-... 45 0.500
...
Listing 5. Sample U-SQL query retrieving data for patients, whose resting
heart rate is good or competitive (athlete’s).
2) Fuzzy selection with the use of linguistic variables This U-SQL query processes and transforms the rowset
(SQ2): In this example, filtering is performed on the basis of @data (line 5) and produces the rowset @hr (line 1) con-
fuzzy search condition that makes use of a linguistic variable. taining selected attributes of the rowset @data (Id of the
In this example, we perform the following operation: Retrieve patient, age, and resting heart rate RestHr, line 2), and the
data for patients, whose cholesterol level is normal. The U- degree of membership of the crisp value of the resting heart
SQL query implementing this operation is shown in Listing 4. rate (RestHr) for a particular row to the fuzzy set Good or
1 @normal = Competitive defined as linguistic values (lines 3–4). Selection
2 SELECT Id , Age , Sex , RestBP , C h o l e s t e r o l , D i s e a s e
3 FROM @data
in the WHERE clause (line 6) is based on the complex fuzzy
4 WHERE Udfs . V a l u e ( Udfs . I s ( C h o l e s t e r o l , search condition that uses the overloaded disjunction operator
5 V a r i a b l e s . C h o l e s t e r o l . V a l u e s . Normal ) ) >= 0 . 5 ; |. Moreover, it makes use of linguistic variable RestHrCondi-
Listing 4. Sample U-SQL query retrieving patients, whose cholesterol level tion (lines 7–8) and its Good and Competitive linguistic values
is normal.
defined in an external, domain-specific library HeartDisease-
This U-SQL query processes and transforms the rowset
Data, implemented in C# programming language. The rowset
@data (line 3) and produces the rowset @normal (line 1)
generated by the query has the following form.
containing selected attributes of the rowset @data (line 2). Se-
lection in the WHERE clause (lines 4–5) is based on the fuzzy Id Age RestHR MemberDg
------------ --- ------ --------
condition that uses the linguistic variable Cholesterol (invoca- 93e06584-... 64 58 1.0
tion: Variables.Cholesterol, line 5) and its Normal linguistic 8e4d191d-... 66 49 1.0
9a249302-... 43 67 0.6
value (invocation: Variables.Cholesterol.Values.Normal, line 0b72a9cf-... 55 69 0.2
5), all defined in an external, domain-specific library called 0a8a261d-... 65 68 0.4
...
HeartDiseaseData, implemented in C# programming lan-
guage. The Udfs.Is function (line 4) is used to assess how 4) Fuzzy grouping with the use of dynamically defined
much a crisp value of the Cholesterol attribute (line 4) of linguistic variable (SQ4): This example focuses on fuzzy
the @data rowset matches the Normal linguistic value of the grouping, which is performed with the use of a linguistic
linguistic variable Cholesterol (line 5) for each row of the variable. Unlike in SQ3, here the linguistic variable and its
processed rowset. The Udfs.Value function (line 4) from the linguistic values are defined dynamically in the U-SQL code.
Udfs module of the Fuzzy Search Library is invoked in order We also show extended projection with T̃L fuzzy transfor-
to convert the Result object returned by the Udfs.Is function mation. In this example, we perform the following operation:
to a numerical value that can be compared with the minimal Show me the report on the number of patients grouped by
membership degree λ, which qualifies rows for retrieval. The the number of cigarettes smoked per day. The U-SQL query
produced rowset has the following form. implementing this operation is shown in Listing 6.
Id Age Sex RestBP Chlrl Disease 1 DECLARE @ c i g a r e t t e s s t r i n g = ” c i g s : None , T , 0 , 0 , 0 , 0 /
------------ --- --- ------ ----- ------- L i t t l e , T , 0 , 1 , 3 , 5 / Average , T , 4 , 5 , 9 , 1 0 / Much , T
474a1bce-... 57 M 140 192 True
, 9 , 1 0 , 2 0 , 2 2 / VeryMuch , T , 1 8 , 2 0 , 1 0 0 , 2 0 0 ” ;
33be448a-... 49 M 120 188 False
ca67a6dc-... 40 M 110 167 False 2 @groups = SELECT
f5e38c59-... 46 F 142 177 True 3 Udl . Get ( @ c i g a r e t t e s ) . Group ( C i g a r e t t e s P e r D a y )
8b0b7450-... 62 F 160 164 False 4 AS CigsPerDay , COUNT( I d ) AS N o O f P a t i e n t s
9d252762-... 51 M 110 175 True 5 FROM @data

1063-6706 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TFUZZ.2018.2812157, IEEE
Transactions on Fuzzy Systems
10

6 GROUP BY Udl . Get ( @ c i g a r e t t e s ) . Group ( TABLE I

CigarettesPerDay ) ; E FFECTIVENESS OF F UZZY Q UERYING IN SQ1 FOR VARIOUS λ
Listing 6. Sample U-SQL code showing the statistics on the number of
patients grouped by the number of cigarettes smoked per day according to λ Selc RCard Flex Conf Uncr FPrec
linguistic variable defined dynamically. 1.0 0.016 0.016 0.00 1.000 0.000 1.000
This U-SQL query processes and transforms the rowset 0.9 0.048 0.047 0.67 0.340 0.660 0.982
@data (line 5) and produces the rowset @groups (line 2) 0.8 0.080 0.076 0.80 0.212 0.788 0.947
containing the name of the group, which is one of the linguistic 0.7 0.112 0.100 0.86 0.160 0.840 0.900
values returned by the T̃L fuzzy transformation (line 3), 0.6 0.143 0.120 0.89 0.133 0.867 0.843
and the number of patients in the particular group (line 4). 0.5 0.174 0.136 0.91 0.118 0.882 0.782
Grouping (line 6) is performed on the basis of cigs linguistic 0.4 0.174 0.136 0.91 0.118 0.882 0.782
variable defined ad hoc in the U-SQL script (@cigarettes, 0.3 0.204 0.147 0.92 0.109 0.891 0.720
line 1). The Udl.Get function (invoked twice, in line 3 and 0.2 0.235 0.155 0.93 0.104 0.896 0.660
6), from the Udl module of the Fuzzy Search Library, is 0.1 0.294 0.163 0.95 0.098 0.902 0.555
used to extract the linguistic variable and its values from U- 0.0 1.000 0.167 0.98 0.096 0.904 0.167
SQL string variable @cigarettes. Then, the Group function Crisp 0.016 0.016 0.00 1.000 0.000 1.000
(from the Linguistics module of the Fuzzy Search Library)
returns the best matching linguistic value for the crisp value TABLE II
of CigarettesPerDay attribute of each row of the processed E FFECTIVENESS OF F UZZY Q UERYING IN SQ2 FOR VARIOUS λ
rowset. The query generates the following rowset.
λ Selc RCard Flex Conf Uncr FPrec
CigsPerDay NoOfPatients
---------- ------------ 1.0 0.305 0.305 0.00 1.00 0.00 1.000
None 390,970,972 0.9 0.308 0.308 0.01 0.99 0.01 0.999
Little 37,623,539
Average 43,628,778 0.8 0.312 0.311 0.02 0.98 0.02 0.996
Much 134,500,127 0.7 0.316 0.313 0.03 0.97 0.03 0.993
VeryMuch 291,459,274
0.6 0.319 0.316 0.05 0.97 0.03 0.988
0.5 0.323 0.318 0.06 0.96 0.04 0.983
D. Effectiveness of Fuzzy Querying 0.4 0.327 0.319 0.07 0.95 0.05 0.976
Fuzzy querying intends to provide more flexibility while 0.3 0.331 0.320 0.08 0.95 0.05 0.968
retrieving relevant data from a data source, like a database 0.2 0.334 0.321 0.09 0.95 0.05 0.960
or a data lake. Effectiveness of using the presented fuzzy 0.1 0.338 0.321 0.10 0.95 0.05 0.950
techniques in querying big data has been examined and 0.0 0.338 0.321 0.10 0.95 0.05 0.950
compared to traditional query methods. In order to examine the Crisp 0.305 0.305 0.00 1.00 0.00 1.000
effectiveness, we used several measures, including: selectivity,
relative cardinality, flexibility, fuzzy precision, confidence, and
uncertainty. All used measures are defined in Appendix A in several rows that satisfy the search condition for λ = 1.0
the Supplement to the paper. (equivalent to age = 50). For queries presented in SQ2 and
Effectiveness of using fuzzy techniques in querying big SQ3 the flexibility degree hardly reaches 6% (F lex = 0.06)
data in data lake with the use of queries presented in query for λ = 0.5, and therefore, these queries provide less flexibility
scenarios SQ1–SQ3 for various λ thresholds is presented in with respect to their crisp counterparts. Uncertainty of the
Tables I–III. Additionally, the last row in each of the Tables I– output rowset increases and the confidence of the rowset
III contains values of the same measures for crisp counterparts decreases proportionally with the growing flexibility and de-
of the fuzzy queries (e.g., crisp age = 50 vs. fuzzy age creasing λ. Low values of the uncertainty and high values of
around 50). As can be observed, with the decreasing value the confidence (SQ2) suggest that the output rowset contains
of the λ-cut threshold, the queries become more flexible, mainly rows with the highest possible membership degree.
which is visible in the growing selectivity (Selc) and relative High values of precision in SQ2 and SQ3 mean that each row
cardinality (RCard). For λ = 1.0 the selectivity and relative of the output rowset is highly relevant (contributes with high
cardinality are identical, since only rows with µRO (t) = 1.0 membership degree, regardless of whether it is the highest or
qualify for the output rowset RO . Then, their values become just close to the highest membership degree). The precision
increasingly divergent. Large differences suggest that rows may drop significantly for queries with very relaxed search
with low membership degree are qualified for the output conditions, like the query presented in SQ1 for λ = 0.1, which
rowset. Flexibility of fuzzy queries grows together with the indicates that the output rowset contains many rows with low
decreasing λ values and the degree of flexibility depends on membership degrees.
the definition of the fuzzy set v used in the fuzzy search This shows that fuzziness may extend the rowset and we can
condition, the λ itself, and the input rowset (source data). control the degree of flexibility by appropriately choosing the
For example, flexibility of the fuzzy query presented in SQ1 fuzzy sets in the fuzzy search conditions and by using properly
reaches a high level of F lex = 0.67 (Table I) already for selected λ similarity threshold. By default the λ is usually set
λ = 0.9, since the age around 50 fuzzy set is defined with to 0.5. However, in real-world applications, setting the value
the use of Gaussian membership function and there are just of the parameter depends on our knowledge of data that are

1063-6706 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TFUZZ.2018.2812157, IEEE
Transactions on Fuzzy Systems
11

TABLE III processing in Azure Data Lake with U-SQL query scenarios
E FFECTIVENESS OF F UZZY Q UERYING IN SQ3 FOR VARIOUS λ SQ1–SQ5. As can be observed, with the growing number of
allocation units, the time needed to complete the whole EPS
λ Selc RCard Flex Conf Uncr FPrec
process for all query scenarios SQ1–SQ5 drops significantly
1.0 0.412 0.412 0.000 1.000 0.000 1.000
(the time is plotted on log10 scale). For example, for query
0.9 0.412 0.412 0.000 1.000 0.000 1.000
scenario SQ4, the execution time was reduced from 3 hours
0.8 0.426 0.423 0.031 0.975 0.025 0.994
0.7 0.426 0.423 0.031 0.975 0.025 0.994
and 35 minutes, exactly 12,910 seconds, (on one AU) to 185
0.6 0.439 0.431 0.061 0.957 0.043 0.982
seconds (on 80 AUs). Eighty allocation units that were in use
0.5 0.439 0.431 0.061 0.957 0.043 0.982
was the maximum number of AUs that were needed to process
0.4 0.453 0.437 0.089 0.945 0.055 0.964 the data set, since the size of the data set did not require
0.3 0.453 0.437 0.089 0.945 0.055 0.964 to engage more computing power. It is worth noting that the
0.2 0.466 0.439 0.115 0.939 0.061 0.942 execution time curves for particular query scenarios SQ1–SQ5
0.1 0.466 0.439 0.115 0.939 0.061 0.942 follow the same pattern — the dynamics of computation speed
0.0 0.466 0.439 0.115 0.939 0.061 0.942 is not constant, and interestingly, changes in the same way
Crisp 0.412 0.412 0.000 1.000 0.000 1.000 for all query scenarios. This is caused by the allocation of
compute units to particular portions of data that are processed
in parallel. The largest of the data sets that we used (109
processed (e.g., distribution of values of an attribute) and the rows) was always divided into the same number of 80 chunks
purpose of processing. If the user is rather precise in terms of (the number depends on the size of data) that were processed
the set of results and given search conditions, but still wants by various numbers of AUs. Each data chunk is processed
to obtain a few similar cases, he should start with high values by a single AU. The allocation of data chunks to the available
of the λ similarity threshold, e.g., λ = 0.9 or λ = 0.95. Then, compute units was then more or less optimally adjusted, which
the value of the parameter can be decreased to relax the search resulted in variable dynamics of the computation speed. When
conditions and include more rows with similar values of an using 80 AUs, each data chunk is assigned to one available
attribute. If the purpose of the fuzzy querying is a preselection, AU, and the processing of the whole data set is performed
a user may start with lower values of the λ parameter (e.g., in one iteration of computations (80 AUs process 80 data
λ = 0.5 or lower), and then gradually increase the value, chunks). However, processing 80 data chunks with the use of
and decide whether the outcome is properly narrowed. On the 40 AUs requires two iterations 40 data chunks are processed
other hand, grouping with the use of linguistic variables may by 40 AUs in each of the two iterations of computations.
narrow the rowset to a representative form that is user friendly Processing the same data set with the use of 60 AUs still
and suitable for presentation, visualization, or further analysis. requires two iterations 60 data chunks are processed by 60
For example, the query presented in SQ4 returns only five AUs in the first iteration, and 20 data chunks by 20 AUs in the
rows (five groups of records, card(RF ) = 5) on the basis of second iteration, leaving 40 AUs idle. Therefore, the execution
arbitrarily defined linguistic variable with five linguistic values time remains the same for processing scenarios performed with
for the number of cigarettes smoked per day. Crisp grouping by 40 to 70 AUs as shown in Fig. 6, since they all need two
CigarettesPerDay attribute returns 197 groups (card(RC ) = iterations for data processing. This dependency is less visible,
197), which is much more, even if the number of cigarettes if the number of AUs is much smaller than the number of data
that a person is able to smoke per day (a domain) is limited. chunks that must be processed.
In order to better illustrate the results, we show the accelera-
tion of fuzzy data processing when scaling the EPS process in
E. Performance Tests Azure Data Lake from 1 to 80 AUs (Fig. 7). The acceleration
We have also conducted a series of tests in order to examine is calculated according to eq. 18 and obtained by comparing
the performance of the presented solution. For the clarity of the execution times to the case in which a particular query scenario
presentation, we show results for query scenarios SQ1–SQ5 SQ1–SQ5 was performed by using only one AU.
explained in Sect. IV-C and in the Supplement. They reflect
T1
a general tendency that we observed, while processing data Sd = , (18)
from Data Lake in the presented environment of the computing Td
cloud. In all cases, performance was assessed on the basis of where d is the number of AUs in use, T1 is the execution time
execution time measurements collected when performing the obtained while performing the U-SQL query scenario with the
whole EPS process (including all phases). We carried out at use of one AU, and Td is the execution time obtained while
least three replicas for each measurement. Then, the obtained performing a particular query scenario with the use of d AUs.
results were averaged, and the averaged values are presented As can be observed in Fig. 7, the speedup is sublinear,
in this section. Averaged values of measurements were also which results from the fact that not all component steps of
used to determine n-fold speedups when scaling computations the ADL job can be equally parallelized — in some steps
in Azure Data Lake. some AUs were idle. All speedup curves follow the same
Fig. 6 shows how the execution time depends on the trend showing that adjusting the degree of parallelism to the
executed query and the number of compute nodes, called data size (and the number of data chunks) by assigning an
allocation units (AUs), when performing parallel, fuzzy data appropriate number of AUs for EPS job execution allows the

1063-6706 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TFUZZ.2018.2812157, IEEE
Transactions on Fuzzy Systems
12

100000 1600
SQ1 SQ1
1400
SQ2 SQ2
1200
SQ3 SQ3

Execution time (s)

Execution time (s)

10000
SQ4 1000 SQ4
SQ5 800 SQ5

600
1000
400

200

100 0
0 10 20 30 40 50 60 70 80 90 0 200 400 600 800 1000 1200
# AUs (compute nodes) # rows (millions)

Fig. 6. Execution time for varying number of allocation units (AUs) for Fig. 8. Execution times for varying volume of data (millions rows of data)
parallel, fuzzy data processing in Azure Data Lake with U-SQL query in the extracted data set for parallel, fuzzy data processing in Azure Data
scenarios SQ1–SQ5. Time plotted on log10 scale. Tests performed with Lake with U-SQL query scenarios SQ1–SQ5. Tests performed with nine
a biomedical data set (extracted rowset) containing a billion (109 ) rows. allocation units (#AUs=9).

90.00 that should be processed in the EPS process, the execution

SQ1 time increases linearly for all query scenarios, and the slope
80.00 SQ2 of the lines depends on the query scenario.
SQ3
70.00
SQ4 F. Coping with Various Data Formats
SQ5 Variety refers to many different formats in which data can
60.00
ideal be stored. Access to data stored in a variety of formats is
n-fold speedup

50.00 performed in the Extract phase of the EPS process. By default,

Data Lake Analytics provides three types of built-in extractors
40.00 to generate rowsets from the input files (as well as three types
of outputters to store data after processing):
30.00
• Text — allows extraction from delimited text files of

20.00 different encodings,

10.00
0.00
0 10 20 30 40 50 60
# AUs (compute nodes)
Fig. 7. n-fold speedup for varying number of allocation units (AUs) for
parallel, fuzzy data processing in Azure Data Lake with U-SQL query
scenarios SQ1–SQ5. Tests performed with a biomedical data set (extracted
rowset) containing a billion (109 ) rows.
best use of available computational resources. The best n-fold
speedup of 69.78 was achieved for the query scenario SQ4
when scaling the execution from 1 to 80 AUs. The worst n-
fold speedup of 49.29 was achieved for query scenario SQ5
for the same scaling scheme, but it is still much better to scale
data processing than perform it on a single compute unit.
Fig. 8 shows how the solution and the entire compute
environment behave in terms of execution time increase, under
the growing amount of data for all query scenarios SQ1–SQ5.
Tests were performed with the use of nine AUs for all data sets,
as this was the maximum number of AUs that were needed for
processing the smallest data set (containing 100 million rows
of data). As can be noticed, with the growing number of rows
• CSV — allows extraction from comma-separated value
(CSV) files of different encodings,
• TSV — allows extraction from tab-separated value (TSV)
files of different encodings.
70 80 90
Sample usage of the CSV extractor in the USING clause of
the EXTRACT U-SQL expression was presented in Listing 2
(Sect. IV-B). Processing data stored in more complex formats
requires custom extractors that must be specific for a particular
format. Custom extractors can be delivered in the Domain
Specific Libraries dedicated for particular data domains. In
Appendix C in the Supplement to this paper we show how
the Variety feature of Big Data is supported by domain-
specific user-defined extractors in two scenarios related to the
processing biological data.
V. D ISCUSSION AND C ONCLUDING R EMARKS
Extending big data analytics with the possibility of fuzzy,
declarative, and scalable querying is very important in the face
of growing volumes of data that are collected by companies,
research centers, economic entities, and other institutions.
This can be especially beneficial for those domains, in which
decisions must be made ad hoc on the basis of large data sets
with uncertainty, collected in Big Data Lake in native formats,
without any pre-processing or schematizing.

1063-6706 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TFUZZ.2018.2812157, IEEE
Transactions on Fuzzy Systems
The solution that we have developed and presented in
the paper satisfies these requirements. It allows performing
full EPS process over the Big Data Lake in which data are
extracted, processed, and transformed, and finally stored in
the selected place and format, suitable for decision makers.
The Fuzzy Search Library for Azure Data Lake makes it
possible to formulate imprecise search criteria, find similar
cases, incorporate domain-specific expert knowledge into the
data analysis, benefit from linguistic terms frequently used
in real life, and change the granularity of data by fuzzy
data grouping. Extensions developed for the U-SQL language
enable declarative processing of the data through querying
rowsets produced during data extraction. Finally, implementa-
tion on the Azure Data Lake located on the cloud guarantees
wide scaling capabilities, which is limited only by a possessed
Azure subscription.
This work complements the collection of existing solutions,
presented in Sect. I-A, by delivering a new functionality for
big data analytics. The novelty of the presented solution lies
mainly in the application of techniques of fuzzy querying
in big data environments, which meets the requirements of
modern information processing and big data challenges. Al-
though, several fuzzy extensions were previously developed
for declarative SQL language and relational database man-
agement systems, they were not dedicated for processing
big data and, likewise some NoSQL solutions, remain on
a strictly defined schema or format. Our approach allows
defining schema on read during the data extraction, and query-
ing produced rowsets in parallel on a highly scalable cloud
infrastructure. On the other hand, those solutions presented in
Sect. I-A, which were dedicated for processing Big Data, were
mainly focused on data clustering and classification with the
use of MapReduce-based procedures executed on Hadoop or
Spark clusters. Both, Hadoop and Spark, are also available as
the HDInsight service in Data Lake Analytics. However, we
decided to provide the scalability of our solution by extending
the functionality of U-SQL code, which enables a declarative
way of data processing and can also be highly parallelized
on many allocation units (compute nodes). The functionality
of our Fuzzy Search Library for Data Lake Analytics also
highly extends the spectrum of methods used for processing
big data. Our solution does not directly extend the set of fuzzy
techniques that can be used, in general, but it extends the
set of operations that can be performed on large, schema-free
(variable) data sets, elevating the value of the outcome through
enrichment provided by fuzzy techniques that are applied.
Presented Fuzzy Search Library for Data Lake Analytics
has many functional advantages over the existing solutions,
which are inherited from the novel technologies that it is based
on. One of the most important advantages is the theoretically
unlimited scalability that allows quick adaption to the growing
volume of data, which is impossible to achieve with relational
databases that provide similar capabilities of fuzzy querying.
The second one is the simplicity of scaling the distributed
fuzzy querying process without the necessity to reconfigure the
whole execution (hardware and software) environment. In con-
trast to Hadoop/Spark-based solutions and NoSQL databases,
which in case of scaling would require reconfiguration of the
1063-6706 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
13
whole cluster (e.g., adding new cluster nodes), the number
of AUs in Azure Data Lake, i.e., the degree of parallelism,
is flexibly controlled by a single runtime parameter. For a U-
SQL developer, it would look like he has got a huge computer
cluster with lots of nodes and he would decide how many
of them should be used when he executes the EPS query
job. Unlike other solutions, the presented one provides the
capability to process not only structured, but also unstructured
and schema-free data stored in the native format in the big data
lake through a unified interface of extractors that represent
the data as a rowset. Our solution also simplifies fuzzy
processing and transformations of data by using declarative U-
SQL SELECT expressions, in contrast to Hadoop/Spark-based
processing, which requires adaptation to particular processing
model, e.g., MapReduce, and implementation of dedicated
processing functions. Fuzzy techniques provided in the form
of functions (UDFs in modules of FSL4DLA) that are invoked
directly from U-SQL queries allow not only to process data in
a declarative way, but also to optimize U-SQL execution plans,
to use less allocation (compute) units, and to decrease the
cost of data processing. Modularization of the FSL4DLA and
separation from the domain knowledge facilitate the extension
of the functionality of the presented solution with the use
of new libraries of user-defined functions. Finally, there is
no need to maintain any (relational/NoSQL) database cluster
or (Hadoop/Spark) computational cluster neither on premises,
nor on the cloud, since Azure Data Lake is maintained by a
cloud provider. Moreover, U-SQL developers pay only for the
computation time related to U-SQL query execution on the
cloud and the number of used allocation units, not just for
having the cluster alive, even if nothing is processed on it.
It is also worth mentioning how our DLA-based solution
solves the problem of Big Data and addresses Vs character-
istics. By massive parallelization of computations related to
data extraction, processing, and storing on the Cloud, we can
largely reduce the time of data processing without decreasing
the workload. This task parallelism addresses the volume
characteristic of Big Data. A part of the Big Data Lake,
which is currently being processed, is divided into smaller
pieces, then each such a piece is processed separately in
parallel, and results are merged together and stored again in
the Data Lake. In terms of the variety, our solution allows for
the structuralization of data stored in Data Lake by building
table-like rowsets over the unstructured data kept in various
formats. The Data Lake does not have to be fully described
both, in terms of attribute names and data types, as the rowset
schema is defined dynamically, on data read. Moreover, for
complex data, like images coming from medical imaging or
genetic data from DNA next-generation sequencing exper-
iments, custom extractors for building rowsets provide the
way for representing data in a structured form. The presented
solution is also highly scalable on the Azure cloud, which
allows accommodating the growth of data in environments
or domains, in which data are generated fast and decisions
must be made quickly on the basis of the performed analyses.
This feature supports the velocity characteristic of Big Data.
We also strongly believe that the presented fuzzy extensions
to U-SQL language will allow taking better decisions on the
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TFUZZ.2018.2812157, IEEE
Transactions on Fuzzy Systems
basis of more valuable information mined from the Data Lake.
The value that is hidden in the big data can be revealed
by finding and associating similar cases with the use of
flexible filtering and fuzzy grouping methods delivered by the
Fuzzy Search Library. These operations, performed as a part
of data processing phase, allow incorporating the knowledge
of experts not only in the analysis of biomedical data, as
presented here, but also in other domains.
Among limitations of the presented solution, we have to
mention the necessity to operate on the specific cloud platform
and to adapt to the specificity of the Azure Data Lake
environment. FSL4DLA is fully a cloud-dedicated solution.
There is no possibility to use the FSL4DLA on large scale on
the hardware kept on premises, although, local deployments
are also possible for testing purposes. Moreover, U-SQL
developers have to pay for computations related to U-SQL
query execution. However, on the other hand, they do not
have to cover the costs of maintenance of the whole hardware
infrastructure kept on premises.
Our intention was to develop the Fuzzy Search Library for
Big Data Lake as a universal tool that delivers methods for
scalable, fuzzy data processing on Azure cloud for various
domains of analyzed data. In the presented solution, experts’
knowledge is modularized in domain-specific libraries that
can be provided separately for various domains by various
experts. These domain-specific libraries are independent from
the Fuzzy Search Library. However, the Fuzzy Search Library
provides common methods for translating the experts’ knowl-
edge into efficient data analysis of Big Data. This comes as
a cost of fitting in to our implementation model for domain-
specific libraries, but we believe it can be beneficial for many
domains and for the future reuse of once created libraries of
experts’ knowledge. With this paper, we hope to attract experts
and specialists, who use fuzzy techniques in processing data
in their areas of interests, to create domain-specific libraries
and make them freely available for the community of users.
ACKNOWLEDGMENT
This work was supported by Microsoft Research within
Microsoft Azure for Research Award grant, and Statutory Re-
search funds of Institute of Informatics, Silesian University of
Technology, Gliwice, Poland (grant No BK/213/RAU2/2018).
Fuzzy Search Library for Data Lake Analytics (FSL4DLA)
is available at (http://www.zti.aei.polsl.pl/w3/dmrozek/science/
fsl4dla.htm). Users must use their own Microsoft Azure sub-
scription in order to run the FSL4DLA on the cloud.
R EFERENCES
[1] G. B. Davis and K. M. Carley, “Clearing the fog: Fuzzy, overlapping
groups for social networks,” Social Networks, vol. 30, no. 3, pp. 201 –
212, 2008.
[2] C. De Maio, G. Fenza, V. Loia, and S. Senatore, “Hierarchical web
resources retrieval by exploiting fuzzy formal concept analysis,” Infor-
mation Processing & Management, vol. 48, no. 3, pp. 399 – 418, 2012.
[3] Z. Wang, L. Tu, Z. Guo, L. T. Yang, and B. Huang, “Analysis of
user behaviors by mining large network data sets,” Future Generation
Computer Systems, vol. 37, pp. 429 – 437, 2014.
[4] S. Kundu and S. K. Pal, “FGSN: Fuzzy granular social networks model
and applications,” Information Sciences, vol. 314, pp. 100–117, 2015.
1063-6706 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
14
[5] S. Kundu and S. Pal, “Fuzzy-rough community in social networks,”
Pattern Recognition Letters, vol. 67, Part 2, pp. 145 – 152, 2015, granular
Mining and Knowledge Discovery.
[6] X. Wei, X. Luo, Q. Li, J. Zhang, and Z. Xu, “Online comment-based
hotel quality automatic assessment using improved fuzzy comprehensive
evaluation and fuzzy cognitive map,” IEEE Transactions on Fuzzy
Systems, vol. 23, no. 1, pp. 72–84, Feb 2015.
[7] C. De Maio, G. Fenza, V. Loia, and M. Parente, “Time aware knowledge
extraction for microblog summarization on Twitter,” Information Fusion,
vol. 28, pp. 60 – 74, 2016.
[8] G. Ghosh, S. Banerjee, and N. Y. Yen, “State transition in com-
munication under social network: An analysis using fuzzy logic and
density based clustering towards Big Data paradigm,” Future Generation
Computer Systems, vol. 65, pp. 207–220, 2016.
[9] B. K. Tripathy and D. Mittal, “Hadoop based uncertain possibilistic ker-
nelized c-means algorithms for image segmentation and a comparative
analysis,” Appl. Soft Comput., vol. 46, pp. 886–923, 2016.
[10] H. Chang, N. Mishra, and C. Lin, “IoT Big-Data centred knowledge
granule analytic and cluster framework for BI applications: a case base
analysis,” Plos One, vol. 10, pp. 1–23, 2015.
[11] H. Lu, Z. Sun, and W. Qu, “Big Data-driven based real-time traffic flow
state identification and prediction,” Discrete Dynamics in Nature and
Society, vol. 2015, pp. 1–11, 2015.
[12] K. Guo, R. Zhang, and L. Kuang, “TMR: Towards an efficient semantic-
based heterogeneous transportation media Big Data retrieval,” Neuro-
computing, vol. 181, pp. 122–131, 2016.
[13] C. Wang, X. Li, X. Zhou, A. Wang, and N. Nedjah, “Soft computing
in Big Data intelligent transportation systems,” Applied Soft Computing,
vol. 38, pp. 1099–1108, 2016.
[14] H. Lu, Z. Sun, W. Qu, and L. Wang, “Real-time corrected traffic
correlation model for traffic flow forecasting,” Math. Probl. Eng., vol.
2015, pp. 1–7, 2015.
[15] D. Mrozek, P. Kasprowski, B. Małysiak-Mrozek, and S. Kozielski, “Life
sciences data analysis,” Information Sciences, vol. 384, pp. 86–89, 2017.
[16] C. Bai, D. Dhavale, and J. Sarkis, “Complex investment decisions using
rough set and fuzzy c-means: An example of investment in green supply
chains,” European Journal of Operational Research, vol. 248, no. 2, pp.
507 – 521, 2016.
[17] L. Meng, A. Tan, and D. Wunsch, “Adaptive scaling of cluster bound-
aries for large-scale social media data clustering,” IEEE Trans. Neur.
Net. Lear., vol. 27, no. 12, pp. 2656–2669, 2015.
[18] Y. Zhong, L. Zhang, S. Xing, F. Li, and B. Wan, “The Big Data
processing algorithm for water environment monitoring of the three
gorges reservoir area,” Abstract and Applied Analysis, vol. 2014, 2014.
[19] National Research Council, Frontiers in Massive Data Analysis. Wash-
ington, D.C.: National Academy Press, 2013.
[20] V. Kreinovich, H. T. Nguyen, and S. Sriboonchitta, “Need for data
processing naturally leads to fuzzy logic (and neural networks): Fuzzy
beyond experts and beyond probabilities,” Departmental Technical Re-
ports (CS), p. Paper 870, 2014.
[21] X. Wang and Y. He, “Learning from uncertainty for Big Data: Future an-
alytical challenges and strategies,” IEEE Systems, Man, and Cybernetics
Magazine, vol. 2, no. 2, pp. 26–31, 2016.
[22] L. Zadeh, “Fuzzy sets,” Information and Control, vol. 8, no. 3, pp. 338–
353, 1965.
[23] T. White, Hadoop – The Definitive Guide: Storage and Analysis at
Internet Scale, 3rd ed. Ireland: OReilly, 2012.
[24] Official web page of Apache Spark, “Lightning-fast cluster computing,”
http://spark.apache.org/ (accessed on April 14, 2017).
[25] P. Mell and T. Grance, “The NIST definition of Cloud Computing.
Special Publication 800-145 /accessed on March 24, 2017/,” 2011, http://
nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-145.pdf.
[26] P. Bosc and O. Pivert, SQLf Query Functionality on Top of a Regular
Relational Database Management System. Heidelberg: Physica-Verlag
HD, 2000, pp. 171–190.
[27] J. Kacprzyk and S. Zadrożny, Data Mining via Fuzzy Querying over the
Internet. Heidelberg: Physica-Verlag HD, 2000, pp. 211–233.
[28] G. Bordogna and G. Psaila, “Customizable flexible querying in classical
relational databases,” in Handbook of Research on Fuzzy Information
Processing in Databases, J. Galindo, Ed. IGI Global, 2008, pp. 191–
217.
[29] M. Hudec, “An approach to fuzzy database querying, analysis and
realisation,” Computer Science and Information Systems, no. 12, pp.
127–140, 2009. [Online]. Available: http://eudml.org/doc/253504
[30] B. Małysiak-Mrozek, S. Kozielski, and D. Mrozek, “Modern software
tools for researching and teaching fuzzy logic incorporated into database
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TFUZZ.2018.2812157, IEEE
Transactions on Fuzzy Systems
systems,” in Proceedings of the iNEER International Conference on
Engineering Education, Gliwice, Poland. iNEER, July 2010, pp. 1–8.
[31] B. Małysiak, D. Mrozek, and S. Kozielski, “Processing fuzzy SQL
queries with flat, context-dependent and multidimensional membership
functions,” in IASTED International Conference on Computational In-
telligence, Calgary, Alberta, Canada, July 4-6, 2005, M. H. Hamza, Ed.
IASTED/ACTA Press, 2005, pp. 36–41.
[32] B. Małysiak-Mrozek, D. Mrozek, and S. Kozielski, Data Grouping
Process in Extended SQL Language Containing Fuzzy Elements. Berlin,
Heidelberg: Springer Berlin Heidelberg, 2009, pp. 247–256.
[33] L. Portinale and S. Montani, “A fuzzy logic approach to case matching
and retrieval suitable to SQL implementation,” in Proc. of the 20th IEEE
Inter. Conf. on Tools with Artificial Intelligence. Washington, DC, USA:
IEEE Computer Society, 2008, pp. 241–245.
[34] K. Myszkorowski, Inference Rules for Fuzzy Functional Dependencies
in Possibilistic Databases. Cham: Springer International Publishing,
2016, pp. 181–191.
[35] G. Appelgren Lara, M. Delgado, and N. Marı́n, Fuzzy Multidimen-
sional Modelling for Flexible Querying of Learning Object Repositories.
Berlin, Heidelberg: Springer Berlin Heidelberg, 2013, pp. 112–123.
[36] B. Małysiak-Mrozek, D. Mrozek, and S. Kozielski, “Processing of crisp
and fuzzy measures in the Fuzzy Data Warehouse for global natural
resources,” in Trends in Applied Intelligent Systems, ser. Lect Notes
Comput Sci, N. Garca-Pedrajas and et al., Eds. Springer Berlin
Heidelberg, 2010, vol. 6098, pp. 616–625.
[37] A. Castelltort and A. Laurent, Fuzzy Queries over NoSQL Graph
Databases: Perspectives for Extending the Cypher Language. Cham:
Springer International Publishing, 2014, pp. 384–395.
[38] A. Castelltort and A. Laurent, “Extracting fuzzy summaries from NoSQL
graph databases,” in Flexible Query Answering Systems, ser. AISC,
T. Andreasen et al., Ed. Springer, 2016, vol. 400, pp. 189–200.
[39] A. Castelltort and A. Laurent, “Exploiting NoSQL graph databases and
in memory architectures for extracting graph structural data summaries,”
International Journal of Uncertainty, Fuzziness and Knowledge-Based
Systems, vol. 25, no. 01, pp. 81–109, 2017.
[40] M. S. Hidri, I. BenAli-Sougui, and A. Grissa-Touzi, “No-FSQL: A
graph-based fuzzy NoSQL querying model,” Int J Fuzzy Syst Appl,
vol. 5, no. 2, pp. 54–63, 2016.
[41] A. B. Kacem and A. G. Touzi, “Towards fuzzy querying of NoSQL
document-oriented databases,” in The Seventh Int. Conf. on Advances
in Databases, Knowledge, and Data Applications, DBKDA 2015, Rome,
Italy, 2015, pp. 153–158.
[42] M. E. Bakry, S. Safwat, and O. Hegazy, “Big Data classification
using Fuzzy K-Nearest Neighbor,” International Journal of Computer
Applications, vol. 132, no. 10, pp. 8–13, 2015.
[43] O. Hegazy, S. Safwat, and M. E. Bakry, “A MapReduce fuzzy techniques
of Big Data classification,” in 2016 SAI Computing Conference (SAI),
July 2016, pp. 118–128.
[44] P. Ducange, F. Marcelloni, and A. Segatori, “A MapReduce-based
fuzzy associative classifier for big data,” in 2015 IEEE International
Conference on Fuzzy Systems (FUZZ-IEEE), Aug 2015, pp. 1–8.
[45] S. del Rı́o, V. López, J. M. Benı́tez, and F. Herrera, “A MapReduce
approach to address Big Data classification problems based on the
fusion of linguistic fuzzy rules,” International Journal of Computational
Intelligence Systems, vol. 8, no. 3, pp. 422–437, 2015.
[46] V. López, S. del Rı́o, J. M. Benı́tez, and F. Herrera, “Cost-sensitive
linguistic fuzzy rule based classification systems under the MapReduce
framework for imbalanced Big Data,” Fuzzy Sets and Systems, vol. 258,
pp. 5–38, 2015.
[47] A. Segatori, F. Marcelloni, and W. Pedrycz, “On distributed fuzzy
decision trees for Big Data,” IEEE Transactions on Fuzzy Systems,
vol. PP, no. 99, pp. 1–1, 2017.
[48] I. Timón, J. Soto, H. Pérez-Sánchez, and J. M. Cecilia, “Parallel
implementation of fuzzy minimals clustering algorithm,” Expert Systems
with Applications, vol. 48, pp. 35 – 41, 2016.
[49] Y. Li, R. Wang, and S. Shiu, “Interval extreme learning machine for
Big Data based on uncertainty reduction,” J. Intell. Fuzzy Syst., vol. 28,
no. 5, pp. 2391–2403, 2015.
[50] S. A. Ludwig, “MapReduce-based fuzzy c-means clustering algo-
rithm: implementation and scalability,” International Journal of Machine
Learning and Cybernetics, vol. 6, no. 6, pp. 923–934, 2015.
[51] T. C. Havens, J. C. Bezdek, C. Leckie, L. O. Hall, and M. Palaniswami,
“Fuzzy c-means algorithms for very large data,” IEEE Transactions on
Fuzzy Systems, vol. 20, no. 6, pp. 1130–1146, Dec 2012.
[52] P. Su, C. Shang, and Q. Shen, “A hierarchical fuzzy cluster ensemble ap-
proach and its application to Big Data clustering,” Journal of Intelligent
& Fuzzy Systems, vol. 28, no. 6, pp. 2409–2421, 2015.
1063-6706 (c) 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
15
[53] S. Prabha and P. Kola Sujatha, “Reduction of Big Data sets using fuzzy
clustering,” IJARCET, vol. 3, no. 6, pp. 2235–2238, 2014.
[54] N. Bharill, A. Tiwari, and A. Malviya, “Fuzzy based clustering algo-
rithms to handle Big Data with implementation on Apache Spark,” in
2016 IEEE Second International Conference on Big Data Computing
Service and Applications (BigDataService), March 2016, pp. 95–104.
[55] M. Prasad, Y. Lin, C. Lin, M. Er, and O. Prasad, “A new data-driven
neural fuzzy system with collaborative fuzzy clustering mechanism,”
Neurocomputing, vol. 167, pp. 558 – 568, 2015.
[56] G. Peters and R. Weber, “DCC: a framework for dynamic granular
clustering,” Granular Computing, vol. 1, no. 1, pp. 1–11, 2016.
[Online]. Available: http://dx.doi.org/10.1007/s41066-015-0012-z
[57] B. Zhang, S. Qin, W. Wang, D. Wang, and L. Xue, “Data stream
clustering based on Fuzzy C-Mean algorithm and entropy theory,” Signal
Process., vol. 126, no. C, pp. 111–116, Sep. 2016.
[58] V. Chitraa and A. Thanamani, “Web log data analysis by enhanced Fuzzy
C Means clustering,” International Journal on Computational Sciences
& Applications (IJCSA), vol. 4, no. 2, pp. 81–95, 2014.
[59] B. K. Tripathy, D. Mittal, and D. P. Hudedagaddi, Hadoop with Intuition-
istic Fuzzy C-Means for Clustering in Big Data. Singapore: Springer,
2016, pp. 599–610.
[60] G. A. Papakostas, E. I. Papageorgiou, and V. G. Kaburlasos, “Linguistic
Fuzzy Cognitive Map (lfcm) for pattern recognition,” in 2015 IEEE Int.
Conf. on Fuzzy Systems (FUZZ-IEEE), 2015, pp. 1–7.
[61] Y. Choi, H. Lee, and Z. Irani, “Big data-driven fuzzy cognitive map
for prioritising it service procurement in the public sector,” Annals of
Operations Research, 2016.
[62] J. Liu, Y. Chi, and C. Zhu, “A dynamic multiagent genetic algorithm for
gene regulatory network reconstruction based on fuzzy cognitive maps,”
IEEE Transactions on Fuzzy Systems, vol. 24, no. 2, pp. 419–431, 2016.
[63] S. D’Onofrio, M. Wehrle, E. Portmann, and T. Myrach, “Striving for
semantic convergence with fuzzy cognitive maps and graph databases,”
in 2017 IEEE Int. Conf. on Fuzzy Systems (FUZZ-IEEE), 2017, pp. 1–6.
[64] S. Ramachandramurthy, S. Subramaniam, and C. Ramasamy, “Distilling
Big Data: refining quality information in the era of yottabytes,” The
Scientific World Journal, vol. 2015, pp. 1–9, 2015.
[65] Y. Cai, Q. Li, H. Xie, and H. Min, “Exploring personalized searches
using tag-based user profiles and resource profiles in folksonomy,”
Neural Networks, vol. 58, pp. 98 – 110, 2014.
[66] Z. Liu, J. Li, J. Li, C. Jia, J. Yang, and K. Yuan, “SQL-based fuzzy
query mechanism over encrypted database,” International Journal of
Data Warehousing and Mining, vol. 10, no. 4, pp. 71–87, 2014.
[67] D. D. Wang, W. Zhou, and H. Yan, “Mining of protein–protein interfacial
residues from massive protein sequential and spatial data,” Fuzzy Sets
and Systems, vol. 258, pp. 101–116, 2015.
[68] B. Novikov, N. Vassilieva, and A. Yarygina, “Querying Big Data,” in
Proceedings of the 13th International Conference on Computer Systems
and Technologies. ACM, 2012, pp. 1–10.
[69] K. Sundharakumar, S. Dhivya, S. Mohanavalli, and R. V. Chander,
“Cloud based fuzzy healthcare system,” Procedia Computer Science,
vol. 50, pp. 143 – 148, 2015.
[70] A. T. Azar and A. E. Hassanien, “Dimensionality reduction of medical
Big Data using neural-fuzzy classifier,” Soft Computing, vol. 19, no. 4,
pp. 1115–1127, 2015.
[71] O. Behadada, M. Trovati, M. A. Chikh, and N. Bessis, “Big Data-
based extraction of fuzzy partition rules for heart arrhythmia detection:
a semi-automated approach,” Concurrency and Computation: Practice
and Experience, vol. 28, no. 2, pp. 360–373, 2016.
[72] J. McEntyre and D. Lipman, “PubMed: bridging the information gap,”
CMAJ, vol. 164, no. 9, pp. 1317–1319, 2001.
[73] H. Wang, Z. Xu, and W. Pedrycz, “An overview on the roles of
fuzzy set techniques in Big Data processing: Trends, challenges and
opportunities,” Knowledge-Based Systems, vol. 118, pp. 15 – 30, 2017.
[74] M. Detyniecki, R. R. Yager, and B. Bouchon-Meunier, “Specifying t-
norms based on the value of t(1/2,1/2),” Mathware & Soft Computing,
vol. 7, no. 1, pp. 77 – 35 387, 2000.
[75] Microsoft Azure, “Overview of Microsoft Azure Data Lake Analytics,”
https://docs.microsoft.com/en-us/azure/data-lake-analytics/
data-lake-analytics-overview (accessed on April 14, 2017).
[76] R. Detrano, A. Janosi, W. Steinbrunn, M. Pfisterer, J. Schmid, S. Sandhu,
K. Guppy, S. Lee, and V. Froelicher, “International application of a
new probability algorithm for the diagnosis of coronary artery disease,”
American Journal of Cardiology, vol. 64, pp. 304–310, 1989.