RDM: Rapid Deduplication for Mobile Cloud Storage
Ryan N. S. Widodo
Department of Ubiquitous IT
Dongseo University, 617-716 Busan, South Korea
ryannswidodo@gmail.com
ABSTRACT
Cloud storages are used to expand the storage on mobile devices.
It can be accessed through an internet connection. On mobile
devices, network data can be expensive in terms of tariff and
power consumption. Deduplication can be used to reduce the
amount of data to be transferred. This paper proposes a fast
deduplication system for mobile devices named rapid
deduplication for mobile cloud storage (RDM). RDM is based on
smart deduplication for mobile cloud storage (SDM) and rapid
asymmetric extremum (RAM). RDM used RAM as the chunking
algorithm instead of Rabin which is used in SDM. Our
experimental results show RDM is at least 39.5% to 44.7% faster
at the cost of 4.2% to 9.1% higher size after deduplication.
CCS Concepts
• Information system ➝ Mobile information processing
systems • Theory of computation➝Data compression
Keywords
Cloud storage; mobile devices; content-defined chunking; multiple
deduplication methods; network access
1. INTRODUCTION
Mobile data traffic grows rapidly year by year. It is predicted that
mobile data traffic will grow by tenfold in the next five years [14].
Higher data traffic will require higher bandwidth communication
which can be costly. One of the main contributors to the growth is
cloud storage services. Cloud storage is accessible from anywhere
and anytime as long as it is accessible. Cloud storage popularity
on mobile devices grows fast because of the limited storage
capacity on mobile devices. To further improve cloud storage
service, multiple cloud storages can also be combined [19].
However, cloud storages rely on network connection to be
accessed from mobile devices and network traffic can be costly in
terms of both energy consumption and data tariff. Data
compression technique is often used to reduce the amount of data.
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. Copyrights
for components of this work owned by others than ACM must be
honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior
specific permission and/or a fee. Request permissions from
Permissions@acm.org.
ICCMS '17, January 20 - 23, 2017, Canberra, Australia
Copyright is held by the owner/author(s). Publication rights licensed to
ACM.
ACM 978-1-4503-4816-4/17/01 …$15.00
DOI: http://dx.doi.org/10.1145/3036331.3036357
14
Hyotaek Lim*
Division of Information Engineering
Dongseo University, 617-716 Busan, South Korea
htlim@dongseo.ac.kr
Data deduplication is a technique to save storage by only saving
one copy of redundant data. In addition of saving storage,
deduplication also reduces data transfers for uploading files when
it is done at the client side. Deduplication is commonly applied on
storage servers. However, applying deduplication on mobile
devices can also reduce the amount of data to be stored [16].
Deduplication is useful for saving storage and bandwidth and can
be a solution for mobile devices which utilize cloud storage to
expand its storage capability.
In our previous work [16], we proposed smart deduplication for
mobile cloud storage (SDM). It utilizes multiple deduplication
methods, file-level and block-level to reduce the process time.
The deduplication method is chosen by the learning system based
on the data collected during the deduplication process. The main
drawback of SDM is the long chunking process for block-level
deduplication. Rabin rolling hash [13], the hashing algorithm used
as the chunking algorithm for the block-level, took significant
amount of time in the deduplication process.
Hash based chunking algorithms are computational heavy,
because it needs to calculate a new hash each time the sliding
window moves. Hashless chunking algorithms solve this by
finding the cut-point not based on the hash value of the data
stream. Local maximum chunking (LMC) [2] uses two
symmetrical sliding window and treats byte as a number and
compares it to find the cut point. It does a lot of comparisons
which makes the process long. Asymmetric Extremum [20],
another hashless chunking algorithm, uses two non-sliding
window to find the cut-point. Because the windows are not
moving, AE does fewer comparisons compared to LMC. Rapid
asymmetric maximum [15] (RAM) is based on AE and uses the
same windows with different configuration on the windows
position. The different configuration allows RAM to do fewer
comparisons and have higher chunking throughput. In this paper,
we propose rapid deduplication for mobile cloud storage (RDM)
which is based on SDM and RAM. RAM is used as the block-
level chunking algorithm instead of Rabin rolling hash. Our
implementation results show at least 39.5% to 44.7% reduction on
deduplication process time and a drawback of 4.2% to 9.1% more
size after deduplication.
The rest of this paper is organized as follows. Section 2 contains
the related work on chunking algorithm and mobile deduplication.
Section 3 discusses the design of RDM. In Section 4, we compare
RDM with SDM and we discussed the results in Section 5. Lastly,
we conclude our paper in Section 6.
2. RELATED WORK
This section discusses the background of content-defined
chunking (CDC) algorithm, related work, the limitations, and our
motivation.
Deduplication eliminates duplicate data by comparing the
fingerprint of the data. The fingerprint can be taken by using
mathematical hash function. Deduplication application is not
limited to storage data [18]. It can also be applied to virtual disk
images [7, 17], memory [1, 8, 9], and network traffic [4, 6, 12].
Based on the deduplication method, deduplication can be
categorized into file-level and block-level.
The simplest deduplication method is file-level deduplication, or
can also be called whole file chunking. On file-level, the
deduplication process is fast because there is no chunking process
involved. File-level deduplication is prone to modification,
because a small modification will result in a completely different
hash.
Block-level deduplication solves this by splitting the file into
chunks. When there is a byte change, only a few chunks are
affected. Block-level deduplication can be done with fix-sized
blocks or variable-sized blocks. Deduplication with fix-sized
blocks is faster than deduplication with variable-sized blocks
because it does not require additional processing. However, fix-
sized blocks have a weakness. Shifting the file by adding a byte or
bytes at the header of the file will result in a completely different
fingerprint for all blocks. The variable sized block or CDC is
more resistant to byte shifting because it uses the fixed chunking
offset. A CDC algorithm can be used to determine the fixed
chunking offset.
CDC algorithms can be categorized into two, hash based CDC
algorithms and hashless CDC algorithms. Hash based CDC
algorithms uses hash function to find the cut-point. The most
common used hash function is Rabin rolling hash [13]. Rabin
rolling hash uses a sliding window and calculates the hash of the
window for each time the window slid. A cut-point is found when
the hash matches a predefined pattern. Hashless CDC algorithms
use different approach to find the cut-point. LMC [2], AE [20],
and RAM [15] treat each byte as a number. When the cut-point
condition is fulfilled, a cut-point is found. Commonly, the
condition is the cut-point byte must be bigger than every byte in
the window.
According to Meister et al. [11] some deduplication methods and
chunking algorithms are more suitable for some file types. To take
the advantage of this fact, some deduplication systems are
proposed to utilize multiple deduplication methods. Marques et al.
[10] and Haustein et al. [5] proposed deduplication systems with
multiple deduplication methods. The deduplication method for
each file type is predefined. The deduplication method of a file
type is rotated in Haustein et al. system if the target deduplication
ratio is not reached. Both of the systems have a problem with new
file type that is not in the configuration and the deduplication
method rotation used in Haustein et al. systems may cause hash
incompatibility because of changing the deduplication methods.
We addressed this issues with SDM [16]. SDM uses a learning
system to decide the deduplication method for each file type. The
learning system also solved the hash incompatibility problem by
using all deduplication method in the learning phase instead of
rotating the deduplication methods.
The main issue with SDM is the low block-level deduplication
throughput. Most of the deduplication process time is spent on the
15
chunking process. In this paper, we proposed rapid deduplication
for mobile cloud storage (RDM). RAM, a hashless chunking
algorithm is used to lower the computational overhead for block-
level deduplication.
3. RAPID DEDUPLICATION FOR
MOBILE CLOUD STORAGE
Rapid deduplication for mobile cloud storage (RDM) is based on
SDM [16] and RAM [15]. RAM is used as block-level chunking
algorithm in place of Rabin. The purpose of using RAM instead of
Rabin is to improve the deduplication process time. In this section,
we discussed the design of SDM and RAM.
3.1 SDM design
SDM is a deduplication system designed for mobile devices
proposed in our previous work [16]. SDM utilizes multiple
deduplication methods to optimize the process time and the
duplicates detection. Meister et al. shows that some deduplication
methods are better than the other deduplication methods for
different file type. SDM used this idea to reduce the processing
time while removing similar amount of duplicates.
Figure 1 illustrated the design of SDM. SDM uses two
deduplication method, file-level and block-level. In [16], we used
Rabin for the block-level deduplication chunking algorithm. The
learning system manages the deduplication method assigned for
each file type. After the file is processed by the chunking
algorithm, the chunk is processed in the chunk management to
find duplicate data. The chunk management used by SDM are
composed of two level, bloom filter [3] and hash table. The bloom
filter performs as the fast lookup because there is no false
negative. To cover the false positive part of the bloom filter, hash
table is used.
Cloud Storage
Chunk management
Learning System
Mobile device
File level Block level
deduplication deduplication
Local Storage
Figure 1. Overview of the SDM.
The learning system of SDM: The learning system of SDM
chooses the best deduplication method for each file type. The
decision on the deduplication method is based on a calculation
made from the block-level chunking throughput, the amount of
duplicates detected, the file size, and the upload speed. The
learning system calculates the amount of duplicates eliminated by
using block-level deduplication and the amount of time saved for
not uploading the duplicate. When the amount of time saved for
uploading less data is bigger than the amount of time used for the
deduplication process (∑𝐹𝐹/𝑆𝑆𝑏𝑏 < ∑𝑓𝑓/𝑈𝑈𝑠𝑠 ), the learning system
will assign block-level deduplication for the file type. Where ∑F
is total file size (kB), ∑f is total duplicate length for the file type
(kB), Sb is the block-level throughput (kBps), and Us is the upload
speed (kBps). The decision is made after processing a few files for
a file type. Because of space constrains, the details can be seen in RAM is faster than AE and Rabin in our experiment [15] because
[16]. of the fewer comparisons [15] at the cost of higher chunk variance
which leads to lower number of duplicates detected. RAM is more
3.2 RAM algorithm design preferable for deduplication process on low computation devices
such as mobile devices and IoT because of the low computation
RAM is a hashless content-defined chunking (CDC) algorithm
overhead. Our analysis shows that RAM has lower probability or
[15]. It treats each byte as a number with a value which allows the
long chunks than Rabin. This makes the chunk distribution for
algorithm to find a cut-point without high computational overhead.
RAM much closer to the mean which may allow lower chunk
RAM is similar to AE [20] because it also uses two windows, fix-
variance. The probability of long chunks is presented in Table 1
sized and variable-sized windows. The windows configuration can
Because of space limitations, our complete analysis on RAM can
be seen in Figure 2 (a) for RAM and (b) for AE. The placement of
be seen in [15].
the windows is different from AE. In RAM, the fix-sized window
is located at the beginning of the chunk and followed by the Table 1. The probability of long chunks for RAM and Rabin.
variable-sized window and the maximum-valued byte. As we can
observe in Figure 2, the maximum-valued byte is included in the 𝑀𝑀 AE RAM Rabin
chunk at the end of the chunk. The windows used in RAM and AE 1 1
𝑒𝑒 −𝑀𝑀
are not sliding windows. The condition for a byte to become the [(𝑒𝑒 − 1)𝑀𝑀]! [2𝑀𝑀]!
cut-point is when the byte‘s value is larger than all of the bytes in 2 0.166667 0.041667 0.135335
the fixed window. The algorithm searches for a byte that fulfills
3 0.008333 0.001389 0.049787
the condition. The pseudo code for the algorithm can be seen in
Algorithm 1. 4 0.001389 2.48E-05 0.018316
5 2.48E-05 2.76E-07 0.006738
Algorithm 1: Algorithm for RAM chunking
Input: input string, Str, length of the string, L; 6 2.76E-07 2.09E-09 0.002479
Output: cut-point I; 7 2.51E-08 1.15E-11 0.000912
Predefined values: window size, w;
function RAMChunkning (Str, L) 8 1.61E-10 4.78E-14 0.000335
i=1;
while (i<L) 4. PERFORMANCE EVALUATION
if Str[i].value>=max.value then
if i>w then This section discusses the performance of RDM in an
return i experimental environment. RDM is installed as an Android
end if application and ran on an Android device, Samsung Galaxy S3.
The Samsung Galaxy S3 uses 1.4 GHz quad core CPU, 1.786 GB
max.value=Str[i].value of RAM, Android version 4.4.4, and build number
max.position = i KTU84P.E210SKSUKNK3.
end if The systems tested in this experiment are as follows:
i=i+1
end while 1. File-level deduplication system (FDS)
end function 2. Rabin block-level with deduplication system (RabinBDS)
3. Rapid Asymmetric Extremum block-level with deduplication
RAM is a content-defined chunking algorithm because it can re- system (RAMBDS)
adjust the chunk size when there is a byte inserted in the chunk. 4. Smart Deduplication for Mobile with Rabin (SDM)
When a byte is inserted in any position in the chunk and the size 5. RDM (RDM)
of the byte is less than the cut-point byte, only one chunk is
affected. If the size of the byte is larger than the cut-point byte As the names indicate, File-level deduplication system (FDS) is a
size and is inserted in the variable window, then the number deduplication system that performs deduplication at file-level and
affected of chunk will increase. If a byte is inserted in the fix- Block-level deduplication system (BDS) is a deduplication system
sized window and the value is larger than the previous maximum that performs deduplication on block-level. There are two variants
in the fixed window, the number of chunks affected by the byte of BDS, Rabin BDS (RabinBDS) and RAM BDS (RAMBDS).
insertion will stop increasing until the next cut-point byte is larger. RabinBDS only uses Rabin for the chunking algorithm while
RAMBDS only uses RAM. SDM uses both file-level and block-
Maximum valued byte Cut-point level deduplication to reduce deduplication process time. SDM
Fixed Variable has two variants, SDM with Rabin (SDM) and SDM with RAM
Data (RDM). The purpose of using these deduplication systems for
RAM comparisons is to show the performance of RAM when used with
(a) low computation devices compared to Rabin based deduplication
Extreme value Cutpoint system.
Variable Fixed
Data Datasets: In our implementation, we used three datasets to show
AE
the performances of each system in different conditions. The
(b) content of each dataset is shown in Table 2. Dataset 1 contains
audio files which are unique; dataset 2 consists of Android
Figure 2. (a) RAM chunk structure, (b) AE chunk structure. application data with OBB extension; dataset 3 includes dataset 1,
dataset 2, and additional JPG and PDF files. Each dataset

16
represent a different condition. Unique files is represented by

54.37
60.00

48.37
dataset 1, redundant files is represented as dataset 2, and standard FDM

Processing time (s)

backup which may consists of unique files and redundant files is 50.00

34.09
represented by dataset 3.

29.28
40.00 RabinBDS

22.23
20.99
Table 2. The datasets used in the implementation.
30.00

18.22

16.36
RAMBDS

13.43

12.29
11.96
Datasets Content File Size
20.00

8.70
8.30
1 Audio (MP3 files) 560 MB (128 files)

7.56
7.41
2 Android application data (OBB 524 MB (20 files) SDM
10.00
files)
3 Mixed (JPG, PDF, MP3, OBB) 1.53 GB (226 files) 0.00 RDM
Dataset 1 Dataset 2 Dataset 3
Datasets

Evaluation Methodology: We used five deduplication systems to Figure 3. Time consumed for deduplication process (lower is
evaluate the performance of RAM on a mobile device. Processing better).
time and size after deduplication are the performance metrics in
1,800.00

1,532.37
1,428.38
the evaluation. The processing time represents the deduplication
FDM

Size after deduplication (MB)

1,164.08
1,141.61
1,600.00

1,100.21
throughput and the time taken by a system to complete a

1,077.44
deduplication process. The size after deduplication shows the final 1,400.00
size of the datasets after a deduplication process is applied and the RabinBDS
1,200.00
deduplication accuracy. Lower size after deduplication means 1,000.00 RAMBDS

560.90
526.45

525.63
525.55

524.29
508.23
503.09
higher deduplication accuracy because more duplicates are

482.34
800.00
eliminated. This evaluation methodology is also used in our

234.94
234.94
600.00 SDM

previous work [16] and other papers [2, 20].
Configurations: The average block' size needs to be at equal size
for all algorithms for a valid comparison. Rabin’s average chunk
size can be configured by changing the average size, minimum
size, maximum size, and the window size. While for RAM, we
can change the window size to adjust the average chunk size.
However, configuring Rabin’s average chunk size is harder than
configuring RAM because of the lower probability of long chunks
of RAM. In our implementation, we ran a test with Rabin and
configured RAM to match the average chunk size of Rabin. The
configuration used for Rabin is as follows: 16 KB, 4 KB, 32 KB,
and 48 bytes for average, minimum, maximum chunk size, and
the window size respectively. The window size used for RAM is
18 KB. FDS and BDS are SDM with file-level or block-level
deduplication disabled.
Evaluation results: The processing time and the size after
deduplication results are displayed in Figure 3 and 4. As can be
seen in Figure 3, RAM consumes 26.3% (dataset 1) to 43%
(dataset 2) less time compared to Rabin when used in block-level
deduplication system (BDS) on mobile devices. When used in
SDM, RAM showed small improvement in dataset 1 compared to
RAMBDS because when processing dataset 1 SDM utilized file-
level deduplication. On dataset 2 and dataset 3, RAM improved
the processing time by 44.7% and 39.5% respectively.
Although RAM is faster than Rabin, Rabin can eliminate more
duplicates than RAM in dataset 2 and dataset 3 as can be seen in
Figure 4. Rabin eliminated 9.1% or 51.07 MB and 4.2% or 64.18
MB more duplicate in dataset 2 and 3 respectively. In dataset 1,
RAM eliminates 5.14 MB more duplicates data than Rabin. The
results are similar when RAM is utilized in SDM. To ease the
comparison between deduplication systems, we included ND
which stands for no deduplication in Figure 4. ND size is the full
size of the dataset.
183.87
183.87
400.00
200.00 RDM
0.00
ND
Dataset 1 Dataset 2 Dataset 3
Datasets
Figure 4. Size of the dataset after deduplication (lower is
better).
5. DISCUSSION
On mobile devices, RAM performs 26.3% to 44.7% better than
Rabin in terms of processing time. However the improvement is
not as big as when RAM is applied on desktop PC. In our other
work [15], RAM is 500% faster than Rabin when the algorithm is
ran on an Intel i7 6700 and an SSD [15] where the deduplication
throughput is similar to the SSD read speed. This shows the
storage read speed on our mobile device bottlenecks the
deduplication process. With better storage read speed, RAM
shows higher improvement over Rabin.
In some datasets, Rabin eliminates more duplicate data than RAM.
This is similar our evaluation done in [15]. On mobile phone
devices, there are some cases where RAM is more preferable to
Rabin. Those cases are when the upload speed is high enough for
uploading the extra bytes to be negligible, local deduplication
where there is no upload involved and processing time is
important for battery, and applications where processor consumes
more power than the network adapter. To improve the duplicates
elimination, the average chunk size of RAM can be modified to
lower value. However, this will also increases the amount of
chunk metadata.
6. CONCLUSION
In this paper, we have discussed how deduplication can improve
the feasibility of cloud storage usage on mobile devices and how
using a better chunking algorithm can reduce the deduplication
process time. We proposed a deduplication system based on SDM
and rapid asymmetric maximum (RAM). The deduplication
system used RAM instead of Rabin for block-level deduplication.

17
The main advantage of RAM is its low computation overhead
which allows faster deduplication process. When RAM is used on
SDM as the block-level chunking algorithm (RDM), the process
time is reduced in all datasets but the amount of duplicates found
is also reduced in some datasets. In some cases, using RAM on
SDM offers 39.5% to 44.7% less processing time with an increase
of 4.2% to 9.1% in size after deduplication than using Rabin.
7. ACKNOWLEDGEMENT
This work was supported by the National Research Foundation of
Korea under Grant 2016R1D1A1A09916932.
8. REFERENCES
[1] Ahn, J. and Shin, D. 2014. Optimizing Power Consumption
of Memory Deduplication Scheme. (2014), 1–2.
[2] Bjørner, N. et al. 2010. Content-dependent chunking for
differential compression , the local maximum approach.
Journal of Computer and System Sciences. 76, 3-4 (2010),
154–203.
[3] Bloom, B.H. 1970. Space / Time Trade-offs in Hash Coding
with Allowable Errors. Communications of the ACM 13.7
(1970). 13, 7 (1970), 422–426.
[4] Bremler-barr, A. et al. Leveraging Traffic Repetitions for
High-Speed Deep Packet Inspection.
[5] Haustein, N. et al. 2009. Method of and system for adaptive
selection of a deduplication chunking technique. United
States Patent. 2, 12 (2009), 1–7.
[6] Hua, K.A. et al. 2015. Redundancy Control through Traffic
Deduplication. (2015), 10–18.
[7] Jin, K. and Miller, E.L. 2009. The Effectiveness of
Deduplication on Virtual Machine Disk Images. Proceedings
of SYSTOR 2009: The Israeli Experimental Systems
Conference. May (2009), 1–12.
[8] Kim, S.H. et al. 2014. Selective memory deduplication for
cost efficiency in mobile smart devices. IEEE Transactions
on Consumer Electronics. 60, 2 (2014), 276–284.
18
[9] Lee, B. et al. 2015. MemScope : Analyzing Memory
Duplication on Android Systems. Proceedings of the 6th
Asia-Pacific Workshop on Systems. ACM. (2015), 19.
[10] Marques, L. and Costa, C.J. 2011. Secure Deduplication on
Mobile Devices. Proceedings of the 2011 Workshop on Open
Source and Design of Communication - OSDOC ’11. (2011),
19.
[11] Meister, D. and Brinkmann, A. 2009. Multi-level comparison
of data deduplication in a backup scenario. Proceedings of
SYSTOR 2009: The Israeli Experimental Systems Conference.
(2009), 8.
[12] Papapanagiotou, I. et al. 2012. Chunk and Object Level
Deduplication for Web Optimization : A Hybrid Approach.
(2012), 1393–1398.
[13] Rabin, M.O. 1981. Fingerprinting by random polynomials.
Center for Research in Computing Techn., Aiken
Computation Laboratory, Univ.
[14] Summary, E. 2016. Cisco Visual Networking Index :
Forecast and Methodology , 2014 – 2019. (2016), 2014–2019.
[15] Widodo, R.N.S. et al. 2016. A New Content-Defined
Chunking Algorithm for Data Deduplication in Cloud
Storage. Manuscript. (2016).
[16] Widodo, R.N.S. et al. 2016. SDM : Smart deduplication for
mobile cloud storage. Future Generation Computer Systems.
(2016).
[17] Xu, J. et al. 2016. Clustering-based acceleration for virtual
machine image deduplication in the cloud environment. The
Journal of Systems & Software. 121, (2016), 144–156.
[18] Yang, C. et al. 2015. Provable ownership of files in
deduplication cloud storage. July 2013 (2015), 2457–2468.
[19] Yeo, H.S. et al. 2014. Leveraging client-side storage
techniques for enhanced use of multiple consumer cloud
storage services on resource-constrained mobile devices.
Journal of Network and Computer Applications. 43, (2014),
142–156.
[20] Zhang, Y. et al. 2015. AE : An Asymmetric Extremum
Content Defined Chunking Algorithm for Fast and
Bandwidth-Efficient Data Deduplication. (2015), 1337–1345.