Record-aware two-level compression for big textual data analysis acceleration

Dapeng Dong; John Herbert

doi:10.1109/CloudCom.2015.32

Record-aware two-level compression for big textual data analysis acceleration

Dapeng Dong, John Herbert

University College Cork

Research output: Chapter in Book or Report/Conference proceeding › Conference Proceeding › peer-review

2 Citations (Scopus)

Abstract

An increasing volume of data puts MapReduce data analytic platforms such as Hadoop under constant resource pressure. A new two-phase text compression scheme has been specially designed to accelerate data analysis and reduce cluster resource usage, and this has been implemented for Hadoop. The scheme consists of two levels of compression. The first level compression allows a Hadoop program to consume the compressed data directly, thus reducing the data transmission cost within a cluster during analysis. The second level packages data into fixed-size blocks that respect the logical data records. This further reduces the data size to a size similar to that achieved by a higher-order entropy encoder while also making the compressed data splittable for the HDFS. The use of the compression scheme is made transparent to Hadoop developers by the provided utility functions. The compression scheme is evaluated using a set of standard MapReduce jobs for a selection of real-world datasets. The experimental results show an improvement on analysis performance of up to 72% and compression ratios close to that achieved by a standard compressor such as Bzip.

Original language	English
Title of host publication	Proceedings - IEEE 7th International Conference on Cloud Computing Technology and Science, CloudCom 2015
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	9-16
Number of pages	8
ISBN (Electronic)	9781467395601
DOIs	https://doi.org/10.1109/CloudCom.2015.32
Publication status	Published - 1 Feb 2016
Externally published	Yes
Event	7th IEEE International Conference on Cloud Computing Technology and Science, CloudCom 2015 - Vancouver, Canada Duration: 30 Nov 2015 → 3 Dec 2015

Publication series

Name	Proceedings - IEEE 7th International Conference on Cloud Computing Technology and Science, CloudCom 2015

Conference

Conference	7th IEEE International Conference on Cloud Computing Technology and Science, CloudCom 2015
Country/Territory	Canada
City	Vancouver
Period	30/11/15 → 3/12/15

Keywords

Big Data
Compression
Content-aware
Hadoop
MapReduce
Record-aware

Access to Document

10.1109/CloudCom.2015.32

Cite this

Dong, D., & Herbert, J. (2016). Record-aware two-level compression for big textual data analysis acceleration. In Proceedings - IEEE 7th International Conference on Cloud Computing Technology and Science, CloudCom 2015 (pp. 9-16). Article 7396132 (Proceedings - IEEE 7th International Conference on Cloud Computing Technology and Science, CloudCom 2015). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/CloudCom.2015.32

Dong, Dapeng ; Herbert, John. / Record-aware two-level compression for big textual data analysis acceleration. Proceedings - IEEE 7th International Conference on Cloud Computing Technology and Science, CloudCom 2015. Institute of Electrical and Electronics Engineers Inc., 2016. pp. 9-16 (Proceedings - IEEE 7th International Conference on Cloud Computing Technology and Science, CloudCom 2015).

@inproceedings{3fc1d5a58d5c495e94cc6e057d0ecb55,

title = "Record-aware two-level compression for big textual data analysis acceleration",

abstract = "An increasing volume of data puts MapReduce data analytic platforms such as Hadoop under constant resource pressure. A new two-phase text compression scheme has been specially designed to accelerate data analysis and reduce cluster resource usage, and this has been implemented for Hadoop. The scheme consists of two levels of compression. The first level compression allows a Hadoop program to consume the compressed data directly, thus reducing the data transmission cost within a cluster during analysis. The second level packages data into fixed-size blocks that respect the logical data records. This further reduces the data size to a size similar to that achieved by a higher-order entropy encoder while also making the compressed data splittable for the HDFS. The use of the compression scheme is made transparent to Hadoop developers by the provided utility functions. The compression scheme is evaluated using a set of standard MapReduce jobs for a selection of real-world datasets. The experimental results show an improvement on analysis performance of up to 72\% and compression ratios close to that achieved by a standard compressor such as Bzip.",

keywords = "Big Data, Compression, Content-aware, Hadoop, MapReduce, Record-aware",

author = "Dapeng Dong and John Herbert",

note = "Publisher Copyright: {\textcopyright} 2015 IEEE.; 7th IEEE International Conference on Cloud Computing Technology and Science, CloudCom 2015 ; Conference date: 30-11-2015 Through 03-12-2015",

year = "2016",

month = feb,

day = "1",

doi = "10.1109/CloudCom.2015.32",

language = "English",

series = "Proceedings - IEEE 7th International Conference on Cloud Computing Technology and Science, CloudCom 2015",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "9--16",

booktitle = "Proceedings - IEEE 7th International Conference on Cloud Computing Technology and Science, CloudCom 2015",

}

Dong, D & Herbert, J 2016, Record-aware two-level compression for big textual data analysis acceleration. in Proceedings - IEEE 7th International Conference on Cloud Computing Technology and Science, CloudCom 2015., 7396132, Proceedings - IEEE 7th International Conference on Cloud Computing Technology and Science, CloudCom 2015, Institute of Electrical and Electronics Engineers Inc., pp. 9-16, 7th IEEE International Conference on Cloud Computing Technology and Science, CloudCom 2015, Vancouver, Canada, 30/11/15. https://doi.org/10.1109/CloudCom.2015.32

Record-aware two-level compression for big textual data analysis acceleration. / Dong, Dapeng; Herbert, John.
Proceedings - IEEE 7th International Conference on Cloud Computing Technology and Science, CloudCom 2015. Institute of Electrical and Electronics Engineers Inc., 2016. p. 9-16 7396132 (Proceedings - IEEE 7th International Conference on Cloud Computing Technology and Science, CloudCom 2015).

Research output: Chapter in Book or Report/Conference proceeding › Conference Proceeding › peer-review

TY - GEN

T1 - Record-aware two-level compression for big textual data analysis acceleration

AU - Dong, Dapeng

AU - Herbert, John

PY - 2016/2/1

Y1 - 2016/2/1

N2 - An increasing volume of data puts MapReduce data analytic platforms such as Hadoop under constant resource pressure. A new two-phase text compression scheme has been specially designed to accelerate data analysis and reduce cluster resource usage, and this has been implemented for Hadoop. The scheme consists of two levels of compression. The first level compression allows a Hadoop program to consume the compressed data directly, thus reducing the data transmission cost within a cluster during analysis. The second level packages data into fixed-size blocks that respect the logical data records. This further reduces the data size to a size similar to that achieved by a higher-order entropy encoder while also making the compressed data splittable for the HDFS. The use of the compression scheme is made transparent to Hadoop developers by the provided utility functions. The compression scheme is evaluated using a set of standard MapReduce jobs for a selection of real-world datasets. The experimental results show an improvement on analysis performance of up to 72% and compression ratios close to that achieved by a standard compressor such as Bzip.

AB - An increasing volume of data puts MapReduce data analytic platforms such as Hadoop under constant resource pressure. A new two-phase text compression scheme has been specially designed to accelerate data analysis and reduce cluster resource usage, and this has been implemented for Hadoop. The scheme consists of two levels of compression. The first level compression allows a Hadoop program to consume the compressed data directly, thus reducing the data transmission cost within a cluster during analysis. The second level packages data into fixed-size blocks that respect the logical data records. This further reduces the data size to a size similar to that achieved by a higher-order entropy encoder while also making the compressed data splittable for the HDFS. The use of the compression scheme is made transparent to Hadoop developers by the provided utility functions. The compression scheme is evaluated using a set of standard MapReduce jobs for a selection of real-world datasets. The experimental results show an improvement on analysis performance of up to 72% and compression ratios close to that achieved by a standard compressor such as Bzip.

KW - Big Data

KW - Compression

KW - Content-aware

KW - Hadoop

KW - MapReduce

KW - Record-aware

UR - https://www.scopus.com/pages/publications/84964329718

U2 - 10.1109/CloudCom.2015.32

DO - 10.1109/CloudCom.2015.32

M3 - Conference Proceeding

AN - SCOPUS:84964329718

T3 - Proceedings - IEEE 7th International Conference on Cloud Computing Technology and Science, CloudCom 2015

SP - 9

EP - 16

BT - Proceedings - IEEE 7th International Conference on Cloud Computing Technology and Science, CloudCom 2015

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 7th IEEE International Conference on Cloud Computing Technology and Science, CloudCom 2015

Y2 - 30 November 2015 through 3 December 2015

ER -

Dong D, Herbert J. Record-aware two-level compression for big textual data analysis acceleration. In Proceedings - IEEE 7th International Conference on Cloud Computing Technology and Science, CloudCom 2015. Institute of Electrical and Electronics Engineers Inc. 2016. p. 9-16. 7396132. (Proceedings - IEEE 7th International Conference on Cloud Computing Technology and Science, CloudCom 2015). doi: 10.1109/CloudCom.2015.32

Record-aware two-level compression for big textual data analysis acceleration

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this