Record-aware compression for big textual data analysis acceleration

Dapeng Dong; John Herbert

doi:10.1109/BigData.2015.7363872

Record-aware compression for big textual data analysis acceleration

Dapeng Dong, John Herbert

University College Cork

Research output: Chapter in Book or Report/Conference proceeding › Conference Proceeding › peer-review

7 Citations (Scopus)

Abstract

Big data analysis technologies are becoming more widely used in industry. The ever-increasing data volume, however, puts data analytic platforms such as Hadoop under constant pressure. Several compression methods have been made available on the Hadoop platform to effectively reduce data size and efficiently deliver data between cluster nodes. In the Hadoop context, compressed data can be categorized as splittable or non-splittable. Working with non-splittable data conflicts with the goal of parallelism. In addition, the current realization of splittable data by indexing is potentially harmful to the data locality property. To this end, we introduce the Record-aware Compression (RaC) scheme that makes the compressed contents splittable, uses a lightweight Hadoop Record Reader, and preserves the parallelism and data locality properties as much as possible. We evaluate RaC using a set of classical MapReduce jobs with a collection of well-known datasets from companies such as Google, Yahoo!, and Amazon. The experimental results show an average 24% improvement on analysis performance and up to 75% data size reduction.

Original language	English
Title of host publication	Proceedings - 2015 IEEE International Conference on Big Data, IEEE Big Data 2015
Editors	Feng Luo, Kemafor Ogan, Mohammed J. Zaki, Laura Haas, Beng Chin Ooi, Vipin Kumar, Sudarsan Rachuri, Saumyadipta Pyne, Howard Ho, Xiaohua Hu, Shipeng Yu, Morris Hui-I Hsiao, Jian Li
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	1183-1190
Number of pages	8
ISBN (Electronic)	9781479999255
DOIs	https://doi.org/10.1109/BigData.2015.7363872
Publication status	Published - 22 Dec 2015
Externally published	Yes
Event	3rd IEEE International Conference on Big Data, IEEE Big Data 2015 - Santa Clara, United States Duration: 29 Oct 2015 → 1 Nov 2015

Publication series

Name	Proceedings - 2015 IEEE International Conference on Big Data, IEEE Big Data 2015

Conference

Conference	3rd IEEE International Conference on Big Data, IEEE Big Data 2015
Country/Territory	United States
City	Santa Clara
Period	29/10/15 → 1/11/15

Keywords

Big Data
Compression
Hadoop
MapReduce
Record-aware

Access to Document

10.1109/BigData.2015.7363872

Cite this

Dong, D., & Herbert, J. (2015). Record-aware compression for big textual data analysis acceleration. In F. Luo, K. Ogan, M. J. Zaki, L. Haas, B. C. Ooi, V. Kumar, S. Rachuri, S. Pyne, H. Ho, X. Hu, S. Yu, M. H.-I. Hsiao, & J. Li (Eds.), Proceedings - 2015 IEEE International Conference on Big Data, IEEE Big Data 2015 (pp. 1183-1190). Article 7363872 (Proceedings - 2015 IEEE International Conference on Big Data, IEEE Big Data 2015). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/BigData.2015.7363872

Dong, Dapeng ; Herbert, John. / Record-aware compression for big textual data analysis acceleration. Proceedings - 2015 IEEE International Conference on Big Data, IEEE Big Data 2015. editor / Feng Luo ; Kemafor Ogan ; Mohammed J. Zaki ; Laura Haas ; Beng Chin Ooi ; Vipin Kumar ; Sudarsan Rachuri ; Saumyadipta Pyne ; Howard Ho ; Xiaohua Hu ; Shipeng Yu ; Morris Hui-I Hsiao ; Jian Li. Institute of Electrical and Electronics Engineers Inc., 2015. pp. 1183-1190 (Proceedings - 2015 IEEE International Conference on Big Data, IEEE Big Data 2015).

@inproceedings{5094c1d42c754792b26c10e3eb4aaf44,

title = "Record-aware compression for big textual data analysis acceleration",

abstract = "Big data analysis technologies are becoming more widely used in industry. The ever-increasing data volume, however, puts data analytic platforms such as Hadoop under constant pressure. Several compression methods have been made available on the Hadoop platform to effectively reduce data size and efficiently deliver data between cluster nodes. In the Hadoop context, compressed data can be categorized as splittable or non-splittable. Working with non-splittable data conflicts with the goal of parallelism. In addition, the current realization of splittable data by indexing is potentially harmful to the data locality property. To this end, we introduce the Record-aware Compression (RaC) scheme that makes the compressed contents splittable, uses a lightweight Hadoop Record Reader, and preserves the parallelism and data locality properties as much as possible. We evaluate RaC using a set of classical MapReduce jobs with a collection of well-known datasets from companies such as Google, Yahoo!, and Amazon. The experimental results show an average 24% improvement on analysis performance and up to 75% data size reduction.",

keywords = "Big Data, Compression, Hadoop, MapReduce, Record-aware",

author = "Dapeng Dong and John Herbert",

note = "Publisher Copyright: {\textcopyright} 2015 IEEE.; 3rd IEEE International Conference on Big Data, IEEE Big Data 2015 ; Conference date: 29-10-2015 Through 01-11-2015",

year = "2015",

month = dec,

day = "22",

doi = "10.1109/BigData.2015.7363872",

language = "English",

series = "Proceedings - 2015 IEEE International Conference on Big Data, IEEE Big Data 2015",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "1183--1190",

editor = "Feng Luo and Kemafor Ogan and Zaki, {Mohammed J.} and Laura Haas and Ooi, {Beng Chin} and Vipin Kumar and Sudarsan Rachuri and Saumyadipta Pyne and Howard Ho and Xiaohua Hu and Shipeng Yu and Hsiao, {Morris Hui-I} and Jian Li",

booktitle = "Proceedings - 2015 IEEE International Conference on Big Data, IEEE Big Data 2015",

}

Dong, D & Herbert, J 2015, Record-aware compression for big textual data analysis acceleration. in F Luo, K Ogan, MJ Zaki, L Haas, BC Ooi, V Kumar, S Rachuri, S Pyne, H Ho, X Hu, S Yu, MH-I Hsiao & J Li (eds), Proceedings - 2015 IEEE International Conference on Big Data, IEEE Big Data 2015., 7363872, Proceedings - 2015 IEEE International Conference on Big Data, IEEE Big Data 2015, Institute of Electrical and Electronics Engineers Inc., pp. 1183-1190, 3rd IEEE International Conference on Big Data, IEEE Big Data 2015, Santa Clara, United States, 29/10/15. https://doi.org/10.1109/BigData.2015.7363872

Record-aware compression for big textual data analysis acceleration. / Dong, Dapeng; Herbert, John.
Proceedings - 2015 IEEE International Conference on Big Data, IEEE Big Data 2015. ed. / Feng Luo; Kemafor Ogan; Mohammed J. Zaki; Laura Haas; Beng Chin Ooi; Vipin Kumar; Sudarsan Rachuri; Saumyadipta Pyne; Howard Ho; Xiaohua Hu; Shipeng Yu; Morris Hui-I Hsiao; Jian Li. Institute of Electrical and Electronics Engineers Inc., 2015. p. 1183-1190 7363872 (Proceedings - 2015 IEEE International Conference on Big Data, IEEE Big Data 2015).

Research output: Chapter in Book or Report/Conference proceeding › Conference Proceeding › peer-review

TY - GEN

T1 - Record-aware compression for big textual data analysis acceleration

AU - Dong, Dapeng

AU - Herbert, John

PY - 2015/12/22

Y1 - 2015/12/22

N2 - Big data analysis technologies are becoming more widely used in industry. The ever-increasing data volume, however, puts data analytic platforms such as Hadoop under constant pressure. Several compression methods have been made available on the Hadoop platform to effectively reduce data size and efficiently deliver data between cluster nodes. In the Hadoop context, compressed data can be categorized as splittable or non-splittable. Working with non-splittable data conflicts with the goal of parallelism. In addition, the current realization of splittable data by indexing is potentially harmful to the data locality property. To this end, we introduce the Record-aware Compression (RaC) scheme that makes the compressed contents splittable, uses a lightweight Hadoop Record Reader, and preserves the parallelism and data locality properties as much as possible. We evaluate RaC using a set of classical MapReduce jobs with a collection of well-known datasets from companies such as Google, Yahoo!, and Amazon. The experimental results show an average 24% improvement on analysis performance and up to 75% data size reduction.

AB - Big data analysis technologies are becoming more widely used in industry. The ever-increasing data volume, however, puts data analytic platforms such as Hadoop under constant pressure. Several compression methods have been made available on the Hadoop platform to effectively reduce data size and efficiently deliver data between cluster nodes. In the Hadoop context, compressed data can be categorized as splittable or non-splittable. Working with non-splittable data conflicts with the goal of parallelism. In addition, the current realization of splittable data by indexing is potentially harmful to the data locality property. To this end, we introduce the Record-aware Compression (RaC) scheme that makes the compressed contents splittable, uses a lightweight Hadoop Record Reader, and preserves the parallelism and data locality properties as much as possible. We evaluate RaC using a set of classical MapReduce jobs with a collection of well-known datasets from companies such as Google, Yahoo!, and Amazon. The experimental results show an average 24% improvement on analysis performance and up to 75% data size reduction.

KW - Big Data

KW - Compression

KW - Hadoop

KW - MapReduce

KW - Record-aware

UR - http://www.scopus.com/inward/record.url?scp=84963760558&partnerID=8YFLogxK

U2 - 10.1109/BigData.2015.7363872

DO - 10.1109/BigData.2015.7363872

M3 - Conference Proceeding

AN - SCOPUS:84963760558

T3 - Proceedings - 2015 IEEE International Conference on Big Data, IEEE Big Data 2015

SP - 1183

EP - 1190

BT - Proceedings - 2015 IEEE International Conference on Big Data, IEEE Big Data 2015

A2 - Luo, Feng

A2 - Ogan, Kemafor

A2 - Zaki, Mohammed J.

A2 - Haas, Laura

A2 - Ooi, Beng Chin

A2 - Kumar, Vipin

A2 - Rachuri, Sudarsan

A2 - Pyne, Saumyadipta

A2 - Ho, Howard

A2 - Hu, Xiaohua

A2 - Yu, Shipeng

A2 - Hsiao, Morris Hui-I

A2 - Li, Jian

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 3rd IEEE International Conference on Big Data, IEEE Big Data 2015

Y2 - 29 October 2015 through 1 November 2015

ER -

Dong D, Herbert J. Record-aware compression for big textual data analysis acceleration. In Luo F, Ogan K, Zaki MJ, Haas L, Ooi BC, Kumar V, Rachuri S, Pyne S, Ho H, Hu X, Yu S, Hsiao MHI, Li J, editors, Proceedings - 2015 IEEE International Conference on Big Data, IEEE Big Data 2015. Institute of Electrical and Electronics Engineers Inc. 2015. p. 1183-1190. 7363872. (Proceedings - 2015 IEEE International Conference on Big Data, IEEE Big Data 2015). doi: 10.1109/BigData.2015.7363872

Record-aware compression for big textual data analysis acceleration

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this