Compressed domain-specific data processing and analysis

Dapeng Dong; John Herbert

doi:10.1109/BigData.2017.8257941

Compressed domain-specific data processing and analysis

Dapeng Dong, John Herbert

University College Cork

Research output: Chapter in Book or Report/Conference proceeding › Conference Proceeding › peer-review

Abstract

Domain specific data such as sensor outputs and server trace logs have low levels of symbol richness, and so they can be represented in a very compact format. In this paper, we present a bit-oriented compression scheme designed not only to represent the data compactly but also to allow MapReduce programs to perform analysis and processing directly on the compressed data, and to do so in parallel. The core of the compression scheme is a novel hybrid data structure supporting bit pattern searching in constant time, and a scheme for making a block-splittable compressed file. Supporting software allows developers to work transparently with the compressed data. Experimental results demonstrate that the proposed compression scheme can significantly reduce data size and improve MapReduce analysis performance.

Original language	English
Title of host publication	Proceedings - 2017 IEEE International Conference on Big Data, Big Data 2017
Editors	Jian-Yun Nie, Zoran Obradovic, Toyotaro Suzumura, Rumi Ghosh, Raghunath Nambiar, Chonggang Wang, Hui Zang, Ricardo Baeza-Yates, Ricardo Baeza-Yates, Xiaohua Hu, Jeremy Kepner, Alfredo Cuzzocrea, Jian Tang, Masashi Toyoda
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	325-330
Number of pages	6
ISBN (Electronic)	9781538627143
DOIs	https://doi.org/10.1109/BigData.2017.8257941
Publication status	Published - 1 Jul 2017
Externally published	Yes
Event	5th IEEE International Conference on Big Data, Big Data 2017 - Boston, United States Duration: 11 Dec 2017 → 14 Dec 2017

Publication series

Name	Proceedings - 2017 IEEE International Conference on Big Data, Big Data 2017
Volume	2018-January

Conference

Conference	5th IEEE International Conference on Big Data, Big Data 2017
Country/Territory	United States
City	Boston
Period	11/12/17 → 14/12/17

Keywords

Algorithm
Big Data
Compression
MapReduce

Access to Document

10.1109/BigData.2017.8257941

Cite this

Dong, D., & Herbert, J. (2017). Compressed domain-specific data processing and analysis. In J.-Y. Nie, Z. Obradovic, T. Suzumura, R. Ghosh, R. Nambiar, C. Wang, H. Zang, R. Baeza-Yates, R. Baeza-Yates, X. Hu, J. Kepner, A. Cuzzocrea, J. Tang, & M. Toyoda (Eds.), Proceedings - 2017 IEEE International Conference on Big Data, Big Data 2017 (pp. 325-330). (Proceedings - 2017 IEEE International Conference on Big Data, Big Data 2017; Vol. 2018-January). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/BigData.2017.8257941

Dong, Dapeng ; Herbert, John. / Compressed domain-specific data processing and analysis. Proceedings - 2017 IEEE International Conference on Big Data, Big Data 2017. editor / Jian-Yun Nie ; Zoran Obradovic ; Toyotaro Suzumura ; Rumi Ghosh ; Raghunath Nambiar ; Chonggang Wang ; Hui Zang ; Ricardo Baeza-Yates ; Ricardo Baeza-Yates ; Xiaohua Hu ; Jeremy Kepner ; Alfredo Cuzzocrea ; Jian Tang ; Masashi Toyoda. Institute of Electrical and Electronics Engineers Inc., 2017. pp. 325-330 (Proceedings - 2017 IEEE International Conference on Big Data, Big Data 2017).

@inproceedings{063cdae043b148278c6bc614b34165b2,

title = "Compressed domain-specific data processing and analysis",

abstract = "Domain specific data such as sensor outputs and server trace logs have low levels of symbol richness, and so they can be represented in a very compact format. In this paper, we present a bit-oriented compression scheme designed not only to represent the data compactly but also to allow MapReduce programs to perform analysis and processing directly on the compressed data, and to do so in parallel. The core of the compression scheme is a novel hybrid data structure supporting bit pattern searching in constant time, and a scheme for making a block-splittable compressed file. Supporting software allows developers to work transparently with the compressed data. Experimental results demonstrate that the proposed compression scheme can significantly reduce data size and improve MapReduce analysis performance.",

keywords = "Algorithm, Big Data, Compression, MapReduce",

author = "Dapeng Dong and John Herbert",

note = "Publisher Copyright: {\textcopyright} 2017 IEEE.; 5th IEEE International Conference on Big Data, Big Data 2017 ; Conference date: 11-12-2017 Through 14-12-2017",

year = "2017",

month = jul,

day = "1",

doi = "10.1109/BigData.2017.8257941",

language = "English",

series = "Proceedings - 2017 IEEE International Conference on Big Data, Big Data 2017",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "325--330",

editor = "Jian-Yun Nie and Zoran Obradovic and Toyotaro Suzumura and Rumi Ghosh and Raghunath Nambiar and Chonggang Wang and Hui Zang and Ricardo Baeza-Yates and Ricardo Baeza-Yates and Xiaohua Hu and Jeremy Kepner and Alfredo Cuzzocrea and Jian Tang and Masashi Toyoda",

booktitle = "Proceedings - 2017 IEEE International Conference on Big Data, Big Data 2017",

}

Dong, D & Herbert, J 2017, Compressed domain-specific data processing and analysis. in J-Y Nie, Z Obradovic, T Suzumura, R Ghosh, R Nambiar, C Wang, H Zang, R Baeza-Yates, R Baeza-Yates, X Hu, J Kepner, A Cuzzocrea, J Tang & M Toyoda (eds), Proceedings - 2017 IEEE International Conference on Big Data, Big Data 2017. Proceedings - 2017 IEEE International Conference on Big Data, Big Data 2017, vol. 2018-January, Institute of Electrical and Electronics Engineers Inc., pp. 325-330, 5th IEEE International Conference on Big Data, Big Data 2017, Boston, United States, 11/12/17. https://doi.org/10.1109/BigData.2017.8257941

Compressed domain-specific data processing and analysis. / Dong, Dapeng; Herbert, John.
Proceedings - 2017 IEEE International Conference on Big Data, Big Data 2017. ed. / Jian-Yun Nie; Zoran Obradovic; Toyotaro Suzumura; Rumi Ghosh; Raghunath Nambiar; Chonggang Wang; Hui Zang; Ricardo Baeza-Yates; Ricardo Baeza-Yates; Xiaohua Hu; Jeremy Kepner; Alfredo Cuzzocrea; Jian Tang; Masashi Toyoda. Institute of Electrical and Electronics Engineers Inc., 2017. p. 325-330 (Proceedings - 2017 IEEE International Conference on Big Data, Big Data 2017; Vol. 2018-January).

Research output: Chapter in Book or Report/Conference proceeding › Conference Proceeding › peer-review

TY - GEN

T1 - Compressed domain-specific data processing and analysis

AU - Dong, Dapeng

AU - Herbert, John

PY - 2017/7/1

Y1 - 2017/7/1

N2 - Domain specific data such as sensor outputs and server trace logs have low levels of symbol richness, and so they can be represented in a very compact format. In this paper, we present a bit-oriented compression scheme designed not only to represent the data compactly but also to allow MapReduce programs to perform analysis and processing directly on the compressed data, and to do so in parallel. The core of the compression scheme is a novel hybrid data structure supporting bit pattern searching in constant time, and a scheme for making a block-splittable compressed file. Supporting software allows developers to work transparently with the compressed data. Experimental results demonstrate that the proposed compression scheme can significantly reduce data size and improve MapReduce analysis performance.

AB - Domain specific data such as sensor outputs and server trace logs have low levels of symbol richness, and so they can be represented in a very compact format. In this paper, we present a bit-oriented compression scheme designed not only to represent the data compactly but also to allow MapReduce programs to perform analysis and processing directly on the compressed data, and to do so in parallel. The core of the compression scheme is a novel hybrid data structure supporting bit pattern searching in constant time, and a scheme for making a block-splittable compressed file. Supporting software allows developers to work transparently with the compressed data. Experimental results demonstrate that the proposed compression scheme can significantly reduce data size and improve MapReduce analysis performance.

KW - Algorithm

KW - Big Data

KW - Compression

KW - MapReduce

UR - http://www.scopus.com/inward/record.url?scp=85047834946&partnerID=8YFLogxK

U2 - 10.1109/BigData.2017.8257941

DO - 10.1109/BigData.2017.8257941

M3 - Conference Proceeding

AN - SCOPUS:85047834946

T3 - Proceedings - 2017 IEEE International Conference on Big Data, Big Data 2017

SP - 325

EP - 330

BT - Proceedings - 2017 IEEE International Conference on Big Data, Big Data 2017

A2 - Nie, Jian-Yun

A2 - Obradovic, Zoran

A2 - Suzumura, Toyotaro

A2 - Ghosh, Rumi

A2 - Nambiar, Raghunath

A2 - Wang, Chonggang

A2 - Zang, Hui

A2 - Baeza-Yates, Ricardo

A2 - Hu, Xiaohua

A2 - Kepner, Jeremy

A2 - Cuzzocrea, Alfredo

A2 - Tang, Jian

A2 - Toyoda, Masashi

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 5th IEEE International Conference on Big Data, Big Data 2017

Y2 - 11 December 2017 through 14 December 2017

ER -

Dong D, Herbert J. Compressed domain-specific data processing and analysis. In Nie JY, Obradovic Z, Suzumura T, Ghosh R, Nambiar R, Wang C, Zang H, Baeza-Yates R, Baeza-Yates R, Hu X, Kepner J, Cuzzocrea A, Tang J, Toyoda M, editors, Proceedings - 2017 IEEE International Conference on Big Data, Big Data 2017. Institute of Electrical and Electronics Engineers Inc. 2017. p. 325-330. (Proceedings - 2017 IEEE International Conference on Big Data, Big Data 2017). doi: 10.1109/BigData.2017.8257941

Compressed domain-specific data processing and analysis

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this