TY - GEN
T1 - Content-aware partial compression for big textual data analysis acceleration
AU - Dong, Dapeng
AU - Herbert, John
N1 - Publisher Copyright:
© 2014 IEEE.
PY - 2015/2/9
Y1 - 2015/2/9
N2 - Analysing text-based data has become increasingly important due to the importance of text from sources such as social media, web contents, web searches. The growing volume of such data creates challenges for data analysis including efficient and scalable algorithm, effective computing platforms and energy efficiency. Compression is a standard method for reducing data size but current standard compression algorithms are destructive to the organisation of data contents. This work introduces Content-aware, Partial Compression (CaPC) for text using a dictionary-based approach. We simply use shorter codes to replace strings while maintaining the original data format and structure, so that the compressed contents can be directly consumed by analytic platforms. We evaluate our approach with a set of real-world datasets and several classical MapReduce jobs on Hadoop. We also provide a supplementary utility library for Hadoop, hence, existing MapReduce programs can be used directly on the compressed datasets with little or no modification. In evaluation, we demonstrate that CaPC works well with a wide variety of data analysis scenarios, experimental results show ~30% average data size reduction, and up to ~32% performance increase on some I/O intensive jobs on an in-house Hadoop cluster. While the gains may seem modest, the point is that these gains are 'for free' and act as supplementary to all other optimizations.
AB - Analysing text-based data has become increasingly important due to the importance of text from sources such as social media, web contents, web searches. The growing volume of such data creates challenges for data analysis including efficient and scalable algorithm, effective computing platforms and energy efficiency. Compression is a standard method for reducing data size but current standard compression algorithms are destructive to the organisation of data contents. This work introduces Content-aware, Partial Compression (CaPC) for text using a dictionary-based approach. We simply use shorter codes to replace strings while maintaining the original data format and structure, so that the compressed contents can be directly consumed by analytic platforms. We evaluate our approach with a set of real-world datasets and several classical MapReduce jobs on Hadoop. We also provide a supplementary utility library for Hadoop, hence, existing MapReduce programs can be used directly on the compressed datasets with little or no modification. In evaluation, we demonstrate that CaPC works well with a wide variety of data analysis scenarios, experimental results show ~30% average data size reduction, and up to ~32% performance increase on some I/O intensive jobs on an in-house Hadoop cluster. While the gains may seem modest, the point is that these gains are 'for free' and act as supplementary to all other optimizations.
KW - Big data
KW - Compression
KW - Content-aware
KW - Hadoop
KW - MapReduce
UR - http://www.scopus.com/inward/record.url?scp=84937878990&partnerID=8YFLogxK
U2 - 10.1109/CloudCom.2014.76
DO - 10.1109/CloudCom.2014.76
M3 - Conference Proceeding
AN - SCOPUS:84937878990
T3 - Proceedings of the International Conference on Cloud Computing Technology and Science, CloudCom
SP - 320
EP - 325
BT - Proceedings - 2014 IEEE 6th International Conference on Cloud Computing Technology and Science, CloudCom 2014
PB - IEEE Computer Society
T2 - 2014 6th IEEE International Conference on Cloud Computing Technology and Science, CloudCom 2014
Y2 - 15 December 2014 through 18 December 2014
ER -