Content-aware partial compression for big textual data analysis acceleration

Dapeng Dong, John Herbert

Research output: Chapter in Book or Report/Conference proceedingConference Proceedingpeer-review

5 Citations (Scopus)

Abstract

Analysing text-based data has become increasingly important due to the importance of text from sources such as social media, web contents, web searches. The growing volume of such data creates challenges for data analysis including efficient and scalable algorithm, effective computing platforms and energy efficiency. Compression is a standard method for reducing data size but current standard compression algorithms are destructive to the organisation of data contents. This work introduces Content-aware, Partial Compression (CaPC) for text using a dictionary-based approach. We simply use shorter codes to replace strings while maintaining the original data format and structure, so that the compressed contents can be directly consumed by analytic platforms. We evaluate our approach with a set of real-world datasets and several classical MapReduce jobs on Hadoop. We also provide a supplementary utility library for Hadoop, hence, existing MapReduce programs can be used directly on the compressed datasets with little or no modification. In evaluation, we demonstrate that CaPC works well with a wide variety of data analysis scenarios, experimental results show ~30% average data size reduction, and up to ~32% performance increase on some I/O intensive jobs on an in-house Hadoop cluster. While the gains may seem modest, the point is that these gains are 'for free' and act as supplementary to all other optimizations.

Original languageEnglish
Title of host publicationProceedings - 2014 IEEE 6th International Conference on Cloud Computing Technology and Science, CloudCom 2014
PublisherIEEE Computer Society
Pages320-325
Number of pages6
EditionFebruary
ISBN (Electronic)9781479940936
DOIs
Publication statusPublished - 9 Feb 2015
Externally publishedYes
Event2014 6th IEEE International Conference on Cloud Computing Technology and Science, CloudCom 2014 - Singapore, Singapore
Duration: 15 Dec 201418 Dec 2014

Publication series

NameProceedings of the International Conference on Cloud Computing Technology and Science, CloudCom
NumberFebruary
Volume2015-February
ISSN (Print)2330-2194
ISSN (Electronic)2330-2186

Conference

Conference2014 6th IEEE International Conference on Cloud Computing Technology and Science, CloudCom 2014
Country/TerritorySingapore
CitySingapore
Period15/12/1418/12/14

Keywords

  • Big data
  • Compression
  • Content-aware
  • Hadoop
  • MapReduce

Fingerprint

Dive into the research topics of 'Content-aware partial compression for big textual data analysis acceleration'. Together they form a unique fingerprint.

Cite this