SentireCache: Accelerate Sentiment Classification With Saliency-Based Caching

Deep Learning methodologies have demonstrated exceptional efficacy in sentiment classification tasks. However, their extended inference times often impede practical deployment, particularly in resource-constrained environments. This paper addresses the challenge of reducing inference time by introducing a novel in-GPU caching approach, termed SentireCache, specifically designed for sentiment classification tasks. While traditional caching methods with the cosine similarity measurement have shown some reduction in inference time, they suffer from low hit rates and accuracy. To overcome this limitation, we incorporate a token filtering mechanism based on saliency into the caching system, along with simplified similarity calculation methods. The effectiveness of our proposed approach is theoretically analyzed. Moreover, extensive experimentation is conducted to compare SentireCache with other state-of-the-art caching methods. The results demonstrate a significant 37.7% reduction in inference time with an average performance degradation of 4.69%.