TBDB: Token Bucket-Based Dynamic Batching for Resource Scheduling Supporting Neural Network Inference in Intelligent Consumer Electronics

Honghao Gao, Binyang Qiu*, Ye Wang, Si Yu, Yueshen Xu*, Xinheng Wang

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

15 Citations (Scopus)

Abstract

Consumer electronics such as mobile phones, wearable devices, and vehicle electronics use many intelligent applications such as voice commands, machine translation, and face recognition. These applications require large inference workloads to perform intelligent tasks, which are often completed using deep neural network (DNN) models. Traditional approaches rely on pure cloud computing, with consumer devices collecting data and cloud computing platforms completing inference tasks. In real life, the workloads of these applications are not fixed and are likely to exhibit fluctuations or unexpected surges, increasing the workload of cloud computing platforms. Simply increasing server resources often leads to resource waste. Therefore, a dynamic resource scheduling method is needed. In this paper, a token bucket-based dynamic batching (TBDB) algorithm that maintains throughput while reducing latency and increasing device utilization, especially for large volumes of requests, is proposed. Our work includes the following achievements: 1) We employ the token bucket algorithm to determine the workload, considering the concurrency and frequency of the data. We dynamically vary the maximum batch size (MBS) that will trigger the inference process for the next batch. 2) A low-coupling mode architecture that can be embedded into various consumer electronics in a plug-and-play manner is designed. 3) The performance of the electronic devices and the maximum latency are studied to provide guidance for setting hyperparameters. Finally, we evaluate the effectiveness of our method in three consumer electronic scenarios and present a theoretical analysis for setting hyperparameters in different scenarios.

Original languageEnglish
Pages (from-to)1134-1144
Number of pages11
JournalIEEE TRANSACTIONS ON CONSUMER ELECTRONICS
Volume70
Issue number1
DOIs
Publication statusPublished - 1 Feb 2024

Keywords

  • Consumer electronics
  • dynamic batching
  • inference task
  • neural network
  • token bucket
  • workload balance

Fingerprint

Dive into the research topics of 'TBDB: Token Bucket-Based Dynamic Batching for Resource Scheduling Supporting Neural Network Inference in Intelligent Consumer Electronics'. Together they form a unique fingerprint.

Cite this