TBDB: Token Bucket-Based Dynamic Batching for Resource Scheduling Supporting Neural Network Inference in Intelligent Consumer Electronics

Honghao Gao, Binyang Qiu, Ye Wang, Si Yu, Yueshen Xu, Xinheng Wang

Research output: Contribution to journalArticlepeer-review

3 Citations (Scopus)


Consumer electronics such as mobile phones, wearable devices, and vehicle electronics use many intelligent applications such as voice commands, machine translation, and face recognition. These applications require large inference workloads to perform intelligent tasks, which are often completed using deep neural network (DNN) models. Traditional approaches rely on pure cloud computing, with consumer devices collecting data and cloud computing platforms completing inference tasks. In real life, the workloads of these applications are not fixed and are likely to exhibit fluctuations or unexpected surges, increasing the workload of cloud computing platforms. Simply increasing server resources often leads to resource waste. Therefore, a dynamic resource scheduling method is needed. In this paper, a token bucket-based dynamic batching (TBDB) algorithm that maintains throughput while reducing latency and increasing device utilization, especially for large volumes of requests, is proposed. Our work includes the following achievements: 1) We employ the token bucket algorithm to determine the workload, considering the concurrency and frequency of the data. We dynamically vary the maximum batch size (MBS) that will trigger the inference process for the next batch. 2) A low-coupling mode architecture that can be embedded into various consumer electronics in a plug-and-play manner is designed. 3) The performance of the electronic devices and the maximum latency are studied to provide guidance for setting hyperparameters. Finally, we evaluate the effectiveness of our method in three consumer electronic scenarios and present a theoretical analysis for setting hyperparameters in different scenarios.

Original languageEnglish
Pages (from-to)1
Number of pages1
Publication statusAccepted/In press - 2023


  • Computational modeling
  • Consumer Electronics
  • Consumer electronics
  • Dynamic Batching
  • Heuristic algorithms
  • Inference Task
  • Neural Network
  • Performance evaluation
  • Servers
  • Task analysis
  • Throughput
  • Token Bucket
  • Workload Balance


Dive into the research topics of 'TBDB: Token Bucket-Based Dynamic Batching for Resource Scheduling Supporting Neural Network Inference in Intelligent Consumer Electronics'. Together they form a unique fingerprint.

Cite this