TBDB: Token Bucket-Based Dynamic Batching for Resource Scheduling Supporting Neural Network Inference in Intelligent Consumer Electronics

Consumer electronics such as mobile phones, wearable devices, and vehicle electronics use many intelligent applications such as voice commands, machine translation, and face recognition. These applications require large inference workloads to perform intelligent tasks, which are often completed using deep neural network (DNN) models. Traditional approaches rely on pure cloud computing, with consumer devices collecting data and cloud computing platforms completing inference tasks. In real life, the workloads of these applications are not fixed and are likely to exhibit fluctuations or unexpected surges, increasing the workload of cloud computing platforms. Simply increasing server resources often leads to resource waste. Therefore, a dynamic resource scheduling method is needed. In this paper, a token bucket-based dynamic batching (TBDB) algorithm that maintains throughput while reducing latency and increasing device utilization, especially for large volumes of requests, is proposed. Our work includes the following achievements: 1) We employ the token bucket algorithm to determine the workload, considering the concurrency and frequency of the data. We dynamically vary the maximum batch size (MBS) that will trigger the inference process for the next batch. 2) A low-coupling mode architecture that can be embedded into various consumer electronics in a plug-and-play manner is designed. 3) The performance of the electronic devices and the maximum latency are studied to provide guidance for setting hyperparameters. Finally, we evaluate the effectiveness of our method in three consumer electronic scenarios and present a theoretical analysis for setting hyperparameters in different scenarios.