Multi-Model Running Latency Optimization in an Edge Computing Paradigm

Peisong Li, Xinheng Wang*, Kaizhu Huang, Yi Huang, Shancang Li, Muddesar Iqbal

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

3 Citations (Scopus)


Recent advances in both lightweight deep learning algorithms and edge computing increasingly enable multiple model inference tasks to be conducted concurrently on resource-constrained edge devices, allowing us to achieve one goal collaboratively rather than getting high quality in each standalone task. However, the high overall running latency for performing multi-model inferences always negatively affects the real-time applications. To combat latency, the algorithms should be optimized to minimize the latency for multi-model deployment without compromising the safety-critical situation. This work focuses on the real-time task scheduling strategy for multi-model deployment and investigating the model inference using an open neural network exchange (ONNX) runtime engine. Then, an application deployment strategy is proposed based on the container technology and inference tasks are scheduled to different containers based on the scheduling strategies. Experimental results show that the proposed solution is able to significantly reduce the overall running latency in real-time applications.

Original languageEnglish
Article number6097
Issue number16
Publication statusPublished - Aug 2022


  • AI
  • autonomous driving
  • edge computing
  • latency optimization
  • multi-model
  • task scheduling


Dive into the research topics of 'Multi-Model Running Latency Optimization in an Edge Computing Paradigm'. Together they form a unique fingerprint.

Cite this