A Platform Independent Model Design and Adaptation for Edge Intelligence

Along with the rapid developments in deep learning and edge computing technologies, deploying models on mobile devices is a trend for modern applications. However, the resource-constrained and heterogeneous edge devices fail to cope with complicated model inference. To tackle this challenge, this study explores the platform independent model design based on Open Neural Network Exchange. Then we quantize the existing heavy deep learning models into lightweight ones by applying low-precision arithmetic and partitioning the computational graph into sub-graphs for parallel execution and hardware-based acceleration on heterogeneous systems. This study evaluates the efficiency and effectiveness among different quantization settings and analyses the impacts for model acceleration. The experiments show that the quantized models with uint8 type computation perform faster inference while still preserve high performance. This result suggests a novel solution for the research on edge intelligence.