Causal Unstructured Pruning in Linear Networks Using Effective Information

Excessive number of parameters in today's (deep) neural networks demands tremendous computational resources and slows down training speed. The problem also makes it difficult to deploy these neural network models on capability constrained devices such as mobile devices. To address this challenge, we propose an unstructured pruning method that measures the causal structure of neural networks based on effective information (EI). It introduces an intervention to the input and computes the mutual information between the interference and its corresponding output, within a single linear layer measuring the importance of each weight. In the experiments, we found that the sparsity of EI pruning can reach more than 90%. Only 10% of non-zero parameters in the linear layers were needed compared to the benchmark methods without pruning, while ensuring similar level of accuracy and stable training performance in iterative pruning. In addition, as the invariance of the causal structure of the network is exploited, the network after pruning using EI is highly generalizable and interpretable than other methods.