Causal Unstructured Pruning in Linear Networks Using Effective Information

Changyu Zeng, Li Liu, Haocheng Zhao, Yu Zhang, Wei Wang, Ning Cai, Yutao Yue*

*Corresponding author for this work

Research output: Chapter in Book or Report/Conference proceedingConference Proceedingpeer-review

2 Citations (Scopus)

Abstract

Excessive number of parameters in today's (deep) neural networks demands tremendous computational resources and slows down training speed. The problem also makes it difficult to deploy these neural network models on capability constrained devices such as mobile devices. To address this challenge, we propose an unstructured pruning method that measures the causal structure of neural networks based on effective information (EI). It introduces an intervention to the input and computes the mutual information between the interference and its corresponding output, within a single linear layer measuring the importance of each weight. In the experiments, we found that the sparsity of EI pruning can reach more than 90%. Only 10% of non-zero parameters in the linear layers were needed compared to the benchmark methods without pruning, while ensuring similar level of accuracy and stable training performance in iterative pruning. In addition, as the invariance of the causal structure of the network is exploited, the network after pruning using EI is highly generalizable and interpretable than other methods.

Original languageEnglish
Title of host publicationProceedings - 2022 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery, CyberC 2022
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages294-302
Number of pages9
ISBN (Electronic)9798350331547
DOIs
Publication statusPublished - 2022
Event12th International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery, CyberC 2022 - Virtual, Online, China
Duration: 15 Dec 202216 Dec 2022

Publication series

NameProceedings - 2022 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery, CyberC 2022

Conference

Conference12th International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery, CyberC 2022
Country/TerritoryChina
CityVirtual, Online
Period15/12/2216/12/22

Keywords

  • Causal Inference
  • Deep Learning
  • Effective Information
  • Unstructured Pruning

Fingerprint

Dive into the research topics of 'Causal Unstructured Pruning in Linear Networks Using Effective Information'. Together they form a unique fingerprint.

Cite this