Improving deep neural networks by using sparse dropout strategy

Hao Zheng; Mingming Chen; Wenju Liu; Zhanlei Yang; Shan Liang

doi:10.1109/ChinaSIP.2014.6889194

Improving deep neural networks by using sparse dropout strategy

Hao Zheng, Mingming Chen, Wenju Liu, Zhanlei Yang, Shan Liang

CAS - Institute of Automation

Research output: Chapter in Book or Report/Conference proceeding › Conference Proceeding › peer-review

6 Citations (Scopus)

Abstract

Recently, deep neural networks(DNNs) have achieved excellent results on benchmarks for acoustic modeling of speech recognition. By randomly discarding network units, a strategy which is called as dropout can improve the performance of DNNs by reducing the influence of over-fitting. However, the random dropout strategy treats units indiscriminately, which may lose information on distributions of units outputs. In this paper, we improve the dropout strategy by differential treatment to units according to their outputs. Only minor changes to an existing neural network system can achieve a significant improvement. Experiments of phone recognition on TIMIT show that the sparse dropout fine-tuning gets significant performance improvement.

Original language	English
Title of host publication	2014 IEEE China Summit and International Conference on Signal and Information Processing, IEEE ChinaSIP 2014 - Proceedings
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	21-26
Number of pages	6
ISBN (Electronic)	9781479954032
DOIs	https://doi.org/10.1109/ChinaSIP.2014.6889194
Publication status	Published - 3 Sept 2014
Externally published	Yes
Event	2nd IEEE China Summit and International Conference on Signal and Information Processing, IEEE ChinaSIP 2014 - Xi'an, China Duration: 9 Jul 2014 → 13 Jul 2014

Publication series

Name	2014 IEEE China Summit and International Conference on Signal and Information Processing, IEEE ChinaSIP 2014 - Proceedings

Conference

Conference	2nd IEEE China Summit and International Conference on Signal and Information Processing, IEEE ChinaSIP 2014
Country/Territory	China
City	Xi'an
Period	9/07/14 → 13/07/14

Keywords

deep learning
deep neural networks
dropout
sparse dropout

Access to Document

10.1109/ChinaSIP.2014.6889194

Cite this

Zheng, H., Chen, M., Liu, W., Yang, Z., & Liang, S. (2014). Improving deep neural networks by using sparse dropout strategy. In 2014 IEEE China Summit and International Conference on Signal and Information Processing, IEEE ChinaSIP 2014 - Proceedings (pp. 21-26). Article 6889194 (2014 IEEE China Summit and International Conference on Signal and Information Processing, IEEE ChinaSIP 2014 - Proceedings). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ChinaSIP.2014.6889194

Zheng, Hao ; Chen, Mingming ; Liu, Wenju et al. / Improving deep neural networks by using sparse dropout strategy. 2014 IEEE China Summit and International Conference on Signal and Information Processing, IEEE ChinaSIP 2014 - Proceedings. Institute of Electrical and Electronics Engineers Inc., 2014. pp. 21-26 (2014 IEEE China Summit and International Conference on Signal and Information Processing, IEEE ChinaSIP 2014 - Proceedings).

@inproceedings{a85069f009d54f87a07f3479120dfc50,

title = "Improving deep neural networks by using sparse dropout strategy",

abstract = "Recently, deep neural networks(DNNs) have achieved excellent results on benchmarks for acoustic modeling of speech recognition. By randomly discarding network units, a strategy which is called as dropout can improve the performance of DNNs by reducing the influence of over-fitting. However, the random dropout strategy treats units indiscriminately, which may lose information on distributions of units outputs. In this paper, we improve the dropout strategy by differential treatment to units according to their outputs. Only minor changes to an existing neural network system can achieve a significant improvement. Experiments of phone recognition on TIMIT show that the sparse dropout fine-tuning gets significant performance improvement.",

keywords = "deep learning, deep neural networks, dropout, sparse dropout",

author = "Hao Zheng and Mingming Chen and Wenju Liu and Zhanlei Yang and Shan Liang",

note = "Publisher Copyright: {\textcopyright} 2014 IEEE.; 2nd IEEE China Summit and International Conference on Signal and Information Processing, IEEE ChinaSIP 2014 ; Conference date: 09-07-2014 Through 13-07-2014",

year = "2014",

month = sep,

day = "3",

doi = "10.1109/ChinaSIP.2014.6889194",

language = "English",

series = "2014 IEEE China Summit and International Conference on Signal and Information Processing, IEEE ChinaSIP 2014 - Proceedings",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "21--26",

booktitle = "2014 IEEE China Summit and International Conference on Signal and Information Processing, IEEE ChinaSIP 2014 - Proceedings",

}

Zheng, H, Chen, M, Liu, W, Yang, Z & Liang, S 2014, Improving deep neural networks by using sparse dropout strategy. in 2014 IEEE China Summit and International Conference on Signal and Information Processing, IEEE ChinaSIP 2014 - Proceedings., 6889194, 2014 IEEE China Summit and International Conference on Signal and Information Processing, IEEE ChinaSIP 2014 - Proceedings, Institute of Electrical and Electronics Engineers Inc., pp. 21-26, 2nd IEEE China Summit and International Conference on Signal and Information Processing, IEEE ChinaSIP 2014, Xi'an, China, 9/07/14. https://doi.org/10.1109/ChinaSIP.2014.6889194

Improving deep neural networks by using sparse dropout strategy. / Zheng, Hao; Chen, Mingming; Liu, Wenju et al.
2014 IEEE China Summit and International Conference on Signal and Information Processing, IEEE ChinaSIP 2014 - Proceedings. Institute of Electrical and Electronics Engineers Inc., 2014. p. 21-26 6889194 (2014 IEEE China Summit and International Conference on Signal and Information Processing, IEEE ChinaSIP 2014 - Proceedings).

Research output: Chapter in Book or Report/Conference proceeding › Conference Proceeding › peer-review

TY - GEN

T1 - Improving deep neural networks by using sparse dropout strategy

AU - Zheng, Hao

AU - Chen, Mingming

AU - Liu, Wenju

AU - Yang, Zhanlei

AU - Liang, Shan

PY - 2014/9/3

Y1 - 2014/9/3

N2 - Recently, deep neural networks(DNNs) have achieved excellent results on benchmarks for acoustic modeling of speech recognition. By randomly discarding network units, a strategy which is called as dropout can improve the performance of DNNs by reducing the influence of over-fitting. However, the random dropout strategy treats units indiscriminately, which may lose information on distributions of units outputs. In this paper, we improve the dropout strategy by differential treatment to units according to their outputs. Only minor changes to an existing neural network system can achieve a significant improvement. Experiments of phone recognition on TIMIT show that the sparse dropout fine-tuning gets significant performance improvement.

AB - Recently, deep neural networks(DNNs) have achieved excellent results on benchmarks for acoustic modeling of speech recognition. By randomly discarding network units, a strategy which is called as dropout can improve the performance of DNNs by reducing the influence of over-fitting. However, the random dropout strategy treats units indiscriminately, which may lose information on distributions of units outputs. In this paper, we improve the dropout strategy by differential treatment to units according to their outputs. Only minor changes to an existing neural network system can achieve a significant improvement. Experiments of phone recognition on TIMIT show that the sparse dropout fine-tuning gets significant performance improvement.

KW - deep learning

KW - deep neural networks

KW - dropout

KW - sparse dropout

UR - http://www.scopus.com/inward/record.url?scp=84929404573&partnerID=8YFLogxK

U2 - 10.1109/ChinaSIP.2014.6889194

DO - 10.1109/ChinaSIP.2014.6889194

M3 - Conference Proceeding

AN - SCOPUS:84929404573

T3 - 2014 IEEE China Summit and International Conference on Signal and Information Processing, IEEE ChinaSIP 2014 - Proceedings

SP - 21

EP - 26

BT - 2014 IEEE China Summit and International Conference on Signal and Information Processing, IEEE ChinaSIP 2014 - Proceedings

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2nd IEEE China Summit and International Conference on Signal and Information Processing, IEEE ChinaSIP 2014

Y2 - 9 July 2014 through 13 July 2014

ER -

Zheng H, Chen M, Liu W, Yang Z, Liang S. Improving deep neural networks by using sparse dropout strategy. In 2014 IEEE China Summit and International Conference on Signal and Information Processing, IEEE ChinaSIP 2014 - Proceedings. Institute of Electrical and Electronics Engineers Inc. 2014. p. 21-26. 6889194. (2014 IEEE China Summit and International Conference on Signal and Information Processing, IEEE ChinaSIP 2014 - Proceedings). doi: 10.1109/ChinaSIP.2014.6889194

Improving deep neural networks by using sparse dropout strategy

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this