Parallel growing and training of neural networks using output parallelism

Sheng Uei Guan*, Shanchun Li

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

43 Citations (Scopus)

Abstract

In order to find an appropriate architecture for a large-scale real-world application automatically and efficiently, a natural method is to divide the original problem into a set of subproblems. In this paper, we propose a simple neural-network task decomposition method based on output parallelism. By using this method, a problem can be divided flexibly into several subproblems as chosen, each of which is composed of the whole input vector and a fraction of the output vector. Each module (for one subproblem) is responsible for producing a fraction of the output vector of the original problem. The hidden structure for the original problem's output units are decoupled. These modules can be grown and trained in parallel on parallel processing elements. Incorporated with a constructive learning algorithm, our method does not require excessive computation and any prior knowledge concerning decomposition. The feasibility of output parallelism is analyzed and proved. Some benchmarks are implemented to test the validity of this method. Their results show that this method can reduce computational time, increase learning speed and improve generalization accuracy for both classification and regression problems.

Original languageEnglish
Pages (from-to)542-550
Number of pages9
JournalIEEE Transactions on Neural Networks
Volume13
Issue number3
DOIs
Publication statusPublished - May 2002
Externally publishedYes

Keywords

  • Constructive learning algorithm
  • Multilayered feedforward networks
  • Output parallelism
  • Parallel growing
  • Task decomposition

Fingerprint

Dive into the research topics of 'Parallel growing and training of neural networks using output parallelism'. Together they form a unique fingerprint.

Cite this