STOCHASTIC COMPUTING IN NEURAL NETWORKS

Abstract

In conventional information systems uncertainty in information is regarded as "noise", and treated as a problem. In neural networks, however, uncertainty is an intrinsic feature of much of the information processing, with many of the computations involving the deliberate injection of stochastic processes. This paper analyzes uncertainty in information representation and processing in terms of the benefits arising, their theoretical foundations, their practical applications, and their hardware implementation. It gives a new perspective on network computations and the information involved. As an example of this approach the paper analyzes the representation of analog quantities by stochastic sequences of bits. Benefits of this approach are both theoretical and practical. Realization of communication links between units is greatly simplified and the number of wires needed in interconnecting VLSI chips correspondingly reduced. The hardware realization of basic operations on analog quantities is very simple, for example, in some cases a single and-gate can be used as a multiplier. The stochastic approach also gives elegant solutions to problems including learning, representation of system controllers, uncertain inference, and partial differential equations.