gneuralnetwork
[Top][All Lists]

## [Gneuralnetwork] Bayesian models and Monte Carlo methods

 From: Tobias Wessels Subject: [Gneuralnetwork] Bayesian models and Monte Carlo methods Date: Thu, 24 Mar 2016 21:50:05 +0700

Hi everyone,

First a short remark on my previous question and then, more
importantly, an issue which I have with the current implementation of
neural networks.

I have read more about Monte Carlo methods and my previous question is
that
$\sum_{i=1}^L \tilde q(w_i)$
would approximate  the total weight of $\tilde q$, which it doesn't,
because the $w_i$ are already drawn according to the distribution of
$q$.

Now to the more important issue:
As it seems to me, Monte Carlo methods are mainly used for models,
which use Bayesian inference, as in these models you have to calculate
probabilistic integrals frequently, which is a difficult task unless
the model is broken down to a very narrow class of distribution
functions (i.e. Gaussian). Monte Carlo methods are a tool to
approximate these integrals by simple (that is: computationally
inexpensive) means.

The issue that I have now with the current implementation of neural
networks is that it is not adapted to Bayesian inference models. In
these models, the networks are not trained to use a single, most likely
choice of weight vector as in the maximum likelihood method, but
instead these models consider a probability distribution for the
weights of the neurons and then calculates an average (expected value)
of the output, given a new input x. (This is the important step, where
Monte Carlo methods are used, because this expectation is basically an
integral over a very complex probability density).

In the current implementation, however, the nodes have a single
parameter/array, which is set to the most likely choice, rather than
using a distribution of possible values for the weights. I don't see
how I would implement a Bayesian inference model using the current
code.

I hope that my explanations were somewhat clear and somewhat accurate,
as I am still new to this topic myself. So if you see it differently,