[Libann-dev] Re: ANN++

[Dan Pemstein]

> 1.  Whilst your Network class is indeed quite flexible, it seems
>     to be quite clearly a multi-layer perceptron (and possibly a
>     Kohonen net too).  Did you intend ANN++ to support other
>     architectures such as the Boltzmann machine, Hopfield net, ART
>     etc??  If so, was it your intention that they would be derived
>     from ::Network or would they have their own class?
> 

Pretty much all of my experience with neural networks has been with
multi-layer perceptron networks.  Therefore ANN++ is horribly biased
toward such networks.  Ideally, ::Network would be the base class for
all types of neural nets, but I know that isn't possible in its current
form and I'm not sure it is possible at all.

> 2.  Your ::Unit class seems to be similar to my
>     ann::ActivationFunction class.  The difference is in the
>     ownership.  Whereas I have one ActivationFunction per Network, you
>     have one per Neuron.  Whilst it would be feasible to have a
>     network with every neuron having a distinct activation function,
>     I've never heard of such an arrangement being used, and it would
>     certainly make the training algorithm very complex.
> 
>     Was it a conscious decision to permit many different activation
>     functions or is this just how it worked out??
> 

I provided one unit per neuron because I was toying with the idea of
adding support for parallel computation either through pthreads or MPI
and having one activation function object would create a bottleneck in a
parallel implementation.  Also, every design decision in ANN++ defaulted
toward generality rather than efficiency.

> 3.  What tool did you use to create your class diagram?
> 

Dia.  http://www.lysator.liu.se/~alla/dia.

> 4.  Which do you consider to be your better area of expertise:
> 
>     C++ or Neural Networks ?
>

Probably C++.  I wouldn't say I'm an expert in either but my neural net
experience revolves mostly around theoretical coursework on multi-layer
perceptrons.  I'd also note that my C++ has improved quite a bit since
last year when most of ANN++ was written, especially in terms of the
STL.
 
> 5.  Do you envisage ANN++ as 
> 
>     * A tool for persons wanting to write programs needing Neural Nets;
>     * A vehicle to assist the study of Neural Nets themselves; or
>     * None of the above.
> 

Both.  I created ANN++ initially as a back-end for a project I was doing
on temporal difference learning.  I wanted a tool that would allow me to
test many different network structures without much rewriting of the
network code.  I also wanted support for recurrent nets because of a
project in political forecasting I was working on at the time.  At the
same time I wanted something that others would find useful not only for
rapid-prototyping of networks in the applications, but also for
algorithm development.

> 1.  Notably absent from my class diagram (if I had one) is the Neuron
>     and Connection classes.  In fact, in the early development I
>     started out with exactly those, but later dumped a large amount of
>     code in favour of the current implementation based around matrix
>     algebra.  I found that it made the implementation a lot easier and
>     a lot of tricky design problems disappeared.
>

I was considering moving to an underlying matrix representation myself. 
I'm trying to figure out the conjugate gradient algorithm and it is
always presented in terms of matrix manipulation. I'd still like to
provide an interface that gives the user the impression that she is
manipulating neurons and connections.

There are some reasons to maintain this object-heavy paradigm, mostly
relating to generality.  See #6.

> 2.  I'm rather fussy (some would say pedantic) about correct use of
>     C++ and  OO techniques.  For example I'm rather keen to stick to
>     the `Instantiation is Initialisation' principle.  Your code seems
>     not to consider this important.
>

Whenever you create a new network in ANN++ the object created is pretty
useless (unless you fill and connect it), this doesn't technically mean
it isn't a fully initialized object (the object is not in a "broken"
state).  I see what you are getting at, but I found minimal
initialization at instantiation useful.  But that is partly because I
wanted to support dynamically resizing nets (such as in optimal brain
damage) without actual dynamic resizes.  I can't think of any other
object in ANN++ that doesn't follow the instantiation is initialization
principle.

For lack of a better way to deal with things, I did settle on that whole
finalize() function in the network class which I agree is complete crap.
  
> 3.  One of the differences between Libann and ANN++ is that I have a
>     fixed number (1) of hidden layers whereas you have an arbitrary
>     number. I guess this is all part of the flexibility you want.  I
>     chose to stick to a single hidden layer for a) simplicity; b) all
>     the literature says there is little or no benefit from having more
>     than one.
>

A single hidden layer can represent any continuous function.  Multiple
layers may represent discontinuous functions.
     
>     I think that if I was going to allow more than one hidden layer, I
>     would default it to one, so that users need not bother about this
>     unless they really wanted to.
> 

I agree, although the base Network class might not contain this
modification.  As ANN++ is set up, end users will never actually
interact with the base level of the library, only through algorithm
extensions (a design I find appealing).

> 4.  I was toying with the idea of replacing my ann::matrix class with
>     something from the Octave C++ library.  If I did this, then I
>     could provide a facility for users to pre-process their
>     classification data with a PCA algorithm.  Whilst that's not
>     anything to do with Neural Nets per se, it's almost always
>     required for any serious classification application.  
> 
>     The disadvantage would be the dependence upon a third party
>     product.
> 
> 

I think the base library should stand on its own feet.  Perhaps we could
provide a set of tools for converting to and from the octave format.

> 5.  As you will have seen, Libann has a somewhat `higher level' API
>     which uses things like feature maps and output maps to make it
>     easy for the user to apply the NN to real life situations.  Like
>     you said, it's probably better to have a lower level API and make
>     some interface classes available for convenience.
> 

The ANN++ division between "base library" and "extensions" allows me to
maintain a very general API at one level but allow a very constrictive
one at another.  One could even extend the extensions to make really
user-friendly interfaces.  Essentially, I'd like to provide a gradient
of interfaces:  from the really general for algorithm developers to the
really specific for those wishing to apply the library to practical
applications.  Actually, this is sort of my answer to question 5 in the
last section.

> 6.  My multilayer perceptron class has been coded such that it's easy
>     to substitute another activation function instead of the sigmoid.
>     However, if this was done, then the backpropogation algorithm
>     would need to be completely changed.  Currently there is no way
>     for the user to  slot in alternative training algorithms.  I'm
>     thinking about this, but there are some rather tricky issues to be
>     solved.

That is the advantage of implementation in terms of actual neurons and
connections.  My ::Neuron and ::Connection classes take care of
structural issues, while the extending ::BackpropNeuron and
::WeightedConnection classes are algorithm specific.  It ends up being
less difficult to support multiple algorithms using a building-block
approach although there is a greater initial investment in terms of
coding and probably an efficiency hit.

Dan
 

-- 
----------------------------------------------------------------------
| Daniel Pemstein                                                    |
| Email: address@hidden, address@hidden               |
| Web: http://www.danpemstein.com                                    |
| PGP: http://www.danpemstein.com/files/pgp.txt                      |
----------------------------------------------------------------------

signature.asc
Description: This is a digitally signed message part