Re: algorithm for fminunc

[Jose]

Hello.

On 01/21/2013 07:49 PM, Jordi Gutiérrez Hermoso wrote:
> Jaroslav, do you think Nocedal & Wright's "Numerical Optimization" is
> a suitable reference for your implementation of damped BFGS in
> fminunc.m?

I have received a detailed answer from Jaroslav with lots of valuable 
information. I copy-paste it straight here for reference.

-------
Indeed, looks like exactly the same formula for hessian update. It even 
uses the same symbol letters :)
Possibly confusing is "sBs = sumsq (w)" -> that's an optimization taking 
advantage of the fact that the function also computes w = hesr * s, 
where s is the search direction and hesr is the cholesky factor of the 
approximate hessian, i.e. B is recoverable as B = hesr' * hesr at any 
time, but the function never does it and instead operates with the 
factor (to avoid repeated factorization). The two cholupdate calls then 
correspond to the + and - terms in the formula 18.23. The double dogleg 
subproblem solver is pretty much stolen from MINPACK: 
http://www.netlib.org/minpack/dogleg.f and is the analogous to the one 
in fsolve. Other parts of fminunc/fsolve also heavily inspired by 
MINPACK - fsolve is mostly a rewrite, but fminunc has the BFGS update 
instead of the scaled Broyden update that's used for nonlinear systems 
and uses Cholesky factorization instead of QR. As a result, the routines 
can even be used educationally to show the analogy (and the differences) 
between the two problems (nonlinear optimization vs nonlinear least 
squares).
--------

BR
Jose