Re: Contribution to the optimization toolbox

[Jaroslav Hajek]

On Tue, Sep 8, 2009 at 4:20 PM, Jaroslav Hajek<address@hidden> wrote:
> On Tue, Sep 8, 2009 at 1:11 PM, Michael Creel<address@hidden> wrote:
>> Thanks for the new version. Note that when dim is increased, bfgsmin
>> need to have max iters increased, otherwise it chokes before
>> converging. Setting control={10000,0} with dim=100, I get the results
>>
>> octave:4> compare
>> 100/100
>> Results using analytic gradient
>> ans =
>>
>>   3.1157e-01   4.6628e-08   1.2482e-15   4.3526e-01   6.0199e-02   1.6613e-03
>>
>> ans =
>>
>>   2.9021e-01   2.9620e-08   3.5626e-16   4.3474e-01   4.5174e-02   5.0746e-04
>>
>> so bfgsmin is finding the correct answer.
>>
>>
>> When dim is high, it makes sense to try lbfgs. Setting
>> control={10000,0,1,1,5} to use a memory of 5 iterations, the results
>> for dim=100 are
>>
>> octave:6> compare
>> 100/100
>> Results using analytic gradient
>> ans =
>>
>>   9.0097e-02   1.1931e-06   1.6012e-12   4.3407e-01   5.7046e-02   1.5292e-03
>>
>> ans =
>>
>>   8.7479e-02   3.4221e-07   4.5890e-14   4.3370e-01   3.7205e-02   3.5980e-04
>>
>> so lbfgs is considerably faster than bfgs.
>>
>> For setting up a bank of test problems, do you think that this test
>> script captures the important metrics: speed and accuracy? Anything
>> else to add? I may take the project on,
>>
>> Cheers, M.
>>
>
> Looking at the problem from a slightly different perspective, it now
> seems to me that taking the dogleg TR step selection method from
> fsolve may not have been that good idea as I thought initially. For
> fsolve, it shines because fsolve updates the model after each trial
> step, even an unsuccessful one. Uhm. Maybe a line search technique
> would be better. At least that would allow simply employing all three
> updating techniques in fminunc: BFGS, factored dual BFGS, and LBFGS.
>
> Btw., when I noticed that the rosenbrock function is actually a sum of
> squares, just for fun I tried rewriting the problem as a nonlinear
> least squares and feed it to fsolve. Doing thath, fsolve literally
> crushed the problem converging in 7 iterations (8 function and one
> jacobian evaluation).

I should have added; 7 iterations for dim = 100 :D


-- 
RNDr. Jaroslav Hajek
computing expert & GNU Octave developer
Aeronautical Research and Test Institute (VZLU)
Prague, Czech Republic
url: www.highegg.matfyz.cz