FOR TESTING PURPOSES ONLY

Hi all,

in the attached plugin:
- new functions NCGM(...) and NCGM.CD(...) - Nonlinear conjugate gradient method optimization algorithms;
- new function Gradient.CD(...) - central differences based Gradient;
- new function Jacobian.CD(...) - central differences based Jacobian;
- new function Hessian.CD(...) - central differences based Hessian;
- new functions GaussNewton.CD(...) and GaussNewton.CDGSS(...) - central differences based GaussNewton;
- new function LevenbergMarquardt.CD(...) - central differences based LevenbergMarquardt;
- new function NewtonRaphson.CD(...) - central differences based NewtonRaphson;
- new functions NewtonMethod.CD(...) and NewtonMethod.CDGSS(...) - central differences based NewtonMethod;
- increased performances on Gradient/Jacobian/Hessian-based functions;
- fixed issues of Broyden(...) and HRE.B(...) about custom settings;
- minor changes.

from previous BETA:
- new function GaussNewton(...) - Gauss-Newton optimization algorithm (previous BETA - increased performances);
- new functions GoldenSectionSearch.min(...) and GoldenSectionSearch.max(...) - Golden Section Search minimization/maximization algorithms (previous BETA - no changes);
- new functions GradientDescent(...) and GradientDescent.GSS(...) - Gradient Descent optimization algorithm (respectively with fixed step length and GoldenSectionSearch-based step length) (previous BETA - increased performances);
- new function LevenbergMarquardt(...) - Levenberg-Marquardt optimization algorithm (previous BETA - increased performances);
- new functions NewtonMethod(...) and NewtonMethod.GSS(...) - Newton Method optimization algorithm (respectively with fixed step length and GoldenSectionSearch-based step length) (previous BETA - increased performances);
- new function Diag(...) - improved SMath diag() (previous BETA - no changes);
- new function Gradient(...) - 1st order derivatives (previous BETA - increased performances);
- new function Hessian(...) - 2nd order derivatives (previous BETA - increased performances);
- function Jacobian(...) revisited (now returns only a derivative or a mxn Jacobian) (previous BETA - increased performances);
- solver Bisection(...) revisited (the number of iterations is no longer required, as reported by adiaz) (previous BETA - no changes);
- All root-finding algorithms in k variable now accept multiple thresholds (a target precision value for each function) (previous BETA - no changes);
- Fixed "custom decimal symbol" issue of HRE functions. (previous BETA - no changes);

REQUIRES customFunctions plugin

SEE THE ATTACHMENTS FOR MANY INFOS - PLEASE REPORT ANY ISSUE


best regards,

w3b5urf3r

Edited by user 12 January 2013 00:04:46(UTC)  | Reason: requirements

Hi Martin,

No problem, regional settings it's an important feature of SMath and I want the plugin give less problems as possible.

In all the root finding methods the convergence it's set with reference to the function value: f(x.target)<epsilon (BTW any testing file contains the structure of the function, so you can easily understand the behavior of any input argument). The degree in the convergence argument in the picture it's is my forgetfulness, I was checking that the arguments accepts the input in degrees, I had forgotten to remove (screenshot updated)

Bracketed algorithms accept a 2nd convergence criterion on bracket width, further convergence criteria will be added ASAP.

Please be careful with homotopy estimation methods... the 4th arguments represents the homotopy transformations (Δλ1-0)/homotopyTransformations); any transformation involve a root-finding call, so more tranformations->more calls->most computation time; considere that 10 transformations (or less) are usually enough.

regards,

w3b5urf3r

Edited by user 12 October 2012 12:57:46(UTC)  | Reason: Not specified

Hello w3b5urf3r,

I was playing a bit with optimization .CD functions - of course with my "famous" NLS example. The calculation time decreases on some more acceptable level (order of magnitude - few minutes each) and I could afford more runs for this testing. It seems that for this example the best performance had GausNewton.CD, NewtonMethod.CD and NewtonMethod.CDGSS. Morover, NewtonMethod.CDGSS performed the best (with a bit longer calculation time) - just compare with the result obtained by my LMA() function in the same file. I was expecting much more from LevenbergMarquardt() - to be quite good at this example, but it seems I was wrong. It was quite slow and rather inefficient. Actually, CD functions did some quite good job compared to the original ones. Hope that some more improvements could be done here

The attached are two files (worse and better initial conditions).
The picture is for the worse ones.

Regards,
Radovan

Hello w3b5urf3r,

It seems you've done something with the numerical derivatives - it seems quite accurate
Just for comparison see the post and the picture above regarding spline and spline derivatives . Jacobian.CD() - Gradient.CD() did the job quite well

Thank you very much

Regards,
Radovan

Edited by user 12 October 2012 22:50:59(UTC)  | Reason: Not specified

Hi omorr,

thank you for your testings

You're right, the finite difference newton-raphson contains a bug, now is solved (BETA plugin updated).

EDIT Levenberg-Marquardt and Gauss-Newton general algorithms are built to work knowing both the system of equations (the residuals) and the cost function (the sum of squared residuals; the plugin calculated the sum o squares of systems inside any algorithm); the LMA(...) in your scripts accept the sum of squared residuals as input because have a custom Jacobian inside it;
GradientDescent(...), NewtonMethod(...) and NCGM(...) need only the cost function, so you can use as input what do you want.


I'm surprised ; I've done nothing strange, just a "generalization" of Finite difference in several variables... maybe have you changed some parameter in your test (f.e. the perturbation value)?


regards,

w3b5urf3r

Edited by user 14 October 2012 00:41:44(UTC)  | Reason: Not specified

Hi
Just for your reference,

this is a LM coded in Matlab with three numerical examples to test its numerical robustness.
see "Levenberg-Marquardt method: introduction and examples" and "Levenberg-Marquardt method: m-files"
at http://www.duke.edu/~hpgavin/ce281/

And "Iterative Methods for Optimization" by C.T. Kelley has matlab codes for LM and code for numerical derivatives.
Free code at
http://www.siam.org/book...lley/fr18/matlabcode.php

Hi w3b5urf3r,

I'd prefer a list or vector of lists as format for target precision, because the number of functions does not necessarily match the number of variables. Also, giving just one value for all functions or variables would then be done by a one element list (system). Of course, this is perhaps matter of taste and as such the cook's decision.

BTW, in the past I have repeatedly applied the evolution strategy, including the version with covariance matrix adaptation. As far as I understand, this algorithm requires eigenvalues and eigenvectors of the covariance matrix to be calculated. The procedure is quite robust against noise, including inconsistent offspring selection. See CMA ES for theory and code examples.

Best regards, Martin

Hi all,

I'm working to improve the convergence criteria; at the time I found two different ways to do this (see the attachments).

I'd like to know which would be preferred in your opinion.


best regards,

w3b5urf3r

Hello
I was reading about Stiff IVPs and recalled I linked a gpl .NET library.
I've just added one more resource with GEAR BDF code on that post:
http://en.smath.info/for...algorithms.aspx#post7971

Hello w3bsurf3r,

I played around with nonlinear solvers, unfortunately without any success. At least not if units are involved.

Are the solvers supposed to work with units?
- can the unknown variables have units (given via units of limits or start point)?
- need the epsilons have units as well?

do I need to define functions or can I just give expressions as first argument?