Lastly, I've added a new ParameterStatistics class that computes all the key statistical measures for regression parameters, including standard errors, t-statistics, p-values, confidence intervals, and dependency analysis.

Ready for review and merge.

I've just added the ILeastSquaresMinimizer interface, which unifies our API for nonlinear least squares minimizers by providing two overloads for the FindMinimum method (one for Vector and one for double[]).

Please review and merge when ready. Thank you!

In the meantime, I found a critical bug in the Trust Region optimization implementation. The subproblem solvers were using unscaled gradient and Hessian values directly from the IObjectiveModel, while they should have been using the scaled values that NonlinearMinimizerBase.EvaluateJacobian produces.

The last commit fixes this issue by modifying the ITrustRegionSubproblem interface to accept the correctly scaled gradient and Hessian as parameters.

Summary of Key Changes

1. Enhanced Levenberg-Marquardt Algorithm

   • Support for direct residual functions
   • Improved MINPACK-style control flow for damping parameter updates

2. Comprehensive Statistical Analysis

   • New ParameterStatistics class for regression parameters analysis
   • Calculates standard errors, t-statistics, p-values, confidence intervals, and dependencies

3. API Improvements

   • Added ILeastSquaresMinimizer interface to unify nonlinear least squares minimizers

4. Bug Fixes

   • Fixed a bug in Trust Region optimization where subproblem solvers were using unscaled gradient and Hessian values
   • Modified ITrustRegionSubproblem interface to accept correctly scaled gradient and Hessian as parameters

Hi @diluculo,
Thank you for the trying to improve the optimizers here.
There's a few independent changes in this PR which makes review difficult however.

Please could this bit be pulled out to a separate PR for now?

Fixed a bug in Trust Region optimization where subproblem solvers were using unscaled gradient and Hessian values

Hi @febkor

Thanks for your review. You're right, this PR contains several independent changes that should be separated. I've split it into 3 separate PRs:

1. Enhanced Levenberg-Marquardt Algorithm with residual support (including ParameterStatistics)
2. ILeastSquaresMinimizer interface : Add Basin Hopping Global Optimization Algorithm with New Least Squares Interface #1122
3. Trust Region optimization bug fix : Fix and Extend Trust Region Subproblem Implementations #1120
4. ParameterStatistics for parameter inference

I initially planned to separate ParameterStatistics as well, but since it depends on changes to IObjectiveModelEvaluation in the LM enhancements, I've kept it in this PR to maintain functionality.

Edit:
In addition to splitting the PR, I've also added new features that I had been working on:

• The Basin Hopping global optimization algorithm in PR Add Basin Hopping Global Optimization Algorithm with New Least Squares Interface #1122
• The Exact Trust Region subproblem solver in PR Fix and Extend Trust Region Subproblem Implementations #1120

Hi @cdrnet,
This ended up being more work than initially anticipated, but I've completed everything I had prepared. I hope these PRs can be included in v6.0.

All PRs are now ready for review.

Regarding the failing test in AppVeyor: This is unrelated to the Levenberg-Marquardt enhancements - it's a floating-point precision issue in the statistics tests for .NET 8.0 (off by ~1.03E-09).

Hi @febkor

Here is the mathematical background document I've prepared to explain the residual function support. It illustrates the relationship between the objective function (χ²) and the residual vector (R). I hope this helps in understanding the purpose and implementation of this PR.