Implementation of a Levenberg–Marquardt algorithm #131

Deltadahl · 2023-01-22T12:52:33Z

Implementation of a Levenberg–Marquardt algorithm #119, according to the paper Improvements to the Levenberg-Marquardt algorithm for nonlinear least-squares minimization.
The solver in raphson.jl was used as a template for this solver.

codecov · 2023-01-22T12:58:02Z

Codecov Report

Merging #131 (518747a) into master (37034a5) will increase coverage by 2.09%.
The diff coverage is 95.94%.

@@            Coverage Diff             @@
##           master     #131      +/-   ##
==========================================
+ Coverage   88.79%   90.89%   +2.09%     
==========================================
  Files           6        7       +1     
  Lines         348      494     +146     
==========================================
+ Hits          309      449     +140     
- Misses         39       45       +6

Impacted Files	Coverage Δ
src/NonlinearSolve.jl	`60.00% <ø> (ø)`
src/ad.jl	`100.00% <ø> (ø)`
src/levenberg.jl	`95.68% <95.68%> (ø)`
src/trustRegion.jl	`98.40% <100.00%> (+0.09%)`	⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

ChrisRackauckas · 2023-01-22T13:23:46Z

src/ad.jl

@@ -26,7 +26,7 @@ end
 function SciMLBase.solve(prob::NonlinearProblem{<:Union{Number, StaticArraysCore.SVector},
                                                iip,
                                                <:Dual{T, V, P}},
-                         alg::Union{NewtonRaphson, TrustRegion},
+                         alg::Union{NewtonRaphson, TrustRegion, LevenbergMarquardt},


Suggested change

alg::Union{NewtonRaphson, TrustRegion, LevenbergMarquardt},

alg::AbstractNewtonAlgorithm,

ChrisRackauckas · 2023-01-22T13:23:53Z

src/ad.jl

@@ -35,7 +35,8 @@ end
 function SciMLBase.solve(prob::NonlinearProblem{<:Union{Number, StaticArraysCore.SVector},
                                                iip,
                                                <:AbstractArray{<:Dual{T, V, P}}},
-                         alg::Union{NewtonRaphson, TrustRegion}, args...;
+                         alg::Union{NewtonRaphson, TrustRegion, LevenbergMarquardt},


Suggested change

alg::Union{NewtonRaphson, TrustRegion, LevenbergMarquardt},

alg::AbstractNewtonAlgorithm,

ChrisRackauckas · 2023-01-22T13:28:50Z

src/levenberg.jl

+    cache.v = -(JᵀJ .+ λ .* DᵀD) \ mul!(cache.du_tmp, J', fu)
+
+    # Geodesic acceleration (step_size = v + a / 2).
+    @unpack v, alg = cache
+    h = alg.finite_diff_step_geodesic
+    f(cache.fu_tmp, u .+ h .* v, p)
+    cache.a = -J \ ((2 / h) .* ((cache.fu_tmp .- fu) ./ h .- mul!(cache.du_tmp, J, v)))


The two \'s should be changed to linsolve handlings.

ChrisRackauckas · 2023-01-22T13:29:06Z

src/levenberg.jl

+    @unpack u, p, λ, JᵀJ, DᵀD, J = cache
+
+    # Usual Levenberg-Marquardt step ("velocity").
+    cache.v = -(JᵀJ .+ λ .* DᵀD) \ mul!(cache.du_tmp, J', fu)


-(JᵀJ .+ λ .* DᵀD) should be put into a temporary

mul!(cache.du_tmp, J', fu) I'd move that to a different line for clarity. Though it returns cache.du_tmp in most dispatches, it's usually best to mutate in one line and then use in the next

Deltadahl · 2023-01-22T13:44:56Z

Edit:
I will include this algorithm in precompile_algs. I naively thought I would have the same loading time as the 13 sec stated in #128, and when I got 20 sec for all three algorithms I thought the last 7 sec was due to the newly added LevenbergMarquardt.
And when I removed both LevenbergMarquardt and TrustRegion the loading time vanished.
But I now tested with only LevenbergMarquardt and NewtonRaphson and the loading time was good. While for TrustRegion and NewtonRaphson Is was 20 sec. Therefore I'll now include LevenbergMarquardt in the precompile_algs.

Deltadahl · 2023-01-22T16:13:53Z

Testing to remove the precompilation of LevenbergMarquardt to see if it fixes the error in the check "IntegrationTest / ModelingToolkit.jl/All/1 (pull_request)"

Deltadahl · 2023-01-22T17:59:44Z

Think I've found the issue from #128.
Have to go now, unfortunately, but I'll be back in 12h again.

ChrisRackauckas · 2023-01-23T09:34:57Z

src/levenberg.jl

+    linres = dolinsolve(alg.precs, linsolve, A = cache.mat_tmp, b = _vec(cache.fu_tmp),
+                        linu = _vec(cache.du_tmp), p = p, reltol = cache.abstol)
+    cache.linsolve = linres.cache
+    @. cache.v = -cache.du_tmp
+
+    # Geodesic acceleration (step_size = v + a / 2).
+    @unpack v, alg = cache
+    h = alg.finite_diff_step_geodesic
+    f(cache.fu_tmp, u .+ h .* v, p)
+
+    # The following lines do: cache.a = -J \ cache.fu_tmp
+    mul!(cache.du_tmp, J, v)
+    @. cache.fu_tmp = (2 / h) * ((cache.fu_tmp - fu) / h - cache.du_tmp)
+    linres = dolinsolve(alg.precs, linsolve, A = J, b = _vec(cache.fu_tmp),
+                        linu = _vec(cache.du_tmp), p = p, reltol = cache.abstol)


We may want to split the preconditioners between these two. But that's for follow up.

ChrisRackauckas · 2023-01-23T09:35:31Z

This looks good for a first pass. More to be done, but we should split correctness from improvements.

Forward Mode overloads for Least Squares Problem

Deltadahl and others added 6 commits January 20, 2023 17:34

Started to implement Levenberg-Marquardt.

2387fb6

Implemented Levenberg for iip=true

ee5931d

Merge branch 'SciML:master' into master

fbaec12

Added documentation

a166576

format

a6a5bbb

remove from precompilation

18948e8

ChrisRackauckas reviewed Jan 22, 2023

View reviewed changes

Deltadahl added 4 commits January 22, 2023 15:29

added linsolve, and precompile

1ecad15

Clarity fix

2cdb3a5

updating to @.

d23d758

Testing: removing precomp. of Levenberg

6f92187

Fixing load time for TrustRegion

518747a

Deltadahl mentioned this pull request Jan 23, 2023

type instability fix #132

Merged

ChrisRackauckas reviewed Jan 23, 2023

View reviewed changes

ChrisRackauckas merged commit e839519 into SciML:master Jan 23, 2023

avik-pal added a commit that referenced this pull request Nov 1, 2024

Merge pull request #131 from avik-pal/ap/nlls_adjoint

f6cb31f

Forward Mode overloads for Least Squares Problem

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implementation of a Levenberg–Marquardt algorithm #131

Implementation of a Levenberg–Marquardt algorithm #131

Deltadahl commented Jan 22, 2023 •

edited

Loading

codecov bot commented Jan 22, 2023 •

edited

Loading

ChrisRackauckas Jan 22, 2023

ChrisRackauckas Jan 22, 2023

ChrisRackauckas Jan 22, 2023

ChrisRackauckas Jan 22, 2023

ChrisRackauckas Jan 22, 2023

Deltadahl commented Jan 22, 2023

Deltadahl commented Jan 22, 2023

Deltadahl commented Jan 22, 2023

ChrisRackauckas Jan 23, 2023

ChrisRackauckas commented Jan 23, 2023

	alg::Union{NewtonRaphson, TrustRegion, LevenbergMarquardt},
	alg::AbstractNewtonAlgorithm,

Implementation of a Levenberg–Marquardt algorithm #131

Implementation of a Levenberg–Marquardt algorithm #131

Conversation

Deltadahl commented Jan 22, 2023 • edited Loading

codecov bot commented Jan 22, 2023 • edited Loading

Codecov Report

ChrisRackauckas Jan 22, 2023

Choose a reason for hiding this comment

ChrisRackauckas Jan 22, 2023

Choose a reason for hiding this comment

ChrisRackauckas Jan 22, 2023

Choose a reason for hiding this comment

ChrisRackauckas Jan 22, 2023

Choose a reason for hiding this comment

ChrisRackauckas Jan 22, 2023

Choose a reason for hiding this comment

Deltadahl commented Jan 22, 2023

Deltadahl commented Jan 22, 2023

Deltadahl commented Jan 22, 2023

ChrisRackauckas Jan 23, 2023

Choose a reason for hiding this comment

ChrisRackauckas commented Jan 23, 2023

Deltadahl commented Jan 22, 2023 •

edited

Loading

codecov bot commented Jan 22, 2023 •

edited

Loading