update success checks #29
Conversation
There was a problem hiding this comment.
I'm not sure, why should success_by_size be determined with *tol/2, but success be determined by *tol?
How useful is this *tol/2 transformation? Does it really determine success=True/False so strongly? If so, then I guess the user should tune *tol themselves, right?
There was a problem hiding this comment.
you need tighter tolerances for the inner check otherwise the outer check will fail pretty frequently with more complex examples.
Yes ideally the user can tweak the inner tolerances as well, but all that stuff is already pretty time consuming to tweak anyways (which is really the opposite of what you want to achieve with the toolbox). Overall my impression is that it probably makes more sense to rework these heuristics rather than allowing more user tweaking.
changes success checks heuristics a bit. notably using mean rather than first entry for consistency checks, using lower tolerances for by size checks and averaging after by size checks.
Without these changes, I rand into an abysmal number of false rejections for gradient checks with AMICI for the benchmark collection.