Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CVode returns wrong yout if h becomes zero #82

Open
dweindl opened this issue Oct 8, 2021 · 1 comment
Open

CVode returns wrong yout if h becomes zero #82

dweindl opened this issue Oct 8, 2021 · 1 comment

Comments

@dweindl
Copy link
Contributor

dweindl commented Oct 8, 2021

Hi, I stumbled across a problem in CVode which seems to yield a wrong yout if h becomes zero.

After CVode: Internal t = 0 and h = 0 are such that t + h = t on the next step. The solver will continue anyway. and CVodeGetDky: Illegal value for t.t = 900 is not between tcur - hu = 0 and tcur = 0, CVode still returns CV_SUCCESS, but yout is not the correct value for tout, but for the time when h became zero.

The problem seems to be the multiplication with cv_mem->cv_h for checking whether tout was reached and subsequently ignoring the return value of CVodeGetDky (CV_BAD_T), here:

sundials/src/cvodes/cvodes.c

Lines 3298 to 3305 in a9bc74f

/* In NORMAL mode, check if tout reached */
if ( (itask == CV_NORMAL) && (cv_mem->cv_tn-tout)*cv_mem->cv_h >= ZERO ) {
istate = CV_SUCCESS;
cv_mem->cv_tretlast = *tret = tout;
(void) CVodeGetDky(cv_mem, tout, 0, yout);
cv_mem->cv_next_q = cv_mem->cv_qprime;
cv_mem->cv_next_h = cv_mem->cv_hprime;
break;

Similar checks exist in CVodeF here:

/* Return if tout reached */
if ( (*tret - tout)*cv_mem->cv_h >= ZERO ) {

and here:
if ((ttest - tout)*cv_mem->cv_h >= ZERO) {
/* ttest is after tout, interpolate to tout */

CVodeF leads to the same warnings, but returns CV_BAD_T. That is at least not CV_SUCCESS, but it is somewhat misleading, as CVodeGetDky should not have been called in the first place.

This happens in a more complex application and I am not sure I can easily provide a minimal reproducible example.

@balos1 balos1 added the triage label Mar 29, 2024
dweindl added a commit to dweindl/AMICI that referenced this issue Feb 17, 2025
Due to LLNL/sundials#82, we can get exceptions
from `CVodeSolver::getSens`, escaping `FinalStateStorer::~FinalStateStorer`,
that will result in crashes. This is fixed here.

The issue is semi-producible with
[Boehm_JProteomeRes2014](https://github.com/Benchmarking-Initiative/Benchmark-Models-PEtab/tree/15b4f78aff1121870537ad3e1a6b902cb8f73930/Benchmark-Models/Boehm_JProteomeRes2014):

Progamm output:

```
2025-02-17 14:44:33.988 - amici.swig_wrappers - DEBUG - [model1_data1][cvodes:CVode:WARNING] python/sdist/amici/ThirdParty/sundials/src/cvodes/cvodes.c:3482: Internal t = 0 and h = 0 are such that t + h = t on the next step. The solver will continue anyway. (99)
2025-02-17 14:44:33.988 - amici.swig_wrappers - DEBUG - [model1_data1][cvodes:CVodeGetDky:BAD_T] python/sdist/amici/ThirdParty/sundials/src/cvodes/cvodes.c:3783: Illegal value for t.t = 2.5 is not between tcur - hu = 0 and tcur = 0. (-25)
2025-02-17 14:44:33.988 - amici.swig_wrappers - DEBUG - [model1_data1][cvodes:CVodeGetDky:BAD_T] python/sdist/amici/ThirdParty/sundials/src/cvodes/cvodes.c:3783: Illegal value for t.t = 5 is not between tcur - hu = 0 and tcur = 0. (-25)
2025-02-17 14:44:33.988 - amici.swig_wrappers - DEBUG - [model1_data1][cvodes:CVode:ILL_INPUT] python/sdist/amici/ThirdParty/sundials/src/cvodes/cvodes.c:3328: Trouble interpolating at tout = 5. tout too far back in direction of integration (-22)
2025-02-17 14:44:33.988 - amici.swig_wrappers - ERROR - [model1_data1][FORWARD_FAILURE] AMICI forward simulation failed at t = 5: AMICI failed to integrate the forward problem
terminate called after throwing an instance of 'amici::CvodeException'
  what():  CVODE routine CVodeGetSens failed with error code -25.

Thread 1 "python" received signal SIGABRT, Aborted.
```

Stacktrace:

```
AMICI-dev#10 0x00007ffff70811f1 in _Unwind_RaiseException (exc=0x6d5ff10) at ../../../src/libgcc/unwind.inc:136
AMICI-dev#11 0x00007ffff48bb384 in __cxxabiv1::__cxa_throw (obj=<optimized out>, tinfo=0x7fff5c6d6aa8 <typeinfo for amici::CvodeException>, dest=0x7fff5c5476c4 <amici::CvodeException::~CvodeException()>)
    at ../../../../src/libstdc++-v3/libsupc++/eh_throw.cc:93
AMICI-dev#12 0x00007fff5c5bdb64 in amici::CVodeSolver::getSens (this=0x6d130d0) at python/sdist/amici/src/solver_cvodes.cpp:663
AMICI-dev#13 0x00007fffbc5709e8 in amici::Solver::getStateSensitivity (this=0x6d130d0, t=2.5) at python/sdist/amici/src/solver.cpp:1340
AMICI-dev#14 0x00007fffbc570750 in amici::Solver::writeSolution (this=0x6d130d0, t=0x6d26db0, x=..., dx=..., sx=..., xQ=...) at python/sdist/amici/src/solver.cpp:1303
AMICI-dev#15 0x00007fffbc5d376d in amici::ForwardProblem::getSimulationState (this=0x6d26c90) at python/sdist/amici/src/forwardproblem.cpp:421
AMICI-dev#16 0x00007fffbc5d45df in amici::FinalStateStorer::~FinalStateStorer (this=0x7fffffffb308, __in_chrg=<optimized out>) at include/amici/forwardproblem.h:450
AMICI-dev#17 0x00007fffbc5d2285 in amici::ForwardProblem::workForwardProblem (this=0x6d26c90) at python/sdist/amici/src/forwardproblem.cpp:203
AMICI-dev#18 0x00007fffbc54fbf9 in amici::runAmiciSimulation (solver=..., edata=0x5078010, model=..., rethrow=false) at python/sdist/amici/src/amici.cpp:108
```
github-merge-queue bot pushed a commit to AMICI-dev/AMICI that referenced this issue Feb 17, 2025
Due to LLNL/sundials#82, we can get exceptions
from `CVodeSolver::getSens`, escaping `FinalStateStorer::~FinalStateStorer`,
that will result in crashes. This is fixed here.

The issue is semi-producible with
[Boehm_JProteomeRes2014](https://github.com/Benchmarking-Initiative/Benchmark-Models-PEtab/tree/15b4f78aff1121870537ad3e1a6b902cb8f73930/Benchmark-Models/Boehm_JProteomeRes2014):

Progamm output:

```
2025-02-17 14:44:33.988 - amici.swig_wrappers - DEBUG - [model1_data1][cvodes:CVode:WARNING] python/sdist/amici/ThirdParty/sundials/src/cvodes/cvodes.c:3482: Internal t = 0 and h = 0 are such that t + h = t on the next step. The solver will continue anyway. (99)
2025-02-17 14:44:33.988 - amici.swig_wrappers - DEBUG - [model1_data1][cvodes:CVodeGetDky:BAD_T] python/sdist/amici/ThirdParty/sundials/src/cvodes/cvodes.c:3783: Illegal value for t.t = 2.5 is not between tcur - hu = 0 and tcur = 0. (-25)
2025-02-17 14:44:33.988 - amici.swig_wrappers - DEBUG - [model1_data1][cvodes:CVodeGetDky:BAD_T] python/sdist/amici/ThirdParty/sundials/src/cvodes/cvodes.c:3783: Illegal value for t.t = 5 is not between tcur - hu = 0 and tcur = 0. (-25)
2025-02-17 14:44:33.988 - amici.swig_wrappers - DEBUG - [model1_data1][cvodes:CVode:ILL_INPUT] python/sdist/amici/ThirdParty/sundials/src/cvodes/cvodes.c:3328: Trouble interpolating at tout = 5. tout too far back in direction of integration (-22)
2025-02-17 14:44:33.988 - amici.swig_wrappers - ERROR - [model1_data1][FORWARD_FAILURE] AMICI forward simulation failed at t = 5: AMICI failed to integrate the forward problem
terminate called after throwing an instance of 'amici::CvodeException'
  what():  CVODE routine CVodeGetSens failed with error code -25.

Thread 1 "python" received signal SIGABRT, Aborted.
```

Stacktrace:

```
#10 0x00007ffff70811f1 in _Unwind_RaiseException (exc=0x6d5ff10) at ../../../src/libgcc/unwind.inc:136
#11 0x00007ffff48bb384 in __cxxabiv1::__cxa_throw (obj=<optimized out>, tinfo=0x7fff5c6d6aa8 <typeinfo for amici::CvodeException>, dest=0x7fff5c5476c4 <amici::CvodeException::~CvodeException()>)
    at ../../../../src/libstdc++-v3/libsupc++/eh_throw.cc:93
#12 0x00007fff5c5bdb64 in amici::CVodeSolver::getSens (this=0x6d130d0) at python/sdist/amici/src/solver_cvodes.cpp:663
#13 0x00007fffbc5709e8 in amici::Solver::getStateSensitivity (this=0x6d130d0, t=2.5) at python/sdist/amici/src/solver.cpp:1340
#14 0x00007fffbc570750 in amici::Solver::writeSolution (this=0x6d130d0, t=0x6d26db0, x=..., dx=..., sx=..., xQ=...) at python/sdist/amici/src/solver.cpp:1303
#15 0x00007fffbc5d376d in amici::ForwardProblem::getSimulationState (this=0x6d26c90) at python/sdist/amici/src/forwardproblem.cpp:421
#16 0x00007fffbc5d45df in amici::FinalStateStorer::~FinalStateStorer (this=0x7fffffffb308, __in_chrg=<optimized out>) at include/amici/forwardproblem.h:450
#17 0x00007fffbc5d2285 in amici::ForwardProblem::workForwardProblem (this=0x6d26c90) at python/sdist/amici/src/forwardproblem.cpp:203
#18 0x00007fffbc54fbf9 in amici::runAmiciSimulation (solver=..., edata=0x5078010, model=..., rethrow=false) at python/sdist/amici/src/amici.cpp:108
```
dweindl added a commit to AMICI-dev/AMICI that referenced this issue Feb 18, 2025
Due to LLNL/sundials#82, we can get exceptions from `CVodeSolver::getSens`, escaping `FinalStateStorer::~FinalStateStorer`, that will result in crashes. This is fixed here.

The issue is semi-producible with [Boehm_JProteomeRes2014](https://github.com/Benchmarking-Initiative/Benchmark-Models-PEtab/tree/15b4f78aff1121870537ad3e1a6b902cb8f73930/Benchmark-Models/Boehm_JProteomeRes2014) with forward sensitivities:

Program output:

```
2025-02-17 14:44:33.988 - amici.swig_wrappers - DEBUG - [model1_data1][cvodes:CVode:WARNING] python/sdist/amici/ThirdParty/sundials/src/cvodes/cvodes.c:3482: Internal t = 0 and h = 0 are such that t + h = t on the next step. The solver will continue anyway. (99)
2025-02-17 14:44:33.988 - amici.swig_wrappers - DEBUG - [model1_data1][cvodes:CVodeGetDky:BAD_T] python/sdist/amici/ThirdParty/sundials/src/cvodes/cvodes.c:3783: Illegal value for t.t = 2.5 is not between tcur - hu = 0 and tcur = 0. (-25)
2025-02-17 14:44:33.988 - amici.swig_wrappers - DEBUG - [model1_data1][cvodes:CVodeGetDky:BAD_T] python/sdist/amici/ThirdParty/sundials/src/cvodes/cvodes.c:3783: Illegal value for t.t = 5 is not between tcur - hu = 0 and tcur = 0. (-25)
2025-02-17 14:44:33.988 - amici.swig_wrappers - DEBUG - [model1_data1][cvodes:CVode:ILL_INPUT] python/sdist/amici/ThirdParty/sundials/src/cvodes/cvodes.c:3328: Trouble interpolating at tout = 5. tout too far back in direction of integration (-22)
2025-02-17 14:44:33.988 - amici.swig_wrappers - ERROR - [model1_data1][FORWARD_FAILURE] AMICI forward simulation failed at t = 5: AMICI failed to integrate the forward problem
terminate called after throwing an instance of 'amici::CvodeException'
  what():  CVODE routine CVodeGetSens failed with error code -25.

Thread 1 "python" received signal SIGABRT, Aborted.
```

Stack trace:

```
#10 0x00007ffff70811f1 in _Unwind_RaiseException (exc=0x6d5ff10) at ../../../src/libgcc/unwind.inc:136
#11 0x00007ffff48bb384 in __cxxabiv1::__cxa_throw (obj=<optimized out>, tinfo=0x7fff5c6d6aa8 <typeinfo for amici::CvodeException>, dest=0x7fff5c5476c4 <amici::CvodeException::~CvodeException()>)
    at ../../../../src/libstdc++-v3/libsupc++/eh_throw.cc:93
#12 0x00007fff5c5bdb64 in amici::CVodeSolver::getSens (this=0x6d130d0) at python/sdist/amici/src/solver_cvodes.cpp:663
#13 0x00007fffbc5709e8 in amici::Solver::getStateSensitivity (this=0x6d130d0, t=2.5) at python/sdist/amici/src/solver.cpp:1340
#14 0x00007fffbc570750 in amici::Solver::writeSolution (this=0x6d130d0, t=0x6d26db0, x=..., dx=..., sx=..., xQ=...) at python/sdist/amici/src/solver.cpp:1303
#15 0x00007fffbc5d376d in amici::ForwardProblem::getSimulationState (this=0x6d26c90) at python/sdist/amici/src/forwardproblem.cpp:421
#16 0x00007fffbc5d45df in amici::FinalStateStorer::~FinalStateStorer (this=0x7fffffffb308, __in_chrg=<optimized out>) at include/amici/forwardproblem.h:450
#17 0x00007fffbc5d2285 in amici::ForwardProblem::workForwardProblem (this=0x6d26c90) at python/sdist/amici/src/forwardproblem.cpp:203
#18 0x00007fffbc54fbf9 in amici::runAmiciSimulation (solver=..., edata=0x5078010, model=..., rethrow=false) at python/sdist/amici/src/amici.cpp:108
```
dweindl added a commit to dweindl/AMICI that referenced this issue Feb 18, 2025
…dev#2647)

Due to LLNL/sundials#82, we can get exceptions from `CVodeSolver::getSens`, escaping `FinalStateStorer::~FinalStateStorer`, that will result in crashes. This is fixed here.

The issue is semi-producible with [Boehm_JProteomeRes2014](https://github.com/Benchmarking-Initiative/Benchmark-Models-PEtab/tree/15b4f78aff1121870537ad3e1a6b902cb8f73930/Benchmark-Models/Boehm_JProteomeRes2014) with forward sensitivities:

Program output:

```
2025-02-17 14:44:33.988 - amici.swig_wrappers - DEBUG - [model1_data1][cvodes:CVode:WARNING] python/sdist/amici/ThirdParty/sundials/src/cvodes/cvodes.c:3482: Internal t = 0 and h = 0 are such that t + h = t on the next step. The solver will continue anyway. (99)
2025-02-17 14:44:33.988 - amici.swig_wrappers - DEBUG - [model1_data1][cvodes:CVodeGetDky:BAD_T] python/sdist/amici/ThirdParty/sundials/src/cvodes/cvodes.c:3783: Illegal value for t.t = 2.5 is not between tcur - hu = 0 and tcur = 0. (-25)
2025-02-17 14:44:33.988 - amici.swig_wrappers - DEBUG - [model1_data1][cvodes:CVodeGetDky:BAD_T] python/sdist/amici/ThirdParty/sundials/src/cvodes/cvodes.c:3783: Illegal value for t.t = 5 is not between tcur - hu = 0 and tcur = 0. (-25)
2025-02-17 14:44:33.988 - amici.swig_wrappers - DEBUG - [model1_data1][cvodes:CVode:ILL_INPUT] python/sdist/amici/ThirdParty/sundials/src/cvodes/cvodes.c:3328: Trouble interpolating at tout = 5. tout too far back in direction of integration (-22)
2025-02-17 14:44:33.988 - amici.swig_wrappers - ERROR - [model1_data1][FORWARD_FAILURE] AMICI forward simulation failed at t = 5: AMICI failed to integrate the forward problem
terminate called after throwing an instance of 'amici::CvodeException'
  what():  CVODE routine CVodeGetSens failed with error code -25.

Thread 1 "python" received signal SIGABRT, Aborted.
```

Stack trace:

```
AMICI-dev#10 0x00007ffff70811f1 in _Unwind_RaiseException (exc=0x6d5ff10) at ../../../src/libgcc/unwind.inc:136
AMICI-dev#11 0x00007ffff48bb384 in __cxxabiv1::__cxa_throw (obj=<optimized out>, tinfo=0x7fff5c6d6aa8 <typeinfo for amici::CvodeException>, dest=0x7fff5c5476c4 <amici::CvodeException::~CvodeException()>)
    at ../../../../src/libstdc++-v3/libsupc++/eh_throw.cc:93
AMICI-dev#12 0x00007fff5c5bdb64 in amici::CVodeSolver::getSens (this=0x6d130d0) at python/sdist/amici/src/solver_cvodes.cpp:663
AMICI-dev#13 0x00007fffbc5709e8 in amici::Solver::getStateSensitivity (this=0x6d130d0, t=2.5) at python/sdist/amici/src/solver.cpp:1340
AMICI-dev#14 0x00007fffbc570750 in amici::Solver::writeSolution (this=0x6d130d0, t=0x6d26db0, x=..., dx=..., sx=..., xQ=...) at python/sdist/amici/src/solver.cpp:1303
AMICI-dev#15 0x00007fffbc5d376d in amici::ForwardProblem::getSimulationState (this=0x6d26c90) at python/sdist/amici/src/forwardproblem.cpp:421
AMICI-dev#16 0x00007fffbc5d45df in amici::FinalStateStorer::~FinalStateStorer (this=0x7fffffffb308, __in_chrg=<optimized out>) at include/amici/forwardproblem.h:450
AMICI-dev#17 0x00007fffbc5d2285 in amici::ForwardProblem::workForwardProblem (this=0x6d26c90) at python/sdist/amici/src/forwardproblem.cpp:203
AMICI-dev#18 0x00007fffbc54fbf9 in amici::runAmiciSimulation (solver=..., edata=0x5078010, model=..., rethrow=false) at python/sdist/amici/src/amici.cpp:108
```
@dweindl
Copy link
Contributor Author

dweindl commented Feb 18, 2025

This issue still persists at least with SUNDIALS 7.1.1. I can reproduce it, but unfortunately, I cannot provide a self-contained example. I'm happy to provide more information if needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants