Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix crashes due to uncaught exceptions #2647

Merged
merged 1 commit into from
Feb 18, 2025

Conversation

dweindl
Copy link
Member

@dweindl dweindl commented Feb 17, 2025

Due to LLNL/sundials#82, we can get exceptions from CVodeSolver::getSens, escaping FinalStateStorer::~FinalStateStorer, that will result in crashes. This is fixed here.

The issue is semi-producible with Boehm_JProteomeRes2014 with forward sensitivities:

Program output:

2025-02-17 14:44:33.988 - amici.swig_wrappers - DEBUG - [model1_data1][cvodes:CVode:WARNING] python/sdist/amici/ThirdParty/sundials/src/cvodes/cvodes.c:3482: Internal t = 0 and h = 0 are such that t + h = t on the next step. The solver will continue anyway. (99)
2025-02-17 14:44:33.988 - amici.swig_wrappers - DEBUG - [model1_data1][cvodes:CVodeGetDky:BAD_T] python/sdist/amici/ThirdParty/sundials/src/cvodes/cvodes.c:3783: Illegal value for t.t = 2.5 is not between tcur - hu = 0 and tcur = 0. (-25)
2025-02-17 14:44:33.988 - amici.swig_wrappers - DEBUG - [model1_data1][cvodes:CVodeGetDky:BAD_T] python/sdist/amici/ThirdParty/sundials/src/cvodes/cvodes.c:3783: Illegal value for t.t = 5 is not between tcur - hu = 0 and tcur = 0. (-25)
2025-02-17 14:44:33.988 - amici.swig_wrappers - DEBUG - [model1_data1][cvodes:CVode:ILL_INPUT] python/sdist/amici/ThirdParty/sundials/src/cvodes/cvodes.c:3328: Trouble interpolating at tout = 5. tout too far back in direction of integration (-22)
2025-02-17 14:44:33.988 - amici.swig_wrappers - ERROR - [model1_data1][FORWARD_FAILURE] AMICI forward simulation failed at t = 5: AMICI failed to integrate the forward problem
terminate called after throwing an instance of 'amici::CvodeException'
  what():  CVODE routine CVodeGetSens failed with error code -25.

Thread 1 "python" received signal SIGABRT, Aborted.

Stack trace:

#10 0x00007ffff70811f1 in _Unwind_RaiseException (exc=0x6d5ff10) at ../../../src/libgcc/unwind.inc:136
#11 0x00007ffff48bb384 in __cxxabiv1::__cxa_throw (obj=<optimized out>, tinfo=0x7fff5c6d6aa8 <typeinfo for amici::CvodeException>, dest=0x7fff5c5476c4 <amici::CvodeException::~CvodeException()>)
    at ../../../../src/libstdc++-v3/libsupc++/eh_throw.cc:93
#12 0x00007fff5c5bdb64 in amici::CVodeSolver::getSens (this=0x6d130d0) at python/sdist/amici/src/solver_cvodes.cpp:663
#13 0x00007fffbc5709e8 in amici::Solver::getStateSensitivity (this=0x6d130d0, t=2.5) at python/sdist/amici/src/solver.cpp:1340
#14 0x00007fffbc570750 in amici::Solver::writeSolution (this=0x6d130d0, t=0x6d26db0, x=..., dx=..., sx=..., xQ=...) at python/sdist/amici/src/solver.cpp:1303
#15 0x00007fffbc5d376d in amici::ForwardProblem::getSimulationState (this=0x6d26c90) at python/sdist/amici/src/forwardproblem.cpp:421
#16 0x00007fffbc5d45df in amici::FinalStateStorer::~FinalStateStorer (this=0x7fffffffb308, __in_chrg=<optimized out>) at include/amici/forwardproblem.h:450
#17 0x00007fffbc5d2285 in amici::ForwardProblem::workForwardProblem (this=0x6d26c90) at python/sdist/amici/src/forwardproblem.cpp:203
#18 0x00007fffbc54fbf9 in amici::runAmiciSimulation (solver=..., edata=0x5078010, model=..., rethrow=false) at python/sdist/amici/src/amici.cpp:108

Due to LLNL/sundials#82, we can get exceptions
from `CVodeSolver::getSens`, escaping `FinalStateStorer::~FinalStateStorer`,
that will result in crashes. This is fixed here.

The issue is semi-producible with
[Boehm_JProteomeRes2014](https://github.com/Benchmarking-Initiative/Benchmark-Models-PEtab/tree/15b4f78aff1121870537ad3e1a6b902cb8f73930/Benchmark-Models/Boehm_JProteomeRes2014):

Progamm output:

```
2025-02-17 14:44:33.988 - amici.swig_wrappers - DEBUG - [model1_data1][cvodes:CVode:WARNING] python/sdist/amici/ThirdParty/sundials/src/cvodes/cvodes.c:3482: Internal t = 0 and h = 0 are such that t + h = t on the next step. The solver will continue anyway. (99)
2025-02-17 14:44:33.988 - amici.swig_wrappers - DEBUG - [model1_data1][cvodes:CVodeGetDky:BAD_T] python/sdist/amici/ThirdParty/sundials/src/cvodes/cvodes.c:3783: Illegal value for t.t = 2.5 is not between tcur - hu = 0 and tcur = 0. (-25)
2025-02-17 14:44:33.988 - amici.swig_wrappers - DEBUG - [model1_data1][cvodes:CVodeGetDky:BAD_T] python/sdist/amici/ThirdParty/sundials/src/cvodes/cvodes.c:3783: Illegal value for t.t = 5 is not between tcur - hu = 0 and tcur = 0. (-25)
2025-02-17 14:44:33.988 - amici.swig_wrappers - DEBUG - [model1_data1][cvodes:CVode:ILL_INPUT] python/sdist/amici/ThirdParty/sundials/src/cvodes/cvodes.c:3328: Trouble interpolating at tout = 5. tout too far back in direction of integration (-22)
2025-02-17 14:44:33.988 - amici.swig_wrappers - ERROR - [model1_data1][FORWARD_FAILURE] AMICI forward simulation failed at t = 5: AMICI failed to integrate the forward problem
terminate called after throwing an instance of 'amici::CvodeException'
  what():  CVODE routine CVodeGetSens failed with error code -25.

Thread 1 "python" received signal SIGABRT, Aborted.
```

Stacktrace:

```
AMICI-dev#10 0x00007ffff70811f1 in _Unwind_RaiseException (exc=0x6d5ff10) at ../../../src/libgcc/unwind.inc:136
AMICI-dev#11 0x00007ffff48bb384 in __cxxabiv1::__cxa_throw (obj=<optimized out>, tinfo=0x7fff5c6d6aa8 <typeinfo for amici::CvodeException>, dest=0x7fff5c5476c4 <amici::CvodeException::~CvodeException()>)
    at ../../../../src/libstdc++-v3/libsupc++/eh_throw.cc:93
AMICI-dev#12 0x00007fff5c5bdb64 in amici::CVodeSolver::getSens (this=0x6d130d0) at python/sdist/amici/src/solver_cvodes.cpp:663
AMICI-dev#13 0x00007fffbc5709e8 in amici::Solver::getStateSensitivity (this=0x6d130d0, t=2.5) at python/sdist/amici/src/solver.cpp:1340
AMICI-dev#14 0x00007fffbc570750 in amici::Solver::writeSolution (this=0x6d130d0, t=0x6d26db0, x=..., dx=..., sx=..., xQ=...) at python/sdist/amici/src/solver.cpp:1303
AMICI-dev#15 0x00007fffbc5d376d in amici::ForwardProblem::getSimulationState (this=0x6d26c90) at python/sdist/amici/src/forwardproblem.cpp:421
AMICI-dev#16 0x00007fffbc5d45df in amici::FinalStateStorer::~FinalStateStorer (this=0x7fffffffb308, __in_chrg=<optimized out>) at include/amici/forwardproblem.h:450
AMICI-dev#17 0x00007fffbc5d2285 in amici::ForwardProblem::workForwardProblem (this=0x6d26c90) at python/sdist/amici/src/forwardproblem.cpp:203
AMICI-dev#18 0x00007fffbc54fbf9 in amici::runAmiciSimulation (solver=..., edata=0x5078010, model=..., rethrow=false) at python/sdist/amici/src/amici.cpp:108
```
Copy link

codecov bot commented Feb 17, 2025

Codecov Report

Attention: Patch coverage is 81.81818% with 2 lines in your changes missing coverage. Please review.

Project coverage is 77.16%. Comparing base (8366b87) to head (21e28e2).
Report is 1 commits behind head on develop.

Files with missing lines Patch % Lines
include/amici/forwardproblem.h 81.81% 2 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff             @@
##           develop    #2647      +/-   ##
===========================================
- Coverage    77.20%   77.16%   -0.05%     
===========================================
  Files          332      332              
  Lines        23139    23142       +3     
  Branches      1478     1480       +2     
===========================================
- Hits         17864    17857       -7     
- Misses        5264     5274      +10     
  Partials        11       11              
Flag Coverage Δ
cpp 74.09% <81.81%> (-0.05%) ⬇️
cpp_python 33.38% <45.45%> (-0.01%) ⬇️
petab 39.40% <55.55%> (-0.01%) ⬇️
python 72.21% <72.72%> (-0.04%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
include/amici/forwardproblem.h 94.73% <81.81%> (-5.27%) ⬇️

... and 2 files with indirect coverage changes

@dweindl dweindl marked this pull request as ready for review February 17, 2025 14:49
@dweindl dweindl requested a review from a team as a code owner February 17, 2025 14:49
Copy link
Member

@FFroehlich FFroehlich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to add a regression test?

@dweindl
Copy link
Member Author

dweindl commented Feb 17, 2025

Is it possible to add a regression test?

Not so easy, because it requires very specific parameter values. For fixed parameters, this wouldn't be reproducible on a different system. It took me on average >20min to get the crash. To confidently say it's fixed, we'd have to sample random parameters for several times that long. Not really worth it, I'd say.

@dweindl dweindl added this pull request to the merge queue Feb 17, 2025
@dweindl dweindl removed this pull request from the merge queue due to the queue being cleared Feb 17, 2025
@dweindl dweindl merged commit 2c117a5 into AMICI-dev:develop Feb 18, 2025
19 of 20 checks passed
@dweindl dweindl deleted the fix_finalstate branch February 18, 2025 07:51
dweindl added a commit to dweindl/AMICI that referenced this pull request Feb 18, 2025
…dev#2647)

Due to LLNL/sundials#82, we can get exceptions from `CVodeSolver::getSens`, escaping `FinalStateStorer::~FinalStateStorer`, that will result in crashes. This is fixed here.

The issue is semi-producible with [Boehm_JProteomeRes2014](https://github.com/Benchmarking-Initiative/Benchmark-Models-PEtab/tree/15b4f78aff1121870537ad3e1a6b902cb8f73930/Benchmark-Models/Boehm_JProteomeRes2014) with forward sensitivities:

Program output:

```
2025-02-17 14:44:33.988 - amici.swig_wrappers - DEBUG - [model1_data1][cvodes:CVode:WARNING] python/sdist/amici/ThirdParty/sundials/src/cvodes/cvodes.c:3482: Internal t = 0 and h = 0 are such that t + h = t on the next step. The solver will continue anyway. (99)
2025-02-17 14:44:33.988 - amici.swig_wrappers - DEBUG - [model1_data1][cvodes:CVodeGetDky:BAD_T] python/sdist/amici/ThirdParty/sundials/src/cvodes/cvodes.c:3783: Illegal value for t.t = 2.5 is not between tcur - hu = 0 and tcur = 0. (-25)
2025-02-17 14:44:33.988 - amici.swig_wrappers - DEBUG - [model1_data1][cvodes:CVodeGetDky:BAD_T] python/sdist/amici/ThirdParty/sundials/src/cvodes/cvodes.c:3783: Illegal value for t.t = 5 is not between tcur - hu = 0 and tcur = 0. (-25)
2025-02-17 14:44:33.988 - amici.swig_wrappers - DEBUG - [model1_data1][cvodes:CVode:ILL_INPUT] python/sdist/amici/ThirdParty/sundials/src/cvodes/cvodes.c:3328: Trouble interpolating at tout = 5. tout too far back in direction of integration (-22)
2025-02-17 14:44:33.988 - amici.swig_wrappers - ERROR - [model1_data1][FORWARD_FAILURE] AMICI forward simulation failed at t = 5: AMICI failed to integrate the forward problem
terminate called after throwing an instance of 'amici::CvodeException'
  what():  CVODE routine CVodeGetSens failed with error code -25.

Thread 1 "python" received signal SIGABRT, Aborted.
```

Stack trace:

```
AMICI-dev#10 0x00007ffff70811f1 in _Unwind_RaiseException (exc=0x6d5ff10) at ../../../src/libgcc/unwind.inc:136
AMICI-dev#11 0x00007ffff48bb384 in __cxxabiv1::__cxa_throw (obj=<optimized out>, tinfo=0x7fff5c6d6aa8 <typeinfo for amici::CvodeException>, dest=0x7fff5c5476c4 <amici::CvodeException::~CvodeException()>)
    at ../../../../src/libstdc++-v3/libsupc++/eh_throw.cc:93
AMICI-dev#12 0x00007fff5c5bdb64 in amici::CVodeSolver::getSens (this=0x6d130d0) at python/sdist/amici/src/solver_cvodes.cpp:663
AMICI-dev#13 0x00007fffbc5709e8 in amici::Solver::getStateSensitivity (this=0x6d130d0, t=2.5) at python/sdist/amici/src/solver.cpp:1340
AMICI-dev#14 0x00007fffbc570750 in amici::Solver::writeSolution (this=0x6d130d0, t=0x6d26db0, x=..., dx=..., sx=..., xQ=...) at python/sdist/amici/src/solver.cpp:1303
AMICI-dev#15 0x00007fffbc5d376d in amici::ForwardProblem::getSimulationState (this=0x6d26c90) at python/sdist/amici/src/forwardproblem.cpp:421
AMICI-dev#16 0x00007fffbc5d45df in amici::FinalStateStorer::~FinalStateStorer (this=0x7fffffffb308, __in_chrg=<optimized out>) at include/amici/forwardproblem.h:450
AMICI-dev#17 0x00007fffbc5d2285 in amici::ForwardProblem::workForwardProblem (this=0x6d26c90) at python/sdist/amici/src/forwardproblem.cpp:203
AMICI-dev#18 0x00007fffbc54fbf9 in amici::runAmiciSimulation (solver=..., edata=0x5078010, model=..., rethrow=false) at python/sdist/amici/src/amici.cpp:108
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants