-
Notifications
You must be signed in to change notification settings - Fork 31
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix crashes due to uncaught exceptions #2647
Conversation
Due to LLNL/sundials#82, we can get exceptions from `CVodeSolver::getSens`, escaping `FinalStateStorer::~FinalStateStorer`, that will result in crashes. This is fixed here. The issue is semi-producible with [Boehm_JProteomeRes2014](https://github.com/Benchmarking-Initiative/Benchmark-Models-PEtab/tree/15b4f78aff1121870537ad3e1a6b902cb8f73930/Benchmark-Models/Boehm_JProteomeRes2014): Progamm output: ``` 2025-02-17 14:44:33.988 - amici.swig_wrappers - DEBUG - [model1_data1][cvodes:CVode:WARNING] python/sdist/amici/ThirdParty/sundials/src/cvodes/cvodes.c:3482: Internal t = 0 and h = 0 are such that t + h = t on the next step. The solver will continue anyway. (99) 2025-02-17 14:44:33.988 - amici.swig_wrappers - DEBUG - [model1_data1][cvodes:CVodeGetDky:BAD_T] python/sdist/amici/ThirdParty/sundials/src/cvodes/cvodes.c:3783: Illegal value for t.t = 2.5 is not between tcur - hu = 0 and tcur = 0. (-25) 2025-02-17 14:44:33.988 - amici.swig_wrappers - DEBUG - [model1_data1][cvodes:CVodeGetDky:BAD_T] python/sdist/amici/ThirdParty/sundials/src/cvodes/cvodes.c:3783: Illegal value for t.t = 5 is not between tcur - hu = 0 and tcur = 0. (-25) 2025-02-17 14:44:33.988 - amici.swig_wrappers - DEBUG - [model1_data1][cvodes:CVode:ILL_INPUT] python/sdist/amici/ThirdParty/sundials/src/cvodes/cvodes.c:3328: Trouble interpolating at tout = 5. tout too far back in direction of integration (-22) 2025-02-17 14:44:33.988 - amici.swig_wrappers - ERROR - [model1_data1][FORWARD_FAILURE] AMICI forward simulation failed at t = 5: AMICI failed to integrate the forward problem terminate called after throwing an instance of 'amici::CvodeException' what(): CVODE routine CVodeGetSens failed with error code -25. Thread 1 "python" received signal SIGABRT, Aborted. ``` Stacktrace: ``` AMICI-dev#10 0x00007ffff70811f1 in _Unwind_RaiseException (exc=0x6d5ff10) at ../../../src/libgcc/unwind.inc:136 AMICI-dev#11 0x00007ffff48bb384 in __cxxabiv1::__cxa_throw (obj=<optimized out>, tinfo=0x7fff5c6d6aa8 <typeinfo for amici::CvodeException>, dest=0x7fff5c5476c4 <amici::CvodeException::~CvodeException()>) at ../../../../src/libstdc++-v3/libsupc++/eh_throw.cc:93 AMICI-dev#12 0x00007fff5c5bdb64 in amici::CVodeSolver::getSens (this=0x6d130d0) at python/sdist/amici/src/solver_cvodes.cpp:663 AMICI-dev#13 0x00007fffbc5709e8 in amici::Solver::getStateSensitivity (this=0x6d130d0, t=2.5) at python/sdist/amici/src/solver.cpp:1340 AMICI-dev#14 0x00007fffbc570750 in amici::Solver::writeSolution (this=0x6d130d0, t=0x6d26db0, x=..., dx=..., sx=..., xQ=...) at python/sdist/amici/src/solver.cpp:1303 AMICI-dev#15 0x00007fffbc5d376d in amici::ForwardProblem::getSimulationState (this=0x6d26c90) at python/sdist/amici/src/forwardproblem.cpp:421 AMICI-dev#16 0x00007fffbc5d45df in amici::FinalStateStorer::~FinalStateStorer (this=0x7fffffffb308, __in_chrg=<optimized out>) at include/amici/forwardproblem.h:450 AMICI-dev#17 0x00007fffbc5d2285 in amici::ForwardProblem::workForwardProblem (this=0x6d26c90) at python/sdist/amici/src/forwardproblem.cpp:203 AMICI-dev#18 0x00007fffbc54fbf9 in amici::runAmiciSimulation (solver=..., edata=0x5078010, model=..., rethrow=false) at python/sdist/amici/src/amici.cpp:108 ```
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #2647 +/- ##
===========================================
- Coverage 77.20% 77.16% -0.05%
===========================================
Files 332 332
Lines 23139 23142 +3
Branches 1478 1480 +2
===========================================
- Hits 17864 17857 -7
- Misses 5264 5274 +10
Partials 11 11
Flags with carried forward coverage won't be shown. Click here to find out more.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it possible to add a regression test?
Not so easy, because it requires very specific parameter values. For fixed parameters, this wouldn't be reproducible on a different system. It took me on average >20min to get the crash. To confidently say it's fixed, we'd have to sample random parameters for several times that long. Not really worth it, I'd say. |
…dev#2647) Due to LLNL/sundials#82, we can get exceptions from `CVodeSolver::getSens`, escaping `FinalStateStorer::~FinalStateStorer`, that will result in crashes. This is fixed here. The issue is semi-producible with [Boehm_JProteomeRes2014](https://github.com/Benchmarking-Initiative/Benchmark-Models-PEtab/tree/15b4f78aff1121870537ad3e1a6b902cb8f73930/Benchmark-Models/Boehm_JProteomeRes2014) with forward sensitivities: Program output: ``` 2025-02-17 14:44:33.988 - amici.swig_wrappers - DEBUG - [model1_data1][cvodes:CVode:WARNING] python/sdist/amici/ThirdParty/sundials/src/cvodes/cvodes.c:3482: Internal t = 0 and h = 0 are such that t + h = t on the next step. The solver will continue anyway. (99) 2025-02-17 14:44:33.988 - amici.swig_wrappers - DEBUG - [model1_data1][cvodes:CVodeGetDky:BAD_T] python/sdist/amici/ThirdParty/sundials/src/cvodes/cvodes.c:3783: Illegal value for t.t = 2.5 is not between tcur - hu = 0 and tcur = 0. (-25) 2025-02-17 14:44:33.988 - amici.swig_wrappers - DEBUG - [model1_data1][cvodes:CVodeGetDky:BAD_T] python/sdist/amici/ThirdParty/sundials/src/cvodes/cvodes.c:3783: Illegal value for t.t = 5 is not between tcur - hu = 0 and tcur = 0. (-25) 2025-02-17 14:44:33.988 - amici.swig_wrappers - DEBUG - [model1_data1][cvodes:CVode:ILL_INPUT] python/sdist/amici/ThirdParty/sundials/src/cvodes/cvodes.c:3328: Trouble interpolating at tout = 5. tout too far back in direction of integration (-22) 2025-02-17 14:44:33.988 - amici.swig_wrappers - ERROR - [model1_data1][FORWARD_FAILURE] AMICI forward simulation failed at t = 5: AMICI failed to integrate the forward problem terminate called after throwing an instance of 'amici::CvodeException' what(): CVODE routine CVodeGetSens failed with error code -25. Thread 1 "python" received signal SIGABRT, Aborted. ``` Stack trace: ``` AMICI-dev#10 0x00007ffff70811f1 in _Unwind_RaiseException (exc=0x6d5ff10) at ../../../src/libgcc/unwind.inc:136 AMICI-dev#11 0x00007ffff48bb384 in __cxxabiv1::__cxa_throw (obj=<optimized out>, tinfo=0x7fff5c6d6aa8 <typeinfo for amici::CvodeException>, dest=0x7fff5c5476c4 <amici::CvodeException::~CvodeException()>) at ../../../../src/libstdc++-v3/libsupc++/eh_throw.cc:93 AMICI-dev#12 0x00007fff5c5bdb64 in amici::CVodeSolver::getSens (this=0x6d130d0) at python/sdist/amici/src/solver_cvodes.cpp:663 AMICI-dev#13 0x00007fffbc5709e8 in amici::Solver::getStateSensitivity (this=0x6d130d0, t=2.5) at python/sdist/amici/src/solver.cpp:1340 AMICI-dev#14 0x00007fffbc570750 in amici::Solver::writeSolution (this=0x6d130d0, t=0x6d26db0, x=..., dx=..., sx=..., xQ=...) at python/sdist/amici/src/solver.cpp:1303 AMICI-dev#15 0x00007fffbc5d376d in amici::ForwardProblem::getSimulationState (this=0x6d26c90) at python/sdist/amici/src/forwardproblem.cpp:421 AMICI-dev#16 0x00007fffbc5d45df in amici::FinalStateStorer::~FinalStateStorer (this=0x7fffffffb308, __in_chrg=<optimized out>) at include/amici/forwardproblem.h:450 AMICI-dev#17 0x00007fffbc5d2285 in amici::ForwardProblem::workForwardProblem (this=0x6d26c90) at python/sdist/amici/src/forwardproblem.cpp:203 AMICI-dev#18 0x00007fffbc54fbf9 in amici::runAmiciSimulation (solver=..., edata=0x5078010, model=..., rethrow=false) at python/sdist/amici/src/amici.cpp:108 ```
Due to LLNL/sundials#82, we can get exceptions from
CVodeSolver::getSens
, escapingFinalStateStorer::~FinalStateStorer
, that will result in crashes. This is fixed here.The issue is semi-producible with Boehm_JProteomeRes2014 with forward sensitivities:
Program output:
Stack trace: