Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Writing of omx files slows down #4

Open
evtrb opened this issue Feb 14, 2024 · 1 comment
Open

Writing of omx files slows down #4

evtrb opened this issue Feb 14, 2024 · 1 comment

Comments

@evtrb
Copy link

evtrb commented Feb 14, 2024

Hi,
I use omx-java to write out a lot of omx files. But at some point save() in OmxFile.java slows down extremely, finishing maybe 24-48 hours later. Next file begins writing - same thing again. The slow-down happens after approx. 2700 files (each omx file contains 1 double matrix à 10-70 MB having roughly 5500x5500 entries). It does not happen at a specific file. Meaning the same file for which writing slowed down will be generated without any problem after a restart of the code.
I think the issue is within the HDF5 library, since the slow-down happens in hdf5File.save().

Do you maybe know this problem and do you have a solution for this?

I read that some HDF5 config parameters could be changed, but unfortunately I have no clue what the real problem is or how to even change these HDF5 config parameters in java. My solution so far is to automatically catch the "stuck" situation (using a thread for save() and thereby excluding a java garbage collector problem, since nothing else slows down) and restart the code.
Thanks in advance!

@evtrb
Copy link
Author

evtrb commented Mar 13, 2024

I found a solution. Calling HDF5.close() from time to time avoids this slow-down, since it cleans up memory, etc.

See explanation:

  • H5close flushes all data to disk, closes all file identifiers, and cleans
  • up all memory used by the library.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant