Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support lock when writing pkl (implemented by rename) #660

Merged
merged 1 commit into from
Mar 14, 2025

Conversation

itutu-tienday
Copy link
Collaborator

@itutu-tienday itutu-tienday commented Mar 14, 2025

Problem

  • When generating a large pkl file (over 1GB, for example), there is a delay in writing the file (probably a few seconds). If there is a file read access (checking the end of the workflow) during writing, a pkl read error occurs.
  • Reproducibility is not 100% (depending on the environment and timing)

Countermeasure

Until the pkl file has been completely written, it cannot be read (this is handled by a temporary file)

Testcase

Normal Workflow

  1. Run All (Tutorial 1)
    • 1. mac - native
    • 2. mac - docker
    • 3. ubuntu - native
    • 4. ubuntu - docker
    • 5. windows - native

Workflow with large output

  1. Run All (Microscope(.nd2 > 1GB) + suite2p_file_convert)
    • 1. mac - native (microscope node is not supported)
    • 2. mac - docker
    • 3. ubuntu - native
    • 4. ubuntu - docker
    • 5. windows - native

Copy link
Collaborator

@tsuchiyama-araya tsuchiyama-araya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Windows + Native
Mac + Docker
Ubuntu + Docker
Ubuntu + Native

こちらの環境で動作確認できました。

@itutu-tienday itutu-tienday merged commit f03da2e into develop-main Mar 14, 2025
5 checks passed
tsuchiyama-araya added a commit that referenced this pull request Mar 27, 2025
barebone develop-main -> oist develop-main
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants