Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modified BlockCacheManager to not create caches when size is zero. #5369

Open
wants to merge 2 commits into
base: 2.1
Choose a base branch
from

Conversation

dlmarion
Copy link
Contributor

No description provided.

@dlmarion dlmarion added this to the 2.1.4 milestone Feb 28, 2025
@dlmarion dlmarion self-assigned this Feb 28, 2025
@dlmarion
Copy link
Contributor Author

I found this while working on #5302. I added a data block cache to the Compactor, but not index and summary caches. I noticed in the log that these caches were still being created.

@keith-turner
Copy link
Contributor

keith-turner commented Mar 3, 2025

The RFile client code has this NoopCache w/ a comment about leaking decompressors and it seems to use the NoopCache in lieu of null. I have no idea if this is relevant outside the client code. Depends on how things are closed in the server code overall, maybe the server code always cleans stuff up and closes decompressors and this is not a concern.

// This cache exist as a hack to avoid leaking decompressors. When the RFile code is not given a
// cache it reads blocks directly from the decompressor. However if a user does not read all data
// for a scan this can leave a BCFile block open and a decompressor allocated.
//
// By providing a cache to the RFile code it forces each block to be read into memory. When a
// block is accessed the entire thing is read into memory immediately allocating and deallocating
// a decompressor. If the user does not read all data, no decompressors are left allocated.
private static class NoopCache implements BlockCache {

@dlmarion
Copy link
Contributor Author

dlmarion commented Mar 3, 2025

I noticed the NoopCache, but didn't think it would apply to the server side code. Your comment above made me look, and I'm not quite sure at this point. The code is pretty convoluted, but I think the comment in NoopCache still be correct on the client side. On the server side? no idea.

@dlmarion
Copy link
Contributor Author

dlmarion commented Mar 3, 2025

I also don't see where the existing BlockCache's close the Block such that BCFile.Reader.BlockReader.close is called.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants