Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix out of memory error with very large RetentionTimes table #3389

Merged
merged 7 commits into from
Mar 13, 2025

Conversation

nickshulman
Copy link
Contributor

@nickshulman nickshulman commented Mar 7, 2025

Mike has a .blib file with 60 million rows in the RetentionTimes table.
That .blib file was hitting an OutOfMemoryException in "BiblioSpecLiteLibrary.CreateCache" because that method uses a MemoryStream, and the MemoryStream was more than 2 billion bytes.

This pull request fixes the OutOfMemoryException by writing the MemoryStream out to disk as soon as it grows to larger than 10MB.

If it is not possible to write to disk (because the folder is read-only) then it will keep the entire cache file in memory so the user will still be able to use the .blib file (so long as the cache file is not larger than 2GB).

Next version we should get rid of the .slc file.
I have some ideas about how to speed up reading from the SQLite database using multiple threads so that there won't be any speed improvement having a cache file.

@nickshulman nickshulman requested review from chambm and brendanx67 March 7, 2025 03:19
@brendanx67
Copy link
Contributor

Score being a double also seems like overkill to me.

@nickshulman nickshulman marked this pull request as draft March 7, 2025 16:49
@nickshulman nickshulman marked this pull request as ready for review March 12, 2025 04:06
@nickshulman nickshulman marked this pull request as draft March 12, 2025 05:24
@nickshulman nickshulman marked this pull request as ready for review March 12, 2025 21:22
@nickshulman nickshulman merged commit 581e01f into master Mar 13, 2025
13 checks passed
@nickshulman nickshulman deleted the Skyline/work/20250306_BlibOutOfMemory branch March 13, 2025 20:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants