Skip to content

Fast restart on busy parents #19964

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 21 commits into from
Mar 28, 2025
Merged

Fast restart on busy parents #19964

merged 21 commits into from
Mar 28, 2025

Conversation

ktsaou
Copy link
Member

@ktsaou ktsaou commented Mar 25, 2025

Prepopulate UUIDs to MRG to improve startup times.

  • Pre-populate UUIDs to MRG on start (from SQLite)
  • Cleanup obsolete UUIDs from MRG once finished

This improves startup times by 3x to 5x.

@ktsaou ktsaou requested review from vkalintiris, a team, Ferroin and thiagoftsm as code owners March 25, 2025 20:29
@github-actions github-actions bot added area/packaging Packaging and operating systems support area/build Build system (autotools and cmake). labels Mar 25, 2025
@github-actions github-actions bot added area/database and removed area/packaging Packaging and operating systems support labels Mar 25, 2025
@vkalintiris
Copy link
Contributor

The idea is to dump MRG to disk (LMDB) when exiting and load it back when starting.

  • make LMDB available to Netdata
  • Store metadata about the database files to LMDB
  • Map MRG sections to DBENGINE tiers
  • Save MRG to LMDB on exit
  • Load MRG from LMDB on load

@ktsaou we are at >= 1.5K LOC already. It's better to work on the rest of the items in another PR.

Copy link
Member

@Ferroin Ferroin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ktsaou we are at >= 1.5K LOC already. It's better to work on the rest of the items in another PR.

I also strongly second this point. We need to be better about not making huge PRs. They’re problematic to deal with when they cause a major issue, and they’re impossible to sanely backport for patch releases.

ktsaou and others added 7 commits March 26, 2025 23:36
Imported version is LMDB_0.9.31.

We only rely on 4 files. However, considering
that project is very small and everything's in a
single directory, I opted for simply copying
everything.
Remove save to file and watcher
Remove files / rename mrg-dump-load to mrg-load.c
Update CMakeLists.txt
stelfrag
stelfrag previously approved these changes Mar 28, 2025
@ktsaou ktsaou merged commit 3d0c2b9 into netdata:master Mar 28, 2025
60 of 83 checks passed
stelfrag added a commit to stelfrag/netdata that referenced this pull request Mar 31, 2025
Prepopulate UUIDs to MRG to improve startup times.

- [x] Pre-populate UUIDs to MRG on start (from SQLite)
- [x] Cleanup obsolete UUIDs from MRG once finished

This improves startup times by 3x to 5x.

---------

Co-authored-by: vkalintiris <vasilis@netdata.cloud>
Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>
(cherry picked from commit 3d0c2b9)
@stelfrag stelfrag mentioned this pull request Mar 31, 2025
Ferroin pushed a commit that referenced this pull request Apr 2, 2025
Prepopulate UUIDs to MRG to improve startup times.

- [x] Pre-populate UUIDs to MRG on start (from SQLite)
- [x] Cleanup obsolete UUIDs from MRG once finished

This improves startup times by 3x to 5x.

---------

Co-authored-by: vkalintiris <vasilis@netdata.cloud>
Co-authored-by: Stelios Fragkakis <52996999+stelfrag@users.noreply.github.com>
(cherry picked from commit 3d0c2b9)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants