Skip to content

Improve retention calculation after datafile deletion #20350

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 5 commits into from
Jun 3, 2025
Merged

Improve retention calculation after datafile deletion #20350

merged 5 commits into from
Jun 3, 2025

Conversation

stelfrag
Copy link
Collaborator

@stelfrag stelfrag commented May 27, 2025 •
edited
Loading

Summary
  • Reduce bsearch dataset after every match
  • Detect when all metrics have been processed to avoid scanning all datafiles
  • Detect agent shutdown during retention calculation
    • Cancel retention calculation to speedup shutdown

@stelfrag stelfrag marked this pull request as ready for review May 28, 2025 06:40
@stelfrag stelfrag force-pushed the improve_retention_calculation branch from 1cabebf to eb33d17 Compare May 29, 2025 15:06
thiagoftsm
thiagoftsm previously approved these changes May 29, 2025
View reviewed changes
Copy link
Contributor

@thiagoftsm thiagoftsm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR is working as expected, LGTM!

@stelfrag stelfrag marked this pull request as draft May 30, 2025 12:10
@stelfrag stelfrag force-pushed the improve_retention_calculation branch from dd6d82e to 9eccd3c Compare June 3, 2025 13:56
@stelfrag stelfrag marked this pull request as ready for review June 3, 2025 13:56
thiagoftsm
thiagoftsm approved these changes Jun 3, 2025
View reviewed changes
Copy link
Contributor

@thiagoftsm thiagoftsm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After last commits, netdata is running as expected. LGTM!

@stelfrag stelfrag merged commit 17320a5 into netdata:master Jun 3, 2025
108 checks passed
@stelfrag stelfrag deleted the improve_retention_calculation branch June 3, 2025 20:06
mario-campos pushed a commit to mario-campos/netdata that referenced this pull request Jun 5, 2025
@stelfrag @mario-campos
Improve retention calculation after datafile deletion (netdata#20350)
8abe966 
* Improve retention calculation after datafile deletion

* No more entries in this journalfile, remaining source uuids do not match

* Quick check if we are on the uuid we want to avoid bsearch

* Make sure we have a datafile before releasing

* Add checks to prevent indexing and database rotation during agent shutdown
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@thiagoftsm thiagoftsm thiagoftsm approved these changes

@vkalintiris vkalintiris Awaiting requested review from vkalintiris vkalintiris is a code owner

Assignees
No one assigned
Labels
area/database
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@stelfrag @thiagoftsm