Preserves every sliver tool in memcache, whether it exists in monitoring data or not. #146
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Reviewed 1 of 1 files at r1. Comments from Reviewable |
|
Issue created. #147. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, if a slivertool exists in mlab-ns's memcache, but does not exist in the results returned by Prometheus, it will be implicitly removed from memcache. If there is some sort of major failure in Prometheus, or a bug in our PromQL queries, that causes some or all slivertools to not be present in the monitoring data returned by a query, memcache may only know about some slivertools. If the failure is large, memcache may get close to empty and global traffic will possibly be directed at just the few remaining slivertools.
This PR ensures that all slivertools in memcache at the time the
check_statuscron job is run will remain in memcache, even if it means that the true status of the slivertool is unknown.This change is