Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Give warning of incipient overload in the mlab-ns output #140

Open
5 tasks
pboothe opened this issue May 15, 2018 · 0 comments
Open
5 tasks

Give warning of incipient overload in the mlab-ns output #140

pboothe opened this issue May 15, 2018 · 0 comments

Comments

@pboothe
Copy link
Contributor

pboothe commented May 15, 2018

We would like to provide a little nuance in our JSON output, by adding an extra field which indicates whether the system is currently near overload. Tests can still be performed when the system is near overload, but an interactive client will likely not want to run a test due to the increased probability of providing a bad user experience.

We will need to plumb this new status of through the monitoring and mlab-ns systems, which will likely be a lot of work. The very first step in this work, however, is hopefully easy: add the extra key/value pair "status": "healthy" to the JSON output. Then, development of better clients can begin ASAP, because the API will be already there, and our work on building up all the backend plumbing can proceed asynchronously in parallel.

The sequence of feature changes is:

  • 1. Add the extra key/value pair "status": "healthy" to the JSON output
  • 2. Add a new status to the monitoring system of "near-overload" to track not just whether the system is up, but also whether testing is a good idea right now.
  • 3. Add this new status to the data model for each sliver.
  • 4. Parse the new monitoring data into datastore and memcache
  • 5. Change the JSON output to use the newly-dynamic value for "status" instead of being hardcoded to "healthy"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants