-
Notifications
You must be signed in to change notification settings - Fork 4.6k
VAULT-28192 fix Agent and Proxy consuming large amounts of CPU for auto-auth self-healing #27518
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
229f74e
73655a3
a5eaea2
9418def
27f13f3
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| ```release-note:bug | ||
| agent: Fixed an issue causing excessive CPU usage during normal operation | ||
| ``` | ||
|
|
||
| ```release-note:bug | ||
| proxy: Fixed an issue causing excessive CPU usage during normal operation | ||
| ``` |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -246,31 +246,24 @@ func (ts *Server) Run(ctx context.Context, incoming chan string, templates []*ct | |
| ts.runner.Stop() | ||
| return nil | ||
| } | ||
| default: | ||
| // We are using default instead of a new case block to prioritize the case where <-incoming has a new value over | ||
| // receiving an error message from the consul-template server | ||
| select { | ||
| case err := <-ts.runner.ServerErrCh: | ||
| var responseError *api.ResponseError | ||
| ok := errors.As(err, &responseError) | ||
| if !ok { | ||
| ts.logger.Error("template server: could not extract error response") | ||
| continue | ||
| } | ||
| if responseError.StatusCode == 403 && strings.Contains(responseError.Error(), logical.ErrInvalidToken.Error()) && !tokenRenewalInProgress.Load() { | ||
| ts.logger.Info("template server: received invalid token error") | ||
|
|
||
| // Drain the error channel before sending a new error | ||
| select { | ||
| case <-invalidTokenCh: | ||
| default: | ||
| } | ||
| invalidTokenCh <- err | ||
| } | ||
| default: | ||
| case err := <-ts.runner.ServerErrCh: | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I wonder if this is a possible scenario -
We are now stuck in a loop where we always honor the token one behind valid token.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Hmm, that's a good point! I do think it's likely in this scenario that both tokens will be valid, but it's still not a great state to be in. I'll rework this to drain the incoming channel in the same place we drain the invalid token channel. I think that should prevent any looping
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I definitely understand why we had it the way we had it before though, but I do think this might be the best fix, and the only situation it would struggle is if we have the two channels filled exactly simultaneously
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I like this!! Thanks for adding Violet! |
||
| var responseError *api.ResponseError | ||
| ok := errors.As(err, &responseError) | ||
| if !ok { | ||
| ts.logger.Error("template server: could not extract error response") | ||
| continue | ||
| } | ||
|
|
||
| if responseError.StatusCode == 403 && strings.Contains(responseError.Error(), logical.ErrInvalidToken.Error()) && !tokenRenewalInProgress.Load() { | ||
| ts.logger.Info("template server: received invalid token error") | ||
|
|
||
| // Drain the error channel and incoming channel before sending a new error | ||
| select { | ||
| case <-invalidTokenCh: | ||
| case <-incoming: | ||
| default: | ||
| } | ||
| invalidTokenCh <- err | ||
| } | ||
| } | ||
| } | ||
| } | ||
|
|
||
Uh oh!
There was an error while loading. Please reload this page.