Skip to content

Add retry logic for bad gateway response errors#149

Merged
tr4nt0r merged 1 commit intomainfrom
bad_gateway_retry
Feb 8, 2025
Merged

Add retry logic for bad gateway response errors#149
tr4nt0r merged 1 commit intomainfrom
bad_gateway_retry

Conversation

@tr4nt0r
Copy link
Collaborator

@tr4nt0r tr4nt0r commented Feb 8, 2025

Bring are using AWS Elastic Load Balancer to distribute the load to the Bring servers. Sporadically they respond with a 502 bad gateway error. My research for the cause of this error has shown that the cause may be a misconfiguration of their servers. The keep-alive of the API servers is shorter than the keep-alive of the load balancers, which is causing the load balancers not being able route the request because the connection was already closed.
See - kubernetes-sigs/aws-load-balancer-controller#2272 (comment)

I can't fix that as it is outside of my control, but I can add a retry logic for this error.
The retry will be happen after 502, 503 and 504 status after waiting randomly for 1 ms to 5 s

@tr4nt0r tr4nt0r added the ✨ code quality Code quality improvements label Feb 8, 2025
@tr4nt0r tr4nt0r requested a review from miaucl February 8, 2025 03:13
@codecov-commenter
Copy link

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 94.04%. Comparing base (99c9e6e) to head (8d2536b).

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #149      +/-   ##
==========================================
+ Coverage   94.00%   94.04%   +0.04%     
==========================================
  Files           6        6              
  Lines         584      588       +4     
  Branches       30       31       +1     
==========================================
+ Hits          549      553       +4     
  Misses         27       27              
  Partials        8        8              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@tr4nt0r tr4nt0r added 👻 bug Something isn't working and removed ✨ code quality Code quality improvements labels Feb 8, 2025
@tr4nt0r tr4nt0r mentioned this pull request Feb 8, 2025
@miaucl
Copy link
Owner

miaucl commented Feb 8, 2025

Wow good research, lets hope it helps!👍

@tr4nt0r tr4nt0r merged commit 7ae97ce into main Feb 8, 2025
32 checks passed
@tr4nt0r tr4nt0r deleted the bad_gateway_retry branch February 8, 2025 14:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

👻 bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants