fix: use hickory-dns to avoid musl getaddrinfo failures in Kubernetes#2054
Open
bsquizz wants to merge 1 commit into
Open
fix: use hickory-dns to avoid musl getaddrinfo failures in Kubernetes#2054bsquizz wants to merge 1 commit into
bsquizz wants to merge 1 commit into
Conversation
The supervisor binary is statically linked with musl libc, whose getaddrinfo sends A+AAAA queries simultaneously on a single UDP socket. With Kubernetes' default ndots:5 and multiple search domains, external FQDNs get expanded through all search domains first, generating 12+ concurrent queries whose responses arrive out of order — causing musl to return zero usable addresses and the router to emit 503 "inference service unavailable". Enable the `hickory-dns` feature on reqwest so all HTTP clients use a pure-Rust DNS resolver instead of musl's getaddrinfo. This bypasses the concurrent A+AAAA issue entirely and aligns with the supervisor's static-binary design (no libc dependency for DNS). Fixes NVIDIA#2053 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
bsquizz
requested review from
a team,
derekwaynecarr,
maxamillion and
mrunalp
as code owners
June 30, 2026 06:36
|
Thank you for your interest in contributing to OpenShell, @bsquizz. This project uses a vouch system for first-time contributors. Before submitting a pull request, you need to be vouched by a maintainer. To get vouched:
See CONTRIBUTING.md for details. |
|
All contributors have signed the DCO ✍️ ✅ |
Author
|
I have read the DCO document and I hereby sign the DCO. |
|
Label |
Collaborator
|
/ok to test 9a2b333 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Enable the
hickory-dnsfeature on reqwest so the supervisor's router uses a pure-Rust DNS resolver instead of musl'sgetaddrinfo.Problem
The supervisor is statically linked with musl libc. musl's
getaddrinfosends A and AAAA DNS queries simultaneously on a single UDP socket. With Kubernetes' defaultndots:5and 5 search domains, external FQDNs likeaiplatform.googleapis.com(3 dots < 5) get expanded through all search domains first — 12+ concurrent queries whose responses arrive out of order, causing musl to return zero usable addresses. This manifests as 503"inference service unavailable"on every inference call to external providers (Vertex AI, Anthropic API, etc.).Fix
One-line change in workspace
Cargo.toml:hickory-dns(formerly trust-dns) is a pure-Rust async DNS resolver. It bypasses musl'sgetaddrinfoentirely, eliminating the concurrent A+AAAA query issue regardless ofndotsconfiguration. This also aligns with the supervisor's static-binary design — no libc dependency for DNS resolution.Verification
cargo check -p openshell-sandbox— compiles cleanlycargo check -p openshell-router— compiles cleanlyFixes #2053