feat: enable localdns hosts plugin to cache critical AKS FQDNs

[saewoni]

What type of PR is this?
/kind feature

What this PR does / why we need it:
This is adding the aks-setup-hosts.sh, aks-setup-hosts.service, and aks-service.hosts-timer to resolve the AKS critical FQDNs every 15 minutes, which are written to /etc/localdns/hosts file. This file can be mounted onto the LocalDNS corefile if EnableHostsPlugin bool from AKS-RP is passed in.

Which issue(s) this PR fixes:

Fixes #

Requirements:

• uses conventional commit messages
• includes documentation
• adds unit tests
• tested upgrade from previous version
• commits are GPG signed and Github marks them as verified

Special notes for your reviewer:

Release note:

none

[Copilot]

Pull request overview

This PR introduces a periodic “local DNS cache” for mcr.microsoft.com that writes resolved IPs into /etc/hosts.testing and wires CoreDNS/localdns to consult that file before forwarding, improving reliability and latency for MCR image pulls when LocalDNS is enabled (especially in the scriptless path).

Changes:

• Adds mcr-hosts-setup script, systemd service, and timer into the VHD build pipeline and node provisioning flow, including a new shouldEnableMCRHostsSetup helper and CSE wiring to enable the timer when LocalDNS (scriptless) is enabled.
• Updates the localdns CoreDNS template and associated tests to add a hosts /etc/hosts.testing plugin block so MCR lookups can be served from the generated hosts file before going to Azure DNS.
• Adds targeted shellspec coverage for the new mcr-hosts-setup behavior and for enabling its timer, and refreshes baked CustomData blobs used in VHD-related tests to include the new artifacts.

Reviewed changes

Copilot reviewed 31 out of 83 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
vhdbuilder/packer/vhd-image-builder-mariner*.json, vhdbuilder/packer/vhd-image-builder-flatcar*.json, vhdbuilder/packer/vhd-image-builder-cvm.json, vhdbuilder/packer/vhd-image-builder-base.json	Ensures new mcr-hosts-setup.sh, .service, and .timer artifacts are copied into /home/packer during various VHD builds so they can be installed into the image.
vhdbuilder/packer/vhd-image-builder-arm64-gen2.json	Same as above for the ARM64 Gen2 image; one object is slightly misformatted compared to the rest of the JSON.
vhdbuilder/packer/packer_source.sh	Copies mcr-hosts-setup.sh to /opt/azure/containers and installs the corresponding systemd service and timer units into /etc/systemd/system with appropriate permissions.
parts/linux/cloud-init/artifacts/mcr-hosts-setup.sh	New script that resolves A/AAAA records for mcr.microsoft.com via dig and writes them into /etc/hosts.testing, logging both summary counts and the concrete IPs.
parts/linux/cloud-init/artifacts/mcr-hosts-setup.service	Ones-shot systemd unit that runs the mcr-hosts-setup.sh script after network-online.target is reached.
parts/linux/cloud-init/artifacts/mcr-hosts-setup.timer	Systemd timer that triggers mcr-hosts-setup.service at boot and every 5 minutes thereafter, with jitter and ordering relative to localdns.service.
parts/linux/cloud-init/artifacts/cse_config.sh	Adds shouldEnableMCRHostsSetup, which uses systemctlEnableAndStart mcr-hosts-setup.timer 30 to enable/start the timer and logs descriptive messages.
parts/linux/cloud-init/artifacts/cse_main.sh	Integrates shouldEnableMCRHostsSetup into the base provisioning flow, calling it when SHOULD_ENABLE_LOCALDNS is true so the timer is only enabled alongside LocalDNS scriptless corefile generation.
pkg/agent/baker.go	Extends the LocalDNS CoreDNS template so that, when $isRootDomain is true, a hosts /etc/hosts.testing { fallthrough } block is inserted before the Azure DNS forwarder.
pkg/agent/baker_test.go	Updates expected localdns corefile strings in tests to include the new hosts /etc/hosts.testing stanza, ensuring the template change is validated.
spec/parts/linux/cloud-init/artifacts/mcr_hosts_setup_spec.sh	New shellspec tests that (by re-simulating the logic) verify hosts file generation and content based on mocked dig output; currently they do not execute the real script, which has maintainability implications.
spec/parts/linux/cloud-init/artifacts/cse_config_spec.sh	Adds tests to ensure shouldEnableMCRHostsSetup echoes the expected messages and calls systemctlEnableAndStart mcr-hosts-setup.timer 30.
pkg/agent/testdata/CustomizedImage*/CustomData	Refreshes gzipped CustomData payloads to include the new artifacts and behavior, keeping VHD-related tests aligned with the new provisioning logic.

[Copilot]

Pull request overview

Copilot reviewed 46 out of 170 changed files in this pull request and generated 3 comments.

---

You can also share your feedback on Copilot code review. Take the survey.

[Copilot]

Pull request overview

Copilot reviewed 47 out of 170 changed files in this pull request and generated 4 comments.

Comments suppressed due to low confidence (1)

e2e/validation.go:59

• for i := range maxRetries does not compile (cannot range over an int). This should be a standard counted loop (e.g., for i := 0; i < maxRetries; i++ { ... }).

func ValidatePodRunningWithRetry(ctx context.Context, s *Scenario, pod *corev1.Pod, maxRetries int) {
	var err error
	for i := range maxRetries {
		err = validatePodRunning(ctx, s, pod)
		if err != nil {
			time.Sleep(1 * time.Second)
			s.T.Logf("retrying pod %q validation (%d/%d)", pod.Name, i+1, maxRetries)
			continue

---

You can also share your feedback on Copilot code review. Take the survey.

[Copilot]

Pull request overview

Copilot reviewed 48 out of 180 changed files in this pull request and generated 2 comments.

---

You can also share your feedback on Copilot code review. Take the survey.

[Copilot]

copyPackerFiles() now unconditionally copies /home/packer/aks-hosts-setup.{sh,service,timer}. This will fail VHD builds for templates that don't upload these artifacts (e.g. vhd-image-builder-acl.json currently only uploads localdns.* and not aks-hosts-setup.), because cpAndMode exits on any copy failure. Either add these files to every relevant vhd-image-builder-.json (including ACL) or make the copy conditional on the source files existing to keep older/variant templates building.

[Copilot]

Pull request overview

Copilot reviewed 48 out of 180 changed files in this pull request and generated 2 comments.

---

You can also share your feedback on Copilot code review. Take the survey.

[Copilot]

Pull request overview

Copilot reviewed 48 out of 180 changed files in this pull request and generated 2 comments.

---

You can also share your feedback on Copilot code review. Take the survey.

[Copilot]

select_localdns_corefile waits only 30s for /etc/localdns/hosts to contain at least one mapping. aks-hosts-setup.sh does up to 2 DNS queries (A + AAAA) per FQDN with a 3s timeout each; for the public cloud list this can be ~12 queries, i.e., up to ~36s in the worst case (even before network-online delays). With the current 30s timeout, a healthy but slow first run can cause a permanent fallback to the no-hosts Corefile on boot. Consider increasing the timeout (or triggering an initial synchronous run of aks-hosts-setup.service) to better match the worst-case resolution time.

[Copilot]

The new spec expects enableAKSHostsSetup to attempt an initial run and log "Warning: Initial hosts setup failed" when the setup script exits non-zero, but enableAKSHostsSetup in cse_config.sh currently never runs aks-hosts-setup.sh (it only enables the timer and touches the hosts file). This will make this test (and any similar ones) fail. Either update enableAKSHostsSetup to perform an initial run (and emit the warning on failure) or adjust/remove this test case to match the implemented behavior.

[Copilot]

Pull request overview

Copilot reviewed 48 out of 181 changed files in this pull request and generated 1 comment.

---

You can also share your feedback on Copilot code review. Take the survey.

[Copilot]

With set -euo pipefail, the resolve_ipv4 pipeline can exit non-zero when grep finds no matches (e.g., domains with no A records). Because the pipeline isn’t guarded, that non-zero status can terminate the whole script even though “no records” should be a normal case. Make the pipeline tolerant (e.g., ensure it returns 0 when there are no matches) so missing A records just yield an empty result instead of aborting the run.

Suggested change

	done
	done || true

[Copilot]

Pull request overview

Copilot reviewed 49 out of 195 changed files in this pull request and generated 3 comments.

---

You can also share your feedback on Copilot code review. Take the survey.

[Copilot]

The comment says the generated corefile variants are "written" to specific paths (including localdns-nohosts.corefile), but this helper only returns base64 content; the actual write happens later in shell. This is misleading for future maintainers—either update the comment to reflect that this function only generates/encodes the corefile, or implement writing the no-hosts variant if that’s the intent.

Suggested change

	// The corefile with hosts plugin is written to /opt/azure/containers/localdns/localdns.corefile.
	// The corefile without hosts plugin is written to /opt/azure/containers/localdns/localdns-nohosts.corefile
	// and used as a fallback when enableAKSHostsSetup fails or when older VHDs don't have aks-hosts-setup artifacts.
	// Runtime selection happens in cse_main.sh based on the availability of /etc/localdns/hosts.
	// This helper only generates and encodes the corefile variants; it does not write them to disk.
	// The provisioning scripts (e.g., cse_main.sh) use these values to write the corefile with hosts
	// plugin to /opt/azure/containers/localdns/localdns.corefile and the corefile without hosts plugin
	// to /opt/azure/containers/localdns/localdns-nohosts.corefile, using the latter as a fallback when
	// enableAKSHostsSetup fails or when older VHDs don't have aks-hosts-setup artifacts. Runtime selection
	// happens in cse_main.sh based on the availability of /etc/localdns/hosts.

[Copilot]

enableLocalDNS hard-fails if systemctlEnableAndStart localdns fails. On older VHDs that don't ship localdns.service, this causes provisioning failure (systemctl restart retries then exits). To preserve backward compatibility, add a guard before starting/enabling localdns (e.g., check systemctl list-unit-files for localdns.service or verify the unit file exists) and skip localdns setup with a warning instead of exiting when the unit is missing.

[Copilot]

aks-hosts-setup.service is a oneshot intended to be triggered by the timer. Having WantedBy=... kubelet.service means enabling the service could also hook it to kubelet restarts/ordering, which is surprising and can cause extra DNS resolution runs unrelated to the timer. Consider limiting WantedBy to multi-user.target (or removing the install section entirely if the service is never meant to be enabled directly) and rely on enabling aks-hosts-setup.timer only.

Suggested change

	WantedBy=multi-user.target kubelet.service
	WantedBy=multi-user.target