Fix monit api iptables rules to prevent LAST-ACK

[selzoc]

Description

This PR fixes a regression introduced in commit 4dfe37a31 (Add cgroups v2 support for Jammy stemcells) that causes intermittent Cannot connect to the monit daemon errors during BOSH deployments.

# cat /var/vcap/bosh/etc/stemcell_version  && echo
1.1123
# sudo iptables -t mangle -L POSTROUTING -n -v --line-numbers
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1      318 22831 ACCEPT     tcp  --  *      *       0.0.0.0/0            127.0.0.1            tcp dpt:2822 state RELATED,ESTABLISHED
2        0     0 DROP       tcp  --  *      *       0.0.0.0/0            127.0.0.1            tcp dpt:2822 cgroup ! 2958295041

# cat /var/vcap/bosh/etc/stemcell_version  && echo
1.1298
# sudo iptables -t mangle -L POSTROUTING -n -v --line-numbers
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 DROP       tcp  --  *      *       0.0.0.0/0            127.0.0.1            tcp dpt:2822 cgroup ! 2958295041
2      602 43163 ACCEPT     tcp  --  *      *       0.0.0.0/0            127.0.0.1            tcp dpt:2822 state RELATED,ESTABLISHED

The Bug:
The previous commit refactored the iptables rules in restrict-monit-api-access to be idempotent, but accidentally reversed the insertion order. Both rules used -I (insert at position 1), which caused the DROP rule to be evaluated before the ESTABLISHED,RELATED -j ACCEPT rule.
Because of this, TCP RESET packets sent by the kernel to clean up orphaned connections were dropped (since the kernel does not belong to the monit-api-access cgroup). This left sockets lingering in the LAST-ACK state for ~106 seconds. If a subsequent monit command was assigned the same ephemeral port, it would fail with a "Connection Refused" error.

This results in deployments failing with:

Stopping Monitored Services: Stop all services: Running command: 'monit stop -g vcap', stdout: '', stderr: 'monit-actual: Cannot connect to the monit daemon. Did you start it with http support?
monit-actual: Cannot connect to the monit daemon. Did you start it with http support?

We can observe the LAST-ACK connections in a real-vm deployment scenario. In certain CI environments, this makes the deployment fail.

The Fix:
This PR updates the script to insert the rules in the correct order

Testing

Verified on a live VM that:

1. The iptables rules are now in the correct order (ACCEPT is rule 1, DROP is rule 2).
2. We reduced the amount of available ephemeral ports to 5, and ran bosh stop and bosh start in a loop, and observed several connections stuck in LAST-ACK that resulted in some failures (watch "ss -tan '( sport = :2822 or dport = :2822 )'")

AI analysis of why this happens

Here is the exact sequence of events that causes the kernel to send the TCP RST (which then gets dropped by the firewall):

1. The Keep-Alive Pool
   When the BOSH agent polls monit (e.g., during StopAndWait), it makes an HTTP GET request to http://127.0.0.1:2822/_status2.

Because it uses Go's default http.Client, it uses HTTP Keep-Alive. After the BOSH agent receives the XML response, it does not close the TCP connection. Instead, the Go runtime puts that idle TCP connection into a connection pool, hoping to reuse it for the next poll 500ms later.

2. The Server Closes the Connection
   monit is a very simple HTTP server. It does not support HTTP Keep-Alive by default, or it has a very aggressive timeout for idle connections.

Shortly after sending the XML response, the monit daemon decides it is done with the transaction and initiates a TCP connection close.

monit sends a TCP FIN packet to the BOSH agent.
The BOSH agent's OS kernel receives the FIN, sends an ACK, and puts the socket into CLOSE-WAIT.
The monit daemon's socket goes into FIN-WAIT-2.
3. The Client Discovers the Dead Connection
Meanwhile, the Go runtime inside the BOSH agent still thinks this connection is sitting happily in its idle connection pool.

When the BOSH agent wakes up 500ms later to poll monit again, the Go runtime pulls that connection out of the pool and tries to write the new HTTP GET request to it.

4. The Kernel Sends the RST
   When the Go runtime tries to write data to a socket that the server has already started closing (the socket is in CLOSE-WAIT), the Linux kernel immediately realizes the connection is invalid.

The kernel's TCP stack reacts by aborting the connection. It generates a TCP RST (Reset) packet and sends it to monit to say, "Abort! The application tried to write to a closed pipe!"

5. The Firewall Drops the RST
   Because this RST packet is generated asynchronously by the kernel's TCP stack in response to an invalid write, the packet is not tagged with the BOSH agent's cgroup (monit-api-access).

The packet hits the iptables rules. Because the rules are out of order (the DROP rule is evaluated first), the firewall drops the RST packet.

6. The Server Gets Stuck in LAST-ACK
   Because the RST packet was dropped, the monit daemon never receives it.

monit eventually sends its final FIN packet, transitioning its socket to LAST-ACK, waiting for an acknowledgment that will never come (because the firewall is dropping it). The socket sits there for ~106 seconds until the kernel's TCP retransmission timer finally gives up and clears it.

[coderabbitai]

Walkthrough

The script now documents the required ordering of loopback Monit firewall rules. The ESTABLISHED,RELATED ACCEPT rule must precede the cgroup-based DROP rule. When either rule is missing, the script appends it with iptables -A. Existing-rule checks remain idempotent.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the monit iptables rule fix and the LAST-ACK issue addressed by the pull request.
Description check	✅ Passed	The description explains the regression, root cause, fix, impact, and testing with relevant command output and technical detail.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/monit-2822-rule-order

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@stemcell_builder/stages/bosh_monit/assets/restrict-monit-api-access`:
- Around line 27-36: Update the managed iptables rule logic in
restrict-monit-api-access so each run guarantees the ESTABLISHED,RELATED ACCEPT
rule precedes the cgroup-based DROP rule, rather than only checking rule
presence. Remove and re-add the managed DROP rule or rebuild both managed rules
in the correct order, while preserving idempotency; add coverage for
pre-existing reversed and DROP-only states.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 1e53023c-09ae-47dc-b63a-aa20e97c2d89

📥 Commits

Reviewing files that changed from the base of the PR and between f40c278 and 382ac0b.

📒 Files selected for processing (1)

• stemcell_builder/stages/bosh_monit/assets/restrict-monit-api-access

[selzoc]

Confirmed on a candidate stemcell that these changes result in the correct order of the rules:

# cat /var/vcap/bosh/etc/stemcell_version  && echo
1.1330
# sudo iptables -t mangle -L POSTROUTING -n -v --line-numbers
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1      271 19531 ACCEPT     tcp  --  *      *       0.0.0.0/0            127.0.0.1            tcp dpt:2822 state RELATED,ESTABLISHED
2        0     0 DROP       tcp  --  *      *       0.0.0.0/0            127.0.0.1            tcp dpt:2822 cgroup ! 2958295041