File Descriptor leak and deadlock when BIRD termination string 0000 \n crosses the 4096-byte buffer boundary

[jbrunk1966]

this issue is linked to czerwonk/bird_exporter#139

Environment
OS: Ubuntu 26.04 LTS (resolute)

Kernel: Linux 7.0.0-27-generic x86_64

BIRD version: 2.18

Exporter version: 1.4.4 (Ubuntu Repo) AND 1.5.0 (GitHub release)

Symptom
The prometheus-bird-exporter causes a massive File Descriptor (FD) leak on the BIRD unix socket, eventually leading to a complete hang of the BIRD daemon once it hits its soft limit of 1024 open files.
The exporter process shows 0% CPU usage but continuously accumulates open FDs with every Prometheus scrape.

Root Cause Analysis (Buffer Boundary Bug)
After extensive debugging using strace, lsof, and curl, we identified the root cause as a buffer boundary parsing issue.

The exporter reads the BIRD socket in fixed chunks of 4096 bytes. BIRD terminates its responses (e.g., to show protocols all) with the string 0000 \n.
If the total byte length of BIRD's output happens to align perfectly with the buffer size (e.g., around 8192 bytes), the termination string is split across two read operations.

Here is the actual strace output of the exporter hitting the bug:

Code-Snippet
[pid 302649] read(56, "2002-Name...", 4096) = 4096 <-- Block 1
[pid 302649] read(56, "...", 4096) = 4096 <-- Block 2 (ends exactly after the first two zeros of the termination string)
[pid 302649] read(56, "00 \n", 4096) = 4 <-- Block 3 (receives the split remainder)
The Chain of Events:

The exporter receives 00 at the end of block 2, and 00 \n in block 3.

Because the termination string 0000 \n is never read as a continuous sequence in a single buffer, the exporter fails to recognize the end of the transmission.

The exporter calls epoll_pwait and waits indefinitely for more data from BIRD (causing the 0% CPU deadlock).

After a few seconds, the HTTP client (Prometheus) times out and drops the connection.

The exporter aborts the scrape but fails to close the Unix socket to BIRD, resulting in an FD leak on both sides (Exporter and BIRD).

Steps to Reproduce
This is highly dependent on the exact length of the BIRD output. In our case, the bug was triggered dynamically by BIRD's internal Since timestamp formatting:

When BIRD protocols run for >24h, the timestamp is short (2026-06-30).

When BIRD is freshly restarted, the timestamp includes milliseconds (05:21:23.226).

This formatting difference added exactly enough bytes to push the termination string precisely onto the 8192-byte boundary (2x 4096 bytes), immediately triggering the deadlock on every scrape.

Workaround
We verified this bug by artificially padding a description field in the bird.conf by a few characters (e.g., adding FIX_BUFFER_DEADLOCK_123). This shifted the BIRD output length away from the 4096-byte boundary. After a birdc configure, the exporter read the termination string in one piece, closed the sockets correctly, and the FD leak disappeared completely.

Suggested Fix
The socket reading logic should use a buffered reader (like bufio.Reader or bufio.Scanner) that safely searches for the delimiter across chunk boundaries, or properly inspect the combined buffer rather than relying on the termination string appearing completely within a single raw 4096-byte chunk. Additionally, a timeout context should ensure that the unix socket is closed (defer conn.Close()) even if the parsing hangs or the HTTP client disconnects.

[jbrunk1966]

probably this should fix the issue

--- a/bird_socket.go
+++ b/bird_socket.go
@@ -3,6 +3,7 @@
 import (
 	"net"
 	"regexp"
 	"strings"
+	"time"
 )
 
 var birdReturnCodeRegex *regexp.Regexp
@@ -87,14 +88,23 @@
 	b := make([]byte, 0)
 	buf := make([]byte, s.bufferSize)
 
-	done := false
-	for !done {
+	for {
+		// Set a read deadline to prevent indefinite hanging (epoll_pwait deadlock)
+		// ensuring the socket is eventually released if BIRD stops sending data.
+		err := conn.SetReadDeadline(time.Now().Add(10 * time.Second))
+		if err != nil {
+			return nil, err
+		}
+
 		n, err := conn.Read(buf[:])
 		if err != nil {
 			return nil, err
 		}
 
 		b = append(b, buf[:n]...)
-		done = containsActionCompletedCode(buf[:n])
+		
+		// Evaluate the aggregated buffer 'b' instead of the isolated chunk 'buf[:n]'.
+		// This ensures the regex matches even if the termination string is split across chunks.
+		if containsActionCompletedCode(b) {
+			break
+		}
 	}
 
+	// Reset the read deadline after successfully reading the data
+	conn.SetReadDeadline(time.Time{})
+
 	return b, nil
 }

[jbrunk1966]

i patched the current version and compiled bird_exporter 1.5.0 with it, looks promising so far

MacBook-Pro:bird_socket-master jbrunk$ diff -bBu bird_socket.go.bak bird_socket.go
--- bird_socket.go.bak	2026-07-08 08:11:08
+++ bird_socket.go	2026-07-08 08:15:56
@@ -4,6 +4,7 @@
 	"net"
 	"regexp"
 	"strings"
+	"time"
 )
 
 var birdReturnCodeRegex *regexp.Regexp
@@ -103,16 +104,30 @@
 	b := make([]byte, 0)
 	buf := make([]byte, s.bufferSize)
 
-	done := false
-	for !done {
+	for {
+		// Set a read deadline to prevent indefinite hanging (epoll_pwait deadlock)
+		// ensuring the socket is eventually released if BIRD stops sending data.
+		err := conn.SetReadDeadline(time.Now().Add(10 * time.Second))
+		if err != nil {
+			return nil, err
+		}
+
 		n, err := conn.Read(buf[:])
 		if err != nil {
 			return nil, err
 		}
 
 		b = append(b, buf[:n]...)
-		done = containsActionCompletedCode(buf[:n])
+
+		// Evaluate the aggregated buffer 'b' instead of the isolated chunk 'buf[:n]'.
+		// This ensures the regex matches even if the termination string is split across chunks.
+		if containsActionCompletedCode(b) {
+			break
 		}
+	}
+
+	// Reset the read deadline after successfully reading the data
+	conn.SetReadDeadline(time.Time{})
 
 	return b, nil
 }

$ go version -m bird_exporter | grep bird_socket
	dep	github.com/czerwonk/bird_socket	v1.0.1
	=>	../bird_socket-master	(devel)	

$ go list -m github.com/czerwonk/bird_socket
github.com/czerwonk/bird_socket v1.0.1 => ../bird_socket-master

# systemctl status  prometheus-bird-exporter.service -l --no-pager
● prometheus-bird-exporter.service - Prometheus exporter for the BIRD routing daemon
     Loaded: loaded (/usr/lib/systemd/system/prometheus-bird-exporter.service; enabled; preset: enabled)
     Active: active (running) since Wed 2026-07-08 06:40:22 UTC; 6min ago
 Invocation: eb31dcd339f84e38aa65cde7ec2be81d
       Docs: https://github.com/czerwonk/bird_exporter
             man:prometheus-bird-exporter(1)
   Main PID: 463090 (prometheus-bird)
      Tasks: 19 (limit: 135369)
     Memory: 15.3M (peak: 17.5M)
        CPU: 579ms
     CGroup: /system.slice/prometheus-bird-exporter.service
             └─463090 /usr/bin/prometheus-bird-exporter -bird.v2 -bird.socket=/var/run/bird/bird.ctl

Jul 08 06:40:22 myserver systemd[1]: Started prometheus-bird-exporter.service - Prometheus exporter for the BIRD routing daemon.
Jul 08 06:40:22 myserver prometheus-bird-exporter[463090]: time="2026-07-08T06:40:22Z" level=info msg="Starting bird exporter (Version: 1.5.0-bugfix)"
Jul 08 06:40:22 myserver prometheus-bird-exporter[463090]: time="2026-07-08T06:40:22Z" level=info msg="Listening for /metrics on :9324 (TLS: false)"

then i watched the FDs of bird and the bird-exporter

$ BPID=$(pgrep -x bird) ; EPID=$(pgrep -x prometheus-bird) ; watch -n 2 "echo 'BIRD FDs:'; ls /proc/$BPID/fd 2>/dev/null | wc -l; echo 'Exporter FDs:'; ls /proc/$EPID/fd 2>/dev/null | wc -l"

and performed 100 queries against the bird-exporter

$ for i in $(seq 1 100); do echo $i;  curl -s --max-time 3 http://127.0.0.1:9324/metrics > /dev/null || echo TIMEOUT; sleep 0.5; done

with the above bird-socket fix the FDs will not increase!

[jbrunk1966]

additionally you can trace the bird-exporter
# strace -ff -e read,write -s 10000 -p $(pgrep -f prometheus-bird-exporter | head -n 1)