Implement TUN-based packet processing engine#112
Implement TUN-based packet processing engine#112DanielLavrushin wants to merge 1 commit intomainfrom
Conversation
- Added a new engine package with PacketVerdict type and Engine interface. - Introduced nfq package for engine-agnostic packet processing logic. - Created tun package to manage TUN device creation, routing, and packet processing. - Implemented routeManager for setting up and tearing down routing rules for TUN mode. - Enhanced Engine to handle TUN device operations and integrate with existing NFQUEUE logic. - Added methods for reading packets from TUN device and forwarding them based on verdicts.
|
There was a problem hiding this comment.
Pull request overview
Adds a new packet-processing backend that routes traffic through a TUN device while reusing the existing NFQUEUE worker logic, enabling “tun” mode configuration alongside the current “nfqueue” mode.
Changes:
- Introduces a TUN engine (device creation, routing/policy routing setup, TUN read loops, raw-socket forwarding).
- Refactors NFQUEUE handling to reuse a new engine-agnostic
Worker.ProcessPacket()verdict path. - Extends configuration with
queue.modeandqueue.tun.*, and updates startup/shutdown logic to select the appropriate engine.
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| src/tun/tun.go | New TUN engine: opens TUN, starts read loops, forwards packets, manages lifecycle |
| src/tun/route.go | New routing/policy-routing manager for TUN mode (setup/teardown + helpers) |
| src/tun/device.go | New low-level TUN device creation via /dev/net/tun ioctl |
| src/nfq/process.go | New engine-agnostic packet processing core returning accept/drop verdicts |
| src/nfq/nfq.go | Refactors NFQUEUE callback to delegate to ProcessPacket() and apply verdict |
| src/nfq/inc.go | Adjusts incoming handler to return engine verdict rather than setting NFQUEUE verdict directly |
| src/nfq/dns.go | Adjusts DNS handler to return engine verdict rather than setting NFQUEUE verdict directly |
| src/main.go | Chooses between NFQUEUE and TUN mode at runtime; updates graceful shutdown signature/behavior |
| src/engine/engine.go | Introduces engine.Engine interface and PacketVerdict type |
| src/config/types.go | Adds QueueConfig.Mode and TUNConfig schema |
| src/config/methods.go | Validates queue.mode and TUN-required fields |
| src/config/config.go | Adds defaults for queue.mode and queue.tun.* |
| src/config/bind.go | Removes queue-related CLI flag bindings (queue-num/threads/mark/ipv4/ipv6) |
Comments suppressed due to low confidence (1)
src/config/bind.go:12
- Queue-related CLI flags (queue-num/threads/mark/ipv4/ipv6) were removed from BindFlags and I don't see replacements elsewhere. This is a user-facing regression: running b4 with CLI overrides will no longer affect QueueConfig. Re-add these flags (and consider adding new flags for queue.mode and tun.*) or move them to another binding function and call it from init().
func (c *Config) BindFlags(cmd *cobra.Command) {
// Config path
cmd.Flags().StringVar(&c.ConfigPath, "config", c.ConfigPath, "Path to config file")
// System configuration
cmd.Flags().IntVar(&c.System.Tables.MonitorInterval, "tables-monitor-interval", c.System.Tables.MonitorInterval, "Tables monitor interval in seconds (default 10, 0 to disable)")
cmd.Flags().BoolVar(&c.System.Tables.SkipSetup, "skip-tables", c.System.Tables.SkipSetup, "Skip iptables/nftables setup on startup")
cmd.Flags().BoolVar(&c.System.Tables.Masquerade, "masquerade", c.System.Tables.Masquerade, "Enable NAT masquerade (useful for containers)")
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if err := e.routes.setup(); err != nil { | ||
| e.sender.Close() | ||
| e.tunFile.Close() | ||
| return err | ||
| } |
There was a problem hiding this comment.
If routes.setup() returns an error after partially applying system changes (addr add/link up/ip rule/route/default route replace), Engine.Start() currently returns without attempting any rollback. This can leave the host routing in a broken state when startup fails. Consider calling e.routes.teardown() (or adding a rollback mechanism inside setup()) on error before returning.
| for _, w := range e.pool.Workers { | ||
| if err := w.InitSender(); err != nil { | ||
| return err | ||
| } | ||
| } |
There was a problem hiding this comment.
If InitSender() fails for one of the workers, Start() returns immediately but any senders already initialized for earlier workers are left open. Track which workers were initialized and close their senders on error (or defer a cleanup that runs unless Start completes).
| // Save current default route for restoration | ||
| out, err := run("ip", "route", "show", "default") | ||
| if err != nil { | ||
| return fmt.Errorf("failed to read current default route: %w", err) | ||
| } | ||
| r.savedDefault = strings.TrimSpace(out) | ||
| log.Infof("TUN: saved default route: %s", r.savedDefault) |
There was a problem hiding this comment.
Saving the default route as a raw string from ip route show default can produce multiple lines (multiple default routes) or include fields that strings.Fields later flattens incorrectly. This can make teardown fail to restore routing (and potentially break connectivity). Consider capturing/restoring only the first line, or parsing the output into a structured representation and restoring each default route line-by-line.
| // 1. Configure TUN device | ||
| if _, err := run("ip", "addr", "add", r.tunAddr, "dev", r.tunName); err != nil { | ||
| return fmt.Errorf("ip addr add: %w", err) | ||
| } | ||
| if _, err := run("ip", "link", "set", r.tunName, "up"); err != nil { | ||
| return fmt.Errorf("ip link set up: %w", err) | ||
| } | ||
| if _, err := run("ip", "link", "set", r.tunName, "mtu", "1500"); err != nil { | ||
| log.Warnf("TUN: failed to set MTU: %v", err) | ||
| } | ||
|
|
||
| // 2. Policy routing: marked packets use a separate table that routes via the real interface | ||
| markStr := fmt.Sprintf("0x%x", r.mark) | ||
| tableStr := fmt.Sprintf("%d", r.routeTable) | ||
|
|
||
| // Clean up stale rules/routes from a previous run that didn't shut down cleanly | ||
| run("ip", "rule", "del", "fwmark", markStr, "lookup", tableStr) | ||
| run("ip", "route", "flush", "table", tableStr) | ||
|
|
||
| if _, err := run("ip", "rule", "add", "fwmark", markStr, "lookup", tableStr, "priority", "100"); err != nil { | ||
| return fmt.Errorf("ip rule add: %w", err) | ||
| } | ||
| if _, err := run("ip", "route", "add", "default", "via", r.outGateway, "dev", r.outIface, "table", tableStr); err != nil { | ||
| return fmt.Errorf("ip route add table: %w", err) | ||
| } | ||
|
|
||
| // 3. Replace default route to go through TUN, preserving the original source IP | ||
| // so the kernel doesn't assign the TUN address (e.g. 10.255.0.1) as source | ||
| srcIP := extractField(r.savedDefault, "src") | ||
| if srcIP != "" { | ||
| if _, err := run("ip", "route", "replace", "default", "dev", r.tunName, "src", srcIP); err != nil { | ||
| return fmt.Errorf("ip route replace default: %w", err) | ||
| } | ||
| } else { | ||
| if _, err := run("ip", "route", "replace", "default", "dev", r.tunName); err != nil { | ||
| return fmt.Errorf("ip route replace default: %w", err) | ||
| } | ||
| } |
There was a problem hiding this comment.
TUN routing setup currently only manipulates IPv4 routes (ip route ...) and assigns a default IPv4-only TUN address. If IPv6 is enabled in config, IPv6 traffic won't be routed through the TUN device and won't be processed. Either add equivalent ip -6 route/rule setup (and a configurable IPv6 TUN address), or explicitly validate/force IPv6Enabled=false in TUN mode and document that limitation.
| // Make a copy since buf is reused | ||
| raw := make([]byte, n) | ||
| copy(raw, buf[:n]) | ||
|
|
There was a problem hiding this comment.
readLoop allocates a fresh raw := make([]byte, n) for every packet. In TUN mode this will likely be on the hot path and can create significant GC pressure. Consider using a buffer pool (sync.Pool) and passing pooled buffers through processing, or using a single reader goroutine feeding a worker channel to reduce allocations.
| // extractField parses a route line for a keyword and returns the next token. | ||
| // e.g. extractField("default via 1.2.3.4 dev eth0 src 10.0.0.1", "via") => "1.2.3.4" | ||
| func extractField(routeLine, keyword string) string { | ||
| parts := strings.Fields(routeLine) | ||
| for i, p := range parts { | ||
| if p == keyword && i+1 < len(parts) { | ||
| return parts[i+1] | ||
| } | ||
| } | ||
| return "" | ||
| } | ||
|
|
||
| // extractGateway parses "default via X.X.X.X dev Y" to get the gateway IP. | ||
| func extractGateway(routeLine string) string { | ||
| return extractField(routeLine, "via") | ||
| } |
There was a problem hiding this comment.
Route parsing/restoration (extractField/extractGateway + savedDefault handling) is critical and easy to break with real-world route formats (metrics, multiple defaults, missing src, etc.), but there are no unit tests for these helpers. Add tests covering representative ip route show default outputs to prevent regressions and ensure teardown restores correctly.
| if tunEngine != nil { | ||
| log.Infof("Stopping TUN engine...") | ||
| tunEngine.Stop() | ||
| } else { | ||
| log.Infof("Stopping netfilter queue pool...") | ||
| pool.Stop() | ||
| } |
There was a problem hiding this comment.
In TUN mode the shutdown path stops tunEngine but never stops the nfq.Pool. The pool was still created (and starts DHCP + cleanup goroutines) and, more importantly, the workers' raw socket senders initialized by tunEngine.Start will never be closed and in-flight worker goroutines won't be waited on. Consider always calling pool.Stop() during shutdown (even in TUN mode), or add a dedicated pool shutdown for the non-NFQUEUE resources used by TUN mode.


feature to enable b4 work through
tun