Packets are not held until tunnels are open

This needs further investigation. Ideally the tool should be entirely transparent. Delays (especially in initial packets which SSH tunnels are set up) are acceptable, but no packet should ever be incorrectly dropped, unless a s severe timeout has occurred.

danny commented

2026-07-07 14:35:01 +01:00

(Migrated from gitlab.keyop.co.uk)

mentioned in issue #31

danny commented

2026-07-07 14:35:20 +01:00

(Migrated from gitlab.keyop.co.uk)

mentioned in issue #32

danny commented

2026-07-07 14:35:36 +01:00

(Migrated from gitlab.keyop.co.uk)

mentioned in issue #33

danny commented

2026-07-07 14:36:03 +01:00

(Migrated from gitlab.keyop.co.uk)

Investigation findings and root cause.

Traced the full first-packet path through getNfCallback (internal/nfq_forwarder/nfq_forwarder.go) and internal/sshtunnel. The local tunnel listener and the DNAT rule are both genuinely, synchronously ready (listener bound+listening, DNAT rule actually present in the kernel's nat table) before the current packet's verdict is issued - there's no listener-vs-DNAT timing race in this code.

The real problem is structural: the NFQUEUE interception rule lives in filter/OUTPUT (priority 0), while the DNAT rule is inserted into nat/OUTPUT (priority -100). For a single packet's traversal of the OUTPUT hook, Linux netfilter evaluates registered hook priorities in a fixed order - nat runs before filter. Once a packet reaches the NFQUEUE target in filter/OUTPUT, nat/OUTPUT has already run for it (with no match, since the DNAT rule didn't exist yet) and cannot run again for that same packet - a verdict (ACCEPT/REPEAT/QUEUE) can only continue forward or repeat the same hook stage, never rewind to an earlier-priority table.

So the very first packet of every new destination is, structurally, unrecoverable once queued - no verdict this code can issue will make that packet benefit from the DNAT rule just added. The connection only survives today because the client's TCP stack retransmits the SYN a moment later, and that genuinely new packet correctly traverses nat -> filter and gets forwarded. Any traffic that doesn't retransmit (short client connect-timeouts, non-retrying application logic) fails outright on the first attempt, every time - not intermittently.

Proposed fix: move the NFQUEUE interception rule from filter/OUTPUT to mangle/OUTPUT (priority -150, which runs before nat/OUTPUT's -100). Once the tunnel is open and the DNAT rule is inserted, verdict with NfAccept instead of requeueing - since we're now upstream of nat/OUTPUT in the same traversal, continuing via ACCEPT lets the packet actually flow through the just-inserted DNAT rule as intended. This is a small, surgical change (one table name, one verdict constant) rather than a redesign.

Caveat: I can't verify netfilter hook-ordering behaviour against a real kernel in this sandbox - I want to confirm this with a packet capture (tcpdump + iptables -t nat -L OUTPUT -v --line-numbers timing) on a real host as part of testing.

Related bugs spun out to their own tickets (found along the way, each independently provable and worth fixing regardless of how this one is resolved): #31 (unconditional drop with no retry + a 5s blanket cooldown that drops unrelated destinations' traffic too), #32 (a malformed packet permanently kills that queue's entire read loop - silently ending all packet processing for every destination on that connection), #33 (the NfQeueue verdict this codebase relies on for redelivery actually always requeues to queue 0, not the real queue number, so it silently fails regardless of the hook-ordering issue above).

Planned order of work (agreed with Danny): this ticket (#4) first - most severe, causes guaranteed packet loss on every new connection; then #32 - narrower trigger but the worst blast radius (permanently kills a whole connection's packet processing); then #31 - real but session-bounded impact; then #33 - likely made moot for the primary path once #4's fix removes reliance on NfQeueue, but still worth auditing/fixing for any remaining call sites.

**Investigation findings and root cause.** Traced the full first-packet path through `getNfCallback` (`internal/nfq_forwarder/nfq_forwarder.go`) and `internal/sshtunnel`. The local tunnel listener and the DNAT rule are both genuinely, synchronously ready (listener bound+listening, DNAT rule actually present in the kernel's nat table) before the current packet's verdict is issued - there's no listener-vs-DNAT timing race in this code. The real problem is structural: the NFQUEUE interception rule lives in `filter/OUTPUT` (priority 0), while the DNAT rule is inserted into `nat/OUTPUT` (priority -100). For a single packet's traversal of the `OUTPUT` hook, Linux netfilter evaluates registered hook priorities in a fixed order - nat runs *before* filter. Once a packet reaches the NFQUEUE target in `filter/OUTPUT`, `nat/OUTPUT` has already run for it (with no match, since the DNAT rule didn't exist yet) and **cannot run again for that same packet** - a verdict (`ACCEPT`/`REPEAT`/`QUEUE`) can only continue forward or repeat the same hook stage, never rewind to an earlier-priority table. So the very first packet of every new destination is, structurally, unrecoverable once queued - no verdict this code can issue will make *that packet* benefit from the DNAT rule just added. The connection only survives today because the client's TCP stack retransmits the SYN a moment later, and that genuinely new packet correctly traverses nat -> filter and gets forwarded. Any traffic that doesn't retransmit (short client connect-timeouts, non-retrying application logic) fails outright on the first attempt, every time - not intermittently. **Proposed fix**: move the NFQUEUE interception rule from `filter/OUTPUT` to `mangle/OUTPUT` (priority -150, which runs *before* nat/OUTPUT's -100). Once the tunnel is open and the DNAT rule is inserted, verdict with `NfAccept` instead of requeueing - since we're now upstream of `nat/OUTPUT` in the same traversal, continuing via ACCEPT lets the packet actually flow through the just-inserted DNAT rule as intended. This is a small, surgical change (one table name, one verdict constant) rather than a redesign. Caveat: I can't verify netfilter hook-ordering behaviour against a real kernel in this sandbox - I want to confirm this with a packet capture (tcpdump + `iptables -t nat -L OUTPUT -v --line-numbers` timing) on a real host as part of testing. **Related bugs spun out to their own tickets** (found along the way, each independently provable and worth fixing regardless of how this one is resolved): #31 (unconditional drop with no retry + a 5s blanket cooldown that drops unrelated destinations' traffic too), #32 (a malformed packet permanently kills that queue's entire read loop - silently ending all packet processing for every destination on that connection), #33 (the `NfQeueue` verdict this codebase relies on for redelivery actually always requeues to queue 0, not the real queue number, so it silently fails regardless of the hook-ordering issue above). **Planned order of work** (agreed with Danny): this ticket (#4) first - most severe, causes guaranteed packet loss on every new connection; then #32 - narrower trigger but the worst blast radius (permanently kills a whole connection's packet processing); then #31 - real but session-bounded impact; then #33 - likely made moot for the primary path once #4's fix removes reliance on `NfQeueue`, but still worth auditing/fixing for any remaining call sites.

danny commented

2026-07-07 14:45:40 +01:00

(Migrated from gitlab.keyop.co.uk)

mentioned in commit b8717269e9

mentioned in commit b8717269e9287a56991b891f78f02abf6a7d5986

danny commented

2026-07-07 14:46:08 +01:00

(Migrated from gitlab.keyop.co.uk)

mentioned in merge request !70

danny commented

2026-07-07 14:46:19 +01:00

(Migrated from gitlab.keyop.co.uk)

Implementation is up for review: !70 (branch issue4_hold_packets_until_tunnel_open). Moves the NFQUEUE rule from filter/OUTPUT to mangle/OUTPUT and switches the success/already-open verdicts from NfQeueue to NfAccept, per the root-cause analysis above. Waiting on review before merging.

Implementation is up for review: !70 (branch `issue4_hold_packets_until_tunnel_open`). Moves the NFQUEUE rule from `filter/OUTPUT` to `mangle/OUTPUT` and switches the success/already-open verdicts from `NfQeueue` to `NfAccept`, per the root-cause analysis above. Waiting on review before merging.

danny commented

2026-07-07 14:48:32 +01:00

(Migrated from gitlab.keyop.co.uk)

Real-host verification procedure for this fix.

Goal: prove the very first packet of a brand-new connection now actually reaches the tunnel, rather than being silently dropped and only surviving via a kernel-level TCP retransmit ~1s later (the old, broken behaviour). Needs a real host with the fixed binary running (--debug/--verbose recommended so getNfCallback's log lines are visible) and root for tcpdump/iptables -L -v.

0. Confirm rule placement

sudo iptables -t mangle -L OUTPUT -v -n --line-numbers | grep NFQUEUE   # should show our rule
sudo iptables -t filter -L OUTPUT -v -n --line-numbers                  # should NOT show an nfq_forwarder-tagged rule (unless cleaning up a stale one from an old binary)

1. Start a capture covering both the real egress interface and loopback, for the duration of the test:

sudo tcpdump -i any -n -w /tmp/nfq_test.pcap 'tcp and port <DEST_PORT>'

2. Pick a destination:port from your config that's genuinely cold - i.e. nfq_forwarder hasn't tunnelled it since it last started (restart the service, or use an entry you know hasn't been touched yet), so there's no already-open tunnel/DNAT rule to short-circuit the test.

3. Note the 'before' packet counters on both rules:

sudo iptables -t mangle -L OUTPUT -v -n --line-numbers   # note the NFQUEUE rule's packet count
sudo iptables -t nat -L OUTPUT -v -n --line-numbers      # note count for this destination's DNAT rule (won't exist yet on a cold destination)

4. From the SAME host (nfq_forwarder only intercepts locally-originated packets, via the OUTPUT chain - the 'client' has to be a process on this same box), time a single connection attempt:

time nc -zv -w 5 <DEST_IP> <DEST_PORT>

(or curl --max-time 5 -v telnet://<DEST_IP>:<DEST_PORT> if nc isn't available)

Expected with the fix: connects in well under a second - bounded only by the SSH dial + tunnel-open time (typically tens to a couple hundred ms against a healthy jump host). A time of ~1s or more (Linux's default initial SYN retransmit timeout) would indicate the first SYN was still lost and only a retransmit got through - i.e. the bug is still present.

5. Inspect the capture for the SYNs to this destination:

tcpdump -r /tmp/nfq_test.pcap -n 'tcp[tcpflags] & tcp-syn != 0 and tcp[tcpflags] & tcp-ack == 0'

Expect exactly one SYN to the real destination IP (captured before DNAT rewrite, on the egress interface) and its SYN-ACK coming back - no second, retransmitted SYN a second or so later with the same source port/sequence number. A retransmitted SYN here means the first attempt was still lost.

6. Re-check the 'after' packet counters from step 3 - both the mangle NFQUEUE rule and the newly-inserted nat DNAT rule for this destination should show exactly +1 packet, confirming the same packet was counted by both rules in sequence - only possible if it genuinely traversed mangle -> nat in one pass, which is the whole mechanism this fix relies on.

7. Sanity-check kernel queue drops (unrelated to this fix specifically, but worth a glance given the queue-overflow risk discussed under #31):

cat /proc/net/netfilter/nfnetlink_queue

queue_dropped/queue_user_dropped (the last two numeric columns) shouldn't increase across the test.

8. Repeat step 2-6 against a second, different cold destination:port to confirm it's not a fluke.

9. Confirm the trade-off doesn't regress steady-state use: immediately open a second new connection to the same now-already-tunnelled destination:port from step 2-6, and confirm it still succeeds quickly, with getNfCallback's "Mapping already exists for ..." log line appearing (the fast userspace round-trip path) rather than any error/delay.

Optional but the strongest possible evidence: check out main at the commit just before this fix (or the running production binary, if it predates this change), repeat steps 0-6 once, and confirm you do see the ~1s+ delay and the retransmitted SYN in that case - a clean before/after contrast.

**Real-host verification procedure for this fix.** Goal: prove the *very first* packet of a brand-new connection now actually reaches the tunnel, rather than being silently dropped and only surviving via a kernel-level TCP retransmit ~1s later (the old, broken behaviour). Needs a real host with the fixed binary running (`--debug`/`--verbose` recommended so `getNfCallback`'s log lines are visible) and root for `tcpdump`/`iptables -L -v`. **0. Confirm rule placement** ``` sudo iptables -t mangle -L OUTPUT -v -n --line-numbers | grep NFQUEUE # should show our rule sudo iptables -t filter -L OUTPUT -v -n --line-numbers # should NOT show an nfq_forwarder-tagged rule (unless cleaning up a stale one from an old binary) ``` **1. Start a capture covering both the real egress interface and loopback**, for the duration of the test: ``` sudo tcpdump -i any -n -w /tmp/nfq_test.pcap 'tcp and port <DEST_PORT>' ``` **2. Pick a destination:port from your config that's genuinely cold** - i.e. nfq_forwarder hasn't tunnelled it since it last started (restart the service, or use an entry you know hasn't been touched yet), so there's no already-open tunnel/DNAT rule to short-circuit the test. **3. Note the 'before' packet counters** on both rules: ``` sudo iptables -t mangle -L OUTPUT -v -n --line-numbers # note the NFQUEUE rule's packet count sudo iptables -t nat -L OUTPUT -v -n --line-numbers # note count for this destination's DNAT rule (won't exist yet on a cold destination) ``` **4. From the SAME host** (nfq_forwarder only intercepts locally-originated packets, via the OUTPUT chain - the 'client' has to be a process on this same box), time a single connection attempt: ``` time nc -zv -w 5 <DEST_IP> <DEST_PORT> ``` (or `curl --max-time 5 -v telnet://<DEST_IP>:<DEST_PORT>` if nc isn't available) **Expected with the fix**: connects in well under a second - bounded only by the SSH dial + tunnel-open time (typically tens to a couple hundred ms against a healthy jump host). A time of ~1s or more (Linux's default initial SYN retransmit timeout) would indicate the first SYN was still lost and only a retransmit got through - i.e. the bug is still present. **5. Inspect the capture** for the SYNs to this destination: ``` tcpdump -r /tmp/nfq_test.pcap -n 'tcp[tcpflags] & tcp-syn != 0 and tcp[tcpflags] & tcp-ack == 0' ``` Expect exactly **one** SYN to the real destination IP (captured before DNAT rewrite, on the egress interface) and its SYN-ACK coming back - no second, retransmitted SYN a second or so later with the same source port/sequence number. A retransmitted SYN here means the first attempt was still lost. **6. Re-check the 'after' packet counters** from step 3 - both the `mangle` NFQUEUE rule and the newly-inserted `nat` DNAT rule for this destination should show exactly **+1 packet**, confirming the *same* packet was counted by both rules in sequence - only possible if it genuinely traversed mangle -> nat in one pass, which is the whole mechanism this fix relies on. **7. Sanity-check kernel queue drops** (unrelated to this fix specifically, but worth a glance given the queue-overflow risk discussed under #31): ``` cat /proc/net/netfilter/nfnetlink_queue ``` `queue_dropped`/`queue_user_dropped` (the last two numeric columns) shouldn't increase across the test. **8. Repeat step 2-6 against a second, different cold destination:port** to confirm it's not a fluke. **9. Confirm the trade-off doesn't regress steady-state use**: immediately open a *second* new connection to the *same* now-already-tunnelled destination:port from step 2-6, and confirm it still succeeds quickly, with `getNfCallback`'s "Mapping already exists for ..." log line appearing (the fast userspace round-trip path) rather than any error/delay. **Optional but the strongest possible evidence**: check out `main` at the commit just before this fix (or the running production binary, if it predates this change), repeat steps 0-6 once, and confirm you *do* see the ~1s+ delay and the retransmitted SYN in that case - a clean before/after contrast.

danny commented

2026-07-07 15:23:01 +01:00

(Migrated from gitlab.keyop.co.uk)

mentioned in commit 68a4eaef87

mentioned in commit 68a4eaef87b92661807a27c567ae841978592639

danny commented

2026-07-07 15:23:45 +01:00

(Migrated from gitlab.keyop.co.uk)

Real-host verification results (from testing.txt) - the fix checks out. Thanks for running this.

Rule placement (step 0/3): confirmed - the NFQUEUE rule is in mangle/OUTPUT (queue 21086, matching 10.66.16.0/20:9100), and filter/OUTPUT is clean (no leftover rule).

Packet counters (step 6) - the cleanest evidence in the whole test:

mangle NFQUEUE rule: 0 -> 1 packet
nat DNAT rule:      (didn't exist) -> 1 packet

Both show exactly +1 packet for the same connection attempt. That's only possible if the identical first packet was queued in mangle, accepted, and then actually traversed nat/OUTPUT and got DNAT'd there in the same pass - precisely the mechanism this fix is supposed to guarantee. This on its own is solid confirmation the fix works as designed.

Packet capture (step 5) - my test methodology had a gap, not a bug: the empty result for a SYN filtered on port 9100 isn't a problem - it's expected, and I should have anticipated it. DNAT rewrites the packet's destination before it's ever actually put on a real interface within that same netfilter traversal, so there's no wire-level capture point where the pre-DNAT form of that first packet (destined to port 9100) ever exists to be captured - by the time it would hit a device for real transmission, its destination is already 127.0.0.1:<tunnel-port>. The three packets that did show up (SYN-ACK/ACK/FIN between 10.66.16.2:9100 and 10.79.24.110:56562 on lo) are consistent with a working connection, just reflecting loopback+conntrack's un-DNAT view for the return direction - not something to chase further. The packet-counter comparison above is the reliable way to confirm this on a real host; I'll drop the SYN-capture step from the procedure for next time.

Timing - this is what actually nails it down. Your first run (2.999s) understandably looked like a possible problem, but once you controlled for it by restarting nfq_forwarder (forcing a genuinely cold tunnel) with the jump host's SSSD auth cache already warm, a very clean pattern emerged:

Binary	Run 1	Run 2
`main` (v1.1.0, pre-fix)	1.06s	1.01s
this branch (fix)	0.28s	0.24s

~1.0-1.06s is essentially Linux's default initial TCP SYN retransmission timeout - that's the signature of a silently dropped first SYN, only rescued by the kernel's automatic retransmit a second later. That's exactly the bug this ticket describes. The fix consistently completes in ~0.25s - no retransmit-shaped delay at all, roughly 4x faster, and consistent across both fixed-branch runs.

The initial 2.999s reading was a real but unrelated confound: an SSSD lookup delay on the jump host's own PAM auth after a period of inactivity, not anything in this code - it's why the clean comparison above always restarts nfq_forwarder (forcing a fresh tunnel dial) rather than the jump host's own auth state.

Conclusion: both lines of evidence (packet counters and timing) independently confirm the fix resolves the reported bug. I've recorded this verification in CLAUDE.md's Known Issues checklist (commit 68a4eae on this branch) so it's not re-litigated later, and noted the tcpdump methodology gap there too. Ready for your final review/merge whenever you're happy.

**Real-host verification results (from `testing.txt`) - the fix checks out.** Thanks for running this. **Rule placement (step 0/3):** confirmed - the NFQUEUE rule is in `mangle`/OUTPUT (queue 21086, matching `10.66.16.0/20:9100`), and `filter`/OUTPUT is clean (no leftover rule). **Packet counters (step 6) - the cleanest evidence in the whole test:** ``` mangle NFQUEUE rule: 0 -> 1 packet nat DNAT rule: (didn't exist) -> 1 packet ``` Both show exactly **+1 packet** for the same connection attempt. That's only possible if the *identical* first packet was queued in `mangle`, accepted, and then actually traversed `nat`/OUTPUT and got DNAT'd there in the same pass - precisely the mechanism this fix is supposed to guarantee. This on its own is solid confirmation the fix works as designed. **Packet capture (step 5) - my test methodology had a gap, not a bug:** the empty result for a SYN filtered on port 9100 isn't a problem - it's expected, and I should have anticipated it. DNAT rewrites the packet's destination *before* it's ever actually put on a real interface within that same netfilter traversal, so there's no wire-level capture point where the pre-DNAT form of that first packet (destined to port 9100) ever exists to be captured - by the time it would hit a device for real transmission, its destination is already `127.0.0.1:<tunnel-port>`. The three packets that *did* show up (SYN-ACK/ACK/FIN between `10.66.16.2:9100` and `10.79.24.110:56562` on `lo`) are consistent with a working connection, just reflecting loopback+conntrack's un-DNAT view for the return direction - not something to chase further. The packet-counter comparison above is the reliable way to confirm this on a real host; I'll drop the SYN-capture step from the procedure for next time. **Timing - this is what actually nails it down.** Your first run (2.999s) understandably looked like a possible problem, but once you controlled for it by restarting `nfq_forwarder` (forcing a genuinely cold tunnel) with the jump host's SSSD auth cache already warm, a very clean pattern emerged: | Binary | Run 1 | Run 2 | |---|---|---| | `main` (v1.1.0, pre-fix) | 1.06s | 1.01s | | this branch (fix) | 0.28s | 0.24s | ~1.0-1.06s is essentially Linux's default initial TCP SYN retransmission timeout - that's the signature of a *silently dropped first SYN*, only rescued by the kernel's automatic retransmit a second later. That's exactly the bug this ticket describes. The fix consistently completes in ~0.25s - no retransmit-shaped delay at all, roughly 4x faster, and consistent across both fixed-branch runs. The initial 2.999s reading was a real but unrelated confound: an SSSD lookup delay on the jump host's own PAM auth after a period of inactivity, not anything in this code - it's why the clean comparison above always restarts `nfq_forwarder` (forcing a fresh tunnel dial) rather than the jump host's own auth state. **Conclusion**: both lines of evidence (packet counters and timing) independently confirm the fix resolves the reported bug. I've recorded this verification in `CLAUDE.md`'s Known Issues checklist (commit `68a4eae` on this branch) so it's not re-litigated later, and noted the tcpdump methodology gap there too. Ready for your final review/merge whenever you're happy.

danny commented

2026-07-07 15:28:23 +01:00

(Migrated from gitlab.keyop.co.uk)

mentioned in commit a6eaee7e01

mentioned in commit a6eaee7e0195abdfbfd745288a382835f87acb16

danny (Migrated from gitlab.keyop.co.uk) closed this issue

2026-07-07 15:28:23 +01:00

danny commented

2026-07-07 15:29:32 +01:00

(Migrated from gitlab.keyop.co.uk)

Merged via !70 (merge commit a6eaee7) and released as v1.1.1 (a patch bump - a bug fix, unlike v1.1.0's feature addition). Confirmed on a real host via packet-counter and timing evidence (see above).

Merged via !70 (merge commit `a6eaee7`) and released as **v1.1.1** (a patch bump - a bug fix, unlike v1.1.0's feature addition). Confirmed on a real host via packet-counter and timing evidence (see above).

danny commented

2026-07-07 15:33:39 +01:00

(Migrated from gitlab.keyop.co.uk)

mentioned in merge request !71