Program routes and policy rules using netlink, not iptables binary #391

danderson · 2020-05-15T17:56:52Z

The linux router currently programs the linux network stack with ip and iptables. iptables is unavoidable because the corresponding netlink APIs are not safe for direct use. However, everything we do with ip we could just do directly over a netlink socket.

This would in particular help embedded systems that ship with the Busybox version of ip, which is stripped down and doesn't understand advanced commands like policy routing rules, which Tailscale requires.

The text was updated successfully, but these errors were encountered:

stapelberg · 2020-05-27T17:08:18Z

iptables is unavoidable because the corresponding netlink APIs are not safe for direct use.

Can you expand on this? Which APIs do you need, and in which way are they unsafe?

I was about to open an issue suggesting to use https://github.com/google/nftables/ where available to avoid the iptables dependency. This would help make tailscale work on https://gokrazy.org/, where only Go is available (no C userland, i.e. no iptables).

danderson · 2020-05-27T17:49:12Z

For one, we can't depend on nftables, because we need to support OSes that haven't switched to nftables under the hood (e.g. pre-Buster debian), and we have to be cross-compatible with other software that doesn't support nftables (e.g. Docker, Kubernetes). The main way we can do that safely is to continue using the iptables binary, which will do "the right thing" based on OS/administrator policy.

And the legacy iptables is unsafe to use over netlink, because the API does replacement of the entire firewall at once, and doesn't have a way of preventing concurrent writes from squashing each other. The userspace tools work around this (poorly) with file-based locks, which already goes wrong in situations like containers.

If we can (a) positively detect that nftables, and only nftables is in use on the system; (b) we can manipulate the "legacy iptables converted to nftables" rules in nftables without conflicting with iptables itself... Then we could possibly make that work. But it's a bunch of work that, frankly, only makes sense for something like gokrazy, and not really for any other OS. So, that's going to be pretty low priority from my POV.

stapelberg · 2020-05-27T18:24:32Z

Thanks for the additional details, I agree with your view point.

I expect that using nftables directly might become more attractive in a few years as more and more installations run on the legacy-converted-to-nftables model.

Would an nftables fallback code path be something that you would accept if people contributed it, or is there a lot of churn and hence maintenance burden in that part of the tailscale code base?

danderson · 2020-05-27T18:32:31Z

I would love to not depend on iptables where possible. The part I'm unsure about is whether we can reliably detect that yes, it's definitely safe to use nftables directly without breaking the system.

If we can figure out how to make that part work reliably, I'd be very happy to have logic that uses nftables directly where possible, and falls back to shelling out to iptables otherwise.

There is definitely a maintenance burden, and given that we support distros like RHEL that take nearly a decade to drop support, I don't think iptables is going anywhere any time soon, sadly. I still think it's worth doing, if you want to open that particular can of worms :). Otherwise, I think we'll get to it "someday", but once we get the iptables logic stable it's going to be low priority.

bradfitz · 2021-10-14T16:33:01Z

Before I lose this, adding @stapelberg's pointers from https://twitter.com/zekjur/status/1427356540072255501:

https://github.com/rtr7/router7/blob/5869922efbe39c4911b18cb417f9e266458be879/internal/netconfig/netconfig.go#L301

Which uses https://pkg.go.dev/github.com/vishvananda/netlink

Updates #3060 Updates #391 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>

Updates #3060 Updates #391 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com> (cherry picked from commit 14f9c75)

Converts up, down, add/del addresses, add/del non-throw routes. Not yet done: throw routes, rules. Updates #391 Change-Id: I02554ca07046d18f838e04a626ba99bbd35266fb Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>

Converts up, down, add/del addresses, add/del routes. Not yet done: rules. Updates #391 Change-Id: I02554ca07046d18f838e04a626ba99bbd35266fb Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>

No non-test changes. Updates #391 Change-Id: Ia88610c08e07a119d002e58250463cb4659b9f54 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>

Pull out the list of policy routing rules to a data structure now shared between the add & delete paths, but to also be shared by the netlink paths in a future change. Updates #391 Change-Id: I119ab1c246f141d639006c808b61c585c3d67924 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>

bradfitz · 2021-10-29T05:04:05Z

I sent vishvananda/netlink#711 to add the support for the unreachable rule we use. After that we're down to "just" iptables/nftables. 🥲

Updates #391 Change-Id: I6e1de96cf0750ccba53dabff670aca0c56dffb7c Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>

Updates #391 Change-Id: Ifef196b31dd145f424fb0c0d0bb04565cc22c717 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>

Due to the conflict between our nftables implementation and ufw, which is a common utility used on linux. We now want to take a step back to prevent regression. This will give us more chance to let users to test our nftables support and heuristic. Updates: #391 Signed-off-by: KevinLiang10 <kevinliang@tailscale.com> (cherry picked from commit 93cab56)

This commit tries to mimic the way iptables-nft work with the filewall rules. We follow the convention of using tables like filter, nat and the conventional chains, to make our nftables implementation work with ufw. Updates: #391 Signed-off-by: KevinLiang10 <kevinliang@tailscale.com> (cherry picked from commit b040094)

Updates #391 Change-Id: Ifef196b31dd145f424fb0c0d0bb04565cc22c717 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com> (cherry picked from commit 282dad1)

* wgengine/router: fall back and set iptables as default again Due to the conflict between our nftables implementation and ufw, which is a common utility used on linux. We now want to take a step back to prevent regression. This will give us more chance to let users to test our nftables support and heuristic. Updates: #391 Signed-off-by: KevinLiang10 <kevinliang@tailscale.com> (cherry picked from commit 93cab56) * util/linuxfw: reorganize nftables rules to allow it to work with ufw This commit tries to mimic the way iptables-nft work with the filewall rules. We follow the convention of using tables like filter, nat and the conventional chains, to make our nftables implementation work with ufw. Updates: #391 Signed-off-by: KevinLiang10 <kevinliang@tailscale.com> (cherry picked from commit b040094) * tailcfg: update docs on NetInfo.FirewallMode Updates #391 Change-Id: Ifef196b31dd145f424fb0c0d0bb04565cc22c717 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com> (cherry picked from commit 282dad1) --------- Co-authored-by: KevinLiang10 <kevinliang@tailscale.com> Co-authored-by: Brad Fitzpatrick <bradfitz@tailscale.com>

raggi · 2023-08-21T20:36:49Z

An update on status:

v1.48.1 has been released and will default to iptables again.

You can test out the detection logic which will pick between nftables and iptables based on usage, by setting: TS_DEBUG_FIREWALL_MODE=auto for example in /etc/default/tailscaled. You can also explicitly force the mode with TS_DEBUG_FIREWALL_MODE=nftables or TS_DEBUG_FIREWALL_MODE=iptables.

There is additionally a fix in 1.48.1 that resolves the conflict with ufw that lead to forwarding traffic (subnet routers and exit nodes) from functioning properly under nftables mode. We would appreciate wider testing of the nftables mode to ensure broad coverage, as the integration requires coordination with conventions that are not standards in the platform. As of 1.48.1, in ntables mode Tailscale inserts rules into the tables used by iptables-nft and ufw in order to coordinate with those ecosystems.

We hope to re-enable firewall mode auto-detection in a future release.

hj-collab · 2023-08-22T12:41:05Z

@hj-collab, we're not able to reproduce. Can you file a new bug with a lot more details?

Not able to reproduce with 1.48.1. I'll report back if I face this issue again.

zlepper · 2023-08-24T07:16:57Z

It is working for me on version v1.48.1-t0e9f04c83 now, with TS_DEBUG_FIREWALL_MODE=auto env set when running on Azure kubernetes (AKS) and the tailscale k8s operator.

Signed-off-by: Zephyr Lykos <git@mochaa.ws>

Add an explicit accept rule for input to the tun interface, as a mirror to the explicit rule to accept output from the tun interface. The rule matches any packet in to our tun interface and accepts it, and the rule is positioned and prioritized such that it should be evaluated prior to conventional ufw/iptables/nft rules. Updates #391 Fixes #7332 Updates #9084 Signed-off-by: James Tucker <james@tailscale.com>

adrianmihalko · 2023-08-31T09:27:00Z

In https://tailscale.com/kb/1294/firewall-mode/ guide

"When neither the iptables binary nor the nftables Netlink API are available, Tailscale will fall back to a degraded operation that may result in reduced performance or increased CPU usage. "

this is still true? For me it doesn't fall back to degraded operation, instead it just exiting.

wgengine.NewUserspaceEngine(tun "tailscale0") error: creating router: exec: "iptables": executable file not found in $PATH

rodrigc · 2023-09-02T17:57:37Z

I ran a quick test.

Spun up a Kubernetes cluster in Oracle Cloud
Used image: docker.io/tailscale/tailscale:unstable-v1.49.189
Created a simple tailscale-test.yaml for testing:

---
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: tailscale
  name: tailscale
spec:
  containers:
    - args:
        - /bin/sh
        - -c
        - echo hello;sleep 36000
      image: docker.io/tailscale/tailscale:unstable-v1.49.189
      name: tailscale
      securityContext:
        privileged: true
      resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Never
status: {}

Applied the yaml:

kubectl apply -f tailscale-test.yaml

Jumped into the container:

kubectl exec -t -i tailscale -- sh

Set the heuristic to auto and started tailscaled:

export TS_DEBUG_FIREWALL_MODE=auto
tailscaled

I saw that the heurstic correctly identified nftables:

logpolicy: using system state directory "/var/lib/tailscale"
logpolicy.ConfigFromFile /var/lib/tailscale/tailscaled.log.conf: open /var/lib/tailscale/tailscaled.log.conf: no such file or directory
logpolicy.Config.Validate for /var/lib/tailscale/tailscaled.log.conf: config is nil
wgengine.NewUserspaceEngine(tun "tailscale0") ...
router: nftables rule count: 0, iptables rule count: 0
router: nftables is available
router: using nftables
router: disabling tunneled IPv6 due to system IPv6 config: %!w(*errors.errorString=&{disable_ipv6 is set})

Verified that this image was being used on the Oracle side:
Oracle-Linux-8.8-2023.06.30-0-OKE-1.26.2-632

Add an explicit accept rule for input to the tun interface, as a mirror to the explicit rule to accept output from the tun interface. The rule matches any packet in to our tun interface and accepts it, and the rule is positioned and prioritized such that it should be evaluated prior to conventional ufw/iptables/nft rules. Updates #391 Fixes #7332 Updates #9084 Signed-off-by: James Tucker <james@tailscale.com>

This change is introducing new netfilterRunner interface and moving iptables manipulation to a lower leveled iptables runner. For tailscale#391 Signed-off-by: KevinLiang10 <kevinliang@tailscale.com> Signed-off-by: Alex Paguis <alex@windscribe.com>

Exclide GOARCHs including: mips, mips64, mips64le, mipsle, riscv64. These archs are not supported by gvisor.dev/gvisor/pkg/hostarch. Fixes: tailscale#391 Signed-off-by: KevinLiang10 <kevinliang@tailscale.com> Signed-off-by: Alex Paguis <alex@windscribe.com>

This commit adds nftable rule injection for tailscaled. If tailscaled is started with envknob TS_DEBUG_USE_NETLINK_NFTABLES = true, the router will use nftables to manage firewall rules. Updates: tailscale#391 Signed-off-by: KevinLiang10 <kevinliang@tailscale.com> Signed-off-by: Alex Paguis <alex@windscribe.com>

Due to the conflict between our nftables implementation and ufw, which is a common utility used on linux. We now want to take a step back to prevent regression. This will give us more chance to let users to test our nftables support and heuristic. Updates: tailscale#391 Signed-off-by: KevinLiang10 <kevinliang@tailscale.com> Signed-off-by: Alex Paguis <alex@windscribe.com>

This commit tries to mimic the way iptables-nft work with the filewall rules. We follow the convention of using tables like filter, nat and the conventional chains, to make our nftables implementation work with ufw. Updates: tailscale#391 Signed-off-by: KevinLiang10 <kevinliang@tailscale.com> Signed-off-by: Alex Paguis <alex@windscribe.com>

Updates tailscale#391 Change-Id: Ifef196b31dd145f424fb0c0d0bb04565cc22c717 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com> Signed-off-by: Alex Paguis <alex@windscribe.com>

Add an explicit accept rule for input to the tun interface, as a mirror to the explicit rule to accept output from the tun interface. The rule matches any packet in to our tun interface and accepts it, and the rule is positioned and prioritized such that it should be evaluated prior to conventional ufw/iptables/nft rules. Updates tailscale#391 Fixes tailscale#7332 Updates tailscale#9084 Signed-off-by: James Tucker <james@tailscale.com> Signed-off-by: Alex Paguis <alex@windscribe.com>

danderson added L2 Few Likelihood P3 Can't get started Priority level T8 Crash Issue type labels May 15, 2020

This was referenced May 27, 2020

Improve support for running as a non-root user #77

Open

Implement WiFi setup gokrazy/gokrazy#13

Closed

DentonGentry added T0 New feature Issue type and removed T8 Crash Issue type labels May 20, 2021

bradfitz added a commit that referenced this issue Oct 14, 2021

wgengine/router: ignore Linux ip route error adding dup route

4a86cee

Updates #3060 Updates #391 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>

bradfitz added a commit that referenced this issue Oct 14, 2021

wgengine/router: ignore Linux ip route error adding dup route

14f9c75

Updates #3060 Updates #391 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>

maisem pushed a commit that referenced this issue Oct 14, 2021

wgengine/router: ignore Linux ip route error adding dup route

9df2516

Updates #3060 Updates #391 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com> (cherry picked from commit 14f9c75)

bradfitz added the OS-linux label Oct 28, 2021

bradfitz mentioned this issue Oct 28, 2021

wgengine/router: start using netlink instead of 'ip' on Linux #3209

Merged

bradfitz added a commit that referenced this issue Oct 28, 2021

wgengine/router: add debug test to enumerate rules

d36c0d3

No non-test changes. Updates #391 Change-Id: Ia88610c08e07a119d002e58250463cb4659b9f54 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>

bradfitz mentioned this issue Oct 28, 2021

wgengine/router: refactor in prep for Linux netlink-ification #3213

Merged

bradfitz added a commit that referenced this issue Oct 29, 2021

wgengine/router: use netlink for ip rules on Linux

b4dcac3

Updates #391 Change-Id: I6e1de96cf0750ccba53dabff670aca0c56dffb7c Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>

bradfitz mentioned this issue Oct 29, 2021

wgengine/router: use netlink for ip rules on Linux #3218

Merged

bradfitz added a commit that referenced this issue Nov 1, 2021

wgengine/router: use netlink for ip rules on Linux

c0840a8

Updates #391 Change-Id: I6e1de96cf0750ccba53dabff670aca0c56dffb7c Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>

bradfitz added a commit that referenced this issue Aug 20, 2023

tailcfg: update docs on NetInfo.FirewallMode

282dad1

Updates #391 Change-Id: Ifef196b31dd145f424fb0c0d0bb04565cc22c717 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>

awly pushed a commit that referenced this issue Aug 21, 2023

tailcfg: update docs on NetInfo.FirewallMode

6dc0d51

Updates #391 Change-Id: Ifef196b31dd145f424fb0c0d0bb04565cc22c717 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com> (cherry picked from commit 282dad1)

DentonGentry mentioned this issue Aug 24, 2023

Tailscale-operator not working on k3s cluster #8733

Closed

yfhyou referenced this issue in openwrt/packages Aug 24, 2023

tailscale: Update to 1.48.1

a9373d6

Signed-off-by: Zephyr Lykos <git@mochaa.ws>

raggi mentioned this issue Aug 25, 2023

netfilter: add missing input accepts for both tailscale0 and the bound port #9084

Open

This was referenced Aug 25, 2023

util/linuxfw: add missing input rule to the tailscale tun #9085

Merged

FR: Support Firewalld adding tailscale0 to a zone #3416

Open

oliverpool mentioned this issue Sep 5, 2023

FR: debian package: move iptables from Depends to Recommends #9236

Closed

chennin mentioned this issue Sep 11, 2023

Docs: "Guidance for hardening Tailscale nodes" systemd issues #9344

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Program routes and policy rules using netlink, not iptables binary #391

Program routes and policy rules using netlink, not iptables binary #391

danderson commented May 15, 2020

stapelberg commented May 27, 2020

danderson commented May 27, 2020

stapelberg commented May 27, 2020

danderson commented May 27, 2020

bradfitz commented Oct 14, 2021

bradfitz commented Oct 29, 2021

raggi commented Aug 21, 2023

hj-collab commented Aug 22, 2023

zlepper commented Aug 24, 2023 •

edited

adrianmihalko commented Aug 31, 2023 •

edited

rodrigc commented Sep 2, 2023 •

edited

Program routes and policy rules using netlink, not iptables binary #391

Program routes and policy rules using netlink, not iptables binary #391

Comments

danderson commented May 15, 2020

stapelberg commented May 27, 2020

danderson commented May 27, 2020

stapelberg commented May 27, 2020

danderson commented May 27, 2020

bradfitz commented Oct 14, 2021

bradfitz commented Oct 29, 2021

raggi commented Aug 21, 2023

hj-collab commented Aug 22, 2023

zlepper commented Aug 24, 2023 • edited

adrianmihalko commented Aug 31, 2023 • edited

rodrigc commented Sep 2, 2023 • edited

zlepper commented Aug 24, 2023 •

edited

adrianmihalko commented Aug 31, 2023 •

edited

rodrigc commented Sep 2, 2023 •

edited