I suspect that these issues might be related:

#8244
#5621
#391

@rodrigc Awesome! Thanks for the links. I'm gonna do some reading up on those, and report back here with my findings

@tiesmaster Exact same issue with AKS and tailscale proxy

@wadhah101 Thanks for the +1! I suspected it would brake for others on AKS (should be, right?), though, good to know, and good to know that others would be helped with getting this fixed (and perhaps pushed upstream). I looked into the other issues briefly, but didn't manage to do a write-up here.

What I found out is that it looked like that AKS is not using iptables-ng, as opposed to what you would be expecting from the other issues. So I cannot explain this, so far. I going to look into this over the course of the next week. I'll keep you posted when I have some results.

@tiesmaster Thanks for your response, I lack the skills to debug this unfortunately. Thanks for keeping everyone posted 🙏

@wadhah101 Quick update here, as I didn't manage to look into this the last couple of weeks, and I will be on holiday from now on, for over 2 weeks. Next, I don't know if I'll be able to work on this even after that for the short term, so I cannot give any guarantees. I'll keep you posted when I do.

@tiesmaster @wadhah101 wadhah101 Major update posted by @KevinLiang10 here:
#391 (comment)

It may or may not help with your issue, but is worth testing

I'm having a similar issue on a k3s cluster #8733 wondering if this could be related.

Based on my testing, this issue appears to be resolve by the experimental NFTABLES support that was recently merged. I developed a PR to enable this in the k8s-operator and was able to get the operator working in an AKS cluster correctly as well as in my k3s case above. If you want to try my fix, you can patch the operator.yaml file as follows to pick up my privates and enable NFT mode. Pertinent changes are operator image, PROXY_IMAGE and new OPERATOR_USENFT env var:

 containers:
         - name: operator
           image: clarkezone/tsoperatornftfix:latest
           resources:

…

          - name: CLIENT_SECRET_FILE
             value: /oauth/client_secret
          - name: PROXY_IMAGE
             value: "clarkezone/tsclientnftfix:latest"
          - name: PROXY_TAGS
…

          - name: OPERATOR_USENFT
             value: "true"

@clarkezone That's great that you confirmed the fix.
Even better you submitted a patch!

In your patch to the operator, is it possible to somehow detect that nftables exists on the
system, and then spit out a log informing the user that they should set this particular variable to get things working?

Bravo for your work and @KevinLiang10 's original patch!

@rodrigc Hi I'm working on detecting wether nftables/iptables available or used on machine, and just use it. Tho the implementation is not targeting to solve k8s problems, rather an implementation to relief user from having to explicit set env var to use the new nftables feature.

spit out a log informing the user that they should set this particular variable to get things working

Thanks for this advice we will discuss about adding it to our run time logging. It's because we are detecting nftables/iptables support at runtime.

I'll link these k8s issues when I put the pr up and test if the solution would help on this issue!

@KevinLiang10 Nice work. Long term the heuristics you are implementing as part of #8762 to auto-select
iptables/nftables and choose iptables or nftables is the right approach.
Long term, having the user depend on too many weird knobs is not ideal.

In #8762, if you can add good logging, so that as you develop the feature, end-users can get some
idea how the feature is working in their environment, that would be good.
That will also help identify any weird corner cases where your heuristic logic breaks down.

@rodrigc I've added logs for which firewall tool is being used, and the tool's condition. Apart from that what logs do you think are particularly interesting for you?

Really would love to hear what you are interested, your opinions will definitely help us to improve user experience!

@KevinLiang10 These are some ideas I have of things to log:

1. What firewall tool is being used?
2. What is the version of the firewall tool?
3. What is the version of the kernel?
4. What firewall capability is available in the kernel?
5. What is the version of the firewall capabilty compiled into the kernel?
6. If it is available, output of lsb_release -a is sometimes helpful for quickly figuring out version/distro information

Basically, anything that can help the user debug
when things go wrong, with iptables / nftables

@KevinLiang10 the other perspective I would suggest to you, is
from your perspective at Tailscale,
what kinds of information/logs would you want the user to provide to you in a bug report,
to better troubleshoot this particular feature?

I gave you a list of what I think are good pieces of information to log,
but as the author of this feature, you might have more ideas.