Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tailscale-operator not working on k3s cluster #8733

Closed
clarkezone opened this issue Jul 27, 2023 · 3 comments
Closed

Tailscale-operator not working on k3s cluster #8733

clarkezone opened this issue Jul 27, 2023 · 3 comments
Labels
bug Bug needs-triage

Comments

@clarkezone
Copy link

clarkezone commented Jul 27, 2023 •
edited

### What is the issue?

I’m attempting a simple helloworld level scenario for tailscale-operator on k3s on ubnuntu 22.04 and the scenario isn't working correctly.

Steps to reproduce

  1. backup kubeconfig:
mv ~/.kube/config ~/.kube/backup
  1. install k3s and confirm running:
curl -sfL https://get.k3s.io | sh -
sudo mv /etc/rancher/k3s/k3s.yaml ~/.kube/config
sudo chmod 777 ~/.kube/config
➜  tailscaleoperator git:(c4updates) ✗ k get nodes
NAME                   STATUS   ROLES                  AGE   VERSION
clarkezonedevbox5-tr   Ready    control-plane,master   94s   v1.27.3+k3s1
  1. install tailscale operator from instructions here: https://tailscale.com/kb/1236/kubernetes-operator/
➜  tailscaleoperator git:(c4updates) ✗ k apply -f operator.yaml
namespace/tailscale created
serviceaccount/proxies created
role.rbac.authorization.k8s.io/proxies created
rolebinding.rbac.authorization.k8s.io/proxies created
serviceaccount/operator created
clusterrole.rbac.authorization.k8s.io/tailscale-operator created
clusterrolebinding.rbac.authorization.k8s.io/tailscale-operator created
role.rbac.authorization.k8s.io/operator created
rolebinding.rbac.authorization.k8s.io/operator created
secret/operator-oauth created
deployment.apps/operator created
➜  tailscaleoperator git:(c4updates) ✗ k get pods -n tailscale
NAME                       READY   STATUS    RESTARTS   AGE
operator-74cdfb6f5-gj7dw   1/1     Running   0          8s

# verify operator showing in tailnet
➜  tailscaleoperator git:(c4updates) ✗ tailscale status | grep operator
100.82.108.42   tailscale-operator   tagged-devices linux   -
  1. install test workload:
k apply -f https://gist.github.com/clarkezone/b22a5851f2e4229f5fd29f1115ddee32/raw/766708eee8f614d846dc12afe4dfaa819a678ee9/tailscaletest.yaml

➜  tailscaleoperatortest git:(master) k get pods -n tailscaletest
NAME                              READY   STATUS    RESTARTS   AGE
nginx-tailscale-7bbbb87bf-2f7lc   1/1     Running   0          30s
nginx-tailscale-7bbbb87bf-wkm24   1/1     Running   0          30s
➜  tailscaleoperatortest git:(master) k get services -n tailscaletest
NAME              TYPE           CLUSTER-IP     EXTERNAL-IP                                      PORT(S)        AGE
nginx-tailscale   LoadBalancer   10.43.167.59   tailscaletest-nginx-tailscale.tail967d8.ts.net   80:30110/TCP   37s
  1. Attempt to curl / ping the tailscale URL for the service (Result: it isn't working)
➜  tailscaleoperatortest git:(master) curl tailscaletest-nginx-tailscale.tail967d8.ts.net

^C
➜  tailscaleoperatortest git:(master) ping tailscaletest-nginx-tailscale.tail967d8.ts.net

PING tailscaletest-nginx-tailscale.tail967d8.ts.net (100.70.204.105) 56(84) bytes of data.
^C
--- tailscaletest-nginx-tailscale.tail967d8.ts.net ping statistics ---
11 packets transmitted, 0 received, 100% packet loss, time 10223ms

Logs from proxy pod:
logs.txt

Are there any recent changes that introduced the issue?

First time I’ve tried hence can’t comment

OS

Linux

OS version

Ubuntu 22.04.2 LTS

Tailscale version

1.47.36

Other software

v1.27.3+k3s1

Bug report

No response
@clarkezone
Copy link
Author

At end or repro uninstall k3s and restore config,

/usr/local/bin/k3s-uninstall.sh
mv ~/.kube/backup ~/.kube/config

@clarkezone clarkezone mentioned this issue Jul 29, 2023
Closed
clarkezone added a commit to clarkezone/tailscale that referenced this issue Jul 30, 2023
@clarkezone
cmd/containerboot, cmd/k8s-operator: Add flag to k8s-operator to enab…
8730689 
…le TS_DEBUG_USE_NETLINK_NFTABLES in tailscaled that was introduced in tailscale#8555

Fixes tailscale#8111, tailscale#8733
clarkezone added a commit to clarkezone/tailscale that referenced this issue Jul 30, 2023
@clarkezone
cmd/containerboot, cmd/k8s-operator: Add flag to k8s-operator to enab…
069508a 
…le TS_DEBUG_USE_NETLINK_NFTABLES in tailscaled that was introduced in tailscale#8555

Fixes tailscale#8111, tailscale#8733
clarkezone added a commit to clarkezone/tailscale that referenced this issue Jul 30, 2023
@clarkezone
cmd/containerboot, cmd/k8s-operator: Add flag to k8s-operator to enab…
e50ab2b 
…le TS_DEBUG_USE_NETLINK_NFTABLES in tailscaled that was introduced in tailscale#8555

Fixes tailscale#8111, tailscale#8733
clarkezone added a commit to clarkezone/tailscale that referenced this issue Jul 30, 2023
@clarkezone
cmd/containerboot, cmd/k8s-operator:
32da9b5 
Add flag to k8s-operator to enable TS_DEBUG_USE_NETLINK_NFTABLES in tailscaled that was introduced in tailscale#8555

Fixes tailscale#8111, tailscale#8733
@clarkezone clarkezone mentioned this issue Jul 30, 2023
Open
clarkezone added a commit to clarkezone/tailscale that referenced this issue Jul 30, 2023
@clarkezone
cmd/containerboot, cmd/k8s-operator:
41dc497 
Add flag to k8s-operator to enable TS_DEBUG_USE_NETLINK_NFTABLES in tailscaled that was introduced in tailscale#8555

Fixes tailscale#8111, tailscale#8733

Signed-off-by: James Clarke <james@clarkezone.net>
@KevinLiang10 KevinLiang10 mentioned this issue Aug 1, 2023
Merged
@DentonGentry
Copy link
Contributor 

2023/07/29 00:12:16 health("router"): error: setting up filter/ts-input: running [/sbin/ip6tables -t filter -N ts-input --wait]: exit status 3: modprobe: can't change directory to '/lib/modules': No such file or directory ip6tables v1.8.8 (legacy): can't initialize ip6tables table `filter': Table does not exist (do you need to insmod?)                                                                                                           Perhaps ip6tables or your kernel needs to be upgraded.

I think the fix for this is the auto-detection tracked in #391
Closing as a duplicate of #391

@spasche
Copy link

spasche commented Sep 28, 2023

I had the same issue: on a k3s cluster, impossible to reach a service exposed on tailscale.

The error from the proxy Pod:

ts-nginx-ts-8f2zt-0 tailscale 2023/09/28 21:09:21 health("router"): error: setting up filter/ts-input: running [/sbin/ip6tables -t filter -N ts-input --wait]: exit status 3: modprobe: can't change directory to '/lib/modules': No such file or directory
ts-nginx-ts-8f2zt-0 tailscale ip6tables v1.8.8 (legacy): can't initialize ip6tables table `filter': Table does not exist (do you need to insmod?)
ts-nginx-ts-8f2zt-0 tailscale Perhaps ip6tables or your kernel needs to be upgraded.

I could fix it by loading the ip6table_filter module on the node:

modprobe ip6table_filter

And to make it persist on reboots:

echo ip6table_filter | sudo tee -a /etc/modules-load.d/k3s.conf

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Bug needs-triage
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@spasche @clarkezone @DentonGentry