Klipper fails when k3s runs inside LXC container

[Marx2]

Hi
I'm running k3s inside LXC container. It starts ok, the only missing bit is pod named svclb-traefik-xxx. I see it's from image rancher/klipper-lb:v0.1.2. It doesn't start, showing error in logs:

2020-12-07T07:14:51.134808188Z + trap exit TERM INT
2020-12-07T07:14:51.135173011Z /usr/bin/entry: line 6: can't create /proc/sys/net/ipv4/ip_forward: Read-only file system
2020-12-07T07:14:51.135212862Z + echo 1
2020-12-07T07:14:51.135229374Z + true
2020-12-07T07:14:51.135381090Z + cat /proc/sys/net/ipv4/ip_forward
2020-12-07T07:14:51.136264347Z + '[' 0 '!=' 1 ]
2020-12-07T07:14:51.136442799Z + exit 1

Looking at command output from inside container

# cat /proc/sys/net/ipv4/ip_forward
1

shows, that forwarding is already enabled.

Looking at the source

klipper-lb/entry

Lines 6 to 10 in 824f44a

	echo 1 > /proc/sys/net/ipv4/ip_forward || true
	if [ `cat /proc/sys/net/ipv4/ip_forward` != 1 ]; then
	exit 1
	fi

I think that this check doesn't only check, but try to set forwarding even, if it's already enabled.

I think solution to my problem would be to first check if forwarding is already enabled, and then set forwarding only, if it's not enabled (not always, like above)

Edit: I've just found similar issue:
#4

[marcbachmann]

Oh thanks, I just wanted to report the same. I've set up k3s yesterday and I'm seeing those errors when I try to launch a service using a Loadbalancer.

As quick fix, I've changed the container to privileged in the DeamonSet config:

          securityContext:
            privileged: true
            capabilities:
              add:
                - NET_ADMIN

I'm also not sure whether the ip_forward config is already set to 1. I think it's actually wrong (kubernetes doesn't inherit the sysctl configs of the system) and then it can't set it.

edit: Ooops, I did the regular setup, not within LXC. But I'm having the same issue running on Debian 10.

[marcbachmann]

This issue is probably similar like k3s-io/k3s#201
There are different defaults and then k3s/klipper can't set up the ip_forward because of missing permissions.

[Marx2]

It looks similar. But my report is about fixing order in starting script. Instead of trying to always set ip forwarding, pod should first check, if ip forwarding is already enabled, and in such case just continue. It will allow pod to start in environment, where setting ip forwarding is read only

[marcbachmann]

👍 Initially I also thought I'm running into the same thing. But it looks really easy to fix. I'm sure they'd accept a pr 😉

[marcbachmann]

Ah, there's already a fallback to verify that it doesn't fail with the || true condition.
It looks like there's really something wrong in your setup and ip_forward isn't configured.

This ensures that the write is optional:

echo 1 > /proc/sys/net/ipv4/ip_forward || true

And it then only exits if the value is wrong: https://github.com/k3s-io/klipper-lb/blob/master/entry#L8-L10

Maybe some better logs would be nice.

[Marx2]

I'm not bash expert - sorry for bad suspicion - so this startup script looks ok.
Then, if I check from command line and it says "1", but similar check from script in pod says "0", it means that pod environment doesn't inherit this value, and it looks like duplicate of k3s-io/k3s#201

[marcbachmann]

I suggest to use istio with hostPort bindings: k3s-io/k3s#1652 (comment)

either as DaemonSet or Deployment.
in case you only need to expose http and https

[Marx2]

I've resolved this issue by permanently enabling forwarding on host with:
sysctl -w net.ipv4.ip_forward=1
Message about readonly filesystem is still there but pod is able to continue and starts