Parameterize kubelet root directory (/var/lib/kubelet)

[ant31]

What type of PR is this?

/kind cleanup

What this PR does / why we need it:

In some situation, users wants to use a different directory than the default "/var/lib/kubelet'.
It was previously possible by setting the kubelet_custom_flags but many roles were still hardcoded to "/var/lib/kubelet".

Which issue(s) this PR fixes:

Fixes #7087

Special notes for your reviewer:

Not sure the best way to handle users that were using the kubelet_custom_flags.

Does this PR introduce a user-facing change?:

Add new variable: `kubelet_root_dir`  to change the kubelet root default "/var/lib/kubelet" to a custom directory

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ant31

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [ant31]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ant31]

/retest

[ant31]

/ok-to-test

[ant31]

/hold

It's not so easy to change the default. Too many operator, csi and other software in the ecosystem have hardcoded /var/lib/kubelet.
One way to deal with this is to mount bind the new root to /var/lib/kubelet.

[ant31]

/unhold

PR is ready.
I've added a test
It's not ideal but there's no other good options AFAIK.

[VannTen]

/hold

It's not so easy to change the default. Too many operator, csi and other software in the ecosystem have hardcoded /var/lib/kubelet. One way to deal with this is to mount bind the new root to /var/lib/kubelet.

In light of this, do we actually want to support that ? That's quite the corner case. The original issue has 0 actual use case for this, so I don't see (yet, at least) what's the purpose would be.

[ant31]

Yes,
Some provider offer little flexibility to configure the boot volume partitions. It was the case on equinix and it s the case on oracle cloud too.

This PR provide enough to workaround that.

[k8s-ci-robot]

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ant31]

@vanteen up ^

[VannTen]

Sorry, I though I replied but I must had forgot to hit Send.

Yes, Some provider offer little flexibility on the boot volume partition, or with some complexity. It was the case on equinix and it s the case on oracle cloud. For equinix I went with long and manual workaround for Oracle cloud, I ve parameterized kubelet root and containerd to use faster and bigger volume

I had not seen that you changed to a bind-mount implementation, sorry. I'm still not sure we should do this in Kubespray, honestly. The current implementation use a bind-mount, but another frequent use cases is to have a specific FS volume mounted here. Basically this is rather general, and rather easy to setup outside of kubespray (e.g. in a parent playbook importing kubespray). Maybe we could instead have an extra_playbook doing that ? Also, I'm not sure the migrate part is safe. Newly opened files will be fine, but process holding file descriptors to files or directories there will have a no longer updated view of the world, which could cause subtle bugs

[ant31]

ok for me