Fix worker machine creation

[xrstf]

After deploying the machine-controller v0.9.9 to a cluster and then creating a MachineDeployment like so:

apiVersion: cluster.k8s.io/v1alpha1
kind: MachineDeployment
metadata:
  name: worker-machines-deployment
  namespace: kube-system
spec:
  replicas: 5
  selector:
    matchLabels:
      workerset: worker-machines
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      labels:
        workerset: worker-machines
      namespace: kube-system
    spec:
      providerConfig:
        value:
          cloudProvider: aws
          cloudProviderSpec:
            ami: ami-5970c436 # Ubuntu 18.04
            availabilityZone: eu-central-1a
            diskSize: 50
            diskType: gp2
            instanceType: t2.medium
            region: eu-central-1
            subnetId: subnet-2bff4f43
            vpcId: vpc-819f62e9
          operatingSystem: ubuntu
          operatingSystemSpec:
            distUpgradeOnBoot: false
      versions:
        kubelet: 1.12.2

  # these two are only here to prevent segfaults in the MC
  minReadySeconds: 0
  paused: false

Nothing really happens. The MC is creating a MachineSet and all 5 Machines, but then no instances are spawned on AWS EC2. Killing the MC in the cluster and starting a local one using the current master branch brings the machines up, though. (this problem is tackled in #35)

The machines boot up and create the kubelet-bootstrap services. Unfortunately, the kubelet cannot connect to the masters. The journal logs

systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 18.
systemd[1]: Stopping kubelet-healthcheck.service...
systemd[1]: Stopped kubelet-healthcheck.service.
systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
systemd[1]: Started kubelet: The Kubernetes Node Agent.
systemd[1]: Started kubelet-healthcheck.service.
health-monitor.sh[11895]: Start kubernetes health monitoring for kubelet
health-monitor.sh[11895]: Wait for 2 minutes for kubelet to be functional
kubelet[11894]: Flag --resolv-conf has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
kubelet[11894]: Flag --pod-manifest-path has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
kubelet[11894]: Flag --allow-privileged has been deprecated, will be removed in a future version
kubelet[11894]: Flag --authorization-mode has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
kubelet[11894]: Flag --client-ca-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
kubelet[11894]: Flag --rotate-certificates has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
kubelet[11894]: Flag --authentication-token-webhook has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
kubelet[11894]: Flag --read-only-port has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
kubelet[11894]: Flag --anonymous-auth has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
kubelet[11894]: Flag --protect-kernel-defaults has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
kubelet[11894]: Flag --cluster-dns has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
kubelet[11894]: Flag --cluster-domain has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
systemd[1]: Started Kubernetes systemd probe.
kubelet[11894]: I1119 16:06:54.986978   11894 server.go:408] Version: v1.12.2
kubelet[11894]: I1119 16:06:54.987250   11894 server.go:486] acquiring file lock on "/tmp/kubelet.lock"
kubelet[11894]: I1119 16:06:54.987397   11894 server.go:491] watching for inotify events for: /tmp/kubelet.lock
kubelet[11894]: I1119 16:06:54.987828   11894 aws.go:1042] Building AWS cloudprovider
kubelet[11894]: I1119 16:06:54.990634   11894 aws.go:1004] Zone not specified in configuration file; querying AWS metadata service
kubelet[11894]: E1119 16:06:55.146729   11894 tags.go:94] Tag "KubernetesCluster" nor "kubernetes.io/cluster/..." not found; Kubernetes may behave unexpectedly.
kubelet[11894]: F1119 16:06:55.146786   11894 server.go:262] failed to run Kubelet: could not init cloud provider "aws": AWS cloud failed to find ClusterID
systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
systemd[1]: kubelet.service: Failed with result 'exit-code'.

This is how the systemd units are configured:

ubuntu@ip-172-31-12-1:/etc/systemd/system$ cat kubelet-healthcheck.service
[Unit]
Requires=kubelet.service
After=kubelet.service
[Service]
ExecStart=/opt/bin/health-monitor.sh kubelet
[Install]
WantedBy=multi-user.target


ubuntu@ip-172-31-12-1:/etc/systemd/system$ cat kubelet.service
[Unit]
After=docker.service
Requires=docker.service
Description=kubelet: The Kubernetes Node Agent
Documentation=https://kubernetes.io/docs/home/
[Service]
Restart=always
StartLimitInterval=0
RestartSec=10
Environment="PATH=/opt/bin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin/"
ExecStart=/opt/bin/kubelet $KUBELET_EXTRA_ARGS \
  --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf \
  --kubeconfig=/etc/kubernetes/kubelet.conf \
  --pod-manifest-path=/etc/kubernetes/manifests \
  --allow-privileged=true \
  --network-plugin=cni \
  --cni-conf-dir=/etc/cni/net.d \
  --cni-bin-dir=/opt/cni/bin \
  --authorization-mode=Webhook \
  --client-ca-file=/etc/kubernetes/pki/ca.crt \
  --rotate-certificates=true \
  --cert-dir=/etc/kubernetes/pki \
  --authentication-token-webhook=true \
  --cloud-provider=aws \
  --cloud-config=/etc/kubernetes/cloud-config \
  --read-only-port=0 \
  --exit-on-lock-contention \
  --lock-file=/tmp/kubelet.lock \
  --anonymous-auth=false \
  --protect-kernel-defaults=true \
  --cluster-dns=10.10.10.10 \
  --cluster-domain=cluster.local
[Install]
WantedBy=multi-user.target

From what I've seen, we need to define a "cluster ID" since Kubernetes 1.10, but I could not find out how. It's supposed to be a label kubernetes.io/cluster/<ID>=owned, but this seems to be something either Terraform is supposed to do for the masters and then the MC is supposed to do for the worker instances.

This is a sub-task of #4.

[xrstf]

Yes, you need to do it before the init. The tag must exist on all instances, so doing it with terraform is the way to go.
You can check what the cluster controller is doing on AWS

the tag [for workers] needs to be specified on the machine object. Tags can be specified within the providerConfig

[xrstf]

This was not solely fixed by #36, but also by adding proper tags on Machine objects in 926f008.