Can not create worker nodes

[AndrukhivAndriy]

Hi. I am trying to create k3s cluster. But I am stuck on creating worker nodes. I am not using a passphrase in SSH keys. Tried to connect via the id_ed25519 key to the server - success. This is my config:

hetzner_token: M2bk…….
cluster_name: nice
kubeconfig_path: "./kubeconfig"
k3s_version: v1.32.3+k3s1

networking:
ssh:
port: 22
use_agent: false # set to true if your key has a passphrase
public_key_path: "/root/.ssh/id_ed25519.pub"
private_key_path: "/root/.ssh/id_ed25519"
allowed_networks:
ssh:
- 0.0.0.0/0
api: # this will firewall port 6443 on the nodes
- 0.0.0.0/0
public_network:
ipv4: true
ipv6: false
private_network:
enabled: true
subnet: 10.0.0.0/16
existing_network_name: ""
cni:
enabled: true
encryption: false
mode: flannel
datastore:
mode: etcd # etcd (default) or external
schedule_workloads_on_masters: false
masters_pool:
instance_type: cx22

image: ubuntu-22.04

instance_count: 1 # for HA; you can also create a single master cluster for dev and testing (not recommended for prod
locations: # You can choose a single location for single master clusters or if you prefer to have all masters in the
- fsn1

worker_node_pools:

• name: small-static
  instance_type: cpx11
  instance_count: 2
  location: fsn1

image: debian-11

mbedded_registry_mirror:
enabled: false
protect_against_deletion: false
create_load_balancer_for_the_kubernetes_api:

So, the result was created on Hetzner: master node, SSH key, network. Firewall and worker nodes were not created. The output of the hetzner-k3s create --config cluster_config.yaml

.....
[Instance nice-master1] Instance status: running
Error creating instance: timeout after 00:00:05
Instance creation for nice-master1 failed. Try rerunning the create command.

I am using Version: 2.2.6. 2.2.7 - have the same issue for me. Also changed ssh key algorithms - rsa and ed25519

[vitobotta]

Hi, please use code snippets next time to make the code/config more readable.

Which OS are you using hetzner-k3s on? Have you tried with the agent enabled?

[AndrukhivAndriy]

Thanks. I am using Ubuntu 22 and 24. Yes, i tried agent enabled -- no result

[vitobotta]

Please rerun the create command with DEBUG=true as that might show useful information to figure out the problem.

[AndrukhivAndriy]

[Configuration] Validating configuration...
[Configuration] ...configuration seems valid.
[Placement groups] Deleting unused placement group nicely-small-static-1...
[Placement groups] ...placement group nicely-small-static-1 deleted
[Placement groups] Creating placement group nicely-small-static-1...
[Placement groups] ...placement group nicely-small-static-1 created
[Instance nicely-master1] Instance status: running
Error creating instance: timeout after 00:00:05
Instance creation for nicely-master1 failed. Try rerunning the create command.

[vitobotta]

You need to set the environment variable DEBUG to true.

[AndrukhivAndriy]

ssh: connect to host 159.69.50.XX port 22: Connection timed out
[Instance nicely-master1] debug1: Reading configuration data /etc/ssh/ssh_config
[Instance nicely-master1] debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/.conf matched no files
[Instance nicely-master1] debug1: /etc/ssh/ssh_config line 21: Applying options for *
[Instance nicely-master1] debug1: Connecting to 159.69.50.XX [159.69.50.XX] port 22.
[Instance nicely-master1] debug1: connect to address 159.69.50.XX port 22: Connection timed out
[Instance nicely-master1] ssh: connect to host 159.69.50.XX port 22: Connection timed out
[Instance nicely-master1] SSH command failed: debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to 159.69.50.XX [159.69.50.XX] port 22.
debug1: connect to address 159.69.50.XX port 22: Connection timed out
ssh: connect to host 159.69.50.XX port 22: Connection timed out

The same thing with use_agent: true or use_agent: false

[vitobotta]

That's super weird. The connection time out seems to suggest the firewall is blocking the connection, but you have specified 0.0.0.0/0 as allowed network. Out of curiosity, can you try using hetzner-k3s with the exact same configuration but from another connection? You could use a temp hetzner instance for example.

Also please share the full config (without your token) in a code snippet so I can check it better.

I haven't come across this issue and it hasn't been reported before so it's weird. Also can you confirm the firewall has been configured when you created the cluster?

[AndrukhivAndriy]

Yep. I created VM on Hetzner (ubuntu-22). There, I tried to run your script. No result. Today, I installed it on my Mac - it works very well.

[vitobotta]

Can you share your config properly formatted?

[AndrukhivAndriy]

hetzner_token:
cluster_name: nicely
kubeconfig_path: "./kubeconfig"
k3s_version: v1.32.3+k3s1

networking:
ssh:
port: 22
use_agent: true # I have not tried false, but in Ubuntu I have tried true and false.
public_key_path: "./id_ed25519.pub"
private_key_path: "./id_ed25519"
allowed_networks:
ssh:
- 0.0.0.0/0
api:
- 0.0.0.0/0
public_network:
ipv4: true
ipv6: false
private_network:
enabled: true
subnet: 10.0.0.0/16
existing_network_name: ""
cni:
enabled: true
encryption: false
mode: flannel
atastore:
mode: etcd
schedule_workloads_on_masters: false

masters_pool:
instance_type: cx22
image: ubuntu-22.04
instance_count: 3
locations:
- fsn1
- hel1
- nbg1
worker_node_pools:

• name: small-static
  instance_type: cx32
  instance_count: 2
  location: fsn1
  image: ubuntu-22.04
• name: medium-autoscaled
  instance_type: cx22
  location: fsn1
  autoscaling:
  enabled: true
  min_instances: 0
  max_instances: 3
  mbedded_registry_mirror:
  enabled: false

protect_against_deletion: false

create_load_balancer_for_the_kubernetes_api: true

-------- This configuration works fine on the Mac, but doesn't work on Ubuntu 22. ------

[vitobotta]

Are the ubuntu computer and the mac in the same network?

[AndrukhivAndriy]

No. Ubuntu - Hetzner, Mac - home

[vitobotta]

Can you please share your config in a formatted code snippet (between triple backticks) so it's more readable? I will do a test using Ubuntu 22 from a Hetzner server. Your issue is very weird and I am not sure what's happening.

[AndrukhivAndriy]

Sorry. My mistake. Port 22 on the Hetzner Firewall (outbound) was closed. I forgot about outbound traffic.

[vitobotta]

Thanks for the update.