Host an OpenZiti router in Kubernetes
helm repo add openziti https://docs.openziti.io/helm-charts/The default configuration listens for incoming edge connections and router links. Set a public address for this listener (edge.advertisedHost) or disable it (linkListeners.transport.enabled) to avoid routers continually failing to dial into it.
# get a router enrollment token from the controller's management API
ziti edge create edge-router "router1" \
--tunneler-enabled --jwt-output-file /tmp/router1.jwt
# subscribe to the openziti Helm repo
helm repo add openziti https://openziti.github.io/helm-charts/
# install the router chart with a public address
helm upgrade --install \
"ziti-router-123456789" \
openziti/ziti-router \
--set-file enrollmentJwt=/tmp/router1.jwt \
--set ctrl.endpoint=ctrl.ziti.example.com:443 \
--set edge.advertisedHost=router1.ziti.example.com \All router TLS listeners must terminate TLS, so it's essential that Ingress resources use TLS passthrough.
This example demonstrates creating TLS pass-through Ingress resources for use with ingress-nginx.
Ensure you have the ingress-nginx chart installed with controller.extraArgs.enable-ssl-passthrough=true. You can verify this feature is enabled by running kubectl describe pods {ingress-nginx-controller pod} and checking the args for --enable-ssl-passthrough=true.
If not enabled, then you must patch the ingress-nginx deployment to enable the SSL passthrough option.
kubectl patch deployment "ingress-nginx-controller" \
--namespace ingress-nginx \
--type json \
--patch '[{"op": "add",
"path": "/spec/template/spec/containers/0/args/-",
"value":"--enable-ssl-passthrough"
}]'# subscribe to ingress-nginx
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx/
# install ingress-nginx
helm install \
--namespace ingress-nginx --create-namespace --generate-name \
ingress-nginx/ingress-nginx \
--set controller.extraArgs.enable-ssl-passthrough=trueCreate a Helm chart values file for this router chart.
# router-values.yml
ctrl:
endpoint: ziti-controller-ctrl.ziti-controller.svc:1280
advertisedHost: router1.ziti.example.com
edge:
advertisedPort: 443
service:
type: ClusterIP
ingress:
enabled: true
ingressClassName: nginx
annotations:
kubernetes.io/ingress.allow-http: "false"
nginx.ingress.kubernetes.io/ssl-passthrough: "true"Now upgrade your router chart release with the values file.
helm upgrade --install \
"ziti-router-1" \
openziti/ziti-router \
--set-file enrollmentJwt=/tmp/router1.jwt \
--values /tmp/router-values.ymlDisable the link listener if the router does not have a public address set (edge.advertisedHost). Ziti identities inside the cluster can still use the private router's edge listener ClusterIP service by authorizing them with a Ziti edge router policy.
helm upgrade --install \
"ziti-router-1" \
openziti/ziti-router \
--set-file enrollmentJwt=/tmp/router1.jwt \
--set ctrl.endpoint=ctrl.ziti.example.com:443 \
--set linkListeners.transport.enabled=falseDefault: tunnel.mode=none
Host mode enables a router's identity to reverse proxy Ziti service traffic to a target address on the regular network. Enable this mode by updating the router's identity in the controller to enable tunneling, then set tunnel.mode=host and upgrade the Helm release to start hosting Ziti services.
ziti edge update identity "router1" --tunneler-enabledtunnel.mode=proxy
Proxy mode enables the router to publish Ziti services as Kubernetes services.
Here's an example router values' snippet to merge with your other values:
tunnel:
mode: proxy
proxyServices:
# this will be bound on the "default" proxy Kubernetes service, see below
- zitiService: my-ziti-service.svc
containerPort: 10443
advertisedPort: 10443
# this will be bound on an additionally configured proxy Kubernetes service, see below
- zitiService: my-other-service.svc
containerPort: 10022
advertisedPort: 10022
proxyDefaultK8sService:
enabled: true
type: ClusterIP
proxyAdditionalK8sServices:
- name: myservice
type: LoadBalancer
annotations:
metallb.universe.tf/loadBalancerIPs: 192.168.1.100You can configure an additional edge listener by setting edge.additionalListeners. This is useful for making a WebSocket edge listener available for BrowZer clients that require a trusted server certificate.
This example configures a wss listener and requests a certificate from cert-manager. The alternative certificate must have a DNS SAN that is distinct from the public address of the default edge listener (edge.advertisedHost). This cert-manager approach has the advantage of automatically renewing the certificate and ensuring the DNS SAN of the certificate matches an additional listener's advertised host.
edge:
advertisedHost: router1.ziti.example.com
advertisedPort: 443
ingress:
annotations:
kubernetes.io/ingress.allow-http: "false"
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
enabled: true
ingressClassName: nginx
service:
enabled: true
type: ClusterIP
additionalListeners:
- name: router1-edge-wss
protocol: wss
containerPort: 3023 # must be unique
advertisedHost: router1-wss.ziti.example.com # must be distinct from edge.advertisedHost
advertisedPort: 443
addHostToSan: false # must be false to avoid colliding DNS SANs between listeners
service:
enabled: true
type: ClusterIP
ingress:
enabled: true
annotations:
kubernetes.io/ingress.allow-http: "false"
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
ingressClassName: nginx
identity:
altServerCerts:
- name: alt-server-cert-1
mode: certManager
secretName: ziti-router1-alt-server-certs1
additionalListenerName: router1-edge-wss
mountPath: /etc/ziti/alt-server-cert-1
issuerRef:
group: cert-manager.io
kind: ClusterIssuer
name: cloudflare-dns01-issuer-stagingYou don't have to use cert-manager. If you have a TLS secret named ziti-router1-alt-server-certs1 from some other issuer in the same namespace as the router containing the certificate and key, you can use it by setting values like these. You must also configure the additional listener as in the prior example with an advertisedHost that matches a DNS SAN of the alternative certificate.
# this is an generic approach for mounting configmaps, secrets, csi volumes, etc.
additionalVolumes:
- name: alt-server-cert-2
volumeType: secret
mountPath: /etc/ziti/alt-server-cert-2
secretName: ziti-router1-alt-server-cert-2
# this looks up a TLS secret's mountpoint to configure the router's identity
identity:
altServerCerts:
- mode: secret
secretName: ziti-router1-alt-server-cert-2You may also specify matching file paths for an additional volume and alternative certificate if the volume is not a TLS secret.
additionalVolumes:
- name: alt-server-cert-3
volumeType: csi
driverName: csi.bpfd.dev
attributes: volumeAttributes
mountPath: /etc/ziti/alt-server-cert-3
identity:
altServerCerts:
- mode: localFile
serverCert: /etc/ziti/alt-server-cert-3/server3.crt
serverKey: /etc/ziti/alt-server-cert-3/server3.key| Key | Type | Default | Description |
|---|---|---|---|
| additionalVolumes | list | [] |
additional volumes to mount to ziti-router container |
| affinity | object | {} |
deployment template spec affinity |
| configFile | string | "ziti-router.yaml" |
filename of router config YAML |
| configMountDir | string | "/etc/ziti/config" |
writeable mountpoint where read-only config file is projected to allow router to write ./endpoints statefile in same dir |
| csr | object | {"country":"","locality":"","organization":"","organizationalUnit":"","province":"","sans":{"dns":[],"email":[],"ip":[],"noDefaults":false,"uri":[]}} |
Certificate signing request distinguished name and subject alternative names |
| csr.country | string | "" |
country |
| csr.locality | string | "" |
city |
| csr.organization | string | "" |
organization |
| csr.organizationalUnit | string | "" |
organizational unit |
| csr.province | string | "" |
state |
| csr.sans.dns | list | [] |
additional DNS SANs |
| csr.sans.email | list | [] |
additional email SANs |
| csr.sans.ip | list | [] |
additional IP SANs |
| csr.sans.noDefaults | bool | false |
if true, disable computing default SANs from the advertisedHost, etc. |
| csr.sans.uri | list | [] |
additional URI SANs |
| ctrl.endpoint | string | "" |
required control plane endpoint, e.g., ctrl.ziti.example.com:443 |
| dnsConfig | object | {} |
it allows to override dns options when dnsPolicy is set to None. |
| dnsPolicy | string | "ClusterFirstWithHostNet" |
|
| edge.additionalListeners | list | [] |
additional edge listeners have the same shape as the default edge listener, except they're enabled if defined, and you must specify a unique name for each additional edge listener. The name distinguishes their respective cluster services. This is useful for BrowZer clients that require a trusted certificate for the edge WebSocket and advertisedHost must resolve to a public IP that presents a trusted certificate, e.g., an Ingress with TLS termination. |
| edge.advertisedHost | string | "" |
Domain name that edge clients will use to reach this router's edge listener. Default is cluster-internal service DNS name:port. |
| edge.advertisedPort | int | 443 |
cluster service, node port, load balancer, and ingress port |
| edge.containerPort | int | 3022 |
cluster service target port on the container |
| edge.enabled | bool | true |
enable the edge listener in the router config; usually true because tunnel bindings require the edge which must have at least on listener |
| edge.ingress.annotations | object | {} |
ingress annotations, e.g., to configure ingress-nginx for passthrough TLS |
| edge.ingress.enabled | bool | false |
create an ingress for the cluster service; typically paired with a ClusterIP service type when enabled |
| edge.ingress.ingressClassName | string | "" |
ingress class name |
| edge.ingress.labels | object | {} |
ingress labels |
| edge.options | object | {} |
additional common xgress options |
| edge.protocol | string | "tls" |
edge listener protocol: tls, wss; usually tls because additionalListeners can be used to provide a wss listener |
| edge.service.annotations | object | {} |
service annotations |
| edge.service.enabled | bool | true |
create a cluster service for the edge listener; usually true, but you can disable this to effectively un-publish the edge listener |
| edge.service.labels | object | {} |
service labels |
| edge.service.type | string | "ClusterIP" |
expose the service as a ClusterIP, NodePort, or LoadBalancer; default is ClusterIP, but you could use NodePort or LoadBalancer instead of an ingress controller |
| enrollmentJwt | string | nil |
enrollment one time token from the controller's management API |
| enrollmentJwtFromSecret | bool | false |
allow for using a secret to specify the enrollment token instead of using the enrollmentJwt field if enabled, setting the enrollment token on the enrollmentJwt field has no effect |
| enrollmentJwtSecretName | string | "" |
set the enrollment jwt from a secret The enrollment token secret must be of the following format: apiVersion: v1 kind: Secret metadata: name: myEnrollmentJwtSecret type: Opaque data: enrollmentJwt: |
| env | object | {} |
assign key=value in pod environment |
| execMountDir | string | "/usr/local/bin" |
read-only mountpoint for executables (must be in image's executable search PATH) |
| fabric.metrics.enabled | bool | false |
configure fabric metrics in the router config |
| forwarder.latencyProbeInterval | int | 10 |
|
| forwarder.linkDialQueueLength | int | 1000 |
|
| forwarder.linkDialWorkerCount | int | 32 |
|
| forwarder.rateLimitedQueueLength | int | 5000 |
|
| forwarder.rateLimitedWorkerCount | int | 64 |
|
| forwarder.xgressDialQueueLength | int | 1000 |
|
| forwarder.xgressDialWorkerCount | int | 128 |
|
| ha.enabled | bool | false |
must be enabled if multiple controllers |
| hostNetwork | bool | false |
Host networking requested for a pod if set, i.e. tproxy ports enabled in the host namespace. i.e. egress gateway |
| identity.altServerCerts | list | [] |
|
| identityMountDir | string | "/etc/ziti/identity" |
read-only mountpoint for router identity secret specified in deployment for use by router run container |
| image.additionalArgs | list | [] |
additional arguments can be passed directly to the container to modify ziti runtime arguments |
| image.args | list | ["run","{{ .Values.configMountDir }}/{{ .Values.configFile }}"] |
deployment container command args and opts |
| image.command | list | ["/entrypoint.bash"] |
deployment container command |
| image.pullPolicy | string | "Always" |
deployment image pull policy |
| image.repository | string | "docker.io/openziti/ziti-router" |
container image tag for deployment |
| image.tag | string | nil |
container image tag (default is Chart's appVersion) |
| linkDialers.transport.extraProps | object | {} |
extra properties to define for the transport link dialer, e.g., groups |
| linkListeners.transport.advertisedHost | string | "{{ .Values.edge.advertisedHost }}" |
DNS name that other routers will use to form mesh transport links to this listener |
| linkListeners.transport.advertisedPort | string | "{{ .Values.edge.advertisedPort }}" |
cluster service, node port, load balancer, and ingress port for this listener; default is edge.advertisedPort |
| linkListeners.transport.containerPort | string | "{{ .Values.edge.containerPort }}" |
cluster service target port on the container; default is to share a listener with the edge, providing ziti-link ALPN |
| linkListeners.transport.enabled | bool | true |
enable the transport listener in the router config; set false for a private router that only connects to other routers and does not accept incoming links |
| linkListeners.transport.extraProps | object | {} |
extra properties to define for the transport link listener, e.g., groups |
| linkListeners.transport.ingress.annotations | object | {} |
ingress annotations, e.g., to configure ingress-nginx |
| linkListeners.transport.ingress.enabled | bool | false |
create an ingress for the cluster service |
| linkListeners.transport.ingress.ingressClassName | string | "" |
ingress class name |
| linkListeners.transport.ingress.labels | object | {} |
ingress labels |
| linkListeners.transport.options | object | {} |
link listener options |
| linkListeners.transport.service.annotations | object | {} |
service annotations |
| linkListeners.transport.service.enabled | bool | true |
create a cluster service for the router transport link listener; unnecessary if advertisedHost is shared with edge listener (the default) |
| linkListeners.transport.service.labels | object | {} |
service labels |
| linkListeners.transport.service.type | string | "ClusterIP" |
expose the service as a ClusterIP, NodePort, or LoadBalancer |
| noHelmHooks | bool | false |
disable the execution of hooks, defined within this chart This chart makes use of Helm hooks. Setting this to true will prevent the hooks from being deployed. This is useful when using the chart with tools like ArgoCD |
| nodeSelector | object | {} |
deployment template spec node selector |
| persistence.accessMode | string | "ReadWriteOnce" |
PVC access mode: ReadWriteOnce (concurrent mounts not allowed), ReadWriteMany (concurrent allowed) |
| persistence.annotations | object | {} |
annotations for the PVC |
| persistence.enabled | bool | true |
required: place a storage claim for the ctrl endpoints state file |
| persistence.existingClaim | string | "" |
A manually managed Persistent Volume and Claim Requires persistence.enabled: true If defined, PVC must be created manually before volume will be bound |
| persistence.size | string | "50Mi" |
50Mi is plenty for this state file |
| persistence.storageClass | string | "" |
Storage class of PV to bind. By default it looks for the default storage class. If the PV uses a different storage class, specify that here. |
| persistence.volumeName | string | nil |
PVC volume name |
| podAnnotations | object | {} |
annotations to apply to all pods deployed by this chart |
| podSecurityContext | object | {"fsGroup":2171} |
deployment template spec security context |
| podSecurityContext.fsGroup | int | 2171 |
this is the GID of "ziggy" run-as user in the container that has access to any files created by the router process in the emptyDir volume used to persist the list of ctrl endpoints |
| proxy | object | {} |
Explicit proxy setting in the router configuration. Router can be deployed in a site where all egress traffic is forwarded through an explicit proxy. The enrollment will also be forwarded through the proxy. |
| resources | object | {} |
deployment container resources |
| securityContext | string | nil |
deployment container security context |
| tolerations | list | [] |
deployment template spec tolerations |
| tunnel.diverterPath | string | nil |
the tproxy mode can be switched from iptables based interception to bpf interception by passing the user space bpf program path. bpf kernel space program is expected to be loaded prior or during router deployment, e.g. bpfman agent, hostpath, etc |
| tunnel.dnsSvcIpRange | string | nil |
CIDR range for the internal service fqdn to dynamic intercept IP address resolution (default: 100.64.0.0/10) |
| tunnel.lanIf | string | "lo" |
interface device name for setting up INPUT firewall rules if fw enabled. It must be set but not needed in containers. Thus, it is set to lo by default |
| tunnel.mode | string | "none" |
run mode for the router's built-in tunnel component: host, tproxy, proxy, or none |
| tunnel.proxyAdditionalK8sServices | list | [] |
if tunnel mode is "proxy", create a separate cluster service for each Ziti service listed in "proxyServices" which k8sService == name |
| tunnel.proxyDefaultK8sService | object | {"enabled":true,"type":"ClusterIP"} |
if tunnel mode is "proxy", create the a cluster service named {{ release }}-proxy-default listening on each "advertisedPort" defined in "proxyServices" |
| tunnel.proxyServices | list | [] |
list of Ziti services for which K8s services are to be created by this deployment, default is one cluster service port per Ziti service |
| tunnel.resolver | string | nil |
Ziti nameserver listener where OS must be configured to send DNS queries (default: udp://127.0.0.1:53) |
| websocket.enableCompression | bool | true |
enable compression on websocket |
| websocket.enabled | bool | false |
enable the websocket transport. Also requires an appropriate edge.additionalListeners entry. |
| websocket.handshakeTimeout | int | 10 |
websocket handshake timeout |
| websocket.idleTimeout | int | 5 |
websocket idle timeout |
| websocket.pingInterval | int | 54 |
websocket ping timeout |
| websocket.pongTimeout | int | 60 |
websocket pong timeout |
| websocket.readBufferSize | int | 4096 |
websocket read buffer size |
| websocket.readTimeout | int | 5 |
websocket read timeout |
| websocket.writeBufferSize | int | 4096 |
websocket write buffer size |
| websocket.writeTimeout | int | 10 |
websocket write timeout |