From 27094b76d4e844b6736ddc79d97d9593d67fc4fc Mon Sep 17 00:00:00 2001 From: Mangirdas Judeikis Date: Sat, 28 Oct 2023 12:55:04 +0300 Subject: [PATCH 01/10] WIP for proxy implementation --- keps/core/0002-proxy-workspace.md | 101 ++++++++++++++++++++++++++++++ 1 file changed, 101 insertions(+) create mode 100644 keps/core/0002-proxy-workspace.md diff --git a/keps/core/0002-proxy-workspace.md b/keps/core/0002-proxy-workspace.md new file mode 100644 index 0000000..a2bdd1e --- /dev/null +++ b/keps/core/0002-proxy-workspace.md @@ -0,0 +1,101 @@ +# Proxy workspace enhancement proposal + +## Summary + +To extend spectrum of use cases for kcp, we need to support workspaces that are not +backed by a kcp cluster itself, but are backed by a proxy that can be used to proxy +request to remote cluster directly, without having to go through kcp workspace. + +## Motivation + +This would enable usecases where kcp is used as a proxy to remote clusters, but +some of the infrastructure is still managed by kcp. For example, kcp could be +responsible for ArgoCD, Crossplane, or other infrastructure that is more crd-based. + +At this point in time communication between kcp and remote clusters is done via +workspaces abstractions. + +### Goals + +1. Agree how to make proxy functionality as pluggable as possible. +2. Agree on the scope of the proxy functionality. +3. Agree on the implementation of the proxy functionality. + +### Non-Goals + +N/A + +## Proposal + +Both options below would require a new `VirtualWorkspace` implementation that +would be able to proxy requests to and from remote clusters. It would be in the separate +`go module` to avoid introducing new dependencies to the core kcp and for easier split +of the codebase. + +### Option 1: Virtual workspace as a proxy + +Create a `VirtualWorkspace` implementation that would be able to proxy requests +to and from remote clusters. This would be very similar to previous TMC syncer +reverse tunnels that were used to proxy requests to remote clusters. + +Remote cluster would have an agent that would be responsible for establishing +a connection to the kcp cluster and would be responsible for proxying requests +to and from the remote cluster. + +`https://192.168.1.136:6443/services/clusterproxy/root/cluster.proxy.kcp.io` + +Proxy itself would be represented in a workspace via Cluster wide resource: +``` +apiVersion: proxy.kcp.io/v1alpha1 +kind: ClusterProxy +metadata: + name: cluster-proxy +spec: + ...... + status: + conditions: + - lastTransitionTime: "2023-10-07T15:30:12Z" + status: "True" + type: IdentityValid + - lastTransitionTime: "2023-10-07T15:30:12Z" + status: "True" + type: VirtualWorkspaceURLsReady + virtualWorkspaces: + - url: https://192.168.1.138:6443/services/clusterproxy/root/cluster.proxy.kcp.io +``` + +Once this resource is created, it would be possible to create a pass-through +connection. Where `ws use` could detect presence of the proxy and use it +instead of the workspace itself. This would need to land somehow in the kcp core, +similar how `homeWorkspaces` are handled. + +This basically removed users ability to manipulate the workspace directly, but +we could still allow users to manipulate it via custom header allowing interact with the workspace. + +## Option 2: Sub-workspace + +Create a fake workspace view in the CLI. This would follow same `VirtualWorkspace` +implementation as above, but would be represented as a sub-workspace. This would +show as bellow: +``` +$ kubectl ws tree +root + ├── clusters (workspace) + │ ├── cluster-proxy (proxy object but shown as a workspace in tree) +``` + +This way user can still manipulate the workspace directly, but would be able to +use it as a proxy as well by: `ws use root:clusters:cluster-proxy`. + +This would require some changes in the CLI to be able to detect that the workspace proxy +object is present and use it instead of the workspace itself. + +#### Suggested action items + +1. Agree on the possible implementation options. +2. TBC + + +#### Suggested timeline + +Completed in: EOF Q4 2024 From a1b957549dc4175c272b539200c21a863b848944 Mon Sep 17 00:00:00 2001 From: Mangirdas Judeikis Date: Sat, 28 Oct 2023 12:59:51 +0300 Subject: [PATCH 02/10] add example --- keps/core/0002-proxy-workspace.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/keps/core/0002-proxy-workspace.md b/keps/core/0002-proxy-workspace.md index a2bdd1e..858dd61 100644 --- a/keps/core/0002-proxy-workspace.md +++ b/keps/core/0002-proxy-workspace.md @@ -15,6 +15,24 @@ responsible for ArgoCD, Crossplane, or other infrastructure that is more crd-bas At this point in time communication between kcp and remote clusters is done via workspaces abstractions. +In example: +``` +$ kubectl ws tree +root: +root + ├── clusters (workspace) + │ ├── cluster-proxy (proxy object but shown as a workspace in tree) + │ ├── cluster-1 (workspace) + ├── infra (workspace) + │ ├── argocd (workspace) + │ ├── crossplane (workspace) +``` + +Where `cluster-proxy` is a workspace that is backed by a proxy that can be used +to proxy requests to remote clusters. This way we have unified way of interacting +with remote clusters, but still have ability to manage some of the infrastructure +via kcp. + ### Goals 1. Agree how to make proxy functionality as pluggable as possible. From ccd54a997fa4fcdcc005d427d64d01fbcebd3eb4 Mon Sep 17 00:00:00 2001 From: Mangirdas Judeikis Date: Sat, 28 Oct 2023 13:01:53 +0300 Subject: [PATCH 03/10] add tmc to the mix --- keps/core/0002-proxy-workspace.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/keps/core/0002-proxy-workspace.md b/keps/core/0002-proxy-workspace.md index 858dd61..716f06e 100644 --- a/keps/core/0002-proxy-workspace.md +++ b/keps/core/0002-proxy-workspace.md @@ -26,6 +26,9 @@ root ├── infra (workspace) │ ├── argocd (workspace) │ ├── crossplane (workspace) + ├── tmc (tmc workspace) + │ ├── cluster1 (TMC workspace) + ... ``` Where `cluster-proxy` is a workspace that is backed by a proxy that can be used From 4fe01190f92ac723c35883649d89417b031dc5f2 Mon Sep 17 00:00:00 2001 From: Mangirdas Judeikis Date: Sun, 29 Oct 2023 08:36:10 +0200 Subject: [PATCH 04/10] option 3 --- keps/core/0002-proxy-workspace.md | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/keps/core/0002-proxy-workspace.md b/keps/core/0002-proxy-workspace.md index 716f06e..a3e0661 100644 --- a/keps/core/0002-proxy-workspace.md +++ b/keps/core/0002-proxy-workspace.md @@ -111,6 +111,34 @@ use it as a proxy as well by: `ws use root:clusters:cluster-proxy`. This would require some changes in the CLI to be able to detect that the workspace proxy object is present and use it instead of the workspace itself. +## Option 3: Delegated workspace + +Suggestion is to add new api flag `--delegated-workspace-types` that would allow +to specify which workspace types should be delegated to external process to manage. +This flag would prevent any implementation of the workspace type in the kcp core +and would allow to be managed from outside of the kcp core. + +We would introduce workspace type `Proxy` and add reconcilers for it. +RBAC still need to be handled. For RBAC we still use `VirtualWorkspace` and store +rbac artifacts (secret, service account, role, role binding) in the kcp cluster parent workspace +together with `VirtualWorkspace` backing object. + +This way each child `proxy` workspace would need to correspond to an Proxy object: + +``` +apiVersion: proxy.kcp.io/v1alpha1 +kind: WorkspaceProxy +metadata: + name: cluster-proxy +spec: + type: passthrough +``` + +Where `WorkspaceProxy` object would expose `/tunnel` endpoint that would be used +for reverse dialing to the proxy. The proxying itself would need to be implemented +on Workspace level. + + #### Suggested action items 1. Agree on the possible implementation options. From 821734b8f91ac23da0a29cc6cb47e152df5388c1 Mon Sep 17 00:00:00 2001 From: Mangirdas Judeikis Date: Sat, 16 Mar 2024 13:25:17 +0200 Subject: [PATCH 05/10] add proxy/mount functionality --- keps/core/0002-proxy-workspace.md | 161 +++++++++++++----------------- 1 file changed, 71 insertions(+), 90 deletions(-) diff --git a/keps/core/0002-proxy-workspace.md b/keps/core/0002-proxy-workspace.md index a3e0661..c6013be 100644 --- a/keps/core/0002-proxy-workspace.md +++ b/keps/core/0002-proxy-workspace.md @@ -1,10 +1,11 @@ -# Proxy workspace enhancement proposal +# Workspace mounts enhancement proposal ## Summary To extend spectrum of use cases for kcp, we need to support workspaces that are not -backed by a kcp cluster itself, but are backed by a proxy that can be used to proxy -request to remote cluster directly, without having to go through kcp workspace. +backed by a kcp cluster itself, but are backed by a mount that can be used to proxy +request to remote cluster directly or other workspaces. This enables use cases where +we can start adding external & internal integration into same workspace structure. ## Motivation @@ -26,106 +27,92 @@ root ├── infra (workspace) │ ├── argocd (workspace) │ ├── crossplane (workspace) - ├── tmc (tmc workspace) - │ ├── cluster1 (TMC workspace) ... ``` -Where `cluster-proxy` is a workspace that is backed by a proxy that can be used +Where `cluster-mount` is a workspace that is backed by a mount that can be used to proxy requests to remote clusters. This way we have unified way of interacting with remote clusters, but still have ability to manage some of the infrastructure via kcp. ### Goals -1. Agree how to make proxy functionality as pluggable as possible. -2. Agree on the scope of the proxy functionality. -3. Agree on the implementation of the proxy functionality. +1. Agree how to make mount functionality as pluggable as possible. +2. Agree on the scope of the mount functionality. +3. Agree on the implementation of the mount functionality. ### Non-Goals -N/A +Proxy/Mount functionality is not meant to be a replacement for the kcp clusters. +In addition implementation of the mount functionality itself is out of scope of this +proposal. We should enable the mount functionality to be implemented by 3rd party +using kcp as a platform. ## Proposal -Both options below would require a new `VirtualWorkspace` implementation that -would be able to proxy requests to and from remote clusters. It would be in the separate -`go module` to avoid introducing new dependencies to the core kcp and for easier split -of the codebase. - -### Option 1: Virtual workspace as a proxy - -Create a `VirtualWorkspace` implementation that would be able to proxy requests -to and from remote clusters. This would be very similar to previous TMC syncer -reverse tunnels that were used to proxy requests to remote clusters. - -Remote cluster would have an agent that would be responsible for establishing -a connection to the kcp cluster and would be responsible for proxying requests -to and from the remote cluster. - -`https://192.168.1.136:6443/services/clusterproxy/root/cluster.proxy.kcp.io` - -Proxy itself would be represented in a workspace via Cluster wide resource: -``` -apiVersion: proxy.kcp.io/v1alpha1 -kind: ClusterProxy -metadata: - name: cluster-proxy -spec: - ...... - status: - conditions: - - lastTransitionTime: "2023-10-07T15:30:12Z" - status: "True" - type: IdentityValid - - lastTransitionTime: "2023-10-07T15:30:12Z" - status: "True" - type: VirtualWorkspaceURLsReady - virtualWorkspaces: - - url: https://192.168.1.138:6443/services/clusterproxy/root/cluster.proxy.kcp.io +Introduce experimental `Mount` API to enable 2 use cases: + +1. Mounting remote clusters as workspaces +2. Softlink - mounting other workspaces as sub-workspaces, where changing the sub-workspace + would change the parent workspace as well. + +### Mount API + +```go + +// Mount is a workspace mount that can be used to mount a workspace into another workspace or resource. +// Mounting itself is done at front proxy level. +type Mount struct { + // MountSpec is the spec of the mount. + MountSpec MountSpec `json:"spec,omitempty"` + // MountStatus is the status of the mount. + MountStatus MountStatus `json:"status,omitempty"` +} + +type MountSpec struct { + // Reference is an ObjectReference to the object that is mounted. + Reference *corev1.ObjectReference `json:"ref,omitempty"` + // Type is the type of the mount (URL, Redirect, Workspace). + // +kubebuilder:default=Cluster + Type MountType `json:"type,omitempty"` +} + +// MountStatus is the status of a mount. It is used to indicate the status of a mount, +// potentially managed outside of the core API. +type MountStatus struct { + // Phase of the mount (Initializing, Connecting, Ready, Unknown). + // + // +kubebuilder:default=Initializing + Phase MountPhaseType `json:"phase,omitempty"` + // Conditions is a list of conditions and their status. + // Current processing state of the Mount. + // +optional + Conditions conditionsv1alpha1.Conditions `json:"conditions,omitempty"` + + // URL is the URL of the mount. Mount is considered mountable when URL is set. + // +optional + URL string `json:"url,omitempty"` +} ``` -Once this resource is created, it would be possible to create a pass-through -connection. Where `ws use` could detect presence of the proxy and use it -instead of the workspace itself. This would need to land somehow in the kcp core, -similar how `homeWorkspaces` are handled. +Where `MountType` is an enum that would allow to specify the type of the mount: +- `URL` - mount is backed by a URL and should be used as a proxy to remote cluster. +- `Redirect` - mount is backed by another workspace and should be used as redirect to another workspace. +- `Workspace` - mount is backed by another workspace and should be used as a proxy to another workspace. -This basically removed users ability to manipulate the workspace directly, but -we could still allow users to manipulate it via custom header allowing interact with the workspace. +`URL` implementation should be done outside of the kcp core, but we should provide +via VirtualWorkspace. -## Option 2: Sub-workspace +`Redirect` and `Workspace` should be implemented in the kcp core. -Create a fake workspace view in the CLI. This would follow same `VirtualWorkspace` -implementation as above, but would be represented as a sub-workspace. This would -show as bellow: -``` -$ kubectl ws tree -root - ├── clusters (workspace) - │ ├── cluster-proxy (proxy object but shown as a workspace in tree) -``` - -This way user can still manipulate the workspace directly, but would be able to -use it as a proxy as well by: `ws use root:clusters:cluster-proxy`. - -This would require some changes in the CLI to be able to detect that the workspace proxy -object is present and use it instead of the workspace itself. - -## Option 3: Delegated workspace +`Reference` is a reference to the object that is mounted. It can be a reference to +external object used to back the mount. -Suggestion is to add new api flag `--delegated-workspace-types` that would allow -to specify which workspace types should be delegated to external process to manage. -This flag would prevent any implementation of the workspace type in the kcp core -and would allow to be managed from outside of the kcp core. +Object reference would be read by KCP core mount controller and update the status +of the workspace. This way 3rd party components never need to interact with the +kcp core or workspace directly in the write mode. -We would introduce workspace type `Proxy` and add reconcilers for it. -RBAC still need to be handled. For RBAC we still use `VirtualWorkspace` and store -rbac artifacts (secret, service account, role, role binding) in the kcp cluster parent workspace -together with `VirtualWorkspace` backing object. - -This way each child `proxy` workspace would need to correspond to an Proxy object: - -``` +```yaml apiVersion: proxy.kcp.io/v1alpha1 kind: WorkspaceProxy metadata: @@ -134,17 +121,11 @@ spec: type: passthrough ``` -Where `WorkspaceProxy` object would expose `/tunnel` endpoint that would be used -for reverse dialing to the proxy. The proxying itself would need to be implemented -on Workspace level. - - #### Suggested action items -1. Agree on the possible implementation options. -2. TBC - +1. Agree on the scope of the mount functionality. +2. Agree on the implementation of the mount functionality. #### Suggested timeline -Completed in: EOF Q4 2024 +At the best effort basis. From 1ef703d332e426b712c74eef92aa2b67de818209 Mon Sep 17 00:00:00 2001 From: Mangirdas Judeikis Date: Sat, 16 Mar 2024 13:27:12 +0200 Subject: [PATCH 06/10] nit about annotation vs spec --- keps/core/0002-proxy-workspace.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/keps/core/0002-proxy-workspace.md b/keps/core/0002-proxy-workspace.md index c6013be..9f77055 100644 --- a/keps/core/0002-proxy-workspace.md +++ b/keps/core/0002-proxy-workspace.md @@ -56,6 +56,9 @@ Introduce experimental `Mount` API to enable 2 use cases: 2. Softlink - mounting other workspaces as sub-workspaces, where changing the sub-workspace would change the parent workspace as well. +Initially we would put `Mount` into annotation of the workspace, but in the future +we will promote it to spec of the workspace. + ### Mount API ```go From d791b0a6660e39458983d2e5d91ca65384201e32 Mon Sep 17 00:00:00 2001 From: Mangirdas Judeikis Date: Sun, 28 Apr 2024 17:31:19 +0300 Subject: [PATCH 07/10] addressed the scope and terminology --- keps/core/0002-proxy-workspace.md | 71 +++++++++++++++++++++++-------- 1 file changed, 53 insertions(+), 18 deletions(-) diff --git a/keps/core/0002-proxy-workspace.md b/keps/core/0002-proxy-workspace.md index 9f77055..0580098 100644 --- a/keps/core/0002-proxy-workspace.md +++ b/keps/core/0002-proxy-workspace.md @@ -22,7 +22,7 @@ $ kubectl ws tree root: root ├── clusters (workspace) - │ ├── cluster-proxy (proxy object but shown as a workspace in tree) + │ ├── cluster-mount (normal workspace, with mounted cluster ontop of it) │ ├── cluster-1 (workspace) ├── infra (workspace) │ ├── argocd (workspace) @@ -37,17 +37,25 @@ via kcp. ### Goals -1. Agree how to make mount functionality as pluggable as possible. -2. Agree on the scope of the mount functionality. -3. Agree on the implementation of the mount functionality. +1. Mounting remote clusters as workspaces +2. Softlink - mounting other workspaces as sub-workspaces, where changing the sub-workspace +would change the parent workspace as well. ### Non-Goals -Proxy/Mount functionality is not meant to be a replacement for the kcp clusters. +* Proxy/Mount functionality is not meant to be a replacement for the kcp clusters. In addition implementation of the mount functionality itself is out of scope of this proposal. We should enable the mount functionality to be implemented by 3rd party using kcp as a platform. +* Mount resources. This proposal is about enabling the mount functionality for all +workspaces, not about mounting individual resources inside workspace. + +* AuthN/authZ is out of scope of this proposal. We assume that the mount functionality +implements just forwarding and redirections. Authentification and authorization +is implementers responsibility by either federating identities or using other means +to authenticate and authorize requests. + ## Proposal Introduce experimental `Mount` API to enable 2 use cases: @@ -61,6 +69,16 @@ we will promote it to spec of the workspace. ### Mount API +This is a proposal for mount API. This contains 2 parts: + +1. Experimental mount API used in the annotation of the workspace. +2. Proposal how this could looks like when promoted to workspace structure. + + +#### Experimental mount API + +As experimental API we would put mount into annotation of the workspace. + ```go // Mount is a workspace mount that can be used to mount a workspace into another workspace or resource. @@ -75,9 +93,9 @@ type Mount struct { type MountSpec struct { // Reference is an ObjectReference to the object that is mounted. Reference *corev1.ObjectReference `json:"ref,omitempty"` - // Type is the type of the mount (URL, Redirect, Workspace). - // +kubebuilder:default=Cluster - Type MountType `json:"type,omitempty"` + // Type is the type of the mount (URL, Redirect, Workspace). + // +kubebuilder:default=Cluster + Type MountType `json:"type,omitempty"` } // MountStatus is the status of a mount. It is used to indicate the status of a mount, @@ -92,31 +110,48 @@ type MountStatus struct { // +optional Conditions conditionsv1alpha1.Conditions `json:"conditions,omitempty"` - // URL is the URL of the mount. Mount is considered mountable when URL is set. + // URL is the URL of the mount. Mount is considered mounted when URL is set. // +optional URL string `json:"url,omitempty"` } ``` Where `MountType` is an enum that would allow to specify the type of the mount: -- `URL` - mount is backed by a URL and should be used as a proxy to remote cluster. +- `Proxy` - mount is backed by a URL and should be used as a proxy to remote cluster. - `Redirect` - mount is backed by another workspace and should be used as redirect to another workspace. -- `Workspace` - mount is backed by another workspace and should be used as a proxy to another workspace. +- `LogicalCluster` - mount is backed by another workspace and should be used as a proxy to another workspace/logicalcluster. + +`Proxy` most simple proxy implementation is `URL` where we would just proxy all requests. More advanced +implementation could include `VirtualWorkspace` or external service. But from KCP perspective +its reverse proxy to configured URL. + +`Redirect` and `LogicalCluster` has two main behaviours. These include softlinking of the (external or internal). + +1. Case 1: Local redirect: + +As a user I want to redirect workspace `:users:john` to `:my-org:dev:john` so that I can have +a redirect inside the same instance of kcp. In this case frontoproxy returns `301 Redirect` request +with local (hostless) path to the new workspace. CLI should be able to follow the redirect +and update local kubeconfig to point to the new workspace. -`URL` implementation should be done outside of the kcp core, but we should provide -via VirtualWorkspace. +2. Case 2: External redirect: -`Redirect` and `Workspace` should be implemented in the kcp core. +As a user I want to redirect workspace `:users:john` to `https://kcp.mycompany.com/workspaces/my-org/dev/john` +so that I can have a redirect to another kcp instance. In this case frontoproxy returns `301 Redirect` request +with full URL to the new workspace. CLI should be able to follow the redirect and update local kubeconfig. +This would allow nest different kcp instances. `Reference` is a reference to the object that is mounted. It can be a reference to external object used to back the mount. -Object reference would be read by KCP core mount controller and update the status -of the workspace. This way 3rd party components never need to interact with the -kcp core or workspace directly in the write mode. +Object reference would be read by KCP core mount controller. It will retrieve the +proxy URL from the referenced object's status and copy it into the mount status. +This way 3rd party components never need to interact with the kcp core or workspace +directly in write mode. Moreover, the front-proxy interpreting the mount status +URL does not have to read these mount implementation objects directly either. ```yaml -apiVersion: proxy.kcp.io/v1alpha1 +apiVersion: proxy.contrib.kcp.io/v1alpha1 kind: WorkspaceProxy metadata: name: cluster-proxy From 732fbec6ac92c1a306ff3f36f9f0290688c5a82c Mon Sep 17 00:00:00 2001 From: Mangirdas Judeikis Date: Sun, 28 Apr 2024 18:02:20 +0300 Subject: [PATCH 08/10] add workspace status propagation --- keps/core/0002-proxy-workspace.md | 109 ++++++++++++++++++++++++++++++ 1 file changed, 109 insertions(+) diff --git a/keps/core/0002-proxy-workspace.md b/keps/core/0002-proxy-workspace.md index 0580098..1c4dbb2 100644 --- a/keps/core/0002-proxy-workspace.md +++ b/keps/core/0002-proxy-workspace.md @@ -40,6 +40,8 @@ via kcp. 1. Mounting remote clusters as workspaces 2. Softlink - mounting other workspaces as sub-workspaces, where changing the sub-workspace would change the parent workspace as well. +3. Workspace status propagation - when mounting workspace, there should be a way to +propagate status to the workspace itself, preventing false positives serves. ### Non-Goals @@ -116,6 +118,56 @@ type MountStatus struct { } ``` +#### Promoted Workspace API + +When mount is promoted to workspace spec, it could look like example bellow. +Mount would be just yet another condition on the workspace. Most important change +is that mount would be part of the workspace spec and if mount implementation would +become unhealthy, it would be reflected in the workspace conditions. And this would drive +`Phase` change of the workspace. More on this in the next section. + +```go +// WorkspaceSpec defines the desired state of Workspace +type WorkspaceSpec struct { + ... + // Mount is a workspace mount that can be used to mount a workspace into another workspace or resource. + Mount *Mount `json:"mount,omitempty"` +} + +// Mount is a workspace mount that can be used to mount a workspace into another workspace or resource. +type Mount struct { + // Reference is an ObjectReference to the object that is mounted. + Reference *corev1.ObjectReference `json:"ref,omitempty"` + // Type is the type of the mount (URL, Redirect, Workspace). + // +kubebuilder:default=Cluster + Type MountType `json:"type,omitempty"` +} + +// WorkspaceStatus defines the observed state of Workspace +type WorkspaceStatus struct { + // Phase of the workspace (Scheduling, Initializing, Ready). + // + // +kubebuilder:default=Scheduling + Phase corev1alpha1.LogicalClusterPhaseType `json:"phase,omitempty"` + + // Current processing state of the Workspace. + // +optional + Conditions conditionsv1alpha1.Conditions `json:"conditions,omitempty"` + + // initializers must be cleared by a controller before the workspace is ready + // and can be used. + // + // +optional + Initializers []corev1alpha1.LogicalClusterInitializer `json:"initializers,omitempty"` + + // MountURL is the URL of the mount. Mount is considered mounted when URL is set. + // +optional + MountURL string `json:"mountURL,omitempty"` +} +``` + +#### Type explanations + Where `MountType` is an enum that would allow to specify the type of the mount: - `Proxy` - mount is backed by a URL and should be used as a proxy to remote cluster. - `Redirect` - mount is backed by another workspace and should be used as redirect to another workspace. @@ -159,6 +211,63 @@ spec: type: passthrough ``` +### Workspace & Mount status propagation + +When mounting workspace, there should be a way to propagate status to the workspace itself, +so making sure that the workspace is not falsely marked as ready when the mount is not ready. + +When workspace mount is implemented by external objects: +```go + // Reference is an ObjectReference to the object that is mounted. + Reference *corev1.ObjectReference `json:"ref,omitempty"` +``` + +We need a way to propagate status from the mount object to the workspace object. + +Suggestion is to create `kcp` core controller which would propagate any conditions +from the mount object to the workspace object which starts with `Workspace*` prefix. + +if object, implementing object raises conditions: +```yaml + conditions: + - lastTransitionTime: "2024-04-28T10:43:29Z" + status: "False" + type: ConnectorReady + - lastTransitionTime: "2024-04-28T10:43:29Z" + status: "False" + type: WorkspaceMountReady +``` +it would be propagated to the workspace object as bellow, making workspace +unavailable: +```yaml + status: + conditions: + - lastTransitionTime: "2024-04-28T10:43:29Z" + status: "True" + type: MountReady + - lastTransitionTime: "2024-04-28T10:40:25Z" + status: "True" + type: WorkspaceScheduled + - lastTransitionTime: "2024-04-28T10:43:29Z" + status: "False" + type: MountReady + phase: Unavailable +``` + +Proposal is to make workspace conditions propagated as aggregate to workspace `phase`. + +Current flow: `Creating -> Initializing -> Ready` where once workspace is created and ready +it is marked as ready and never updated. + +Proposed flow: `Creating -> Initializing -> Ready <-> Unavailable.` where if any of the conditions +are not `Ready` the workspace is marked as `Unavailable`. + +This would be achieved my adding new controller into kcp core to reconcile workspace phase +based on its own conditions. + +Existing mount controller would be responsible for propagating conditions from the mount object +to the workspace object. + #### Suggested action items 1. Agree on the scope of the mount functionality. From 9e7630db15e9114d7c91f042126cb46c0095a356 Mon Sep 17 00:00:00 2001 From: Mangirdas Judeikis Date: Sun, 28 Apr 2024 18:19:16 +0300 Subject: [PATCH 09/10] clean the language --- keps/core/0002-proxy-workspace.md | 177 +++++++++++++++--------------- 1 file changed, 90 insertions(+), 87 deletions(-) diff --git a/keps/core/0002-proxy-workspace.md b/keps/core/0002-proxy-workspace.md index 1c4dbb2..49408cf 100644 --- a/keps/core/0002-proxy-workspace.md +++ b/keps/core/0002-proxy-workspace.md @@ -1,25 +1,23 @@ -# Workspace mounts enhancement proposal +# Workspace Mounts Enhancement Proposal ## Summary -To extend spectrum of use cases for kcp, we need to support workspaces that are not -backed by a kcp cluster itself, but are backed by a mount that can be used to proxy -request to remote cluster directly or other workspaces. This enables use cases where -we can start adding external & internal integration into same workspace structure. +To extend the spectrum of use cases for `kcp`, we need to support workspaces that +are not backed by a `kcp` cluster itself but are backed by a mount that can be used +to proxy requests directly to remote clusters or other workspaces. This enables +use cases where external and internal integrations can be added into the same workspace structure. ## Motivation -This would enable usecases where kcp is used as a proxy to remote clusters, but -some of the infrastructure is still managed by kcp. For example, kcp could be -responsible for ArgoCD, Crossplane, or other infrastructure that is more crd-based. +This would enable use cases where `kcp` is used as a proxy to remote clusters, but +some of the infrastructure is still managed by `kcp`. For example, `kcp` could be +responsible for ArgoCD, Crossplane, or other infrastructure that is more CRD-based. -At this point in time communication between kcp and remote clusters is done via -workspaces abstractions. +At this point, communication between `kcp` and remote clusters is done via workspace abstractions. -In example: +For example: ``` $ kubectl ws tree -root: root ├── clusters (workspace) │ ├── cluster-mount (normal workspace, with mounted cluster ontop of it) @@ -30,61 +28,60 @@ root ... ``` -Where `cluster-mount` is a workspace that is backed by a mount that can be used -to proxy requests to remote clusters. This way we have unified way of interacting -with remote clusters, but still have ability to manage some of the infrastructure -via kcp. +Where `cluster-mount` is a workspace backed by a mount that can be used to proxy +requests to remote clusters. This provides a unified way of interacting with +remote clusters while still having the ability to manage some of the infrastructure +via `kcp`. ### Goals -1. Mounting remote clusters as workspaces -2. Softlink - mounting other workspaces as sub-workspaces, where changing the sub-workspace -would change the parent workspace as well. -3. Workspace status propagation - when mounting workspace, there should be a way to -propagate status to the workspace itself, preventing false positives serves. +1. Mounting remote clusters as workspaces. +2. Softlink - mounting other workspaces as sub-workspaces, where changing the + sub-workspace would change the parent workspace as well. +3. Workspace status propagation - when mounting a workspace, there should be a + way to propagate the status to the workspace itself, preventing false positives. ### Non-Goals -* Proxy/Mount functionality is not meant to be a replacement for the kcp clusters. -In addition implementation of the mount functionality itself is out of scope of this -proposal. We should enable the mount functionality to be implemented by 3rd party -using kcp as a platform. +* Proxy/Mount functionality is not meant to replace `kcp` clusters. Additionally, + the implementation of the mount functionality itself is out of the scope of this + proposal. We should enable the mount functionality to be implemented by third + parties using `kcp` as a platform. -* Mount resources. This proposal is about enabling the mount functionality for all -workspaces, not about mounting individual resources inside workspace. +* Mount resources. This proposal is about enabling the mount functionality for + all workspaces, not about mounting individual resources inside a workspace. -* AuthN/authZ is out of scope of this proposal. We assume that the mount functionality -implements just forwarding and redirections. Authentification and authorization -is implementers responsibility by either federating identities or using other means -to authenticate and authorize requests. +* AuthN/AuthZ is out of scope of this proposal. We assume that the mount functionality + implements just forwarding and redirections. Authentication and authorization are + the implementer's responsibility, either by federating identities or using other + means to authenticate and authorize requests. ## Proposal -Introduce experimental `Mount` API to enable 2 use cases: +Introduce an experimental `Mount` API to enable two use cases: -1. Mounting remote clusters as workspaces -2. Softlink - mounting other workspaces as sub-workspaces, where changing the sub-workspace - would change the parent workspace as well. +1. Mounting remote clusters as workspaces. +2. Softlink - mounting other workspaces as sub-workspaces, where changing the + sub-workspace would change the parent workspace as well. -Initially we would put `Mount` into annotation of the workspace, but in the future -we will promote it to spec of the workspace. +Initially, we would put Mount into the annotation of the workspace, but in the +future, we will promote it to the spec of the workspace. ### Mount API -This is a proposal for mount API. This contains 2 parts: +This is a proposal for a mount API. This contains two parts: 1. Experimental mount API used in the annotation of the workspace. -2. Proposal how this could looks like when promoted to workspace structure. +2. Proposal for how this could look when promoted to the workspace structure. -#### Experimental mount API +#### Experimental Mount API -As experimental API we would put mount into annotation of the workspace. +As an experimental API, we would place the mount into the annotation of the workspace. ```go - // Mount is a workspace mount that can be used to mount a workspace into another workspace or resource. -// Mounting itself is done at front proxy level. +// Mounting itself is done at the front proxy level. type Mount struct { // MountSpec is the spec of the mount. MountSpec MountSpec `json:"spec,omitempty"` @@ -112,7 +109,7 @@ type MountStatus struct { // +optional Conditions conditionsv1alpha1.Conditions `json:"conditions,omitempty"` - // URL is the URL of the mount. Mount is considered mounted when URL is set. + // URL is the URL of the mount. Mount is considered mounted when the URL is set. // +optional URL string `json:"url,omitempty"` } @@ -120,11 +117,11 @@ type MountStatus struct { #### Promoted Workspace API -When mount is promoted to workspace spec, it could look like example bellow. -Mount would be just yet another condition on the workspace. Most important change -is that mount would be part of the workspace spec and if mount implementation would -become unhealthy, it would be reflected in the workspace conditions. And this would drive -`Phase` change of the workspace. More on this in the next section. +When the mount is promoted to the workspace spec, it could look like the example +below. Mount would be just yet another condition on the workspace. The most important +change is that the mount would be part of the workspace spec, and if the mount +implementation becomes unhealthy, it would be reflected in the workspace conditions. +This would drive the `Phase` change of the workspace. More on this in the next section. ```go // WorkspaceSpec defines the desired state of Workspace @@ -154,13 +151,13 @@ type WorkspaceStatus struct { // +optional Conditions conditionsv1alpha1.Conditions `json:"conditions,omitempty"` - // initializers must be cleared by a controller before the workspace is ready + // Initializers must be cleared by a controller before the workspace is ready // and can be used. // // +optional Initializers []corev1alpha1.LogicalClusterInitializer `json:"initializers,omitempty"` - // MountURL is the URL of the mount. Mount is considered mounted when URL is set. + // MountURL is the URL of the mount. Mount is considered mounted when the URL is set. // +optional MountURL string `json:"mountURL,omitempty"` } @@ -168,39 +165,43 @@ type WorkspaceStatus struct { #### Type explanations -Where `MountType` is an enum that would allow to specify the type of the mount: -- `Proxy` - mount is backed by a URL and should be used as a proxy to remote cluster. -- `Redirect` - mount is backed by another workspace and should be used as redirect to another workspace. -- `LogicalCluster` - mount is backed by another workspace and should be used as a proxy to another workspace/logicalcluster. +Where `MountType` is an enum that would allow specifying the type of the mount: + +- `Proxy` - the mount is backed by a URL and should be used as a proxy to a remote cluster. +- `Redirect` - the mount is backed by another workspace and should be used as a redirect to another workspace. +- `LogicalCluster` - the mount is backed by another workspace and should be used as a proxy to another workspace/logical cluster. -`Proxy` most simple proxy implementation is `URL` where we would just proxy all requests. More advanced -implementation could include `VirtualWorkspace` or external service. But from KCP perspective -its reverse proxy to configured URL. +`Proxy` most simple proxy implementation is URL where we would just proxy all requests. +A more advanced implementation could include `VirtualWorkspace` or an external +service. But from a `kcp` perspective, it's a reverse proxy to a configured URL. -`Redirect` and `LogicalCluster` has two main behaviours. These include softlinking of the (external or internal). +`Redirect` and `LogicalCluster` have two main behaviors. These include softlinking +of the (external or internal). 1. Case 1: Local redirect: -As a user I want to redirect workspace `:users:john` to `:my-org:dev:john` so that I can have -a redirect inside the same instance of kcp. In this case frontoproxy returns `301 Redirect` request -with local (hostless) path to the new workspace. CLI should be able to follow the redirect -and update local kubeconfig to point to the new workspace. +As a user, I want to redirect workspace `:users:john` to `:my-org:dev:john` so +that I can have a redirect inside the same instance of `kcp`. In this case, the +front proxy returns a `301 Redirect` request with a local (hostless) path to the +new workspace. The CLI should be able to follow the redirect and update the local +kubeconfig to point to the new workspace. -2. Case 2: External redirect: +1. Case 2: External redirect: -As a user I want to redirect workspace `:users:john` to `https://kcp.mycompany.com/workspaces/my-org/dev/john` -so that I can have a redirect to another kcp instance. In this case frontoproxy returns `301 Redirect` request -with full URL to the new workspace. CLI should be able to follow the redirect and update local kubeconfig. -This would allow nest different kcp instances. +As a user, I want to redirect workspace `:users:john` to `https://kcp.mycompany.com/workspaces/my-org/dev/john` +so that I can have a redirect to another `kcp` instance. In this case, the front proxy +returns a `301 Redirect` request with a full URL to the new workspace. The CLI +should be able to follow the redirect and update the local kubeconfig. This +would allow nesting different `kcp` instances. -`Reference` is a reference to the object that is mounted. It can be a reference to -external object used to back the mount. +`Reference` is a reference to the object that is mounted. It can be a reference +to an external object used to back the mount. -Object reference would be read by KCP core mount controller. It will retrieve the -proxy URL from the referenced object's status and copy it into the mount status. -This way 3rd party components never need to interact with the kcp core or workspace -directly in write mode. Moreover, the front-proxy interpreting the mount status -URL does not have to read these mount implementation objects directly either. +Object reference would be read by the `kcp` core mount controller. It will retrieve +the proxy URL from the referenced object's status and copy it into the mount status. +This way, third-party components never need to interact with the `kcp` core or workspace +directly in write mode. Moreover, the front-proxy interpreting the mount status URL +does not have to read these mount implementation objects directly either. ```yaml apiVersion: proxy.contrib.kcp.io/v1alpha1 @@ -211,23 +212,25 @@ spec: type: passthrough ``` -### Workspace & Mount status propagation +### Workspace & Mount Status Propagation -When mounting workspace, there should be a way to propagate status to the workspace itself, -so making sure that the workspace is not falsely marked as ready when the mount is not ready. +When mounting a workspace, there should be a way to propagate the status to the +workspace itself, ensuring that the workspace is not falsely marked as ready when +the mount is not ready. -When workspace mount is implemented by external objects: +When a workspace mount is implemented by external objects: ```go // Reference is an ObjectReference to the object that is mounted. Reference *corev1.ObjectReference `json:"ref,omitempty"` ``` -We need a way to propagate status from the mount object to the workspace object. +We need a way to propagate the status from the mount object to the workspace object. + +The suggestion is to create a `kcp` core controller which would propagate any +conditions from the mount object to the workspace object which starts with the `Workspace*` prefix. -Suggestion is to create `kcp` core controller which would propagate any conditions -from the mount object to the workspace object which starts with `Workspace*` prefix. +If an object, implementing object raises conditions: -if object, implementing object raises conditions: ```yaml conditions: - lastTransitionTime: "2024-04-28T10:43:29Z" @@ -262,7 +265,7 @@ it is marked as ready and never updated. Proposed flow: `Creating -> Initializing -> Ready <-> Unavailable.` where if any of the conditions are not `Ready` the workspace is marked as `Unavailable`. -This would be achieved my adding new controller into kcp core to reconcile workspace phase +This would be achieved my adding new controller into `kcp` core to reconcile workspace phase based on its own conditions. Existing mount controller would be responsible for propagating conditions from the mount object @@ -270,9 +273,9 @@ to the workspace object. #### Suggested action items -1. Agree on the scope of the mount functionality. -2. Agree on the implementation of the mount functionality. +1. Define the scope of the mount functionality. +2. Determine the implementation approach for the mount functionality. #### Suggested timeline -At the best effort basis. +Efforts will proceed on a best-effort basis. From 5d220652793ead5868b7046cfca66315fc195acc Mon Sep 17 00:00:00 2001 From: Mangirdas Judeikis Date: Sun, 28 Apr 2024 18:20:36 +0300 Subject: [PATCH 10/10] tabs vs spaces --- keps/core/0002-proxy-workspace.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/keps/core/0002-proxy-workspace.md b/keps/core/0002-proxy-workspace.md index 49408cf..2e3f466 100644 --- a/keps/core/0002-proxy-workspace.md +++ b/keps/core/0002-proxy-workspace.md @@ -236,9 +236,9 @@ If an object, implementing object raises conditions: - lastTransitionTime: "2024-04-28T10:43:29Z" status: "False" type: ConnectorReady - - lastTransitionTime: "2024-04-28T10:43:29Z" - status: "False" - type: WorkspaceMountReady + - lastTransitionTime: "2024-04-28T10:43:29Z" + status: "False" + type: WorkspaceMountReady ``` it would be propagated to the workspace object as bellow, making workspace unavailable: @@ -251,9 +251,9 @@ unavailable: - lastTransitionTime: "2024-04-28T10:40:25Z" status: "True" type: WorkspaceScheduled - - lastTransitionTime: "2024-04-28T10:43:29Z" - status: "False" - type: MountReady + - lastTransitionTime: "2024-04-28T10:43:29Z" + status: "False" + type: MountReady phase: Unavailable ```