How to aggregate OCM crds in one api

[softcane]

I'm working with the Open Cluster Management Project. It's a powerful tool, but it includes many CRDs, such as ManifestWorkReplicaSet, ManifestWork (in the cluster namespace), PlacementDecision, and ManagedClusterSetBinding.

We're not planning to use this project to manage deployments across clusters. Instead, we see it as a platform control plane component mainly to keep track of clusters and manage tenant configurations, while we handle orchestration ourselves.

I want to create a new custom resource called TenantDeploymentRecord using Kube Builder. This CRD should combine information from all the existing CRDs mentioned above. It should also update automatically whenever any of those CRDs change.

Example snippet

apiVersion: my.control-plane.api.v1alpha1/v1alpha1
kind: TenantDeploymentRecord
metadata:
  name: sample-tenant
spec:
  environment: prod
  clusterType: api
  placementRef:
    - edge-01
    - edge-02
  workload:
    - clusterName: edge-01
      manifest:
        apiVersion: apps/v1
        kind: Deployment
        metadata:
          name: frontend
        spec:
          replicas: 3
          selector:
            matchLabels: {app: frontend}
          template:
            metadata:
              labels: {app: frontend}
            spec:
              containers:
                - name: nginx
                  image: nginx:1.21
    - clusterName: edge-02
      manifest:
      workloads:
        - manifest:
            apiVersion: v1
            kind: ConfigMap
            metadata:
              name: shared-config
            data:
              LOG_LEVEL: info

I’d appreciate any feedback or suggestions from the community.

[dosubot]

There isn't any official documentation or examples within the Open Cluster Management (OCM) project or its website that directly address aggregating multiple CRDs into a single API using Kube Builder, nor are there existing discussions or issues on this topic.

However, based on general Kubernetes and Kube Builder practices, here’s a high-level approach you can take:

1. Define Your Aggregating CRD:
   Use Kube Builder to scaffold a new CRD (e.g., TenantDeploymentRecord) that includes fields referencing or embedding the data you want to aggregate from ManifestWorkReplicaSet, ManifestWork, PlacementDecision, and ManagedClusterSetBinding.

2. Implement a Controller for Aggregation:
   Write a controller (using Kube Builder) that:

   • Watches for changes to the relevant OCM CRDs (using controller-runtime's multi-resource watch capabilities).
   • When any of those CRDs change, the controller fetches the latest state of all relevant resources.
   • Aggregates the data into your custom resource (TenantDeploymentRecord), updating it as needed.

3. Reconciliation Logic:
   In your controller’s Reconcile function, implement logic to:

   • List and read the current state of the OCM CRDs.
   • Transform and combine the data into the structure you want for your custom resource.
   • Update the TenantDeploymentRecord resource accordingly.

4. Automatic Updates:
   By setting up watches on all the relevant CRDs, your controller ensures that any changes in those resources will trigger a reconciliation and keep your aggregated resource up to date.

References for Implementation:

• Kube Builder Book: Writing a Controller
• controller-runtime: Watching Multiple Resources

Key Points:

• This pattern is common for building "summary" or "status" resources in Kubernetes.
• You will need to handle permissions (RBAC) so your controller can read the OCM CRDs.
• There is no built-in OCM support for this aggregation; it will be a custom implementation.

If you need more concrete code examples or have questions about specific parts of the implementation, let me know!

To reply, just mention my name @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}   

[qiujian16]

Hi, thanks for raising this issue.

we see it as a platform control plane component mainly to keep track of clusters and manage tenant configurations, while we handle orchestration ourselves.

This is certainly a valid way to use ocm.

Regarding to the API, would you like to provide more info including

• a list of use cases that you would want to achieve via this API.
• what is main difference from manifestworkreplicaset, e.g. could we enhance manifestworkreplicaset API to achieve the same goal?
  I think this will help us to understand the whole picture :)

[softcane]

@qiujian16
Here are some top-level use cases.

• The ManifestWorkReplicaSet includes a reference to a Placement, which depends on a PlacementDecision. We want to make this simpler by creating a version of the manifest that directly lists all the clusters from the PlacementDecision, without needing to follow references.

• We manage a separate CRD for tenant configurations, because this information can’t be added directly into the ManifestWorkReplicaSet. An aggregator CRD would help us include this information.

• In our setup, the TenantDeploymentRecord is not used by the controller to deploy components directly. Instead, we want to use each tenant’s preferred orchestration method for deployment. The TenantDeploymentRecord just acts as a single API that holds all relevant context in one place, so tenants can easily access and use it.

• The TenantDeploymentRecord also works as an abstraction layer over OCM (Open Cluster Management), which helps reduce our dependency on it and makes things simpler.

We’d appreciate your guidance on this approach.