Skip to content

Commit 4e5b9ad

Browse files
jerqialexcqtan
authored and
gibchikafa
committed
[apache#288] feat(operator): Supports mounting PVC for shuffleserver (apache#1813)
### What changes were proposed in this pull request? Previous pull request is apache#466 Supports mounting PVC for shuffleserver ### Why are the changes needed? Fix: apache#288 ### Does this PR introduce _any_ user-facing change? Yes, added the document. ### How was this patch tested? Add uts and test by hand in the GKE. Co-authored-by: alexcqtan <alexcqtan@tencent.com>
1 parent 7e010cb commit 4e5b9ad

File tree

8 files changed

+437
-0
lines changed

8 files changed

+437
-0
lines changed

‎deploy/kubernetes/operator/api/uniffle/v1alpha1/remoteshuffleservice_types.go

+23
Original file line numberDiff line numberDiff line change
@@ -142,6 +142,29 @@ type ShuffleServerConfig struct {
142142

143143
// UpgradeStrategy defines upgrade strategy of shuffle servers.
144144
UpgradeStrategy *ShuffleServerUpgradeStrategy `json:"upgradeStrategy"`
145+
146+
// volumeClaimTemplates is a list of claims that pods are allowed to reference.
147+
// The StatefulSet controller is responsible for mapping network identities to
148+
// claims in a way that maintains the identity of a pod. Every claim in
149+
// this list must have at least one matching (by name) volumeMount in one
150+
// container in the template. A claim in this list takes precedence over
151+
// any volumes in the template, with the same name.
152+
// +optional
153+
VolumeClaimTemplates []ShuffleServerPersistentVolumeClaimTemplate `json:"volumeClaimTemplates,omitempty" protobuf:"bytes,4,rep,name=volumeClaimTemplates"`
154+
}
155+
156+
type ShuffleServerPersistentVolumeClaimTemplate struct {
157+
// May contain labels and annotations that will be copied into the PVC
158+
// when creating it. No other fields are allowed and will be rejected during
159+
// validation.
160+
//
161+
VolumeNameTemplate *string `json:"volumeNameTemplate"`
162+
163+
// The specification for the PersistentVolumeClaim. The entire content is
164+
// copied unchanged into the PVC that gets created from this
165+
// template. The same fields as in a PersistentVolumeClaim
166+
// are also valid here.
167+
Spec corev1.PersistentVolumeClaimSpec `json:"spec" protobuf:"bytes,2,name=spec"`
145168
}
146169

147170
// ShuffleServerUpgradeStrategy defines upgrade strategy of shuffle servers.

‎deploy/kubernetes/operator/api/uniffle/v1alpha1/zz_generated.deepcopy.go

+28
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎deploy/kubernetes/operator/config/crd/bases/uniffle.apache.org_remoteshuffleservices.yaml

+199
Original file line numberDiff line numberDiff line change
@@ -8362,6 +8362,205 @@ spec:
83628362
required:
83638363
- type
83648364
type: object
8365+
volumeClaimTemplates:
8366+
description: volumeClaimTemplates is a list of claims that pods
8367+
are allowed to reference. The StatefulSet controller is responsible
8368+
for mapping network identities to claims in a way that maintains
8369+
the identity of a pod. Every claim in this list must have at
8370+
least one matching (by name) volumeMount in one container in
8371+
the template. A claim in this list takes precedence over any
8372+
volumes in the template, with the same name.
8373+
items:
8374+
properties:
8375+
spec:
8376+
description: The specification for the PersistentVolumeClaim.
8377+
The entire content is copied unchanged into the PVC that
8378+
gets created from this template. The same fields as in
8379+
a PersistentVolumeClaim are also valid here.
8380+
properties:
8381+
accessModes:
8382+
description: 'accessModes contains the desired access
8383+
modes the volume should have. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#access-modes-1'
8384+
items:
8385+
type: string
8386+
type: array
8387+
dataSource:
8388+
description: 'dataSource field can be used to specify
8389+
either: * An existing VolumeSnapshot object (snapshot.storage.k8s.io/VolumeSnapshot)
8390+
* An existing PVC (PersistentVolumeClaim) If the provisioner
8391+
or an external controller can support the specified
8392+
data source, it will create a new volume based on
8393+
the contents of the specified data source. If the
8394+
AnyVolumeDataSource feature gate is enabled, this
8395+
field will always have the same contents as the DataSourceRef
8396+
field.'
8397+
properties:
8398+
apiGroup:
8399+
description: APIGroup is the group for the resource
8400+
being referenced. If APIGroup is not specified,
8401+
the specified Kind must be in the core API group.
8402+
For any other third-party types, APIGroup is required.
8403+
type: string
8404+
kind:
8405+
description: Kind is the type of resource being
8406+
referenced
8407+
type: string
8408+
name:
8409+
description: Name is the name of resource being
8410+
referenced
8411+
type: string
8412+
required:
8413+
- kind
8414+
- name
8415+
type: object
8416+
x-kubernetes-map-type: atomic
8417+
dataSourceRef:
8418+
description: 'dataSourceRef specifies the object from
8419+
which to populate the volume with data, if a non-empty
8420+
volume is desired. This may be any local object from
8421+
a non-empty API group (non core object) or a PersistentVolumeClaim
8422+
object. When this field is specified, volume binding
8423+
will only succeed if the type of the specified object
8424+
matches some installed volume populator or dynamic
8425+
provisioner. This field will replace the functionality
8426+
of the DataSource field and as such if both fields
8427+
are non-empty, they must have the same value. For
8428+
backwards compatibility, both fields (DataSource and
8429+
DataSourceRef) will be set to the same value automatically
8430+
if one of them is empty and the other is non-empty.
8431+
There are two important differences between DataSource
8432+
and DataSourceRef: * While DataSource only allows
8433+
two specific types of objects, DataSourceRef allows
8434+
any non-core object, as well as PersistentVolumeClaim
8435+
objects. * While DataSource ignores disallowed values
8436+
(dropping them), DataSourceRef preserves all values,
8437+
and generates an error if a disallowed value is specified.
8438+
(Beta) Using this field requires the AnyVolumeDataSource
8439+
feature gate to be enabled.'
8440+
properties:
8441+
apiGroup:
8442+
description: APIGroup is the group for the resource
8443+
being referenced. If APIGroup is not specified,
8444+
the specified Kind must be in the core API group.
8445+
For any other third-party types, APIGroup is required.
8446+
type: string
8447+
kind:
8448+
description: Kind is the type of resource being
8449+
referenced
8450+
type: string
8451+
name:
8452+
description: Name is the name of resource being
8453+
referenced
8454+
type: string
8455+
required:
8456+
- kind
8457+
- name
8458+
type: object
8459+
x-kubernetes-map-type: atomic
8460+
resources:
8461+
description: 'resources represents the minimum resources
8462+
the volume should have. If RecoverVolumeExpansionFailure
8463+
feature is enabled users are allowed to specify resource
8464+
requirements that are lower than previous value but
8465+
must still be higher than capacity recorded in the
8466+
status field of the claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#resources'
8467+
properties:
8468+
limits:
8469+
additionalProperties:
8470+
anyOf:
8471+
- type: integer
8472+
- type: string
8473+
pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
8474+
x-kubernetes-int-or-string: true
8475+
description: 'Limits describes the maximum amount
8476+
of compute resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/'
8477+
type: object
8478+
requests:
8479+
additionalProperties:
8480+
anyOf:
8481+
- type: integer
8482+
- type: string
8483+
pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
8484+
x-kubernetes-int-or-string: true
8485+
description: 'Requests describes the minimum amount
8486+
of compute resources required. If Requests is
8487+
omitted for a container, it defaults to Limits
8488+
if that is explicitly specified, otherwise to
8489+
an implementation-defined value. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/'
8490+
type: object
8491+
type: object
8492+
selector:
8493+
description: selector is a label query over volumes
8494+
to consider for binding.
8495+
properties:
8496+
matchExpressions:
8497+
description: matchExpressions is a list of label
8498+
selector requirements. The requirements are ANDed.
8499+
items:
8500+
description: A label selector requirement is a
8501+
selector that contains values, a key, and an
8502+
operator that relates the key and values.
8503+
properties:
8504+
key:
8505+
description: key is the label key that the
8506+
selector applies to.
8507+
type: string
8508+
operator:
8509+
description: operator represents a key's relationship
8510+
to a set of values. Valid operators are
8511+
In, NotIn, Exists and DoesNotExist.
8512+
type: string
8513+
values:
8514+
description: values is an array of string
8515+
values. If the operator is In or NotIn,
8516+
the values array must be non-empty. If the
8517+
operator is Exists or DoesNotExist, the
8518+
values array must be empty. This array is
8519+
replaced during a strategic merge patch.
8520+
items:
8521+
type: string
8522+
type: array
8523+
required:
8524+
- key
8525+
- operator
8526+
type: object
8527+
type: array
8528+
matchLabels:
8529+
additionalProperties:
8530+
type: string
8531+
description: matchLabels is a map of {key,value}
8532+
pairs. A single {key,value} in the matchLabels
8533+
map is equivalent to an element of matchExpressions,
8534+
whose key field is "key", the operator is "In",
8535+
and the values array contains only "value". The
8536+
requirements are ANDed.
8537+
type: object
8538+
type: object
8539+
x-kubernetes-map-type: atomic
8540+
storageClassName:
8541+
description: 'storageClassName is the name of the StorageClass
8542+
required by the claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#class-1'
8543+
type: string
8544+
volumeMode:
8545+
description: volumeMode defines what type of volume
8546+
is required by the claim. Value of Filesystem is implied
8547+
when not included in claim spec.
8548+
type: string
8549+
volumeName:
8550+
description: volumeName is the binding reference to
8551+
the PersistentVolume backing this claim.
8552+
type: string
8553+
type: object
8554+
volumeNameTemplate:
8555+
description: May contain labels and annotations that will
8556+
be copied into the PVC when creating it. No other fields
8557+
are allowed and will be rejected during validation.
8558+
type: string
8559+
required:
8560+
- spec
8561+
- volumeNameTemplate
8562+
type: object
8563+
type: array
83658564
volumeMounts:
83668565
description: VolumeMounts indicates describes mountings of volumes
83678566
within shuffle servers' container.

‎deploy/kubernetes/operator/examples/pvc-example/README.md

+28
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
<!--
2+
~ Licensed to the Apache Software Foundation (ASF) under one or more
3+
~ contributor license agreements. See the NOTICE file distributed with
4+
~ this work for additional information regarding copyright ownership.
5+
~ The ASF licenses this file to You under the Apache License, Version 2.0
6+
~ (the "License"); you may not use this file except in compliance with
7+
~ the License. You may obtain a copy of the License at
8+
~
9+
~ http://www.apache.org/licenses/LICENSE-2.0
10+
~
11+
~ Unless required by applicable law or agreed to in writing, software
12+
~ distributed under the License is distributed on an "AS IS" BASIS,
13+
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
~ See the License for the specific language governing permissions and
15+
~ limitations under the License.
16+
-->
17+
18+
# Use PVC for Shuffle Servers
19+
20+
If we want to use PVC for shuffle servers, we need to edit the rss object as follows.
21+
22+
+ update `.spec.shuffleServer.volumeMounts` with the mount points of PVC.
23+
+ update `.spec.shuffleServer.volumeClaimTemplates` with the correct volume claim templates.
24+
25+
We can refer to the [example](rss-pvc-on-gce.yaml).
26+
Of course, you should create a storage class first if it's not already created.
27+
You may have to ask your K8S administrator to create one.
28+
A [storage class](gce-storage-class.yml) is also provided in this example dir.

‎deploy/kubernetes/operator/examples/pvc-example/gce-storage-class.yml

+25
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
#
2+
# Licensed to the Apache Software Foundation (ASF) under one or more
3+
# contributor license agreements. See the NOTICE file distributed with
4+
# this work for additional information regarding copyright ownership.
5+
# The ASF licenses this file to You under the Apache License, Version 2.0
6+
# (the "License"); you may not use this file except in compliance with
7+
# the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
#
17+
18+
---
19+
apiVersion: storage.k8s.io/v1
20+
kind: StorageClass
21+
metadata:
22+
name: fast
23+
provisioner: kubernetes.io/gce-pd
24+
parameters:
25+
type: pd-ssd

0 commit comments

Comments
 (0)