TiDB增强型 StatefulSet 控制器--Advanced StatefulSet

K8S内置的 StatefulSet 为 Pods 分配连续的序号。比如 3 个副本时，Pods 分别为 pod-0, pod-1, pod-2。扩缩容时，必须在尾部增加或删除 Pods。比如扩容到 4 个副本时，会新增 pod-3。缩容到 2 副本时，会删除 pod-2。

在使用本地存储时，Pods 与 Nodes 存储资源绑定，无法自由调度。若希望删除掉中间某个 Pod ，以便维护其所在的 Node 但并没有其他 Node 可以迁移时，或者某个 Pod 故障想直接删除，另起一个序号不一样的 Pod 时，无法通过内置 StatefulSet 实现。

增强型 StatefulSet 控制器 基于内置 StatefulSet 实现，新增了自由控制 Pods 序号的功能。本文介绍如何在 TiDB Operator 中使用。

开启

载入 Advanced StatefulSet 的CRD文件：

Kubernetes 1.16 之前版本：

kubectl apply -f https://raw.githubusercontent.com/pingcap/tidb-operator/v1.3.9/manifests/advanced-statefulset-crd.v1beta1.yaml

Kubernetes 1.16 及之后版本:

kubectl apply -f https://raw.githubusercontent.com/pingcap/tidb-operator/v1.3.9/manifests/advanced-statefulset-crd.v1.yaml

具体执行如下：

[root@k8s-master tidb]# kubectl apply -f https://raw.githubusercontent.com/pingcap/tidb-operator/v1.3.9/manifests/advanced-statefulset-crd.v1beta1.yaml Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition customresourcedefinition.apiextensions.k8s.io/statefulsets.apps.pingcap.com configured

在 TiDB Operator chart 的 values.yaml 中启用 AdvancedStatefulSet 特性：

##修改1 features: - AdvancedStatefulSet=true advancedStatefulset: create: true ##修改2： advancedStatefulset: create: true image: pingcap/advanced-statefulset:v0.4.0 imagePullPolicy: IfNotPresent serviceAccount: advanced-statefulset-controller logLevel: 4 replicas: 1 resources: limits: cpu: 500m memory: 300Mi requests: cpu: 200m memory: 50Mi

具体实例操作如下：

生成相关的配置文件[root@k8s-master tidb]# helm inspect values pingcap/tidb-operator --version=v1.3.9 > /home/tidb/tidb-operator/values-tidb-operator.yaml [root@k8s-master tidb]# vim tidb-operator/values-tidb-operator.yaml [root@k8s-master tidb-operator]# cat values-tidb-operator.yaml # Default values for tidb-operator # clusterScoped is whether tidb-operator should manage kubernetes cluster wide tidb clusters # Also see rbac.create, controllerManager.serviceAccount, scheduler.create and controllerManager.clusterPermissions. clusterScoped: true # Also see clusterScoped and controllerManager.serviceAccount rbac: create: true # timezone is the default system timzone timezone: UTC # operatorImage is TiDB Operator image operatorImage: pingcap/tidb-operator:v1.3.9 imagePullPolicy: IfNotPresent # imagePullSecrets: [] # tidbBackupManagerImage is tidb backup manager image tidbBackupManagerImage: pingcap/tidb-backup-manager:v1.3.9 # # Enable or disable tidb-operator features: # # StableScheduling (default: true) # Enable stable scheduling of tidb servers. # # AdvancedStatefulSet (default: false) # If enabled, tidb-operator will use AdvancedStatefulSet to manage pods # instead of Kubernetes StatefulSet. # Its ok to turn it on if this feature is not enabled. However its not ok # to turn it off when the tidb-operator already uses AdvancedStatefulSet to # manage pods. This is in alpha phase. # features: - AdvancedStatefulSet=true advancedStatefulset: create: true # - AdvancedStatefulSet=false # - StableScheduling=true # - AutoScaling=false appendReleaseSuffix: false controllerManager: create: true # With rbac.create=false, the user is responsible for creating this account # With rbac.create=true, this service account will be created # Also see rbac.create and clusterScoped serviceAccount: tidb-controller-manager # clusterPermissions are some cluster scoped permissions that will be used even if `clusterScoped: false`. # the default value of these fields is `true`. if you want them to be `false`, you MUST set them to `false` explicitly. clusterPermissions: nodes: true persistentvolumes: true storageclasses: true logLevel: 2 replicas: 1 resources: requests: cpu: 80m memory: 50Mi # # REF: https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/ # priorityClassName: system-cluster-critical # # REF: https://pkg.go.dev/k8s.io/client-go/tools/leaderelection#LeaderElectionConfig ## leaderLeaseDuration is the duration that non-leader candidates will wait to force acquire leadership # leaderLeaseDuration: 15s ## leaderRenewDeadline is the duration that the acting master will retry refreshing leadership before giving up # leaderRenewDeadline: 10s ## leaderRetryPeriod is the duration the LeaderElector clients should wait between tries of actions # leaderRetryPeriod: 2s ## number of workers that are allowed to sync concurrently. default 5 # workers: 5 # autoFailover is whether tidb-operator should auto failover when failure occurs autoFailover: true # pd failover period default(5m) pdFailoverPeriod: 5m # tikv failover period default(5m) tikvFailoverPeriod: 5m # tidb failover period default(5m) tidbFailoverPeriod: 5m # tiflash failover period default(5m) tiflashFailoverPeriod: 5m # dm-master failover period default(5m) dmMasterFailoverPeriod: 5m # dm-worker failover period default(5m) dmWorkerFailoverPeriod: 5m ## affinity defines pod scheduling rules,affinity default settings is empty. ## please read the affinity document before set your scheduling rule: ## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity affinity: {} ## nodeSelector ensure pods only assigning to nodes which have each of the indicated key-value pairs as labels ## ref:https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector nodeSelector: {} ## Tolerations are applied to pods, and allow pods to schedule onto nodes with matching taints. ## refer to https://kubernetes.io/docs/concepts/configuration/taint-and-toleration tolerations: [] # - key: node-role # operator: Equal # value: tidb-operator # effect: "NoSchedule" ## Selector (label query) to filter on, make sure that this controller manager only manages the custom resources that match the labels ## refer to https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#equality-based-requirement selector: [] # - canary-release=v1 # - k1==v1 # - k2!=v2 # SecurityContext is security config of this component, it will set template.spec.securityContext # Refer to https://kubernetes.io/docs/tasks/configure-pod-container/security-context securityContext: {} # runAsUser: 1000 # runAsGroup: 2000 # fsGroup: 2000 # PodAnnotations will set template.metadata.annotations # Refer to https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/ podAnnotations: {} scheduler: create: true # With rbac.create=false, the user is responsible for creating this account # With rbac.create=true, this service account will be created # Also see rbac.create and clusterScoped serviceAccount: tidb-scheduler logLevel: 2 replicas: 1 schedulerName: tidb-scheduler resources: limits: cpu: 250m memory: 150Mi requests: cpu: 80m memory: 50Mi kubeSchedulerImageName: k8s.gcr.io/kube-scheduler # This will default to matching your kubernetes version # kubeSchedulerImageTag: ## affinity defines pod scheduling rules,affinity default settings is empty. ## please read the affinity document before set your scheduling rule: ## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity affinity: {} ## nodeSelector ensure pods only assigning to nodes which have each of the indicated key-value pairs as labels ## ref:https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector nodeSelector: {} ## Tolerations are applied to pods, and allow pods to schedule onto nodes with matching taints. ## refer to https://kubernetes.io/docs/concepts/configuration/taint-and-toleration tolerations: [] # - key: node-role # operator: Equal # value: tidb-operator # effect: "NoSchedule" # # SecurityContext is security config of this component, it will set template.spec.securityContext # Refer to https://kubernetes.io/docs/tasks/configure-pod-container/security-context securityContext: {} # runAsUser: 1000 # runAsGroup: 2000 # fsGroup: 2000 # PodAnnotations will set template.metadata.annotations # Refer to https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/ podAnnotations: {} # additional annotations for the configmap, mainly to prevent spinnaker versioning the cm configmapAnnotations: {} # When AdvancedStatefulSet feature is enabled, you must install # AdvancedStatefulSet controller. # Note that AdvancedStatefulSet CRD must be installed manually via the following # command: # kubectl apply -f manifests/advanced-statefulset-crd.v1beta1.yaml # k8s version < 1.16.0 # kubectl apply -f manifests/advanced-statefulset-crd.v1.yaml # k8s version >= 1.16.0 advancedStatefulset: create: true image: pingcap/advanced-statefulset:v0.4.0 imagePullPolicy: IfNotPresent serviceAccount: advanced-statefulset-controller logLevel: 4 replicas: 1 resources: limits: cpu: 500m memory: 300Mi requests: cpu: 200m memory: 50Mi ## affinity defines pod scheduling rules,affinity default settings is empty. ## please read the affinity document before set your scheduling rule: ## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity affinity: {} ## nodeSelector ensure pods only assigning to nodes which have each of the indicated key-value pairs as labels ## ref:https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector nodeSelector: {} ## Tolerations are applied to pods, and allow pods to schedule onto nodes with matching taints. ## refer to https://kubernetes.io/docs/concepts/configuration/taint-and-toleration tolerations: [] # - key: node-role # operator: Equal # value: tidb-operator # effect: "NoSchedule" # # SecurityContext is security config of this component, it will set template.spec.securityContext # Refer to https://kubernetes.io/docs/tasks/configure-pod-container/security-context securityContext: {} # runAsUser: 1000 # runAsGroup: 2000 # fsGroup: 2000 admissionWebhook: create: false replicas: 1 serviceAccount: tidb-admission-webhook logLevel: 2 rbac: create: true ## validation webhook would check the given request for the specific resource and operation validation: ## statefulsets hook would check requests for updating tidbclusters statefulsets ## If enabled it, the statefulsets of tidbcluseter would update in partition by tidbclusters annotation statefulSets: false ## validating hook validates the correctness of the resources under pingcap.com group pingcapResources: false ## mutation webhook would mutate the given request for the specific resource and operation mutation: ## defaulting hook set default values for the the resources under pingcap.com group pingcapResources: true ## failurePolicy are applied to ValidatingWebhookConfiguration which affect tidb-admission-webhook ## refer to https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#failure-policy failurePolicy: ## the validation webhook would check the request of the given resources. ## If the kubernetes api-server version >= 1.15.0, we recommend the failurePolicy as Fail, otherwise, as Ignore. validation: Ignore ## the mutation webhook would mutate the request of the given resources. ## If the kubernetes api-server version >= 1.15.0, we recommend the failurePolicy as Fail, otherwise, as Ignore. mutation: Ignore ## tidb-admission-webhook deployed as kubernetes apiservice server ## refer to https://github.com/openshift/generic-admission-server apiservice: ## apiservice config ## refer to https://kubernetes.io/docs/tasks/access-kubernetes-api/configure-aggregation-layer/#contacting-the-extension-apiserver insecureSkipTLSVerify: true ## The Secret includes the TLS ca, cert and key for the `tidb-admission-webook.<Release Namespace>.svc` Service. ## If insecureSkipTLSVerify is true, this would be ignored. ## You can create the tls secret by: ## kubectl create secret generic <secret-name> --namespace=<release-namespace> --from-file=tls.crt=<path-to-cert> --from-file=tls.key=<path-to-key> --from-file=ca.crt=<path-to-ca> tlsSecret: "" ## The caBundle for the webhook apiservice, you could get it by the secret you created previously: ## kubectl get secret <secret-name> --namespace=<release-namespace> -o=jsonpath={.data.ca\.crt} caBundle: "" ## certProvider indicate the key and cert for the webhook configuration to communicate with `kubernetes.default` service. ## If your kube-apiservers version >= 1.13.0, you can leave cabundle empty and the kube-apiserver ## would trust the roots on the apiserver. ## refer to https://github.com/kubernetes/api/blob/master/admissionregistration/v1/types.go#L529 ## or you can get the cabundle by: ## kubectl get configmap -n kube-system extension-apiserver-authentication -o=jsonpath={.data.client-ca-file} | base64 | tr -d \n cabundle: "" # SecurityContext is security config of this component, it will set template.spec.securityContext # Refer to https://kubernetes.io/docs/tasks/configure-pod-container/security-context securityContext: {} # runAsUser: 1000 # runAsGroup: 2000 # fsGroup: 2000 ## nodeSelector ensures that pods are only scheduled to nodes that have each of the indicated key-value pairs as labels ## ref:https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector nodeSelector: {} ## Tolerations are applied to pods, and allow pods to schedule onto nodes with matching taints. ## refer to https://kubernetes.io/docs/concepts/configuration/taint-and-toleration tolerations: [] # - key: node-role # operator: Equal # value: tidb-operator # effect: "NoSchedule" #更新相关的tidb operator配置[root@k8s-master tidb]# helm upgrade tidb-operator pingcap/tidb-operator --namespace=tidb-admin --version=v1.3.9 -f ./tidb-operator/values-tidb-operator.yaml && kubectl get po -n tidb-admin -l app.kubernetes.io/name=tidb-operator Release "tidb-operator" has been upgraded. Happy Helming! NAME: tidb-operator LAST DEPLOYED: Fri Dec 9 16:31:42 2022 NAMESPACE: tidb-admin STATUS: deployed REVISION: 7 TEST SUITE: None NOTES: Make sure tidb-operator components are running: kubectl get pods --namespace tidb-admin -l app.kubernetes.io/instance=tidb-operator NAME READY STATUS RESTARTS AGE advanced-statefulset-controller-67885c5dd9-lfmbl 1/1 Running 0 27m tidb-controller-manager-d5fc64f85-nlv4k 1/1 Running 1 34m tidb-scheduler-566f48d4bd-82rrd 2/2 Running 0 34m [root@k8s-master tidb]# kubectl get pods --namespace tidb-admin -l app.kubernetes.io/instance=tidb-operator NAME READY STATUS RESTARTS AGE advanced-statefulset-controller-67885c5dd9-lfmbl 1/1 Running 0 28m tidb-controller-manager-d5fc64f85-nlv4k 1/1 Running 1 34m tidb-scheduler-566f48d4bd-82rrd 2/2 Running 0 34m

TiDB Operator 通过开启 AdvancedStatefulSet 特性，会将当前 StatefulSet 对象转换成 AdvancedStatefulSet 对象。但是，TiDB Operator 不支持在关闭 AdvancedStatefulSet 特性后，自动从 AdvancedStatefulSet 转换为 Kubernetes 内置的 StatefulSet 对象。

使用

查看AdvancedStatefulSet 对象

AdvancedStatefulSet 数据格式与 StatefulSet 完全一致，但以 CRD 方式实现，别名为 asts ，可通过以下方法查看命名空间下的对象。

[root@k8s-master tidb]# kubectl get asts -ntidb NAME DESIRED CURRENT AGE lqb-pd 1 1 38m lqb-tidb 1 1 38m lqb-tikv 1 1 38m tidbmonitor-monitor 1 38m tidbngmonitoring-yz-ng-monitoring 1 38m yz-pd 1 1 38m yz-tidb 2 2 38m yz-tiflash 2 2 38m yz-tikv 3 3 38m [root@k8s-master tidb]# kubectl get sts -ntidb NAME READY AGE tidbmonitor-monitor 1/1 46h tidbngmonitoring-yz-ng-monitoring 1/1 2d

对集群指定的pod进行缩容

使用AdvancedStatefulSet在对TiDBCluster进行缩容时，除了减少副本数，可同时通过配置annotations（）对指定的PD、TiKV、TiDB组件下任意一个pod的编号进行缩容

metadata:annotations:tikv.tidb.pingcap.com/delete-slots: [1]

如果想要缩容需同时配置修改replicas和annotations两个配置文件

下边已具体缩容tikv1和pd1为例

编辑所要缩容的集群查看执行的结果：总结

支持的annoations为：

pd.tidb.pingcap.com/delete-slots：指定 PD 组件需要删除的 Pod 序号。

tidb.tidb.pingcap.com/delete-slots：指定 TiDB 组件需要删除的 Pod 序号。

tikv.tidb.pingcap.com/delete-slots：指定 TiKV 组件需要删除的 Pod 序号。

其中 Annotation 值为 JSON 的整数数组，比如 [0], [0,1], [1,3] 等。

对集群指定位置进行扩容

如果扩容的话需要对前面进行反向操作即修改replicas和把annoations修改的数组进行删除即可( delete-slots annotations 可留空，也可完全删除)。

编辑要扩容的集群查看执行的结果