Kubernetes Scheduler扩展

Scheduler扩展有几种方式

1. 修改源码中调度和绑定两个部分，重新编译
2. 和默认调度器独立的调度程序，然后显式指定scheduler
3. 通过Extender扩展，作为默认调度算法的补充
4. 通过Framework扩展，也需要重新编译

修改源码

源码的位置在kubernetes/pkg/scheduler/framework/plugins

显式指定scheduler

Pod指定scheduler

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  schedulerName: my-scheduler
  containers:
  - name: nginx
    image: nginx:1.10

官方示例的SHELL进行调度

#!/bin/bash
SERVER='localhost:8001'
while true;
do
    for PODNAME in $(kubectl --server $SERVER get pods -o json | jq '.items[] | select(.spec.schedulerName == "my-scheduler") | select(.spec.nodeName == null) | .metadata.name' | tr -d '"')
;
    do
        NODES=($(kubectl --server $SERVER get nodes -o json | jq '.items[].metadata.name' | tr -d '"'))
        NUMNODES=${#NODES[@]}
        CHOSEN=${NODES[$[ $RANDOM % $NUMNODES ]]}
        curl --header "Content-Type:application/json" --request POST --data '{"apiVersion":"v1", "kind": "Binding", "metadata": {"name": "'$PODNAME'"}, "target": {"apiVersion": "v1", "kind"
: "Node", "name": "'$CHOSEN'"}}' http://$SERVER/api/v1/namespaces/default/pods/$PODNAME/binding/
        echo "Assigned $PODNAME to $CHOSEN"
    done
    sleep 1
done

通过Extender扩展

构建项目

HTTPExtender的send方法会使用json格式，并且将args参数序列化为json然后发送到extender服务

会有四个位置会调用send方法

• Filter：在调度过程中，过滤掉不部署pod的node
• Preempt：在资源不足的时候，抢占低优先级的pod资源，优先保证高优先级pod运行
• Prioritize：node排序，在资源充足的时候选择最优node
• Bind：绑定阶段

我们需要暴露这些对应的接口/filter和/prioritize等

直接通过http协议提供服务即可，例如filter

package controller

import (
    "github.com/gin-gonic/gin"
    v1 "k8s.io/kube-scheduler/extender/v1"
)

func Filter(ctx *gin.Context) {
    var args v1.ExtenderArgs
    err := ctx.BindJSON(&args)
    if err != nil {
        ctx.JSON(200, v1.ExtenderFilterResult{Error: err.Error()})
    }
    date := doFilter(args)
    ctx.JSON(200, date)
}


func doFilter(args v1.ExtenderArgs) v1.ExtenderFilterResult {
    // 简单实现，理论上需要判断node的各种状态
    nodeNames := []string{"node1", "node2"}
    return v1.ExtenderFilterResult{NodeNames: &nodeNames}
}

同理prioritize

func doPrioritize(args v1.ExtenderArgs) v1.HostPriorityList {
    list := make(v1.HostPriorityList, 2)
    p1 := v1.HostPriority{Host: "node1", Score: 100}
    p2 := v1.HostPriority{Host: "node2", Score: 50}
    list = append(list, p1)
    list = append(list, p2)
    return list
}

scheduler配置

参考Extender定义规范

/etc/kubernetesf/sched.yaml

apiVersion: kubescheduler.config.k8s.io/v1alpha2
kind: KubeSchedulerConfiguration
extenders:
  - urlPrefix: http://192.168.179.139/
    filterVerb: filter
    prioritizeVerb: prioritize
    nodeCacheCapable: true
    enableHttps: false
    weight: 100000
leaderElection:
    leaderElect: true
clientConnection:
    kubeconfig: /etc/kubernetes/scheduler.conf

/etc/kubernetes/manifests/kube-scheduler.yaml

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    component: kube-scheduler
    tier: control-plane
  name: kube-scheduler
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-scheduler
    - --authentication-kubeconfig=/etc/kubernetes/scheduler.conf
    - --authorization-kubeconfig=/etc/kubernetes/scheduler.conf
    - --bind-address=127.0.0.1
    - --kubeconfig=/etc/kubernetes/scheduler.conf
    - --leader-elect=true
    - --config=/etc/kubernetes/sched.yaml
    image: registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.18.2
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 127.0.0.1
        path: /healthz
        port: 10259
        scheme: HTTPS
      initialDelaySeconds: 15
      timeoutSeconds: 15
    name: kube-scheduler
    resources:
      requests:
        cpu: 100m
    volumeMounts:
    - mountPath: /etc/kubernetes/scheduler.conf
      name: kubeconfig
      readOnly: true
    - mountPath: /etc/kubernetes/sched.yaml
      name: schedconfig
      readOnly: true
  hostNetwork: true
  priorityClassName: system-cluster-critical
  volumes:
  - hostPath:
      path: /etc/kubernetes/scheduler.conf
      type: FileOrCreate
    name: kubeconfig
  - hostPath:
      path: /etc/kubernetes/sched.yaml
      type: FileOrCreate
    name: schedconfig
status: {}

通过Framework扩展

以后补充