云容器引擎 CCE-NUMA亲和性调度:Volcano开启NUMA亲和性调度

时间:2024-05-31 08:37:52

Volcano开启NUMA亲和性调度

  1. 开启静态(static)CPU管理策略,具体请参考 开启CPU管理策略
  2. 配置CPU拓扑策略。

    1. 登录CCE控制台,单击集群名称进入集群,在左侧选择“节点管理”,在右侧选择“节点池”页签,单击节点池名称后的“ 配置管理”
    2. 将kubelet的拓扑管理策略(topology-manager-policy)的值修改为需要的CPU拓扑策略即可。

      有效拓扑策略为“none”、“best-effort”、“restricted”、“single-numa-node”,具体策略对应的调度行为请参见Pod调度预测

  3. 开启numa-aware插件功能和resource_exporter功能。

    Volcano 1.7.1及以上版本

    1. 登录CCE控制台,单击集群名称进入集群,单击左侧导航栏的“插件中心”,在右侧找到Volcano,单击“编辑”,并在“参数配置”中设置Volcano调度器配置参数。
      {
          "ca_cert": "",
          "default_scheduler_conf": {
              "actions": "allocate, backfill, preempt",
              "tiers": [
                  {
                      "plugins": [
                          {
                              "name": "priority"
                          },
                          {
                              "name": "gang"
                          },
                          {
                              "name": "conformance"
                          }
                      ]
                  },
                  {
                      "plugins": [
                          {
                              "name": "drf"
                          },
                          {
                              "name": "predicates"
                          },
                          {
                              "name": "nodeorder"
                          }
                      ]
                  },
                  {
                      "plugins": [
                          {
                              "name": "cce-gpu-topology-predicate"
                          },
                          {
                              "name": "cce-gpu-topology-priority"
                          },
                          {
                              "name": "cce-gpu"
                          },
                          {
                              // add this also enable resource_exporter
                              "name": "numa-aware", 
                              // the weight of the NUMA Aware Plugin
                              "arguments": { 
                                 "weight": "10"
                              }
                          }
                      ]
                  },
                  {
                      "plugins": [
                          {
                              "name": "nodelocalvolume"
                          },
                          {
                              "name": "nodeemptydirvolume"
                          },
                          {
                              "name": "nodeCSIscheduling"
                          },
                          {
                              "name": "networkresource"
                          }
                      ]
                  }
              ]
          },
          "server_cert": "",
          "server_key": ""
      }
    Volcano 1.7.1以下版本
    1. Volcano插件开启resource_exporter_enable参数,用于收集节点numa拓扑信息。
      {
         "plugins": {
            "eas_service": {
               "availability_zone_id": "",
               "driver_id": "",
               "enable": "false",
               "endpoint": "",
               "flavor_id": "",
               "network_type": "",
               "network_virtual_subnet_id": "",
               "pool_id": "",
               "project_id": "",
               "secret_name": "eas-service-secret"
            }
         },
         "resource_exporter_enable": "true"
      }
      开启后可以查看当前节点的numa拓扑信息。
      kubectl get numatopo 
      NAME              AGE
      node-1            4h8m
      node-2            4h8m
      node-3            4h8m
    2. 启用Volcano numa-aware算法插件。
      kubectl edit cm -n kube-system volcano-scheduler-configmap
      kind: ConfigMap
      apiVersion: v1
      metadata:
        name: volcano-scheduler-configmap
        namespace: kube-system
      data:
        default-scheduler.conf: |-
          actions: "allocate, backfill, preempt"
          tiers:
          - plugins:
            - name: priority
            - name: gang
            - name: conformance
          - plugins:
            - name: overcommit
            - name: drf
            - name: predicates
            - name: nodeorder
          - plugins:
            - name: cce-gpu-topology-predicate
            - name: cce-gpu-topology-priority
            - name: cce-gpu
          - plugins:
            - name: nodelocalvolume
            - name: nodeemptydirvolume
            - name: nodeCSIscheduling
            - name: networkresource
              arguments:
                NetworkType: vpc-router
            - name: numa-aware # add it to enable numa-aware plugin
              arguments:
                weight: 10 # the weight of the NUMA Aware Plugin

support.huaweicloud.com/usermanual-cce/cce_10_0425.html