istio如何使用envoy进行kubernetes流量治理

时间：July 6, 2021 分类：容器

目录：

envoy如何注入

istio的sidecar自动注入是通过Admission Controller的Mutating阶段完成的

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  creationTimestamp: "2021-06-22T07:43:30Z"
  generation: 2
  labels:
    app: sidecar-injector
    install.operator.istio.io/owning-resource: tcm
    istio.io/rev: 1-8-1
    operator.istio.io/component: Pilot
    release: istio
  name: istio-sidecar-injector-1-8-1
  resourceVersion: "2381424"
  selfLink: /apis/admissionregistration.k8s.io/v1/mutatingwebhookconfigurations/istio-sidecar-injector-1-8-1
  uid: ba5d408c-e734-4dc1-8e0e-2d2960eb48d6
webhooks:
- admissionReviewVersions:
  - v1beta1
  - v1
  clientConfig:
    caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk...
    service:
      name: istiod-1-8-1
      namespace: istio-system
      path: /inject
      port: 443
  failurePolicy: Fail
  matchPolicy: Exact
  name: sidecar-injector.istio.io
  namespaceSelector:
    matchExpressions:
    - key: istio-injection
      operator: DoesNotExist
    - key: istio.io/rev
      operator: In
      values:
      - 1-8-1
  objectSelector: {}
  reinvocationPolicy: Never
  rules:
  - apiGroups:
    - ""
    apiVersions:
    - v1
    operations:
    - CREATE
    resources:
    - pods
    scope: '*'
  sideEffects: None
  timeoutSeconds: 30

监听pod创建后先认证鉴权，然后在准入过程中以Rest方式调用配置的sidecar injetor进行容器注入，注入流程为

  1. 解析Webhook REST请求，将AdmissionReview原始数据反序列化
  2. 解析Pod，将AdmissionReview中的AdmissionRequest反序列化
  3. 利用Pod及网格配置渲染Sidecar配置模板
  4. 利用Pod及渲染后的模板创建JSON patch
  5. 构造AdmissionResponse
  6. 构造AdmissionReview，在进行JSON编码后，将其发送给HTTP客户端即Kube-apiserver

sidecar injetor进行容器注入流程

envoy如何代理流量

注入过程中有initContainer和Container istio-proxy

initContainer在pod的时候首先设置iptables规则，用于拦截Inbound/Outbound的流量

  initContainers:
  - args:
    - istio-iptables
    - -p // envoy转发tcp流量端口
    - "15001"
    - -z
    - "15006"
    - -u
    - "1337"
    - -m // iptable模式，还有操作mangle表的TPROXY
    - REDIRECT
    - -i // 需要转发的数据包网段，CIDR形式
    - '*'
    - -x // 不需要转发的数据包网段
    - ""
    - -b // 需要转发的入主机方向端口
    - '*'
    - -d // 不需要转发的入主机方向端口
    - 15090,15021,15020
    env:
    - name: DNS_AGENT
    image: ccr.ccs.tencentyun.com/tcmimages/proxyv2:1.8.1
    imagePullPolicy: Always
    name: istio-init
    resources:
      limits:
        cpu: "2"
        memory: 1Gi
      requests:
        cpu: 100m
        memory: 128Mi
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        add:
        - NET_ADMIN
        - NET_RAW
        drop:
        - ALL
      privileged: false
      readOnlyRootFilesystem: false
      runAsGroup: 0
      runAsNonRoot: false
      runAsUser: 0
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-54wsc
      readOnly: true

initContainer中对securityContext赋予了NET_ADMIN权限，使initContainer有了更新net namespace中iptables的能力

通过owner模块，将1337用户的报文排除，因为envoy是以这个用户运行的

    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      privileged: false
      readOnlyRootFilesystem: true
      runAsGroup: 1337
      runAsNonRoot: true
      runAsUser: 1337

通过REDIRECT模块将请求都转发到本地的15001端口，这个是envoy的监听端口，由envoy进行代理，envoy的报文因为1337用户正常进行发送，不会转发到envoy

$ docker inspect f7d64564a056 | grep Pid
            "Pid": 595029,
            "PidMode": "",
            "PidsLimit": null,
$ nsenter -t 595029 -n /bin/sh
$ iptables-save > /tmp/a  # iptables -t nat -S
$ cat !$
# Generated by iptables-save v1.4.21 on Mon Jun 28 16:05:00 2021
*filter
:INPUT ACCEPT [1370:9457737]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [1124:15918543]
COMMIT
# Completed on Mon Jun 28 16:05:00 2021
# Generated by iptables-save v1.4.21 on Mon Jun 28 16:05:00 2021
*nat
:PREROUTING ACCEPT [522324:22336816]
:INPUT ACCEPT [522324:22336816]
:OUTPUT ACCEPT [104749:9516051]
:POSTROUTING ACCEPT [104808:9519591]
:ISTIO_INBOUND - [0:0]
:ISTIO_IN_REDIRECT - [0:0]
:ISTIO_OUTPUT - [0:0]
:ISTIO_REDIRECT - [0:0]
-A PREROUTING -p tcp -j ISTIO_INBOUND
-A OUTPUT -p tcp -j ISTIO_OUTPUT
-A ISTIO_INBOUND -p tcp -m tcp --dport 15008 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 22 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15090 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15021 -j RETURN
-A ISTIO_INBOUND -p tcp -m tcp --dport 15020 -j RETURN
-A ISTIO_INBOUND -p tcp -j ISTIO_IN_REDIRECT
-A ISTIO_IN_REDIRECT -p tcp -j REDIRECT --to-ports 15006
-A ISTIO_OUTPUT -s 127.0.0.6/32 -o lo -j RETURN
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -m owner --uid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --uid-owner 1337 -j RETURN
-A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -m owner --gid-owner 1337 -j ISTIO_IN_REDIRECT
-A ISTIO_OUTPUT -o lo -m owner ! --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -m owner --gid-owner 1337 -j RETURN
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
-A ISTIO_REDIRECT -p tcp -j REDIRECT --to-ports 15001
COMMIT
# Completed on Mon Jun 28 16:05:00 2021