kubernetes nginx-ingress

时间：Jan. 18, 2019 分类：容器

目录：

ingress用于管理Server外露的问题

安装ingress-nginx

下载

$ git clone https://github.com/kubernetes/ingress-nginx.git
$ cd ingress-nginx/deploy/

修改mandatory.yaml配置文件，spec.hostNetwork设置为true

    spec:
      serviceAccountName: nginx-ingress-serviceaccount
      hostNetwork: true
      containers:

启动ingress

$ kubernetes apply -f mandatory.yaml

创建ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx
spec:
  rules:
  - host: test.whysdomain.com
    http:
      paths:
      - backend:
          serviceName: nginx
          servicePort: 80

目前证书都是配置在LB上，如果需要配置SSL参考配置ssl

把日志打开

$ kubectl exec -it nginx-deployment-76bf4969df-wxjnd /bin/bash
sed -i "s@#access_log@access_log@g" /etc/nginx/conf.d/default.conf
cd /var/log/nginx/
mkdir log
nginx -s reload
tail -F host.access.log

进行请求

$ curl -H "Host: test.whysdomain.com" 127.0.0.1:80

获得到的日志如下

10.244.2.0 - - [17/Jan/2019:08:23:38 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "127.0.0.1"
10.244.2.0 - - [17/Jan/2019:08:23:40 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "127.0.0.1"

ClientIP问题

10.244.2.0为flannel.1的IP地址，由于NAT和SNAT的原因，日志中并不能记录

参考官方的解决方案

即使加上外部LB也会有这个问题

10.244.2.0 - - [17/Jan/2019:08:30:43 +0000] "HEAD / HTTP/1.1" 200 0 "-" "-" "119.28.183.246"
10.244.2.0 - - [17/Jan/2019:08:30:43 +0000] "HEAD / HTTP/1.1" 200 0 "-" "-" "119.28.183.246"
10.244.2.0 - - [17/Jan/2019:08:30:48 +0000] "HEAD / HTTP/1.1" 200 0 "-" "-" "119.28.183.246"
10.244.2.0 - - [17/Jan/2019:08:30:49 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.221 Safari/537.36 SE 2.X MetaSr 1.0" "114.251.228.131"
10.244.2.0 - - [17/Jan/2019:08:30:53 +0000] "HEAD / HTTP/1.1" 200 0 "-" "-" "119.28.183.246"
10.244.2.0 - - [17/Jan/2019:08:30:53 +0000] "HEAD / HTTP/1.1" 200 0 "-" "-" "119.28.183.246"
10.244.2.0 - - [17/Jan/2019:08:30:58 +0000] "HEAD / HTTP/1.1" 200 0 "-" "-" "119.28.183.246"

这边我希望的是从nginx-ingress就获取nginx日志，就不存在获取不到ClientIP的问题了，将Deployment转换为DaemonSet的方式，在用于接入流量的机器上启动服务

自定义的方式

Configuration snippet

Using this annotation you can add additional configuration to the NGINX location. For example:

nginx.ingress.kubernetes.io/configuration-snippet: |
  access_log "Request-Id: $req_id";

配置ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx2
  annotations:
     nginx.ingress.kubernetes.io/enable-access-log: "true"
     nginx.ingress.kubernetes.io/configuration-snippet: |
         access_log /var/log/nginx/test.whysdomain.com.access.log;
         error_log  /var/log/nginx/test.whysdomain.com.error.log;
spec:
  rules:
  - host: test.whysdomain.com
    http:
      paths:
      - backend:
          serviceName: nginx
          servicePort: 80

apply之后可以看到配置的更新，需要保证添加的配置格式等正确，否则是不会更新的

www-data@node-02:/etc/nginx$ cat nginx.conf
            access_log /var/log/nginx/test.whysdomain.com.access.log;
            error_log  /var/log/nginx/test.whysdomain.com.error.log;

然后ingress统计的日志就是正常的了

119.28.183.246 - - [18/Jan/2019:02:46:01 +0000] "HEAD / HTTP/1.1" 200 0 "-" "-"
114.251.228.131 - - [18/Jan/2019:02:46:03 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.221 Safari/537.36 SE 2.X MetaSr 1.0"
114.251.228.131 - - [18/Jan/2019:02:46:05 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.221 Safari/537.36 SE 2.X MetaSr 1.0"
119.28.183.246 - - [18/Jan/2019:02:46:06 +0000] "HEAD / HTTP/1.1" 200 0 "-" "-"

这边调整一下时区，对应日志目录挂上数据盘就可以使用了

$ docker run -it --name look-nginx-ingress-controller quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.22.0 /bin/sh
$ cd /etc/nginx     
$ cat nginx.conf
# A very simple nginx configuration file that forces nginx to start.
pid /tmp/nginx.pid;

events {}
http {}
daemon off;

可以看到配置都是后来自动生成的