coredns使用代理解析失败

时间:Jan. 18, 2021 分类:

目录:

问题介绍

新上kubernetes业务解析部分域名失败

现象参考

$ nslookup api-push.vivo.com.cn
Server:         10.96.0.10
Address:        10.96.0.10#53

** server can't find api-push.vivo.com.cn: SERVFAIL
$ nslookup www.sina.com.cn
Server:         10.96.0.10
Address:        10.96.0.10#53

** server can't find www.sina.com.cn: SERVFAIL
$ nslookup www.baidu.com
Server:         10.96.0.10
Address:        10.96.0.10#53

Non-authoritative answer:
www.baidu.com   canonical name = www.a.shifen.com.
Name:   www.a.shifen.com
Address: 180.101.49.11
Name:   www.a.shifen.com
Address: 180.101.49.12
这里www.baidu.com是能正常解析的,而其他两个域名不能正常解析

在宿主机上,三个域名都能正常解析

问题排查

抓包排查

这里宿主机抓包看到,dns的请求返回的数据是正常的,但是会在一次dns请求的时间内重复请求

$ tcpdump -nnn  port 53
11:16:23.111282 IP 192.168.1.200.53062 > 192.168.1.100.53: 30973+ A? api-push.vivo.com.cn. (38)
11:16:23.113128 IP 192.168.1.100.53 > 192.168.1.200.53062: 30973 20/0/0 A 182.92.255.169, A 112.126.11.70, A 112.126.122.234, A 39.105.250.172, A 112.126.122.207, A 112.126.12.147, A 112.126.123.53, A 39.106.77.118, A 39.105.252.165, A 39.105.250.170, A 112.126.12.58, A 39.107.240.232, A 39.105.251.19, A 39.106.66.100, A 39.105.250.174, A 123.57.207.113, A 112.126.12.13, A 39.105.253.59, A 39.105.180.52, A 39.105.250.169 (758)
11:16:23.113434 IP 192.168.1.200.17143 > 192.168.1.101.53: 30973+ A? api-push.vivo.com.cn. (38)
11:16:23.113518 IP 192.168.1.200.40943 > 192.168.1.100.53: 64593+ NS? . (17)
11:16:23.115875 IP 192.168.1.101.53 > 192.168.1.200.17143: 30973 20/0/0 A 39.105.252.119, A 39.105.252.228, A 39.105.252.165, A 39.106.77.118, A 39.105.180.52, A 182.92.255.169, A 112.126.122.207, A 39.105.253.59, A 39.105.250.169, A 123.57.207.113, A 112.126.12.147, A 39.105.250.172, A 39.106.66.100, A 39.105.250.174, A 112.126.12.13, A 39.105.252.250, A 39.105.251.19, A 39.107.240.232, A 112.126.11.70, A 39.105.250.170 (758)
11:16:23.115899 IP 192.168.1.100.53 > 192.168.1.200.40943: 64593 13/0/0 NS g.root-servers.net., NS k.root-servers.net., NS a.root-servers.net., NS e.root-servers.net., NS d.root-servers.net., NS m.root-servers.net., NS c.root-servers.net., NS b.root-servers.net., NS h.root-servers.net., NS i.root-servers.net., NS l.root-servers.net., NS f.root-servers.net., NS j.root-servers.net. (420)
11:16:23.116168 IP 192.168.1.200.39677 > 192.168.1.102.53: 30973+ A? api-push.vivo.com.cn. (38)
11:16:23.116256 IP 192.168.1.200.53370 > 192.168.1.101.53: 183+ NS? . (17)
11:16:23.118521 IP 192.168.1.102.53 > 192.168.1.200.39677: 30973 20/0/0 A 39.105.253.59, A 39.105.252.250, A 112.126.12.13, A 39.105.250.174, A 39.105.252.165, A 112.126.12.147, A 39.106.77.118, A 39.105.252.228, A 39.105.252.119, A 112.126.122.207, A 112.126.11.70, A 39.105.250.172, A 39.105.250.170, A 39.105.251.19, A 182.92.255.169, A 123.57.207.113, A 39.107.240.232, A 39.106.66.100, A 39.105.180.52, A 39.105.250.169 (758)
11:16:23.118535 IP 192.168.1.101.53 > 192.168.1.200.53370: 183 13/0/0 NS b.root-servers.net., NS h.root-servers.net., NS i.root-servers.net., NS l.root-servers.net., NS f.root-servers.net., NS j.root-servers.net., NS g.root-servers.net., NS k.root-servers.net., NS a.root-servers.net., NS e.root-servers.net., NS d.root-servers.net., NS m.root-servers.net., NS c.root-servers.net. (420)
11:16:23.118745 IP 192.168.1.200.48507 > 192.168.1.100.53: 30973+ A? api-push.vivo.com.cn. (38)
11:16:23.118837 IP 192.168.1.200.27376 > 192.168.1.102.53: 40007+ NS? . (17)
11:16:23.119816 IP 192.168.1.102.53 > 192.168.1.200.27376: 40007 13/0/0 NS j.root-servers.net., NS g.root-servers.net., NS k.root-servers.net., NS a.root-servers.net., NS e.root-servers.net., NS d.root-servers.net., NS m.root-servers.net., NS c.root-servers.net., NS b.root-servers.net., NS h.root-servers.net., NS i.root-servers.net., NS l.root-servers.net., NS f.root-servers.net. (420)
11:16:23.120716 IP 192.168.1.100.53 > 192.168.1.200.48507: 30973 20/0/0 A 39.105.250.174, A 39.105.180.52, A 39.105.252.250, A 112.126.12.147, A 39.107.240.232, A 39.105.250.172, A 39.105.252.228, A 39.105.250.170, A 39.105.251.19, A 39.105.253.59, A 39.105.252.119, A 112.126.122.207, A 123.57.207.113, A 39.105.250.169, A 39.106.66.100, A 39.106.77.118, A 112.126.12.13, A 39.105.252.165, A 112.126.11.70, A 182.92.255.169 (758)
11:16:23.120955 IP 192.168.1.200.4269 > 192.168.1.101.53: 30973+ A? api-push.vivo.com.cn. (38)
11:16:23.120983 IP 192.168.1.200.30322 > 192.168.1.100.53: 9184+ NS? . (17)
11:16:23.121808 IP 192.168.1.100.53 > 192.168.1.200.30322: 9184 13/0/0 NS j.root-servers.net., NS g.root-servers.net., NS k.root-servers.net., NS a.root-servers.net., NS e.root-servers.net., NS d.root-servers.net., NS m.root-servers.net., NS c.root-servers.net., NS b.root-servers.net., NS h.root-servers.net., NS i.root-servers.net., NS l.root-servers.net., NS f.root-servers.net. (420)
11:16:23.122700 IP 192.168.1.101.53 > 192.168.1.200.4269: 30973 20/0/0 A 182.92.255.169, A 112.126.11.70, A 112.126.12.13, A 112.126.123.53, A 39.105.252.165, A 39.105.250.174, A 112.126.12.147, A 112.126.122.207, A 39.105.180.52, A 112.126.122.234, A 39.106.66.100, A 112.126.12.58, A 39.105.250.169, A 39.105.251.19, A 39.105.253.59, A 39.105.250.170, A 39.105.250.172, A 39.107.240.232, A 39.106.77.118, A 123.57.207.113 (758)
11:16:23.122912 IP 192.168.1.200.58375 > 192.168.1.102.53: 30973+ A? api-push.vivo.com.cn. (38)
...

日志

[ERROR] plugin/errors: 2 api-push.vivo.com.cn. A: dns: buffer size too small
[ERROR] plugin/errors: 2 api-push.vivo.com.cn. A: dns: buffer size too small
[ERROR] plugin/errors: 2 api-push.vivo.com.cn. A: dns: buffer size too small
[ERROR] plugin/errors: 2 api-push.vivo.com.cn. A: dns: buffer size too small
[ERROR] plugin/errors: 2 api-push.vivo.com.cn. A: dns: buffer size too small
[ERROR] plugin/errors: 2 api-push.vivo.com.cn. A: dns: buffer size too small
[ERROR] plugin/errors: 2 api-push.vivo.com.cn. A: dns: buffer size too small
[ERROR] plugin/errors: 2 api-push.vivo.com.cn. A: dns: buffer size too small
[ERROR] plugin/errors: 2 api-push.vivo.com.cn. A: dns: buffer size too small
[ERROR] plugin/errors: 2 api-push.vivo.com.cn. A: dns: buffer size too small
[ERROR] plugin/errors: 2 api-push.vivo.com.cn. A: dns: buffer size too small
[ERROR] plugin/errors: 2 api-push.vivo.com.cn. A: dns: buffer size too small
[ERROR] plugin/errors: 2 api-push.vivo.com.cn. A: dns: buffer size too small
[ERROR] plugin/errors: 2 api-push.vivo.com.cn. A: dns: buffer size too small
[ERROR] plugin/errors: 2 api-push.vivo.com.cn. A: dns: buffer size too small
[ERROR] plugin/errors: 2 www.sina.com.cn. A: dns: overflow unpacking uint32
[ERROR] plugin/errors: 2 www.sina.com.cn. A: dns: overflow unpacking uint32
[ERROR] plugin/errors: 2 www.sina.com.cn. A: dns: overflow unpacking uint32
[ERROR] plugin/errors: 2 www.sina.com.cn. A: dns: overflow unpacking uint32
[ERROR] plugin/errors: 2 www.sina.com.cn. A: dns: overflow unpacking uint32
[ERROR] plugin/errors: 2 www.sina.com.cn. A: dns: overflow unpacking uint32
[ERROR] plugin/errors: 2 www.sina.com.cn. A: dns: overflow unpacking uint32
[ERROR] plugin/errors: 2 www.sina.com.cn. A: dns: overflow unpacking uint32
[ERROR] plugin/errors: 2 www.sina.com.cn. A: dns: overflow unpacking uint32
[ERROR] plugin/errors: 2 www.sina.com.cn. A: dns: overflow unpacking uint32
[ERROR] plugin/errors: 2 www.sina.com.cn. A: dns: overflow unpacking uint32
[ERROR] plugin/errors: 2 www.sina.com.cn. A: dns: overflow unpacking uint32
[ERROR] plugin/errors: 2 www.sina.com.cn. A: dns: overflow unpacking uint32
[ERROR] plugin/errors: 2 www.sina.com.cn. A: dns: overflow unpacking uint32
[ERROR] plugin/errors: 2 www.sina.com.cn. A: dns: overflow unpacking uint32
[ERROR] plugin/errors: 2 www.sina.com.cn. A: dns: overflow unpacking uint32

参考buffer size too small问题,可能是因为解析dns返回的数据记录太多导致

问题解决

参考官方文档bufsize

bufsize在1.6.6版本引入,我们是使用的为1.6.7,生效的代码为coredns的plugin/forward/connect.go

func (p *Proxy) Connect(ctx context.Context, state request.Request, opts options) (*dns.Msg, error) {
    ...

    pc, cached, err := p.transport.Dial(proto)
    if err != nil {
        return nil, err
    }

    // Set buffer size correctly for this client.
    pc.c.UDPSize = uint16(state.Size())
    if pc.c.UDPSize < 512 {
        pc.c.UDPSize = 512
    }
    ...
}

使用UDP的原因是,forward这边只做请求的转发,默认不做协议的转换,默认buffersize为512,调整为最大的4096问题恢复

apiVersion: v1
data:
  Corefile: |
    .:53 {
        bufsize 4096
        errors
        health {
           lameduck 5s
        }
        ready
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30 {
           success 10240
           denial 51200
        }
        template ANY AAAA {
           rcode NXDOMAIN
        }
        loop
        reload
        loadbalance
    }
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system

造成一直循环请求的问题是这段代码

/plugin/forward/forward.go

func (f *Forward) ServeDNS(ctx context.Context, w dns.ResponseWriter, r *dns.Msg) (int, error) {
        ...
        opts := f.opts
        for {
            ret, err = proxy.Connect(ctx, state, opts)
            if err == ErrCachedClosed { // Remote side closed conn, can only happen with TCP.
                continue
            }
            // Retry with TCP if truncated and prefer_udp configured.
            if ret != nil && ret.Truncated && !opts.forceTCP && opts.preferUDP {
                opts.forceTCP = true
                continue
            }
            break
        }
        ...

缓冲区溢出导致ret为空,导致一直在for循环直到超时

如果日后出现4096还不能满足的域名解析,需要配置forward的force_tcp参数对域单独进行tcp请求,参考forward

根因调查

线上服务coredns数据包做了报文压缩,大小为400bytes

进行一下进制转换

c00c(十六进制) = 1100 0000 0000 1100(二进制)

  • 前两位(高位)11代表压缩
  • 后边14位为指向的位置1100(二进制转为10进制)为12,代表DNS报文偏移12字节的位置,详见https://www.ietf.org/rfc/rfc1035.txt的4.1.4. Message Compression
  • 08代表这集域名的长度api-push,8个字符,然后61为a,可以python的chr(0x61)

DNS报文偏移的12个字节如下所示

这样2bytes(c0 0c)代替22 bytes(字符对应为8api-push4vivo3com2cn0,长度为22),每个解析节省20bytes,一共20个A记录解析,就是节省400bytes

另一个自研的DNS,数据包没有压缩,大小为800bytes,超过了512B的默认配置,对于400bytes多了400bytes

应该是没有设置压缩的参数Compress,参考https://github.com/miekg/dns/blob/68df4402de4a2303c8dad5aea62450485d589d81/msg.go

type Msg struct {
    MsgHdr
    Compress bool       `json:"-"` // If true, the message will be compressed when converted to wire format.
    Question []Question // Holds the RR(s) of the question section.
    Answer   []RR       // Holds the RR(s) of the answer section.
    Ns       []RR       // Holds the RR(s) of the authority section.
    Extra    []RR       // Holds the RR(s) of the additional section.
}