对于使用clusterIP方式访问

规则通过iptable-save获取，请求链路为

PREROUTING (KUBE-SERVICE -> KUBE-SVC-XXX -> KUBE-SEP-XXX -> KUBE-MARK-MASQ) -> FORWARD（KUBE-FORWARD） -> POSTROUTING（KUBE-POSTROUTING）

OUTPUT(KUBE-SERVICE -> KUBE-SVC-XXX -> KUBE-SEP-XXX) -> POSTROUTING

1.对于请求的数据包PREROUTING或OUTPUT链直接使用KUBE-SERVICES

• pod内部发出来的数据包，进入PREROUTING链
• 本地进程发出的数据包，进入OUTPUT链

# PREROUTING直接使用KUBE-SERVICES
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES

# OUTPUT对新发起的请求使用KUBE-SERVICES
-A OUTPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
# OUTPUT然后使用KUBE-SERVICES限制连接数
-A OUTPUT -j KUBE-FIREWALL
# OUTPUT最后使用KUBE-SERVICES
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES

这里KUBE-FIREWALL

# 对于0x8000/0x8000的进行DROP，目前没发现哪里添加的标记有0x8000/0x8000
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
#
-A KUBE-FIREWALL ! -s 127.0.0.0/8 -d 127.0.0.0/8 -m comment --comment "block incoming localnet connections" -m conntrack ! --ctstate RELATED,ESTABLISHED,DNAT -j DROP

2.对于发往clusterIP对应port的请求使用KUBE-MARK-MASQ添加NAT的标记，并使用KUBE-SVC规则

• 10.243.0.0/16为pod的ip网段
• 10.87.22.200/32为clusterIP

# 请求添加NAT的标记
-A KUBE-SERVICES ! -s 10.243.0.0/16 -d 10.87.22.200/32 -p tcp -m comment --comment "default/flaskapp1:http cluster IP" -m tcp --dport 80 -j KUBE-MARK-MASQ
# 使用KUBE-SVC规则
-A KUBE-SERVICES -d 10.87.22.200/32 -p tcp -m comment --comment "default/flaskapp1:http cluster IP" -m tcp --dport 80 -j KUBE-SVC-MPCMT6MGXSET4UKE

3.根据随机数规则，模拟rr转发的不同的规则

-A KUBE-SVC-MPCMT6MGXSET4UKE -m comment --comment "default/flaskapp1:http" -m statistic --mode random --probability 0.25000000000 -j KUBE-SEP-IR4ERMMC4HNR2YJ7

-A KUBE-SVC-MPCMT6MGXSET4UKE -m comment --comment "default/flaskapp1:http" -m statistic --mode random --probability 0.33333333349 -j KUBE-SEP-GTDJ4WPTJQJC42QW

-A KUBE-SVC-MPCMT6MGXSET4UKE -m comment --comment "default/flaskapp1:http" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-NI6OG4GDPMZ7OKCQ

-A KUBE-SVC-MPCMT6MGXSET4UKE -m comment --comment "default/flaskapp1:http" -j KUBE-SEP-P4JZNZAK77U2CWRH

4. 设置DNAT的目的地址

-A KUBE-SEP-GTDJ4WPTJQJC42QW -s 10.243.3.193/32 -m comment --comment "default/flaskapp1:http" -j KUBE-MARK-MASQ

-A KUBE-SEP-GTDJ4WPTJQJC42QW -p tcp -m comment --comment "default/flaskapp1:http" -m tcp -j DNAT --to-destination 10.243.3.193:80

-A KUBE-SEP-IR4ERMMC4HNR2YJ7 -s 10.243.3.191/32 -m comment --comment "default/flaskapp1:http" -j KUBE-MARK-MASQ

-A KUBE-SEP-IR4ERMMC4HNR2YJ7 -p tcp -m comment --comment "default/flaskapp1:http" -m tcp -j DNAT --to-destination 10.243.3.191:80

-A KUBE-SEP-NI6OG4GDPMZ7OKCQ -s 10.243.3.194/32 -m comment --comment "default/flaskapp1:http" -j KUBE-MARK-MASQ

-A KUBE-SEP-NI6OG4GDPMZ7OKCQ -p tcp -m comment --comment "default/flaskapp1:http" -m tcp -j DNAT --to-destination 10.243.3.194:80

-A KUBE-SEP-P4JZNZAK77U2CWRH -s 10.243.3.195/32 -m comment --comment "default/flaskapp1:http" -j KUBE-MARK-MASQ

-A KUBE-SEP-P4JZNZAK77U2CWRH -p tcp -m comment --comment "default/flaskapp1:http" -m tcp -j DNAT --to-destination 10.243.3.195:80

标记规则

# 添加标记0x4000/0x4000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
# 在FORWARD第一条进行使用
-A FORWARD -m comment --comment "kubernetes forwarding rules" -j KUBE-FORWARD
# 对于标记0x4000/0x4000通过KUBE-FORWARD
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x4000/0x4000 -j ACCEPT
# 对无效数据包进行DROP
-A KUBE-FORWARD -m conntrack --ctstate INVALID -j DROP

1. POSTROUTING链进行伪装

# cni portfwd
-A POSTROUTING -m comment --comment "CNI portfwd requiring masquerade" -j CNI-HOSTPORT-MASQ
# docker出流量
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
# k8s转发
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
# 跳出
-A POSTROUTING -s 10.243.0.0/16 -d 10.243.0.0/16 -j RETURN
# pod进行非组播操作
-A POSTROUTING -s 10.243.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE
# 请求flannel网段的进行请求
-A POSTROUTING ! -s 10.243.0.0/16 -d 10.243.3.0/24 -j RETURN
# 对OUTPUT链的进行伪装
-A POSTROUTING ! -s 10.243.0.0/16 -d 10.243.0.0/16 -j MASQUERADE
# 对于0x4000/0x4000进行伪装（通过PREROUTING和FORWORD链出来的）
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE