目次

Linuxのパケットドロップとは(RX_DROP問題)




Linuxのパケットドロップとは

パケットが破棄されてしまうことです。


Linuxのパケットドロップの原因

【故意にドロップ】未サポートプロトコルパケット


【やむを得ずドロップ】大量のパケットが来たときに溢れる



参考:ifconfig shows dropped rx packets | Support | SUSE

Beginning with kernel 2.6.37, it has been changed the meaning of dropped packet count. Before, dropped packets was most likely due to an error. Now, the rx_dropped counter shows statistics for dropped frames because of:
If any frames meet those conditions, they are dropped before the protocol stack and the rx_dropped counter is incremented.


Linuxで、パケットドロップの確認方法

netstat -i

# netstat -i
Kernel Interface table
Iface      MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0      1500 653090942      0   1481 0      84429784      0      0      0 BMRU
lo       65536  1280117      0      0 0       1280117      0      0      0 LRU
#


# ip -s link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    RX: bytes  packets  errors  dropped overrun mcast
    119101765  1280127  0       0       0       0
    TX: bytes  packets  errors  dropped carrier collsns
    119101765  1280127  0       0       0       0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 02:01:76:1b:xx:xx brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast
    61367859241 653095320 0       1481    0       0
    TX: bytes  packets  errors  dropped carrier collsns
    269612612688 84429834 0       0       0       0


ifconfig

# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet xxx.xxx.xxx.xxx  netmask 255.255.254.0  broadcast xxx.xxx.xxx.xxx
        inet6 fe80::1:xxxx:xxxx:xxxx  prefixlen 64  scopeid 0x20<link>
        ether 02:01:76:xx:xx:xx  txqueuelen 1000  (Ethernet)
        RX packets 653131438  bytes 61371164765 (61.3 GB)
        RX errors 0  dropped 1481  overruns 0  frame 0
        TX packets 84435136  bytes 269624405533 (269.6 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 1280223  bytes 119110409 (119.1 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1280223  bytes 119110409 (119.1 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0


ethtool -S

ifconfigなどのコマンドで、ドロップになるが、ethtool -Sでドロップがないか確認
ethtool -Sでドロップがない場合、未サポートプロトコルの問題で、影響はない。

# ethtool -S eth0
NIC statistics:
     rx_queue_0_packets: 653279140
     rx_queue_0_bytes: 61384496586
     rx_queue_0_drops: 0
     rx_queue_0_xdp_packets: 0
     rx_queue_0_xdp_tx: 0
     rx_queue_0_xdp_redirects: 0
     rx_queue_0_xdp_drops: 0
     rx_queue_0_kicks: 10165
     tx_queue_0_packets: 84452496
     tx_queue_0_bytes: 269671658132
     tx_queue_0_xdp_tx: 0
     tx_queue_0_xdp_tx_drops: 0
     tx_queue_0_kicks: 47507630


ドロップ対策

バッファ拡張

現在のバッファーと利用可能なバッファーを確認

# ethtool -g eth0
Ring parameters for eth0:
Pre-set maximums:
RX:             4094
RX Mini:        0
RX Jumbo:       0
TX:             4094
Current hardware settings:
RX:             256
RX Mini:        0
RX Jumbo:       0
TX:             256


一時的にバッファの拡張

ethtool -G eth0 rx 512


バッファ拡張の永続化

Ubuntu

/etc/network/interfaces

auto eth0
iface eth0 inet static
    address 192.168.1.2
    netmask 255.255.255.0
    gateway 192.168.1.1
    dns-nameservers 192.168.1.1
    post-up ethtool -G eth0 rx 512


ドロップのメモ

RHEL network interface dropping packets

https://access.redhat.com/solutions/21301

RHEL 7 drop statistics cover more reasons for discards of unwanted packets than previous version of RHEL. For details, see this article.
Packets can be dropped at other levels in the network stack. These will not be counted as discards/drops/etc shown by ethtool -S ethX.
ネットワークスタックのその他のレベルで、パケットが破棄される場合があります。そのようなパケットは、ethtool -S ethX で示される discard、drop などには含まれません。


ifconfig and ip commands report packet drops?

https://access.redhat.com/solutions/657483

問題

ip -s link and ifconfig report the rx_drop* counter increasing but no drops are seen with ethtool -S ethX
netstat -i reports RX-ERR and RX-DRP

解決策

This is not a bug. Previous kernels were not accounting for these dropped packets (unknown protocol for example)

バグではない。
不明なパケットでも、前のカーネルではカウントしなかった。
(RHEL7から、カウントするようになった)

原因

Seeing high rx_drop_counters in RHEL 7 from ifconfig|ip -s link is not a definitive conclusion of a network problem. To gain more traction of a network problem “ethtool -S” can be used for Network Card Statistics or “netstat -s” for Network Application statistics.

RHEL7では、ネットワークの問題解決のために、ifconfigip -s linkを使うな。
ethtool -Snetstat -sを使え。

RHEL 7 differentiates packet loss from older o/s (RHEL 6 and RHEL 5), as rx drop counter can increment from the following reasons:

診断

Note: While tcpdump is running, the above frame does not increase rx drop count.

tcpdumpして、rx dropカウンターが上がらないこと。
(上がらなければ、不明なパケットがとどいていてRX Dropがあがっているだけで問題ない。)


promiscモードでドロップが止まる場合

参考:ifconfig shows dropped rx packets | Support | SUSE

Care should be taken to confirm that frames are not being legitimately dropped. A quick way to test this (WARNING: this test does not work for bonding interfaces) is to force the NIC into promiscuous mode:
host:~# ifconfig <interface> promisc
And then watching the rx_dropped counter. If it stops incrementing while the NIC is in promiscuous mode; then it is more than likely showing drops because of the reasons listed earlier.
If additional frames continue to be shown as dropped, further investigation might be warranted. However, since IP communication and/or the applications that use it have methods in place for recovering missing packets, a small number of dropped packets is not expected to have any significant impact, and is not worth investigating. Additionally, some applications, such as video streaming, don't consider a few lost packets to be something that requires any recovery attempts, and they proceed without any delay.


tcpdumpするとドロップが止まる場合

https://community.checkpoint.com/t5/Cloud-Network-Security/Very-high-RX-drops-on-R80-40-on-ESXi/td-p/84873

The “other drops” you are referring to are probably non-IPv4 packets being discarded by the Ethernet driver, does the incrementing of RX-DRP stop for as long as you are actively running a tcpdump? See the following from my book:
RX-DRP Culprit 1: Unknown or Undesired Protocol Type


In every Ethernet frame is a header field called “EtherType”. This field specifies the OSI
Layer 3 protocol that the Ethernet frame is carrying. A very common value for this
header field is 0x0800, which indicates that the frame is carrying an Internet Protocol
version 4 (IPv4) packet. Look at this excerpt from Stage 6 of “A Millisecond in the life
of a frame”:


Stage 6: At a later time the CPU begins SoftIRQ processing and looks in the ring
buffer. If a descriptor is present, the CPU retrieves the frame from the associated
receive socket buffer, clears the descriptor referencing the frame in the ring
buffer, and sends the frame to all “registered receivers” which will be the
SecureXL acceleration driver. If a tcpdump capture is currently running,
libpcap will also be a “registered receiver” in that case and get a copy of the
frame as well. The SoftIRQ processing continues until all ring buffers are
completely emptied, or various packet count or time limits have been reached.


During hardware interrupt processing, the NIC driver will examine the EtherType
field and verify there is a “registered receiver” present for the protocol specified in the
frame header. If there is not, the frame is discarded and RX-DRP is incremented.
Example: an Ethernet frame arrives with an EtherType of 0x86dd indicating the
presence of IPv6 in the Ethernet frame. If IPv6 has not been enabled on the firewall (it is
off by default), the frame will be discarded by the NIC driver and RX-DRP incremented.
What other protocols are known to cause this effect in the real world? Let’s take a look
at a brief sampling of other possible rogue EtherTypes you may see, that is by no means
complete:


- Appletalk (0x809b)
- IPX (0x8137 or 0x8138)
- Ethernet Flow Control (0x8808) if NIC flow control is disabled
- Jumbo Frames (0x8870) if the firewall is not configured to process jumbo frames


The dropping of these protocols for which there is no “registered receiver” does
cause a very small amount of overhead on the firewall during hardware interrupt
processing, but unless the number of frames discarded in this way exceeds 0.1% of all
inbound packets, you probably shouldn’t worry too much about it. An easy way to
confirm that the lack of a registered receiver is the cause of RX-DRPs is to perform the
following test:

1. In a SSH or terminal window, run watch -d netstat -ni and confirm
the constant incrementing of RX-DRP on (interface).
2. In a second SSH session, run tcpdump -ni (interface) host 1.1.1.1

Does the near constant incrementing of RX-DRP on that interface suddenly stop as
long as the tcpdump is still running, and resume when the tcpdump is stopped? If so,
the lack of a registered receiver is indeed the cause of the RX-DRPs. The specified filter
expression (host 1.1.1.1 in our example) does not actually matter, since libpcap will

register to receive all protocols on behalf of the running tcpdump, and then filter the
packets based on the provided tcpdump expression. So as long as the tcpdump is
running, there is essentially a registered received for everything.


参考