leandromoreira /linux-network-performance-parameters

Learn where some of the network sysctl variables fit into the Linux/Kernel network flow. Translations: 🇷🇺

★ 5,832⑂ 531👁 171⑃ 14● 9▤ 0
-1
日增长
5,832
stars
531
forks
171
watchers
14
prs
9
issues
0
releases

README

English | Русский | 中文

 TOC

 Introduction
 Linux network queues overview
 Fitting the sysctl variables into the Linux network flow
   Ingress - they're coming
   Egress - they're leaving
   How to check - perf
 What, Why and How - network and sysctl parameters
   Ring Buffer - rx,tx
   Interrupt Coalescence (IC) - rx-usecs, tx-usecs, rx-frames, tx-frames (hardware IRQ)
   Interrupt Coalescing (soft IRQ) and Ingress QDisc
   Egress QDisc - txqueuelen and default_qdisc
   TCP Read and Write Buffers/Queues
   Honorable mentions - TCP FSM and congestion algorithm
 Network tools
 References

 Introduction

Sometimes people are looking for sysctl cargo cult values that bring high throughput and low latency with no trade-offs and work in every situation. That's not realistic, although we can say that the newer kernel versions are very well tuned by default. In fact, you might hurt performance if you mess with the defaults.

This brief tutorial shows where some of the most used and quoted sysctl/network parameters are located into the Linux network flow, it was heavily inspired by the illustrated guide to the Linux networking stack and many of Marek Majkowski's posts. 

  Feel free to send corrections and suggestions! :)

 Linux network packets flow to memory overview




 Fitting the sysctl variables into the Linux network flow

 Ingress - they're coming
1. Packets arrive at the NIC
2. NIC will verify MAC (if not on promiscuous mode) and FCS and decide to drop or to continue
3. NIC will DMA packets to RAM, in a region previously prepared (mapped) by the driver
4. NIC will enqueue references to the packets at receive ring buffer queue rx until rx-usecs timeout or rx-frames
5. NIC will raise a hard IRQ
6. CPU will run the IRQ handler that runs the driver's code
7. 7. The driver will schedule NAPI, clear the hard IRQ, and return
8. The driver raises a soft IRQ (NET_RX_SOFTIRQ)
9. NAPI will poll data from the receive ring buffer until netdev_budget_usecs timeout or netdev_budget and dev_weight packets
10. Linux will also allocate memory to sk_buff
11. Linux fills the metadata: protocol, interface, setmacheader, removes ethernet
12. Linux will pass the skb to the kernel stack (netif_receive_skb)
13. It will set the network header, clone skb to taps (i.e. tcpdump) and pass it to tc ingress
14. Packets are handled to a qdisc sized netdev_max_backlog with its algorithm defined by default_qdisc
15. It calls ip_rcv and packets are handed to IP
16. It calls netfilter (PREROUTING)
17. It looks at the routing table, if forwarding or local
18. If it's local it calls netfilter (LOCAL_IN)
19. It calls the L4 protocol (for instance tcp_v4_rcv)
20. It finds the right socket
21. It goes to the tcp finite state machine
22. Enqueue the packet to  the receive buffer and sized as tcp_rmem rules
    1. If tcp_moderate_rcvbuf is enabled, the kernel will auto-tune the receive buffer
23. Kernel will signalize that there is data available to apps (epoll or any polling system)
24

Top issues

Snapshot: 2026-11-16T09:23:14Z · Data license Apache-2.0 · JSON