Homa vs TCP: the transport that sends the shortest message first

The tail is the product

A web request in a large datacenter rarely touches one server. It fans out into dozens or hundreds of remote procedure calls (RPCs): cache lookups, index shards, feature stores. The response is ready only when the slowest of them returns. That makes the 99th-percentile latency of a single RPC, not its average, the thing users feel. At a fan-out of 100, almost two requests in three (1 − 0.99¹⁰⁰ ≈ 63%) hit at least one 99th-percentile straggler.

Those RPCs are overwhelmingly small. The Homa paper characterised five workloads, from Facebook's memcached tier to Google's search and Facebook's Hadoop clusters. In three of them more than 85% of messages were under 1,000 bytes.¹ But the bytes don't follow the messages. In the Hadoop workload (W4), 60% of messages are 1,000 bytes or smaller, yet they carry half a percent of the bytes (Figure 1). A transport that treats every byte equally treats the short message, the one a user is waiting on, as a rounding error.

On a 25 Gbps link, a 1 KB message needs about a third of a microsecond of wire time. Everything else in its latency is software overhead or waiting: in a queue, behind a congestion window, or behind another message in the same TCP stream. Homa's argument is that the waiting is mostly self-inflicted.

Five things TCP gets wrong

Ousterhout's 2022 position paper, It's Time to Replace TCP in the Datacenter, argues that "every significant element of TCP, from its stream orientation to its expectation of in-order packet delivery, is wrong for the datacenter".² It names five properties:

1. Stream orientation. A TCP connection is a single ordered byte stream, so a short message queued behind a long one waits for all of it (head-of-line blocking). In traces of TCP RPCs near the 99th percentile, "roughly two-thirds of the traces had head-of-line blocking where a short message was stuck behind a long one in the same stream."³
2. Connection orientation. State per peer, set-up cost, and a lot of connections. The paper puts Linux at about 2,000 bytes of state per TCP socket, against roughly 300 bytes per active RPC and 200 bytes per peer host for Homa.²
3. Bandwidth sharing. TCP congestion control converges on giving every flow an equal share. Equal is not optimal when sizes vary by six orders of magnitude. Measured with TCP at 80% load, "short messages suffer a slowdown almost 10x worse than the longest messages".²
4. Sender-driven congestion control. A sender only learns about congestion once queues have built up, and TCP does not use the priority queues that datacenter switches already have.
5. In-order delivery. Expecting packets in order pushes networks toward flow-consistent routing, which pins each flow to one path. The paper hypothesises that this causes "virtually all" core congestion. It is a hypothesis, not a measurement.

How Homa works

Homa keeps the IP layer and replaces everything above it. The 2018 SIGCOMM paper gives the design in one sentence: "it is receiver-driven; it is message-oriented, rather than stream-oriented; it is connectionless; it uses no explicit acknowledgments; and it implements at-least-once semantics". The last two have since changed (below).¹ The pieces:

• Messages, not streams. The sender declares a message's length up front, and the receiver delivers whole messages. A short message is never stuck behind a long one in the same stream.
• RPCs, not connections. Each RPC carries a globally unique id; there is no handshake and no per-pair connection.
• Blind first bytes. A sender may transmit the first RTTbytes of a message immediately, without asking. That is the amount of data that covers one round trip, "about 10 KB" at 10 Gbps in the original implementation. Most messages fit entirely inside this unscheduled window, so they never wait for permission.
• Grants for the rest. The receiver sees every incoming message, knows how many bytes each has left, and sends GRANT packets that admit more data, favouring the message with the fewest remaining bytes. This approximates the scheduling policy SRPT, short for shortest remaining processing time.
• Switch priorities do the pre-empting. Datacenter switches have a handful of priority queues per port ("typically 8"). Homa assigns the priority of each packet, so a short message's packets jump ahead of a long message's packets already queued at the top-of-rack switch.
• Controlled overcommitment. A receiver grants to several messages at once, not just the top one, so its downlink stays busy when a sender stalls. In the 2018 simulations, granting to only one message at a time let Homa use only about 63% of the bandwidth on W4; with overcommitment of 7 it reached 89%.¹

Several details have changed since 2018. Two matter for describing the protocol, and descriptions of "current Homa" should say so. The Linux implementation switched from at-least-once to at-most-once RPC semantics, with explicit ACK packets, in November 2021. And in September 2026 the reference implementation stopped sending unscheduled packets for messages large enough to need grants; those now announce themselves with a new START_MSG packet.⁶

Why shortest-first wins

Picture one receiver's downlink, the place Homa expects queueing to happen. Two messages want it: a long one, A, and a short one, B, which arrives a moment later. Fair sharing splits the link between them. Shortest-first sends B to completion, then resumes A. Figure 3 draws both.

The long message finishes at the same moment either way: the link was busy the whole time, so the total work done by time t is identical. Only the order changed, and changing the order made B twice as fast at no cost to A. That is the whole intuition behind SRPT. Schrage proved in 1968 that, on a single server, SRPT minimises mean response time for any arrival sequence and job sizes.⁵

The usual worry is starvation: a long message might wait forever behind a stream of short ones. Queueing theory says it is mostly unfounded. To compare policies, use slowdown: the time a message takes divided by the time it would take alone on an idle link.

Here x is the message size and C the link rate, so S = 1 is perfect. For an idealised fair-sharing link (an M/G/1 processor-sharing queue: random arrivals, any size distribution), the expected slowdown is the same for every message size:

where ρ is the load. At 80% load, every message, a 100-byte lookup or a 1 MB shuffle block, takes five times its ideal time on average. That is what "fair" buys. Bansal and Harchol-Balter analysed SRPT against it for heavy-tailed sizes, where the largest 1% of jobs carry more than half the load. They found that "at least 99% of the jobs have a lower expected response time under SRPT than under PS". For any size distribution at load 0.8, no job size's expected response time under SRPT exceeds 3× its fair-sharing value.⁴ Below 50% load, every job size does better under SRPT.

Homa's own measurements show the cost does exist at the far end: on the RAMCloud implementation the very largest messages saw 99th-percentile slowdowns of 100× or more.¹ That is why the Linux version reserves a small slice of bandwidth, "typically 5–10%", for the oldest message rather than the smallest.²

A simulation you can run

To see the size of the effect on realistic traffic, we simulated one receiver downlink fed with Homa's own workload distributions. We took them from the util/dist.cc tables in the Homa Linux repository, truncated at 1 MB as in the Linux paper; the means reproduce that paper's Table 3 exactly.^3,22 Messages arrive at random (Poisson), and we compare four ways of serving them:

• Fair share: every active message gets an equal slice of the link. This is the fluid ideal TCP congestion control aims for, without TCP's real-world overheads.
• 8 priority levels: messages are binned by remaining size into 8 strict priorities, cutoffs evenly spaced in log(size). Within a level they share fairly, as packets from several senders interleave in one switch queue. A cartoon of Homa's switch queues, not Homa's exact rules.
• SRPT: always the message with the fewest bytes left. The policy Homa approximates.
• First come, first served: one queue, no pre-emption. The worst case for head-of-line blocking, kept as a reference.

The simulator checks itself against Equation 2: its fair-share mean slowdown on W4 is within 1% of 1/(1 − ρ) from 30% to 70% load (5.14 against 5.00 at 80%, where the high variance of message sizes makes finite runs noisier). The setup has 400,000 messages per run, the first 10% discarded as warm-up, and one fixed seed for every run.

Figure 4 is the headline. Under fair sharing the tail is flat and high: about 21× for the smallest messages and the largest. Equation 2 predicts a flat mean of 5×. The flat tail has its own explanation: in a processor-sharing queue the number of messages present is geometric, P(N ≥ k) = ρ^k, and a tiny message's slowdown is about that number plus one. Since 0.8²¹ ≈ 0.01, the 99th percentile lands near 21. Under SRPT the smallest 60% of messages have a 99th-percentile slowdown of 1.0: 99 in 100 of them finish as if the link were idle. Even the largest decile, the messages SRPT is supposed to punish, comes out ahead (10.6× against 20.9×), consistent with Bansal and Harchol-Balter. First come, first served is off the chart: 99th-percentile slowdowns of thousands to tens of thousands for small messages, the cost of one queue with no overtaking.

Load matters more than anything else for fair sharing (Figure 5). Its 99th-percentile slowdown grows from 4× at 30% load to 49× at 90%, while SRPT's rises from 1.8× to 7.5×. Eight log-spaced levels stay within about 15% of SRPT up to 50–60% load, then diverge (3.7× against 2.9× at 70%). So how many levels do you need?

Fewer than you'd think (Figure 6). For the shortest half of messages, going from one level (fair sharing) to two cuts the 99th-percentile slowdown from 21.4× to 2.0×. Four levels reach 1.05×. Across all messages the gains come more slowly: 2 levels take the P99 from 21.3× to 14.3×, and 4 levels to 8.2×, against 3.9× for SRPT. The hardware papers broadly agree. RAMCloud was "almost as good with 4 priority levels as with 8" but noticeably worse with only 2; the Linux implementation "does well with 2 priorities".^1,3 That is good news for deployment: you don't need all eight of a switch's queues, and you can leave some for other traffic.

The pattern holds across workloads. At 80% load, the shortest half of messages see a 99th-percentile slowdown of 21–26× under fair sharing, against 1.0–2.0× under SRPT and 1.0–2.5× with 8 levels. (On real hardware W2 could not be run above about 40% load; that row describes the idealised link only.)³

What was measured on real hardware

Simulations of an idealised link are the easy part. There are two hardware implementations, and they answer different questions. The table separates what each actually measured.

The best-case numbers from the Linux paper show where Homa is not better:³

Three things are worth knowing before quoting Homa's numbers:

• "Under 15 µs at the 99th percentile" is the kernel-bypass number at 10 Gbps. The Linux module's short-message P99 on W4 is about 100 µs. The paper attributes almost all of that to software overheads, such as load-balancing hot spots and NAPI bursts, rather than the network.³
• Large single streams are slower. One client sending large messages back-to-back gets half TCP's throughput, because whole-message delivery leaves less room for pipelining.
• Software is expensive. The 1.6 Mops/s peak used 18 of the machine's 20 hardware threads (10 cores, hyperthreaded). The paper estimates "at least 18 cores" to drive 100 Gbps at 80% load, and argues the transport belongs in the NIC.³

Open problems

Homa has drawn a lot of critical papers. Several don't hold up. Aeolus,¹¹ for example, gave each switch port a fixed 200 KB of buffer, which "no modern switch" does, according to the Homa wiki's rebuttal. dcPIM compares against a modified "Homa Aeolus" rather than Homa, and PowerTCP used a third-party simulator that the Homa authors and, reportedly, PowerTCP's authors agree produced incorrect Homa results.¹³ Practitioners have also questioned whether the TCP usage the position paper criticises is representative of production stacks.²⁰ What survives scrutiny, much of it acknowledged by Homa's author, is still substantial:

• Switch buffers. On the 25 Gbps testbed Homa's peak buffer use ranged from 246 KB (W2) to 8.5 MB (W5) of a 13 MB shared pool, and it degraded severely at 6 MB. TCP degraded noticeably at 10 MB and severely at 6 MB; DCTCP was fine down to 2 MB.³ Asked whether Homa has problems with excessive buffer usage, Ousterhout's own concerns page answers "This is an open question" and says that "if packets are dropped frequently, Homa will perform poorly."¹³ SIRD (NSDI 2025) reports 12× less peak top-of-rack buffering than Homa, with near-competitive latency, in simulation.¹² Ousterhout's response: "These results are very promising", with a SIRD-style change on his to-do list.¹³
• Sharing links with TCP. Homa assumes it owns the receiver's downlink, and real hosts run both. At Netdev 0x1A in July 2026, Ousterhout reported that next to TCP, Homa's performance "degrades drastically (because TCP still overloads buffers)". Without mitigation, Homa's short-message P99 rose about 4× alongside TCP. A new queueing discipline, homa_qdisc (January 2026), paces both protocols; with it, Homa "suffers only slight degradation in the face of competing TCP traffic".^6,14 These are the author's own results on one 100 Gbps cluster, not yet independently replicated. Coexistence was a serious obstacle to gradual adoption; it may now be a manageable one.
• The network core. Homa's congestion control is "effective against congestion at the network edge, but it does not address congestion in the core".³ Google's Falcon paper makes the point sharply: receiver-driven schemes "assume no over-subscription within the network, which is often untrue".¹⁵
• One RTTbytes for everyone. ByteDance's Netdev 0x17 evaluation noted that Homa uses one RTTbytes value for all peers, though round-trip times vary. It also found Homa best suited to very low-latency networks, which limits it "primarily to intra-rack servers".²³
• Security. We found no published analysis of forged grants, malicious receivers or floods of unscheduled packets. That is our observation of a gap, not a known flaw, but a protocol that lets a receiver direct other hosts' traffic deserves one.

Where Homa stands in 2026

• Not merged into Linux. As of 3 October 2026, neither mainline (7.3-rc5) nor net-next contains net/homa. The series "Begin upstreaming Homa transport protocol" has been posted since October 2024; the latest revision, posted on 14 September 2026, is labelled v11 in its subject but v20 in its changelog, and is marked "changes requested".⁷ It is deliberately minimal: about 8,000 of the full implementation's 20,000 lines, with performance "about the same as TCP". LWN's assessment in December 2024 that "the initial merge of Homa is not imminent" has held.⁸
• It has a protocol number. IANA assigned IP protocol 146 to Homa.⁹
• The out-of-tree module is alive. HomaModule was last upgraded, per its README, to Linux 6.17.8 (November 2025), and was backported to RHEL 8 and 9.5 in March 2026.⁶
• No public production deployment. We found none. The gRPC integration, the obvious adoption path, has been "no longer under active development" since late 2023. In its README, short RPCs ran about 40% faster than over TCP, 55 vs 90 µs.¹⁰

Meanwhile the transports that did ship took some of Homa's ideas and left others:

The pattern is clear. The industry adopted message-oriented, out-of-order, multipath transports in hardware, which is Homa's critique of TCP in practice. It has not adopted size-based scheduling. Ultra Ethernet's specification doesn't mention SRPT at all, and its receiver-driven mode descends from NDP, not Homa.¹⁶ One more link: Homa's first author, Behnam Montazeri, is a co-author of both Swift and Falcon.

What we'd take from it

The Model 4077's module backplane is not a datacenter, but it has the same shape of problem: a crowd of small control messages sharing links with bulk transfers, and a crew that only notices the slow ones. We are not adopting Homa wholesale. These are the lessons we'd take into any interconnect:

1. Schedule at the receiver. The receiver is the only node that sees everything competing for its downlink.
2. Make messages the unit. If the transport knows a message's length, it can schedule; a byte stream hides exactly the information it needs.
3. Spend priorities on size. Four levels capture most of the benefit, and two already rescue short messages, in our model and on Homa's implementations.
4. Guard the elephants. Reserve a few percent of capacity for the oldest message so large transfers can't starve.
5. Budget for buffers and neighbours. A clean design on a dedicated link can degrade badly when it shares the wire with a protocol that doesn't play by its rules.

Limits of this note

• The simulation models one bottleneck link with no propagation delay, no packets, no sender side and no incast. It isolates the effect of service order, and nothing else. Real protocols lose some of SRPT's benefit to imperfect knowledge, preemption lag and overhead.
• "Fair share" is the per-flow ideal TCP congestion control converges towards, not TCP. Real TCP also has slow start, head-of-line blocking within streams, loss recovery and FIFO switch queues. The TCP tails measured in [3] are much higher than our fair-share line, but they use a different baseline, so don't compare the numbers directly.
• Our 8-level policy uses log-spaced cutoffs by remaining size. The 2018 design derives unscheduled cutoffs from measured traffic; the implementations use precomputed or configured cutoffs. Scheduled messages are ranked by remaining bytes.^1,6
• Workloads W2 and W3 come from an unpublished Google internal memo, and their only public form is the Homa authors' data files. W4 and W5 were estimated from packet traces, which overestimates message sizes.¹
• The main results of the critiques cited here (Aeolus, dcPIM, SIRD) are simulations; Aeolus also has a small testbed running its own Homa prototype. The 2018 comparisons with pFabric, pHost, PIAS and NDP are simulations too. Homa's own simulator "assumes infinite buffer space", per the Homa wiki.

References

1. B. Montazeri, Y. Li, M. Alizadeh, J. Ousterhout. "Homa: A Receiver-Driven Low-Latency Transport Protocol Using Network Priorities." Proc. ACM SIGCOMM 2018, pp. 221–235. doi:10.1145/3230543.3230564. Complete version: arXiv:1803.09615 (section numbers here follow the arXiv version).
2. J. Ousterhout. "It's Time to Replace TCP in the Datacenter." arXiv:2210.00714, v2, January 2023.
3. J. Ousterhout. "A Linux Kernel Implementation of the Homa Transport Protocol." USENIX ATC 2021, pp. 773–787 (USENIX metadata lists 99–115). usenix.org/conference/atc21/presentation/ousterhout. Ratios quoted are from §5.2 of the body, which is more precise than the abstract.
4. N. Bansal, M. Harchol-Balter. "Analysis of SRPT Scheduling: Investigating Unfairness." Proc. ACM SIGMETRICS 2001, pp. 279–290. doi:10.1145/378420.378792.
5. L. Schrage. "Letter to the Editor—A Proof of the Optimality of the Shortest Remaining Processing Time Discipline." Operations Research 16(3):687–690, 1968. doi:10.1287/opre.16.3.687.
6. PlatformLab. HomaModule: README ("Significant changes") and protocol.md. github.com/PlatformLab/HomaModule, accessed 2026-10-03.
7. J. Ousterhout. "Begin upstreaming Homa transport protocol," netdev patch series, latest posting 2026-09-14. patchwork.kernel.org, accessed 2026-10-03.
8. J. Corbet. "The Homa network protocol." LWN.net, 30 December 2024. lwn.net/Articles/1003059.
9. IANA. Assigned Internet Protocol Numbers (146: Homa). iana.org/assignments/protocol-numbers.
10. PlatformLab. grpc_homa README. github.com/PlatformLab/grpc_homa, accessed 2026-10-03.
11. S. Hu, W. Bai, G. Zeng, Z. Wang, B. Qiao, K. Chen, K. Tan, Y. Wang. "Aeolus: A Building Block for Proactive Transport in Datacenters." Proc. ACM SIGCOMM 2020, pp. 422–434. doi:10.1145/3387514.3405878.
12. K. Prasopoulos, R. Kosta, E. Bugnion, M. Kogias. "SIRD: A Sender-Informed, Receiver-Driven Datacenter Transport Protocol." USENIX NSDI 2025. usenix.org/conference/nsdi25/presentation/prasopoulos.
13. J. Ousterhout. Homa Wiki: "Concerns Raised About Homa", and the responses to Aeolus, dcPIM, PowerTCP and SIRD. homa-transport.atlassian.net/wiki/spaces/HOMA, accessed 2026-10-03.
14. J. Ousterhout. "Can Homa and TCP Get Along?" Netdev 0x1A, July 2026. netdevconf.info/0x1A.
15. A. Singhvi, N. Dukkipati, et al. "Falcon: A Reliable, Low Latency Hardware Transport." Proc. ACM SIGCOMM 2025, pp. 248–263. doi:10.1145/3718958.3754353.
16. Ultra Ethernet Consortium. UE Specification v1.0.2, 28 January 2026 (v1.0 released 11 June 2025). ultraethernet.org.
17. L. Shalev, H. Ayoub, N. Bshara, E. Sabbag. "A Cloud-Optimized Transport Protocol for Elastic and Scalable HPC." IEEE Micro 40(6):67–73, 2020. doi:10.1109/MM.2020.3016891.
18. M. Handley, C. Raiciu, A. Agache, A. Voinescu, A. W. Moore, G. Antichi, M. Wójcik. "Re-architecting Datacenter Networks and Stacks for Low Latency and High Performance." Proc. ACM SIGCOMM 2017, pp. 29–42. doi:10.1145/3098822.3098825.
19. M. Alizadeh, S. Yang, M. Sharif, S. Katti, N. McKeown, B. Prabhakar, S. Shenker. "pFabric: Minimal Near-Optimal Datacenter Transport." Proc. ACM SIGCOMM 2013, pp. 435–446. doi:10.1145/2486001.2486031.
20. I. Pepelnjak. "Is It Time to Replace TCP in Data Centers?" ipSpace.net, 10 January 2023. blog.ipspace.net. A sceptical practitioner's reading of [2], with a reply on the Homa Wiki [13].
21. G. Kumar, N. Dukkipati, K. Jang, et al. "Swift: Delay is Simple and Effective for Congestion Control in the Datacenter." Proc. ACM SIGCOMM 2020, pp. 514–528. doi:10.1145/3387514.3406591.
22. Homa workload tables W1–W5, util/dist.cc, HomaModule commit 40891f0 (BSD-2-Clause or GPL-2.0+).
23. X. Lu, Z. Zhang. "Leveraging Homa: Enhancing Datacenter RPC Transport Protocols." Netdev 0x17, 2023. netdevconf.info/0x17.

Reproduce

Everything in Figures 1 and 4–7 is generated from the code below. You need Python 3.10+ and NumPy; the full run takes about a minute and a half.

python code/ys-en-0001/extract_workloads.py   # fetch W1–W5 from HomaModule (pinned commit)
python code/ys-en-0001/queue_sim.py           # run the simulations → results.json
python code/ys-en-0001/build_figures.py       # rebuild the figures in this page

• queue_sim.py: the four service policies, exact event-driven simulation
• extract_workloads.py and the size distributions W2, W3, W4, W5 (and W1)
• results.json: every number in the figures
• build_figures.py and svgchart.py: figure generation
• LICENSE: the scripts are MIT-licensed. Use them freely, keep the notice; they come with no warranty

Changelog

• 2026-04-05Rev 1. First publication.
• 2026-10-03Rev 2. Upstreaming status (§8) brought up to the September 2026 posting and the July 2026 Netdev talk; references re-checked.