[benou]

The approach is interesting but I'm afraid that the benchmarking is bogus, as it's usually the case is in this kind of papers.

Part of the problem I think is that they have to compare to other existing solutions (click, VPP etc) otherwise everybody will ask, but at the same time it is very difficult to do a fair comparison.

I'm a VPP developer hence I'm both biased and a knowledge-domain expert, but focusing on what I know, which is VPP: figure 11b, they compare VPP to PacketMill for some L2 patch workload. They claim their approach is fare because they use automated tooling to benchmark, good and we also do for VPP, but our results don't necessarily match theirs - and, surprisingly, our results are higher than what they claim for VPP.

PacketMill paper figure 11b for VPP for L2 patch for 64-bytes packets at 1.2GHz using DPDK MLX5 driver [1]: ~5Gbps

VPP for similar configuration [2]: ~7.5Gbps (already a 50% error margin?)

VPP using a more optimized NIC driver (native AVF vs DPDK MLX5) [3]: ~18Gbps (almost 2x what they seem to claim for PacketMill...)

All this to say comparing different solutions is terribly hard, and I'm not sure of the value of this kind of benchmarks.

For VPP we built CSIT [4] which is opensource under the Linux Foundation Networking to automate tests in the same environments, to compare between VPP and DPDK between releases and platforms.

[1] https://packetmill.io/docs/packetmill-asplos21.pdf

[2] http://csit.fd.io/report/#eNp1kd0OwiAMhZ8Gb0zN6MRdeaHuPQxidS...

[3] http://csit.fd.io/report/#eNp1kd0OgjAMhZ9m3pgaVkS88ULlPcwcVU...

[4] http://csit.fd.io/

[benou]

Also, the claims they make in the paper is inaccurate relative to VPP: they claim VPP uses "overlay/convert" method, which is true for DPDK drivers, but we also maintain native drivers for NIC we care about (eg. Intel's) - especially because going through the "overlay/convert" method is costly at high-packet rate.

IOW, one of their strong claim is that PacketMill is innovative because it avoids copying/converting uneeded metadata, but VPP is already doing that since years.

Finally, their claim to break the 100Gbps on single core @2.3GHz is cute, but again I'm afraid they're late to the party. They claim 12-13Mpps per core for 64-bytes packets for example but VPP can achieve 20+Mpps per core already for L3 forwarding (routing).

Again, benchmarking is hard, but I keep reading there claims over and over in academic papers when they're factually wrong for area I know about. I can only imagine what is happening for area I don't know :(