[nodja]

> Reportedly the inter-layer transfer sizes are in kilobyte ranges and PCIe x1 is plenty or something.

Not an expert, but napkin math tells me that more often that not this will be in the order of megabytes—not kilobytes—since it scales with sequence length.

Example: Qwen3 30B has a hidden state size of 5120, even if quantized to 8 bits that's 5120 bytes per token. It would pass the MB boundary with just a little over 200 tokens. Still not much of an issue when a single PCIe lane is ~2GB/s.

I think device to device latency is more of an issue here, but I don't know enough to assert that with confidence.

[remexre]

For each token generated, you only send one token’s worth between layers; the previous tokens are in the KV cache.