[ftufek]

Yeah, it's really just trying to get as much compute as possible as cheaply as possible interconnected in a reasonably fast way with low latency. Slow networking would be a bottleneck and expensive high end networking would defeat the purpose of staying cheap.

[buildbot]

You’d be surprised at how cheap high end networking that outperforms PCIE4 x4 is - 100Gb omni-path nics are running for 20$ on ebay! And those will saturate PCIE3 x16.

Though of course with multiple boards/ram/cpu it gets complicated again.

[magicalhippo]

Which cards? Been looking at nics but couldn't find cheap ones past 25-40Gb

[jacquesm]

Very smart approach. I may copy your setup for some project that I've been working on for years but that stalled waiting for more memory in GPUs.