This is an outstanding blog post. Initially, the title did little to captivate me, but the blog post was so well written that I got nerd-sniped. Who knew this little adapter was so fascinating! I wonder if the manufacturer is buying the Mellanox cards used from data center tear-downs. The author claims they can be had for only 20 USD online. That seems too good to be true!
I cannot find anything for less than 285 USD. The blog post gave a price of 174 USD. I have no reason to disbelieve the author, but a bummer to see the current price is 110 USD more!
> The author claims they can be had for only 20 USD online. That seems too good to be true!
In my experience, the cheap eBay MLX cards are DellEMC/HPE/etc OEM cards. However I also encountered zero problems cross-flashing those cards back to generic Mellanox firmware. I'm running several of those cross-flashed CX-4 Lx cards going on six or seven years now and they've been totally bulletproof.
I believe the author is talking about the OCP (2.0) network card itself, that these adapters internally. The OCP nics are quite cheap compared to pcie - here’s 100GBE for 100!
https://ebay.us/m/HMQAph
Ha! Been running these for years on both linux and windows (on lenovo x1 laptops). Using cheap chinese thunderbolt-to-nvme adapters + nvme-to-pcie boards + mellanox cx4 cards (recently got one cx5 and a solarflare x2).
If you don’t mind me asking, what are you using these for? Saturating these seems like it would have reasonably few workloads outside of like cdn or multi-tenant scenarios. Curious what my lack of imagination is hiding here.
Ha. I got one of the 10G Thunderbolt adapters a several years ago. And eventually started having problems with Zoom calls around noon. With dropped connections and stuttering. Zoom restarts usually fixed the problem.
After it happened 3-4 times, I started debugging. It turned out that we usually get at least a bit of sunlight around noon, as it burns away the morning clouds. And my Thunderbolt box was in direct sunlight, and eventually started overheating.
And a Zoom restart made it fall back onto the Wifi connection instead of wired.
I fixed that by adding a small USB-powered fan to the Thunderbolt box as a temporary workaround. I just realized that it's been like this for the last 3 years: https://pics.ealex.net/s/overheat
Thunderbolt is basically external PCIe, so this is not so surprising. High speed NICs do consume a relatively large amount of power. I have a feeling I've seen that logo on the board before.
I don't know how to measure the direct power impact on a MacBook Pro (since it's got a battery), but the typical power consumption of these cards is 9 W, not much more than Aquantia 10 GBit cards.
Also, if you remember where you saw that logo, please let me know!
I'm surprised you are only getting 20gbit/s. I did not expect PCIe to be be the limiting factor here. I've got a 100gbit cx4 card currently in a PCIe3 X4 slot (for reasons, don't judge) and it easily maxes that out. I would have expected the 25g cx4 cards to be at least able to get everything out of it. RDMA is required to achieve that in a useful way though.
Yeah, it's because the network card adapter's heatsink is sandwiched between two PCBs. Not great, not terrible, works for me.
The placement is mostly determined by the design of the OCP 2.0 connector. OCP 3.0 has a connector at the short edge of the card, which allows exposing/extending the heat sink directly to the outer case.
If somebody has the talent, designing a Thunderbolt 5 adapter for OCP 3.0 cards could be a worthwhile project.
Any idea why ethernet stagnated in terms of speed? There was a time it was so much faster compared to usb. Now even wifi seems to be faster.
Sure one can buy nice ethernet cards and cables, but the reality is that if you grab a random laptop/desktop from best buy and a cable, you are looking at best at a 2.5Gb/s speed.
The new low-power Realtek chipsets will definitely push 10 GbE forward because the chipset won't be much more expensive to integrate and run than the 2.5Gbps packages.
It all comes down to performance per Watt, the availability of cheap switching gear, and the actual utility in an office / home environment.
For 10 Gbps, cabling can be an issue. Existing "RJ45"-style Cat 6 cables could still work, but maybe not all of them.
Higher speeds will most likely demand a switch to fiber (for anything longer than a few meters) or Twinax DAC (for inter-device connects). Since Wifi already provides higher speeds, one may be inclined to upgrade just for that (because at some point, Wireless becomes Wired, too).
That change comes with the complexity of running new cabling, fiber splicing, worrying about different connectors (SFP+, SFP28, SFP56, QSFP28, ...), incompatible transceiver certifications, vendor lock-in, etc. Not a problem in the datacenter, but try to explain this to a layman.
Lastly, without a faster pipe to the Internet, what can you do other than NAS and AI? The computers will still get faster chips but most folks won't be able to make use of the bandwidth because they're still stuck on 1Gbps Internet or less.
But that will change. Swiss Init7 has shown that 25GBps Internet at home is not only feasible but also affordable, and China seems to be adding lots of 10G, and fiber in general.
Ethernet did not stagnate. Ethernet on UTP did stagnate due to reaching the limits of the technology, but Ethernet continues to advance over fiber.
For 10 Gbps I find it simpler and cheaper to use fiber or DACs, but motherboards don't provide SFP+, only RJ45 ports. Over 10 Gbps copper is a no go. SFP28 and above would be nice to have on motherboards, but that's a dream with almost zero chances to happen. For most people RJ45 + WiFi 7 is good enough, computer manufacturers will not put SFP+ or SFP28 for a small minority of people.
> Any idea why ethernet stagnated in terms of speed? There was a time it was so much faster compared to usb. Now even wifi seems to be faster.
Practically spoken, a lot of the transfer speed advertised by wifi is marketing hogwash barely backed by reality, especially in congested environments.
> Sure one can buy nice ethernet cards and cables, but the reality is that if you grab a random laptop/desktop from best buy and a cable, you are looking at best at a 2.5Gb/s speed.
For both laptops and desktops, PCI lanes. Intel doesn't provide many lanes, so manufacturers don't want to waste valuable lanes permanently for capabilities most people don't ever need.
For laptops in particular, power draw. The faster you push copper, the more power you need. And laptops have even less PCIe lanes available to waste.
For desktops, it's a question of market demand. Again - most applications don't need ultra high transfer rate, most household connectivity is DSL and (G)PON so 1 GBit/s is enough to max out the uplink. And those few users that do need higher transfer rates can always install a PCIe card, especially as there is a multitude of different options to provide high bandwidth connectivity.
> Any idea why ethernet stagnated in terms of speed? There was a time it was so much faster compared to usb. Now even wifi seems to be faster.
wifi is not faster.
However ethernet is not as critical as it used to be, even at the office. People like the conveniency of having laptops they can move around. Unless you are working from home, having a dedicated office space is now seen as a waste of space. If the speed of the wifi is good enough when you are in a meeting room or in your kitchen, there is no reason to plug your laptop when you move back in another place, especially if most connections are to the internet and not the local network. In the workplace, most NAS have been replaced by onedrive / gdrive, at home NAS use has always been limited to a niche population: nerds/techies, photographers, music or video producers...
We have 400Gbe which is certainly faster than USB.. but;
On consumer devices, I think part of the issue is that we’re still wedded to four-pair twisted copper as the physical medium. That worked well for Gigabit Ethernet, but once you push to 5 or 10 Gb/s it becomes inherently expensive. Twisted pair is simply a poor medium at those data rates, so you end up needing a large amount of complex silicon to compensate for attenuation, crosstalk, and noise.
That's doable but the double whammy is that most people use the network for 'internet' and 1G is simply more than enough, 10G therefore becomes quite niche so there's no enormous volume to overcome the inherent issues at low cost.
Wireless happened, I'd think. People started using wifi and cellular data for everything, so applications had to adapt to this lowest common denominator, and consumer broadband demand for faster-than-wifi speeds isn't there. Plus operators put all their money into cellular infra leaving no money to update broadband infra.
I've had a lot of problems with even 10GbE via Thunderbolt 3/4. Bandwidth-wise it works fine, but latency and jitter are issues. This means that stuff like high-speed cameras that need to be synchronized over Ethernet using Precision Time Protocol (PTP) tend to simply fail with these devices.
I’d heard similar complaints re: TB networking latency & jitter. Did some investigations and tuning on a pair of machines with USB4 ports connected via short TB5-rated cables. Eventually got the thunderbolt links to consistently beat the ether ones on both latency and jitter. And not just switched Ethernet either - even a direct Ethernet P2P link lost out to TB, though the difference there was small.
I used to have an SFP28 Mellanox card in my home server, but went back to a simple 2.5G Ethernet port for the LAN side. The Mellanox card ran hot and needed an extra fan near it to dissipate the heat. It was cool but there was no real benefit other than occasionally when transferring some large files.
Until motherboards include SFP ports it's probably not worth the effort at all in home setting; external adaptors like the one presented here are unreliable and add several ms of latency.
Does this manufacturer's practice pattern of repackaging data center components (e.g. Mellanox) imply any up and coming product creation opportunities?
> the biggest downside of the PX adapter is that it gets really hot, like not touchable hot. Sometimes, either the network connection silently disappeared or (sadly) my Mac crashed with a kernel panic in the network driver. Apple has assured me that this was not a security issue. Other than that, the PX seems to do the job.
That is really cool to read. And here I am, still running my home network on a measly 1Gbit Ethernet. I considered upgrading, but the equipment power consumption even when idle makes it an expensive proposition to consider just for fun.
Please forgive me for my ignorance, but are there currently any ways of being able to write data down at that speed? I see 2026 PCIe 5.0 NVMe advertising theoretical 14gb/s but not sure how feasible even that is.
PCIe 5 drives max out at about 14 GBps (bytes), which is ~112 gbps (bits).
Displayport 2.1 UHBR20 is 80 gbps.
USB4 maxes out at 80 gbps.
As you can see, 1gbps ethernet is starting to look like stone age technology. 2.5gbps becoming the next step seems a bit strange when we were jumping orders of magnitude every few years before. But also, ethernet tends to be used on longer cables than DP or USB, and trying to push it much faster results in exponentially increasing losses to resistance and radiation, the cable starts acting like an antenna even with the twisted pairs. Fiber optics are much better suited to high speed long distance, but too expensive and fragile for consumer use.
Muscle memory for folks who have been doing it since before -i was an option. I still instinctively type `sudo su -` because it worked consistently on older deployments. When you have to operate a fleet of varying ages and distributions, you tend to quickly learn [if only out of frustration] what works everywhere vs only on the newer stuff.
`sudo su - <user>` also seems easier for me to type than `sudo -i -u <user>`
Thanks! Have you tried the boltctl/rescan setup I mentioned in the post? It should get you going, as long as your Thunderbolt/USB4 setup is correct.
If you're using an adapter card to add Thunderbolt functionality, then your mainboard needs to support that, and the card must be connected to a PCIe bus that's wired to the Intel PCH, not to the CPU.
> All other 25 GbE adapter solutions I’ve found so far ... have a spinning fan. ... the biggest downside of the PX adapter is that it gets really hot, like not touchable hot. Sometimes, either the network connection silently disappeared or (sadly) my Mac crashed with a kernel panic in the network driver. ... Other than that, the PX seems to do the job
[+] [-] throwaway2037|1 month ago|reply
Small thing: I just checked Amazon.com: https://www.amazon.com/s?k=thunderbolt+25G&crid=2RHL4ZJL96Z9...
I cannot find anything for less than 285 USD. The blog post gave a price of 174 USD. I have no reason to disbelieve the author, but a bummer to see the current price is 110 USD more!
[+] [-] kohlschuetter|1 month ago|reply
I think, tragically, the blog post has caused this price increase.
The offers on Amazon are most likely all drop shippers trying to gauge a price that works for them.
You might have better luck ordering directly from China for a fraction of the price: https://detail.1688.com/offer/836680468489.html
[+] [-] Lammy|1 month ago|reply
In my experience, the cheap eBay MLX cards are DellEMC/HPE/etc OEM cards. However I also encountered zero problems cross-flashing those cards back to generic Mellanox firmware. I'm running several of those cross-flashed CX-4 Lx cards going on six or seven years now and they've been totally bulletproof.
[+] [-] geerlingguy|1 month ago|reply
I'm going to try a couple other fan assisted cooling options, as I'd like to keep the setup reasonably compact.
I just ran fiber to my desk and I have a more expensive QNAP unit that does 10G SFP+, but this will let me max out the connection to my NAS.
[+] [-] buildbot|1 month ago|reply
[+] [-] nunez|1 month ago|reply
[+] [-] hardwaresofton|1 month ago|reply
[+] [-] MisterTea|1 month ago|reply
[+] [-] unknown|1 month ago|reply
[deleted]
[+] [-] omgtehlion|1 month ago|reply
Pic of a previous cx3 (10 gig on tb3) setup: https://habrastorage.org/r/w780/getpro/habr/upload_files/d3c...
10gig can saturate full speed, 25G in my experience rarely reaches same 20G as the author observed.
[+] [-] kasey_junk|1 month ago|reply
[+] [-] zokier|1 month ago|reply
https://support.apple.com/guide/mac-help/ip-thunderbolt-conn... etc
[+] [-] kohlschuetter|1 month ago|reply
You'd also mostly be limited to short cables (1-2m) and a ring topology.
[+] [-] cyberax|1 month ago|reply
After it happened 3-4 times, I started debugging. It turned out that we usually get at least a bit of sunlight around noon, as it burns away the morning clouds. And my Thunderbolt box was in direct sunlight, and eventually started overheating.
And a Zoom restart made it fall back onto the Wifi connection instead of wired.
I fixed that by adding a small USB-powered fan to the Thunderbolt box as a temporary workaround. I just realized that it's been like this for the last 3 years: https://pics.ealex.net/s/overheat
[+] [-] jamiek88|1 month ago|reply
[+] [-] userbinator|1 month ago|reply
[+] [-] kohlschuetter|1 month ago|reply
Also, if you remember where you saw that logo, please let me know!
[+] [-] consp|1 month ago|reply
Edit: forgot is isn't "true" PCIe but tunneled.
[+] [-] kohlschuetter|1 month ago|reply
[+] [-] Nextgrid|1 month ago|reply
[+] [-] kohlschuetter|1 month ago|reply
The placement is mostly determined by the design of the OCP 2.0 connector. OCP 3.0 has a connector at the short edge of the card, which allows exposing/extending the heat sink directly to the outer case.
If somebody has the talent, designing a Thunderbolt 5 adapter for OCP 3.0 cards could be a worthwhile project.
[+] [-] whatever1|1 month ago|reply
Sure one can buy nice ethernet cards and cables, but the reality is that if you grab a random laptop/desktop from best buy and a cable, you are looking at best at a 2.5Gb/s speed.
[+] [-] kohlschuetter|1 month ago|reply
It all comes down to performance per Watt, the availability of cheap switching gear, and the actual utility in an office / home environment.
For 10 Gbps, cabling can be an issue. Existing "RJ45"-style Cat 6 cables could still work, but maybe not all of them.
Higher speeds will most likely demand a switch to fiber (for anything longer than a few meters) or Twinax DAC (for inter-device connects). Since Wifi already provides higher speeds, one may be inclined to upgrade just for that (because at some point, Wireless becomes Wired, too).
That change comes with the complexity of running new cabling, fiber splicing, worrying about different connectors (SFP+, SFP28, SFP56, QSFP28, ...), incompatible transceiver certifications, vendor lock-in, etc. Not a problem in the datacenter, but try to explain this to a layman.
Lastly, without a faster pipe to the Internet, what can you do other than NAS and AI? The computers will still get faster chips but most folks won't be able to make use of the bandwidth because they're still stuck on 1Gbps Internet or less.
But that will change. Swiss Init7 has shown that 25GBps Internet at home is not only feasible but also affordable, and China seems to be adding lots of 10G, and fiber in general.
Fun times ahead.
[+] [-] AdrianB1|1 month ago|reply
For 10 Gbps I find it simpler and cheaper to use fiber or DACs, but motherboards don't provide SFP+, only RJ45 ports. Over 10 Gbps copper is a no go. SFP28 and above would be nice to have on motherboards, but that's a dream with almost zero chances to happen. For most people RJ45 + WiFi 7 is good enough, computer manufacturers will not put SFP+ or SFP28 for a small minority of people.
[+] [-] mschuster91|1 month ago|reply
Practically spoken, a lot of the transfer speed advertised by wifi is marketing hogwash barely backed by reality, especially in congested environments.
> Sure one can buy nice ethernet cards and cables, but the reality is that if you grab a random laptop/desktop from best buy and a cable, you are looking at best at a 2.5Gb/s speed.
For both laptops and desktops, PCI lanes. Intel doesn't provide many lanes, so manufacturers don't want to waste valuable lanes permanently for capabilities most people don't ever need.
For laptops in particular, power draw. The faster you push copper, the more power you need. And laptops have even less PCIe lanes available to waste.
For desktops, it's a question of market demand. Again - most applications don't need ultra high transfer rate, most household connectivity is DSL and (G)PON so 1 GBit/s is enough to max out the uplink. And those few users that do need higher transfer rates can always install a PCIe card, especially as there is a multitude of different options to provide high bandwidth connectivity.
[+] [-] undersuit|1 month ago|reply
Servers had a reason to spend for the 10G, 25G and 40G cards which used 4 lanes.
There are 10 Gigabit chips that can run off of one PCI-E 4.0 lane now and the 2.5G and 5G speeds are supported(802.3bz).
[+] [-] prmoustache|1 month ago|reply
wifi is not faster.
However ethernet is not as critical as it used to be, even at the office. People like the conveniency of having laptops they can move around. Unless you are working from home, having a dedicated office space is now seen as a waste of space. If the speed of the wifi is good enough when you are in a meeting room or in your kitchen, there is no reason to plug your laptop when you move back in another place, especially if most connections are to the internet and not the local network. In the workplace, most NAS have been replaced by onedrive / gdrive, at home NAS use has always been limited to a niche population: nerds/techies, photographers, music or video producers...
[+] [-] matt-p|1 month ago|reply
On consumer devices, I think part of the issue is that we’re still wedded to four-pair twisted copper as the physical medium. That worked well for Gigabit Ethernet, but once you push to 5 or 10 Gb/s it becomes inherently expensive. Twisted pair is simply a poor medium at those data rates, so you end up needing a large amount of complex silicon to compensate for attenuation, crosstalk, and noise.
That's doable but the double whammy is that most people use the network for 'internet' and 1G is simply more than enough, 10G therefore becomes quite niche so there's no enormous volume to overcome the inherent issues at low cost.
[+] [-] fulafel|1 month ago|reply
[+] [-] wmf|1 month ago|reply
[+] [-] icelancer|1 month ago|reply
[+] [-] pdrayton|1 month ago|reply
[+] [-] drnick1|1 month ago|reply
Until motherboards include SFP ports it's probably not worth the effort at all in home setting; external adaptors like the one presented here are unreliable and add several ms of latency.
[+] [-] thadk|1 month ago|reply
[+] [-] the_real_cher|1 month ago|reply
Made me chuckle.
[+] [-] shantara|1 month ago|reply
[+] [-] bob_theslob646|1 month ago|reply
[+] [-] hnuser123456|1 month ago|reply
Displayport 2.1 UHBR20 is 80 gbps.
USB4 maxes out at 80 gbps.
As you can see, 1gbps ethernet is starting to look like stone age technology. 2.5gbps becoming the next step seems a bit strange when we were jumping orders of magnitude every few years before. But also, ethernet tends to be used on longer cables than DP or USB, and trying to push it much faster results in exponentially increasing losses to resistance and radiation, the cable starts acting like an antenna even with the twisted pairs. Fiber optics are much better suited to high speed long distance, but too expensive and fragile for consumer use.
[+] [-] kalleboo|1 month ago|reply
[+] [-] project2501a|1 month ago|reply
[+] [-] flounder3|1 month ago|reply
`sudo su - <user>` also seems easier for me to type than `sudo -i -u <user>`
[+] [-] Bjartr|1 month ago|reply
[+] [-] madduci|1 month ago|reply
But this is a cool solution
[+] [-] kohlschuetter|1 month ago|reply
If you're using an adapter card to add Thunderbolt functionality, then your mainboard needs to support that, and the card must be connected to a PCIe bus that's wired to the Intel PCH, not to the CPU.
[+] [-] mrbluecoat|1 month ago|reply
> All other 25 GbE adapter solutions I’ve found so far ... have a spinning fan. ... the biggest downside of the PX adapter is that it gets really hot, like not touchable hot. Sometimes, either the network connection silently disappeared or (sadly) my Mac crashed with a kernel panic in the network driver. ... Other than that, the PX seems to do the job
[+] [-] cs02rm0|1 month ago|reply
[+] [-] otterpro|1 month ago|reply
I had to do a double-take when it mentioned Kelvin since That is physically impossible.
[+] [-] maratc|1 month ago|reply
[+] [-] sigio|1 month ago|reply
It 'reduces it by' ... not reduces it TO