25 years ago, I worked for a small ISP (back when there was such a thing.) When I started there, we had one upstream ISP. I was charged with getting us multihomed. I found some tutorials written by Avi Freedman (1). I don't know what I would have done without him. He made an intimidating topic approachable. Thanks to him, I got us a /20 from ARIN and advertised our routes to two different peers. It was fascinating to learn how it all worked. And the more I learned about it, the more amazed I was that it worked at all.
Like the human body the more you study the Internet the more amazing it is not that it sometimes breaks, but that it works at all. Especially for video/phone/etc.
Glad the content was helpful, I have links to some of them at avi.net (tutorials and old Boardwatch articles).
I swear my motive was pure (frustration with the content out there) but it was easy to see back then that helping people out with good content yields rewards ("Can I buy a T1?") or ("Come run my big global network"). So I still encourage everyone to write about what's confusing and frustrating...
That flapping from EpicUp 140.99.244.0/23 prefix should have been subject to route dampening. This is per peer or per prefix rate limiting typically enforced on all peers by ISPs to prevent this exact issue of a single prefix making up a significant portion of the global BGP churn.
I’m unconvinced of the correlation between the updates that the author attributed to knock on effects. It would be pretty janky to have your advertisements be based on the path to other autonomous systems’ prefixes, especially unstable ones.
I don’t think there is a 40 minute periodicity either (at least there wasn’t 8 years ago when I was deep in the BGP world). Smells like what this dataset happened to show either by luck or because of the network the author was getting the BGP feed from.
If you dig into the data and look at which AS’s and prefixes are experiencing changes, you’ll find it’s all over the place and there isn’t really any bigger pattern.
On any given day there are usually a few noisy ISPs because of bad circuits or misconfigurations. Then there are new prefixes flapping in and out as a new thing is brought online for the first time, etc. Then sprinkle in path changes for regular draining maintenance, etc.
It’s simultaneously both fascinating and a little horrifying how a little ISP in Kansas experiencing a fiber consuming backhoe shows up on routers in Perth. Yet the frequency of updates is kept to <10hz globally through tons of hand tuned policies.
Route dampening has mostly fallen out of fashion with networks these days.
Most setups were horribly misconfigured and (most) routers are no longer extremely CPU starved as they once were, That doesn't mean that it does not still exist of course, when I did bgp battleships ( https://blog.benjojo.co.uk/post/bgp-battleships ) I found that 3356 (at the time) was doing route dampening, so play had to be paused for a while.
If anyone wants to learn about BGP (especially day-to-day stuff for peering scenarios), the Network Startup Resource Center out of U.Oregon has a good series of videos going through things:
Based on some quick sleuthing, I would assume that the 0xff reserved BGP attribute is likely a huawei quirk. Almost all of the 0xff's visible to bgp.tools (hi) follow the same format as the one in the post, and some of those networks with them seem to be running huawei kit.
I once designed and configured a satellite-microwave hybrid network for a large US customer with field offices around Borneo. I’ll never forget making the leased line handoffs back in Jakarta. I had exactly zero experience doing this so I googled around and read that BGP is what you would use to connect between our OSPF/UBNT net and their IGRP/Cisco corporate WAN. When asked to configure BGP on their routers, the guys from Tata were like “what do you think you are AT&T or something?” We did kind of feel that way until one season of lightning strikes took out most of our AirFibers.
I wrote a python script to extract data from an MRT file containing BGP routes from [1] and importing it into Neo4j for exploration. This file contained about 56 million extremely redundant routes and Neo4j is great for "merging" this type of data.
I have had a project on the back burner for about a year now to offer a BGP feed via a websocket to facilitate people playing around and doing research without allowing them to accidently spew crap into the DFZ. Shoot me an email if you are interested and I'll try to get it spun up this week.
It would be nice for the Memory Safety initiative[1] which tackles core internet infrastructure security and safety via re-implementing it in Rust to also work on implementing the BGP server implementation.
chriscjcj|1 year ago
(1) http://avi.freedman.net/
Avi
avifreedman|1 year ago
Like the human body the more you study the Internet the more amazing it is not that it sometimes breaks, but that it works at all. Especially for video/phone/etc.
Glad the content was helpful, I have links to some of them at avi.net (tutorials and old Boardwatch articles).
I swear my motive was pure (frustration with the content out there) but it was easy to see back then that helping people out with good content yields rewards ("Can I buy a T1?") or ("Come run my big global network"). So I still encourage everyone to write about what's confusing and frustrating...
nickstinemates|1 year ago
aspenmayer|1 year ago
Somewhat off-topic, I’ve noticed on HN that sometimes comments end with single word on a newline, like you have done here.
Did your comment get truncated somehow, or did you mess up a copy/paste, or what?
Has anyone else noticed this? Is this a telltale sign of generated comments, or other tool usage?
I’ve only noticed this on HN, and maybe Reddit once? It happens somewhat more regularly than I can attribute to mere coincidence or accident.
kortilla|1 year ago
That flapping from EpicUp 140.99.244.0/23 prefix should have been subject to route dampening. This is per peer or per prefix rate limiting typically enforced on all peers by ISPs to prevent this exact issue of a single prefix making up a significant portion of the global BGP churn.
I’m unconvinced of the correlation between the updates that the author attributed to knock on effects. It would be pretty janky to have your advertisements be based on the path to other autonomous systems’ prefixes, especially unstable ones.
I don’t think there is a 40 minute periodicity either (at least there wasn’t 8 years ago when I was deep in the BGP world). Smells like what this dataset happened to show either by luck or because of the network the author was getting the BGP feed from.
If you dig into the data and look at which AS’s and prefixes are experiencing changes, you’ll find it’s all over the place and there isn’t really any bigger pattern.
On any given day there are usually a few noisy ISPs because of bad circuits or misconfigurations. Then there are new prefixes flapping in and out as a new thing is brought online for the first time, etc. Then sprinkle in path changes for regular draining maintenance, etc.
It’s simultaneously both fascinating and a little horrifying how a little ISP in Kansas experiencing a fiber consuming backhoe shows up on routers in Perth. Yet the frequency of updates is kept to <10hz globally through tons of hand tuned policies.
benjojo12|1 year ago
Most setups were horribly misconfigured and (most) routers are no longer extremely CPU starved as they once were, That doesn't mean that it does not still exist of course, when I did bgp battleships ( https://blog.benjojo.co.uk/post/bgp-battleships ) I found that 3356 (at the time) was doing route dampening, so play had to be paused for a while.
throw0101a|1 year ago
* https://learn.nsrc.org/bgp
benjojo12|1 year ago
CJefferson|1 year ago
I’d definitely be interested to read some follow-ups, diving into more details.
clbrmbr|1 year ago
UltraSane|1 year ago
[1] https://data.ris.ripe.net/rrc00/
zokier|1 year ago
throw0101a|1 year ago
* https://www.ripe.net/analyse/internet-measurements/routing-i...
* https://lukasz.bromirski.net/post/bgp-w-labie-3/
UltraSane|1 year ago
mike_d|1 year ago
motohagiography|1 year ago
does this imply the internet has something akin to tides ?
xvilka|1 year ago
[1] https://www.memorysafety.org/
3D42233565|1 year ago
AS15626 GREEN FLOID LLC (russian propaganda) AS34224 Neterra Ltd. (bullet proof) AS44477 STARK INDUSTRIES SOLUTIONS LTD (bullet proof) AS207656 Epinatura LLC (bullet proof)
nhggfu|1 year ago