> While the SLA says 100%, don't expect perfection
When you have an SLA, understand what it is: a financial arrangement whereby you can request a prorated refund for certain types of outages. It is not in any way a guarantee on the part of a provider that you'll experience even average uptime equaling or exceeding the SLA, just that they can pay out the fraction of customer requests for service credits they receive for the covered outages they have and still make money.
The reality for the type of service the author of this post purchased is that for any physical damage to the fiber plant, he will experience hours of outage while a splice crew locates and repairs the damage. Verizon might offer a 100% SLA, but they didn't engineer it to even five nines of availability. That would require redundant equipment and service entrances at his premises along with path diversity end-to-end.
It's still a very high motivator to keep the service up though. It's not a guarantee of anything as you said but I've been on call for this kind of contract.
And then you get very good at pointing fingers too. Not too sure about this though :D
The reality of business is that a contract is only worth what you can enforce. SLAs are usually worthless:
1. Unless you are a large customer who accounts for an important amount of their bottom line, probably you have little financial leverage with the vendor.
2. The amount at stake in the SLA is not worth going to court for. It's unenforceable and in fact, the amount is usually meaningless.
Let's say you pay $10,000 per month for your 100% SLA dedicated circuit, and it goes down for an entire month. Let's say the vendor doesn't get around to paying you. Is it worth hiring a lawyer to collect $10K? Is it worth distracting you from your job emotionally and mentally, and consuming many hours of your time? Probably not.
Let's say your circuit is down for 3 hours. Let's say the SLA even pays you 3x what you pay for the service for any downtime (most I've seen just refund the money for that time). Let's see: ($10,000/month) / (720 hrs/month) = $13.89/hour. The SLA pays $41.67/hr of downtime, or $125 for your downtime. Is it even worth figuring out how to apply and filling out the form? No. You have much bigger issues in business, and if you don't ... well, then you have bigger issues.
3. The cost to the vendor is reputation: You tell your peers how much your service sucks, and word gets around. I've had techs take disinterested attitudes toward their poor uptime on our circuit; when I've called the account managers, they can have a very different response - they want me spreading their name around in a different way. That has nothing to do with the SLA.
> Verizon might offer a 100% SLA, but they didn't engineer it to even five nines of availability. That would require redundant equipment and service entrances at his premises along with path diversity end-to-end.
Agreed. There is no substitute for the physical reality of the circuit and service, which you should understand if you are buying it. Putting a shiny SLA on it will have no effect on the outcome.
The ISP I own will give you dedicated if you desire. We charge an uplift for the installation (due to additional splicing) and a little bit more on the monthly price for the consumption of dedicated ports and cores rather than using the PON. However, as is the case with most provided services like this, the majority of the cost is covering the risk of the SLA. The likelihood of an outage is not too dissimilar to the likelihood of outage on PON, so really it becomes a financial and service guarantee more than an uptime guarantee. As a healthy middle ground, we will also offer BGP on regular services and we do a bit of a referral system with a couple of other friendly ISPs, who will also do BGP. I actually prefer multi homing to “dedicated” from a resilience perspective because it separates you from the entire network stack all the way to the transit and peering.
Perhaps you’ll be able to help. Why do ISPs, including yours, not provide a map of service? Almost every time it is this “check if you have service” API. Every time I have moved I have wanted to look for areas with support under various ISPs and each time it’s an annoying process of sending each address to this API, dealing with rate limits, anti-fraud etc.
One time I contacted the business contact with a bunch of addresses and that was useful but the round trip is so slow I was better off just querying.
(Potential non-commercial customer) Checked the help page - couldn’t see anything about static IP addresses, IPv6 etc?
Edit: Ah, “Our network. You acknowledge that We may change your Internet Protocol (IP) address from time to time without giving notice unless You have purchased a fixed IP address from us;”
I had "real fiber" run to our farm. 18 miles of cable was run and it took a year going back and forth. Originally it was a priced at $4500, which was to be a business expense. After install I had consistent issues with performance, after doing a lot of work to show them the issues and threatening disconnect, they upgraded the circuits for proper speeds and knocked it down to $2100. It's definitely still a car payment but it's much cheaper than it was.
Ultimately I get the SLA, Direct access to cloud providers maximizing performance, i'm also able to host a few IP blocks which allow a couple internet facing machines.
The home we sold recently had it pretty good too though, ended up with 3 5gig AT&T lines (no redundancy obviously) for only $450 a month total. That was pretty darn rad, even if the SLA wasn't the same.
Benefit of working from the farm is that I can also snag some bw for personal use ;D
In my experience it's just a case of knowing who to call and what to ask for. We are served by Spectrum. They will drop fiber into any building in their HFC service areas. We had it into an apartment building for several years then moved to another building with only coax, but they quoted me $2000 install for fiber. Problem for me is that the fiber service is symmetric and so to get the kind of download needed for Netflix you have to pay for 200M+ upload. That's quite expensive for me so I passed. They don't offer 95% billing.
Hey! That's my setup as well! I have one DIA connection and a backup VPN over a shitty Comcast business connection that gets terminated in a nearby datacenter.
Getting an ISP to even _talk_ to me required quite a bit of sleuthing. And I was saying from the outset that I was ready to fully pay for the fiber run.
Apparently, ISPs in my locality actually divide the city into the service areas. How the heck this is legal, I don't understand.
Some tidbits from me: my ISP installed a big honking ADVA optical line terminal on my premises. Getting them to move it to their side and just provide me with an SFP connection is still my work-in-progress.
The support is also outsourced into India, and getting them to understand what you want over the phone is... painful. Fortunately, the web ticket system is good enough.
Isn't it very typical for an ISP to run their line onto your premises and put the demarcation device on your premises with your power supply etc? Usually there isn't a "their side" where it still works. It's your premises, then up to kilometers of wiring, then the central exchange. Equipment required to terminate the wiring at your end obvious has to be at your end, not theirs.
It's possible that the termination equipment could be a bare SFP and not a big box, but the ISP wants to be able to monitor the status of your connection up to the termination equipment, because that is their responsibility to keep online, and they can't do that if it's just an off-the-shelf SFP. They probably wouldn't agree to do it and still have any SLA.
If there's a physical problem with the box (too big/loud) you can try negotiate for a different box but if you just want control over your network, sorry but that just isn't how it works. Your network starts at the demarcation, and you don't want to be responsible for speaking whatever protocols the ISP is using internally, either. Up to the demarcation, it's ISP internal network, and past that, it's your network, with a standard handover interface like Ethernet.
Regarding the city divided into service areas, if you are in the US many cities provided franchise rights to cable providers that gave them exclusive monopoly rights to provide services in exchange for the cable provider spending all the money to install the lines and infrastructure. Most of those franchise deals were done in the 1970s or early 1980s, essentially mimicking the agreements that were in place for AT&T (or RBOCs).
In my very limited experience, this is common even for real business customers. I think it's because going from commercial ISP service to BGP service is not really an upsell. It's a completely different product category (carrier interconnect) and usually results in less revenue for the ISP (greatly reduced bandwidth charges etc.). As a result, sales folks aren't trained on it, and it is difficult to get through regular channels.
If you get it, it can be great. Imagine your ISP calling you when you reboot your router.
This sounds like so much fun. Thank you to the author for writing this up and sharing the blow-by-blow.
While fascinated with the network stack, I've only gotten as deep as reading Illustrated TCP/IP and pretending I understand tcpdump. I would love to rovel around in BGP and, um, all that jazz.
Any suggestions on how to get started? My vague understanding is that most people get apprenticed into this stuff through work. Are the relevant systems involved just too expensive and locked behind corporate walls to be amenable to autodidactism?
Oddly, my recommendation for getting started is non-technical (and assumes you’re in the US): if you don’t have one already, stand up an LLC. Doesn’t need to be anything fancy; your state’s Secretary of State probably accepts an online form. I’m in CO and at its most basic form this costs $25 a year.
You’ll need some kind of business entity to have easier conversations with ARIN, which is the starting point for getting yourself an ASN.
Once you’ve got an ASN, you have an entity that can “own” IP blocks instead of just relying on other networks to handle that for you. Now, have a look at Neptune Networks’ offerings—they’ll rent you an instance for a reasonable monthly cost that they’ll allow transit to and from. Note that their smallest instance size doesn’t have enough RAM to store the global internet’s full BGP table; this will matter only once you know what that means.
That’ll definitely get you started, and you can learn a lot on the cheap before even looking into your local colocation options.
You're right that a lot of this stuff is learned on the job, but you can get really, really far with the right tools and the right resources and only a modicum of financial investment (if you're set on getting some hardware). Even more so now that things have open source versions. It makes it super easy to start out. That's one of the great things about this field, it's all built on more-or-less open standards. You're not far from the original RFCs used to create all of this: RFC 791 (IP), 793 (TCP), 4271 (BGP4), 9499 (overview of DNS), and many others. They can be dry, almost too much, but for the most part that's the gold right 'thar.
Stevens' book is also a stupendous resource for the down-and-dirty, so good work on starting there. Beyond that you need to start just building things.
Almost every virtualization suite allows you to create network resources (or at least it abstracts the low-level OS calls or commands required to do so). Set up two VMs. Make them talk. Break that link and learn how to repair it, using the tools that you've mentioned you are now using, tcpdump in particular. Figure out how ARP works at a low level, or NDP (neighbor discovery) if you're running IPv6. Learn how to subnet, too! Then work your way up the stack. Set up a VLAN interface, set a 802.1q tag on an interface, try to get two or more vlans to talk to each other, route between them. Break that. Set up a basic OSPF area. Set up a BGP adjacency between two private ASNs you have created. Redistribute routes among different protocols. Set up higher-level services like DNS. Set up a play anycast network on your local host. Play around with load balancers and web servers. Play, break, fix, repeat. That's pretty much what the 'professionals' do all day anyway. It all comes from practice. Software like BIRD, quagga, nginx, haproxy, ip/nftables, dnsmasq/powerdns, etc etc.
When you think you've exhausted the software side of the above tools and beyond and want to lay your hands on some actual hardware, look at picking up a 'white box' switch, a cast-off on eBay from the likes of Quanta/QCT, Edge-core or others. Don't spend more than a couple hundred bucks on this. Throw an ONIE network os on them (I suggest Sonic for open source, or if you want to pay, Cumulus) and start using 'real' hardware and play around with that. Learn the basics of sfp transceivers, fibre optics and the different mode types they come in, direct attach cables, port channels and the like. You can find super cheap transceiver hardware, fibre optic patch cables and all that at a discount vendor like fs.com, or from ebay as well. Learn how to interrogate the firmware on those, find out power transmission levels, error rates and other system info.
There's a sibling comment here suggesting you start out by setting up a LLC and going to ARIN and getting an autonomous system number. Please ignore that advice. You will be just wasting your time and your money and be distracted for no reason until you have the most basic of foundations. Use the abundance of resources you have to learn first. If you really feel like you want to take that next step, then be confident and do it!
Like a lot of things in this industry, the complexity can get fractal in nature the more you look at it. Don't let that overwhelm you. Take it one step at a time and don't be afraid to break shit, fixing it is how you learn best.
> ISPs tend to oversubscribe these services as well (where you and your 10 neighbors might all be able to sign up for 1Gbps symmetric service, but not everyone can leverage that full 1Gbps at the same time).
How does this work for FTTH? I know nothing about fibre optic networks. I had the impression that each subscriber has their own wavelength, or rather a range of wavelengths that captures their bandwidth, and that does not overlap with other subscribers.
Otherwise I have no idea how passive optical networks could even work.
In a typical passive optical network, one PON port is connected to 128 clients through the use of PLC splitters - unlike a WDM splitter which will insert or remove a specific wavelength, these simply split the whole signal. Where I work, that is a 4-4-8 configuration using 3 layers of splitters.
The OLT (optical line terminal, head-end) will tell each ONU/ONT (optical network unit/terminal) how much airtime they can use to transmit - each ONT will take their turn in transmitting so as not to interrupt others. Part of this calculation is the distance the ONT is from the OLT - each ONT will be a different distance depending on the geographic location, which means each ONT will have a different latency. The ONU can request additional airtime if it has a large amound of data to transmit. The amount of airtime the OLT will allocate depends upon the CIR (committed information rate i.e. what will the ISP guarantee at a minimum) and the PIR (peak information rate - the maximum rate based on the subscribers service).
> How does this work for FTTH? I know nothing about fibre optic networks. I had the impression that each subscriber has their own wavelength, or rather a range of wavelengths that captures their bandwidth, and that does not overlap with other subscribers.
The keyword to search for is GPON. It's multiplexed, each subscriber receives a few time slots in a shared wavelength. The transmissions from the subscribers don't collide because their time slots don't overlap.
Verizon must make some killer margins on a connection once it’s up and running, given they’re willing to eat the cost of 4 employees and a police detail splicing fiber in manholes for a week.
Back in 2015, Time Warner Cable testified they make a 97 Percent Profit Margin on High-Speed Internet Service for residential customers. I would guess Verizon is doing okay.
Very similar to the process I took getting DIA to a commercial building. They said they spent $60,000 pulling fiber half a mile through a business park (we didn’t pay this). We only have one ISP device in our rack (Verizon truly ships their org chart). We paid our contractor to have fiber installed from the DEMARC to the server room but apparently the ISP would’ve done that for free, oops.
And yeah the quality of customer service we’ve gotten from three different business providers has been exceptional. It’s crazy to have actual engineers you can call who know what’s going on. You get what you pay for.
This was about 15 years ago, back when I was working at $MEGACORP we had an OC-48 running to our lab. We were having some problems with it. Thanks to 20 years of near constant layoffs all details of who was responsible for that circuit on our side were lost to the sands of time.
I went down to the basement and saw a faded UUNET sticker on the demarc, but there was no circuit id on it. Some googling showed that through the years of corporate takeovers they were now owned by Verizon. So I called Verzion Business and explained the situation. The lady spent a hour on the phone with me, but we tracked down the circuit. The address listed on the circuit was a manhole up the street from our building. They dispatched a technician and we were back up and running in about an hour. They also put new circuit ID label on the demarc so we wouldn't go through that again.
[+] [-] marcus0x62|1 year ago|reply
When you have an SLA, understand what it is: a financial arrangement whereby you can request a prorated refund for certain types of outages. It is not in any way a guarantee on the part of a provider that you'll experience even average uptime equaling or exceeding the SLA, just that they can pay out the fraction of customer requests for service credits they receive for the covered outages they have and still make money.
The reality for the type of service the author of this post purchased is that for any physical damage to the fiber plant, he will experience hours of outage while a splice crew locates and repairs the damage. Verizon might offer a 100% SLA, but they didn't engineer it to even five nines of availability. That would require redundant equipment and service entrances at his premises along with path diversity end-to-end.
[+] [-] Arcanum-XIII|1 year ago|reply
[+] [-] mmooss|1 year ago|reply
1. Unless you are a large customer who accounts for an important amount of their bottom line, probably you have little financial leverage with the vendor.
2. The amount at stake in the SLA is not worth going to court for. It's unenforceable and in fact, the amount is usually meaningless.
Let's say you pay $10,000 per month for your 100% SLA dedicated circuit, and it goes down for an entire month. Let's say the vendor doesn't get around to paying you. Is it worth hiring a lawyer to collect $10K? Is it worth distracting you from your job emotionally and mentally, and consuming many hours of your time? Probably not.
Let's say your circuit is down for 3 hours. Let's say the SLA even pays you 3x what you pay for the service for any downtime (most I've seen just refund the money for that time). Let's see: ($10,000/month) / (720 hrs/month) = $13.89/hour. The SLA pays $41.67/hr of downtime, or $125 for your downtime. Is it even worth figuring out how to apply and filling out the form? No. You have much bigger issues in business, and if you don't ... well, then you have bigger issues.
3. The cost to the vendor is reputation: You tell your peers how much your service sucks, and word gets around. I've had techs take disinterested attitudes toward their poor uptime on our circuit; when I've called the account managers, they can have a very different response - they want me spreading their name around in a different way. That has nothing to do with the SLA.
> Verizon might offer a 100% SLA, but they didn't engineer it to even five nines of availability. That would require redundant equipment and service entrances at his premises along with path diversity end-to-end.
Agreed. There is no substitute for the physical reality of the circuit and service, which you should understand if you are buying it. Putting a shiny SLA on it will have no effect on the outcome.
[+] [-] simonjgreen|1 year ago|reply
[+] [-] renewiltord|1 year ago|reply
One time I contacted the business contact with a bunch of addresses and that was useful but the round trip is so slow I was better off just querying.
[+] [-] implements|1 year ago|reply
Edit: Ah, “Our network. You acknowledge that We may change your Internet Protocol (IP) address from time to time without giving notice unless You have purchased a fixed IP address from us;”
[+] [-] NetOpWibby|1 year ago|reply
[+] [-] baby_souffle|1 year ago|reply
I am jealous of them save for the "your monthly ISP bill is similar to a car payment" aspect.
[+] [-] ganoushoreilly|1 year ago|reply
Ultimately I get the SLA, Direct access to cloud providers maximizing performance, i'm also able to host a few IP blocks which allow a couple internet facing machines.
The home we sold recently had it pretty good too though, ended up with 3 5gig AT&T lines (no redundancy obviously) for only $450 a month total. That was pretty darn rad, even if the SLA wasn't the same.
Benefit of working from the farm is that I can also snag some bw for personal use ;D
[+] [-] therein|1 year ago|reply
[+] [-] dboreham|1 year ago|reply
[+] [-] anotheracc88|1 year ago|reply
Anything privacy related, running a TOR node etc. I get it.
[+] [-] cyberax|1 year ago|reply
Getting an ISP to even _talk_ to me required quite a bit of sleuthing. And I was saying from the outset that I was ready to fully pay for the fiber run.
Apparently, ISPs in my locality actually divide the city into the service areas. How the heck this is legal, I don't understand.
Some tidbits from me: my ISP installed a big honking ADVA optical line terminal on my premises. Getting them to move it to their side and just provide me with an SFP connection is still my work-in-progress.
The support is also outsourced into India, and getting them to understand what you want over the phone is... painful. Fortunately, the web ticket system is good enough.
[+] [-] immibis|1 year ago|reply
It's possible that the termination equipment could be a bare SFP and not a big box, but the ISP wants to be able to monitor the status of your connection up to the termination equipment, because that is their responsibility to keep online, and they can't do that if it's just an off-the-shelf SFP. They probably wouldn't agree to do it and still have any SLA.
If there's a physical problem with the box (too big/loud) you can try negotiate for a different box but if you just want control over your network, sorry but that just isn't how it works. Your network starts at the demarcation, and you don't want to be responsible for speaking whatever protocols the ISP is using internally, either. Up to the demarcation, it's ISP internal network, and past that, it's your network, with a standard handover interface like Ethernet.
[+] [-] fallous|1 year ago|reply
[+] [-] fweimer|1 year ago|reply
If you get it, it can be great. Imagine your ISP calling you when you reboot your router.
[+] [-] xelxebar|1 year ago|reply
While fascinated with the network stack, I've only gotten as deep as reading Illustrated TCP/IP and pretending I understand tcpdump. I would love to rovel around in BGP and, um, all that jazz.
Any suggestions on how to get started? My vague understanding is that most people get apprenticed into this stuff through work. Are the relevant systems involved just too expensive and locked behind corporate walls to be amenable to autodidactism?
[+] [-] Arubis|1 year ago|reply
You’ll need some kind of business entity to have easier conversations with ARIN, which is the starting point for getting yourself an ASN.
Once you’ve got an ASN, you have an entity that can “own” IP blocks instead of just relying on other networks to handle that for you. Now, have a look at Neptune Networks’ offerings—they’ll rent you an instance for a reasonable monthly cost that they’ll allow transit to and from. Note that their smallest instance size doesn’t have enough RAM to store the global internet’s full BGP table; this will matter only once you know what that means.
That’ll definitely get you started, and you can learn a lot on the cheap before even looking into your local colocation options.
[+] [-] Arubis|1 year ago|reply
[+] [-] theideaofcoffee|1 year ago|reply
Stevens' book is also a stupendous resource for the down-and-dirty, so good work on starting there. Beyond that you need to start just building things.
Almost every virtualization suite allows you to create network resources (or at least it abstracts the low-level OS calls or commands required to do so). Set up two VMs. Make them talk. Break that link and learn how to repair it, using the tools that you've mentioned you are now using, tcpdump in particular. Figure out how ARP works at a low level, or NDP (neighbor discovery) if you're running IPv6. Learn how to subnet, too! Then work your way up the stack. Set up a VLAN interface, set a 802.1q tag on an interface, try to get two or more vlans to talk to each other, route between them. Break that. Set up a basic OSPF area. Set up a BGP adjacency between two private ASNs you have created. Redistribute routes among different protocols. Set up higher-level services like DNS. Set up a play anycast network on your local host. Play around with load balancers and web servers. Play, break, fix, repeat. That's pretty much what the 'professionals' do all day anyway. It all comes from practice. Software like BIRD, quagga, nginx, haproxy, ip/nftables, dnsmasq/powerdns, etc etc.
When you think you've exhausted the software side of the above tools and beyond and want to lay your hands on some actual hardware, look at picking up a 'white box' switch, a cast-off on eBay from the likes of Quanta/QCT, Edge-core or others. Don't spend more than a couple hundred bucks on this. Throw an ONIE network os on them (I suggest Sonic for open source, or if you want to pay, Cumulus) and start using 'real' hardware and play around with that. Learn the basics of sfp transceivers, fibre optics and the different mode types they come in, direct attach cables, port channels and the like. You can find super cheap transceiver hardware, fibre optic patch cables and all that at a discount vendor like fs.com, or from ebay as well. Learn how to interrogate the firmware on those, find out power transmission levels, error rates and other system info.
There's a sibling comment here suggesting you start out by setting up a LLC and going to ARIN and getting an autonomous system number. Please ignore that advice. You will be just wasting your time and your money and be distracted for no reason until you have the most basic of foundations. Use the abundance of resources you have to learn first. If you really feel like you want to take that next step, then be confident and do it!
Like a lot of things in this industry, the complexity can get fractal in nature the more you look at it. Don't let that overwhelm you. Take it one step at a time and don't be afraid to break shit, fixing it is how you learn best.
[+] [-] rssoconnor|1 year ago|reply
How does this work for FTTH? I know nothing about fibre optic networks. I had the impression that each subscriber has their own wavelength, or rather a range of wavelengths that captures their bandwidth, and that does not overlap with other subscribers.
Otherwise I have no idea how passive optical networks could even work.
[+] [-] chaz6|1 year ago|reply
The OLT (optical line terminal, head-end) will tell each ONU/ONT (optical network unit/terminal) how much airtime they can use to transmit - each ONT will take their turn in transmitting so as not to interrupt others. Part of this calculation is the distance the ONT is from the OLT - each ONT will be a different distance depending on the geographic location, which means each ONT will have a different latency. The ONU can request additional airtime if it has a large amound of data to transmit. The amount of airtime the OLT will allocate depends upon the CIR (committed information rate i.e. what will the ISP guarantee at a minimum) and the PIR (peak information rate - the maximum rate based on the subscribers service).
[+] [-] cesarb|1 year ago|reply
The keyword to search for is GPON. It's multiplexed, each subscriber receives a few time slots in a shared wavelength. The transmissions from the subscribers don't collide because their time slots don't overlap.
[+] [-] mikeocool|1 year ago|reply
[+] [-] 293984j29384|1 year ago|reply
[+] [-] SemioticStandrd|1 year ago|reply
[+] [-] mastax|1 year ago|reply
And yeah the quality of customer service we’ve gotten from three different business providers has been exceptional. It’s crazy to have actual engineers you can call who know what’s going on. You get what you pay for.
[+] [-] pickle-wizard|1 year ago|reply
I went down to the basement and saw a faded UUNET sticker on the demarc, but there was no circuit id on it. Some googling showed that through the years of corporate takeovers they were now owned by Verizon. So I called Verzion Business and explained the situation. The lady spent a hour on the phone with me, but we tracked down the circuit. The address listed on the circuit was a manhole up the street from our building. They dispatched a technician and we were back up and running in about an hour. They also put new circuit ID label on the demarc so we wouldn't go through that again.
[+] [-] doubleg72|1 year ago|reply
[+] [-] ipython|1 year ago|reply
[+] [-] bumbledraven|1 year ago|reply
[+] [-] lol_catz|1 year ago|reply
[+] [-] anotheracc88|1 year ago|reply