WingNews logo WingNews GitHub [2]
top | item 33894933

Tell HN: IPv6-only still pretty much unusable

686 points| 9dev | 3 years ago | reply

Our Hosting provider, Hetzner, has recently started charging for public IPv4 addresses - as they should! Those numbers started getting expensive. This prompted me to try and set up a new server cluster using IPv6 exclusively, and see how far I could get before having to give in and purchase an additional v4 address.

The experiment ended much sooner than I had anticipated. Some of the road blocks I hit along the way:

  - The GitHub API and its code load endpoints are not reachable via IPv6, making it impossible to download release artefacts from many projects, lots of which distribute their software via GitHub exclusively (Prometheus for instance).
  - The default Ubuntu key servers aren't reachable via IPv6, making it difficult to install packages from third-party registries, such as Docker or Grafana. While debugging, I noticed huge swaths of the GPG infrastructure are defunct: There aren't many key servers left at all, and the only one I found actually working via IPv6 was pgpkeys.eu.
  - BitBucket cannot deploy to IPv6 hosts, as pipelines don't support IPv6 at all. You can self-host a pipeline runner and connect to it via v6, BUT it needs to have a dual stack - otherwise the runner won't start.
  - Hetzner itself doesn't even provide their own API via IPv6 (which we talk to for in-cluster service discovery. Oh, the irony.
It seems IPv6 is still not viable, more than a decade after launch. Do you use it in production? If so, how? What issues did you hit?

631 comments

order
[+] redox99|3 years ago|reply
IPv6 has been one of the biggest failures in the last couple of decades.

And I don't mean adoption, I mean the standard itself.

If IPv6 were IPv4 with more octets, then we would all have been using it for like a decade.

Yes, I understand it would still require some breaking changes, but it would have been a million times easier to upgrade, as it would be a kind of superset of IPv4 (1.2.3.4 can be referred as 0.0.0.0.1.2.3.4).

Not having two sets of firewall rules and two sets of everything. I always disable IPv6 because it can bite you so hard when you don't realize that you are wide open to IPv6 connections because of different firewalls.

Edit: To make everything a bit clearer, the idea with this "ipv4+" is that you don't need the complexity of running both ipv4 and ipv6 as you do now.

And regarding compatibility, with ipv4+ if you have a 0.0.0.0.x.x.x.x ip address you would be able to talk to both ipv4+ aware and legacy ipv4 devices natively without any tunneling (because you also own the legacy, non quad 0 ip address). If you don't have such "quad 0 ip" (you are 1.1.1.1.x.x.x.x), only ipv4+ aware devices would be able to to connect to you, and for you to connect to non ipv4+ aware devices you would need either tunneling, or having a secondary, cgnat, "quad 0 ip".

[+] throw0101a|3 years ago|reply
> And regarding compatibility, with ipv4+ if you have a 0.0.0.0.x.x.x.x ip address you would be able to talk to both ipv4+ aware and legacy ipv4 devices natively without any tunneling (because you also own the legacy, non quad 0 ip address).

This exists:

> Addresses in this group consist of an 80-bit prefix of zeros, the next 16 bits are ones, and the remaining, least-significant 32 bits contain the IPv4 address. For example, ::ffff:192.0.2.128 represents the IPv4 address 192.0.2.128. A previous format, called "IPv4-compatible IPv6 address", was ::192.0.2.128; however, this method is deprecated.[61]

* https://en.wikipedia.org/wiki/IPv6#IPv4-mapped_IPv6_addresse...

* https://datatracker.ietf.org/doc/html/rfc4291#section-2-5-5

You still need to upgrade bit of networking kit between the source and destination to understand "IPv4+", and this (lack of) upgrading and enabling is what is hampering deployment.

What makes you think that companies would have been willing to make the effort to deploy "IPv4+" any more than IPv6?

[+] est31|3 years ago|reply
One of the ideas of ipv6 was to reduce routing tables, those tables that backbone providers have to keep in memory and look up for incoming traffic. With ipv4's fragmented allocation scheme, these routing tables are huge. With ipv6, even huge companies like amazon only have a couple of global allocations. A "ipv4 with more octets" scheme would have kept that fragmentation around.

That being said, Amazon currently has 2880 ipv4 allocations and 946 ipv6 allocations... not much gained I guess? :p https://asnlookup.com/asn/AS16509/

Also, there are definitely some horrible ipv6 warts, like that the only standard for local ipv6 addresses forces you to adopt a scheme where your local address is horribly long, for the sake of global uniqueness, which is something that most people don't really need.

[+] fernandomm|3 years ago|reply
Let's not forget about the idea that ISPs would distribute a /56 range to residential users. You could split it in /64 ranges according to your requirements and everything would work fine.

There is only one "minor" issue: all major ISPs in my country ( Brazil ) only provide a single /64. You can't get another /64 unless you upgrade to a very expensive business plan.

That makes IPv6 not only useless but also a huge security issue.

1) I can't use my Mikrotik as a firewall. Trying to split a /64 range breaks things and some devices ( specially IOT ones ) will simply not work.

2) Routers provided by the ISPs here are very limited, specially for things like firewall rules. Some of them will only provide a On/Off switch, with Off option between the default one.

Although IPV4 + NAT had some issues, it ( accidentally? ) created a safe/sane default config for non-technical users. In order to open a port and expose a device, you have to explicitly add a rule on the firewall.

IPv6 is the other way around. In practice, all devices and ports are exposed unless you explicitly block it.

In the last 3 years I've noticed criminals focusing more and more on IPv6 scans to compromise devices and create botnets since it's much easier to find exposed/unpatched devices as most users don't understand how to correctly configure a firewall.

Most of the time, the only viable solution is to disable IPv6.

[+] rnhmjoj|3 years ago|reply
> If IPv6 were IPv4 with more octets, then we would all have been using it for like a decade.

I don't really think so: it woulds still be completely backward incompatible and still require replacing a lot of costly network equipment. I think that's the main reason why large ISPs and enterprises have been postponing the upgrade since forever but operating systems, smartphones and other new devices didn't really have a problem with it.

[+] WastingMyTime89|3 years ago|reply
It's already more or less how IPv4 addresses are embedded in IPv4 (except its 128bits and they use an FFFF prefix between the 0 and the IPv4 address).

IPv6 doesn't solve anything for the sake of it. Anyone who had to debug ARP caused issue on a network knows it's complete garbage for example.

Providers who explain that they are dragging their feet because of the complexity would have said exactly the same thing even it was only IPv4++. They just don't want to invest any money in something which is working for them.

[+] spookthesunset|3 years ago|reply
It’s the firewall rules that always creep me out. The nice thing about NAT is open ports on your internal network are hidden to the outside world by default. You have to think about which ports you want the NAT gateway to forward.

With IPv6 the entire network is reachable outside by default. Granted I assume you can probably create a default DENY rule for inbound traffic and selectively open ports up as exceptions. Right?

[+] RijilV|3 years ago|reply
It's been a quarter of a century since IPv6 launch.

There's some really good lessons learned here. IPv6 requires everyone, everywhere, needs to change their configuration to add IPv6 addresses and network connectivity to every node/endpoint. The madness of course is that all the underlying infrastructure software (routers, OS, standard libraries) all support IPv6. It would seem, at a large enough scale, that software is the easy part, configuration is the hard part.

DJB wrote this up two decades ago[0] and it remains relevant. In particular the comparison between IPv6 and MX records. It took me a little while to wrap my brain around just extending existing software to support sometimes 32bit and sometimes 128. Ultimately it wasn't hard to dream up a few solutions for how that would work.

The other thing, FWIW, which has been slowing IPv6 adoption is business owners not asking for it as a high priority item from their providers. I've seen this happen way more often that I would like where provider asks a customer what they want and the customer never mentions IPv6, and when prompted shrugs it off because they don't see it as a business critical (and in fairness, it hasn't been). Yes provider isn't asking the 'nerds' at their customer's businesses, but those folks also aren't the ones paying the bills so...

0: https://cr.yp.to/djbdns/ipv6mess.html

[+] unquietwiki|3 years ago|reply
Hey all... unquietwiki from r/ipv6; been lead-mod there for a while now (though the rest of the folks are really amazing on the mod-side).

IPv6 has saved my bacon more times in the past 15 years, than IPv4 has fought with me. No clashing of IPv4 ranges. No fighting with NAT. Ability to easily have concurrent networks, for different purposes. Ability to assign multiple network addresses. Internally, it "just works" on basically anything not an older licensing server, or LAN game. Microsoft is using it internally. Apple has been mandating MacOS and iOS apps to support it for several years now. Usage over the Internet is exceeding 50% in major countries.

Where it falls short... a lot of ISPs use equipment that don't handle it well, and/or don't have the staff to implement it properly; and the ISP buyouts and mergers here in the US of the past 15-20 years REALLY messed up the timeline on adoption. Cloud services, a lot of the stuff Google & Microsoft built up their public offerings on, they were designed largely on older virtualization setups that precluded how IPv6 works, until recently; same goes for Docker & Kubernetes. A lot of VPS hosts support it just fine; but a lot of businesses aren't leveraging those, as they could be. And lastly... a lot of IT guys still don't care for IPv6; as those biases show up in these & on Reddit; so that perpetuates the cycle.

[+] ipython|3 years ago|reply
I have to say I’m super disappointed in the ignorance and negativity in the comments on this thread. Ignorance of both the difficulties inherent in upgrading a fixed size wire protocol designed for a research network fifty years ago, and the widespread adoption of ipv6 for real customer deployments. Heck most of you are probably using ipv6 through your mobile carrier and don’t even know it!
[+] arcade79|3 years ago|reply
I was thinking about ipv6 the other day. I concluded in my head that adoption was just around 5-10%. Luckily I went to verify that with statistics.

https://www.google.com/intl/en/ipv6/statistics.html

While price of ipv4 addresses are increasing, the world has slowly been adopting ipv6. From the graph above, I'd say we cross over 50% in about 2-3 years time. At some point the "dash" to adopt ipv6 starts, and brave folks will drop support for ipv4. Then it'll probably evaporate between 2030-2035.

[+] tjoff|3 years ago|reply
That is a very one-sided view of adoption. It ties directly with the rise of mobile and internet in areas that wasn't able to grab IPv4 addresses in time. Such as India. Not sure what France is doing though, maybe something right.

So, from my perspective (which obviously is tied to my location) is that all computers have IPv4 (haven't heard (and I've asked) of a single consumer ISP that offers IPv6) but all mobile phones have IPv6. Whether people in general do most searches on mobile or PC is gonna change that graph dramatically, but won't explain what I would be interested in regarding ipv6 adoption. That is, how much of the world would be broken without IPv4?

And for that, that graph isn't completely useless - but not too far off.

[+] mort96|3 years ago|reply
> At some point the "dash" to adopt ipv6 starts, and brave folks will drop support for ipv4.

I wouldn't be sure about that. I don't see any "dash" to support v6 in our future, when the option to just keep working around issues with v4 is so much easier and cheaper in the moment. Really, what does anyone have to gain by switching to v6?

[+] bombcar|3 years ago|reply
I'd love for some way to measure how much of my network's traffic outbound and inbound is IPv6. I assume there's some way to "count" it on the router, but Mikrotik doesn't seem to expose it directly.
[+] dilyevsky|3 years ago|reply
At previous gig already seen some isps in India that are ipv6 only
[+] bombcar|3 years ago|reply
Setting a website to be available over IPv6 is relatively easy, yet we see:

    ;; QUESTION SECTION:
    ;news.ycombinator.com.  IN AAAA
Why? Because it's not quite as simple as making Apache respond over IPv6; any website of any size has various protections in place to prevent DDoS, spam, etc, and those tools are almost universally basic and at the root is the "ban by IPv4 address". Without that tooling supporting IPv6, it remains a side note not worth supporting.

Solutions? You can make an IPv6 version available that is read-only; or requires you to login via an IPv4-only gateway first (and protect that) and then ban by username as necessary.

And outbound? You have to IPv4 NAT which maybe Hetzner offers? If not there are things like https://nat64.net

[+] justin_oaks|3 years ago|reply
Having "grown up" with IPv4, I'm slow to learn everything necessary to set up an IPv6 infrastructure. The times I did look into it, IPv6 seemed so much more complicated than IPv4, but maybe that's just because I'm just not familiar with it.

Are there any good resources on setting up IPv6 support from first principles?

I still get confused as to the "right" way to set up internal networks for IPv6, especially when DHCPv6 isn't universally supported (I'm looking at you Android).

[+] yesco|3 years ago|reply
A confusing aspect of IPv6 is that it's actually a much simpler protocol than IPv4, you often end up assuming you need to configure a bunch of stuff that you really don't have to.

The most common example would be NAT, despite the complexity it adds to IPv4, people often get comfortable with idea of setting up complex subnet hierarchies and feel lost when that all just disappears with IPv6.

The key things to remember when working with IPv6 are:

- IPv6 is very unidirectional, it's not a giant one way waterfall like IPv4/NAT

- Routers don't assign addresses, they advertise "prefixes", usually multiple

- Routers will usually have a prefix for: Internet, WAN, Link-Local (last one being advertised only to nodes directly connected to it)

- Nodes use prefixes to auto-generate an address

- Auto generated addresses are usually in the form of "prefix - device_id" so even if a node has a lot of addresses, they are all mostly the same

- Usually nodes can easily communicate back and forth across multiple local routers with little configuration or hierarchy

- Internet/non-local IPv6 addresses break the rules a bit and don't use a device_id in their addresses in order to protect user privacy

- Even if every node has an external address now, you can still configure your firewall to ensure they are isolated from external connections (which is usually the default anyway). You don't need NAT to securely isolate things.

- Once you get the hang of it you will realize how easy it makes everything and despair that support for it sucks and everyone makes it harder than they need to

Finally for learning resources I honestly recommend just reading the RFCs, I personally learned this way and believe they provide the most direct understanding of the rational behind everything.

[+] argulane|3 years ago|reply
Everything should get its IPv6 configuration via SLAAC. DHCPv6 is only useful when you plan to provide prefix delegation for extra routers or network boot information.
[+] AdrianB1|3 years ago|reply
Growing up implementing IPv4, then moving to a different part of IT, I never found a reason to learn IPv6; it is something that is out there, but not relevant for me. I cannot tell if this good or bad.
[+] unethical_ban|3 years ago|reply
Bookmarking for future. I want to fill this out but don't have the info in front of me at the moment.
[+] buildbuildbuild|3 years ago|reply
This problem led us to self-hosting Gitlab four years ago, I can't believe it's still an issue in 2022.

It's appalling that a "developer product" like Github remains such a blocker to IPv6 adoption, especially for highly Github-reliant communities like the Golang ecosystem.

Launch an IPv6-only VM and try to build a mainstream Go project.

[+] mappu|3 years ago|reply
Most of Go dependencies these days come from your $GOPROXY, which by default (proxy.golang.org) is available over IPv6 just fine.
[+] maximaximal|3 years ago|reply
We are IPv6-only on our institute-internal CPU compute cluster based on slurm. Only the head node has an IPv4 address, so that it can be reached from IPv4 only clients (sadly, there are still quite a lot). All nodes inside the cluster talk over IPv6. And all other computers with IPv6 access use that to communicate to the head node. We are transitioning to IPv6-only for internal services and try to avoid using IPv4 addresses, only going back to it when something needs to be accessed from the outside.

Sadly, our local ISP is still IPv4-only, meaning we cannot even access our IPv6 hosts while at home, so we need to fall-back to IPv4 quite a lot. Also, the Cisco VPN is still IPv4 only (because of lacking resources to add IPv6 support), so not even the VPN helps. We need to jump over some dual-stack host then.

When speaking to the local ISP, they just reply that it's not planned soon, they don't have resources for it, and "they evaluated IPv6 and don't have a reason to support it". Me/us giving them reasons was not enough it seems.

[+] gurrone|3 years ago|reply
20 years ago it was the lack of IPv6 support on the CPE holding IPv6 on the server side back, nowadays it's a lack of IPv6 at major SaaS providers causing issues. In most of the scenarios I was involved in we made sure that the CDN in front of the product was able to terminate IPv6 and left everything behind it v4 only. About 1/3 to 1/2 of the traffic received was sent via IPv6 on those setups. Maybe time to turn that around and use the CDN to make the product also available via v4? Leaves you with maintaining a NAT gateway for your own infrastructure.

BTW also only one of the office networks I had to deal with in the past 20 years hat experimental IPv6 support, and that was at a small local hosting company. Everything bigger than that also sticks to IPv4 only for now. :(

Strange how things change but still stay the same.

[+] KomoD|3 years ago|reply
> Our Hosting provider, Hetzner, has recently started charging for public IPv4 addresses

Well actually, they were charging for IPv4 addresses for a while, but just changed pricing... to outrageous pricing. The setup fees are actually insane!

And yeah IPv6-only is quite a terrible experience, I tried out a few days ago. I suggest just using nat64.net if you want to access IPv4 sites over IPv6.

One quite funny thing, I recently asked my ISP if they were ever going to add IPv6, they told me no, there's "not enough demand"... well there's "not enough demand" because there's barely any IPv6 websites, and then websites refuse to add support for IPv6 because... you guessed it, "not enough demand".

[+] MindTooth|3 years ago|reply
In Norway, it's required[0] for all public sectors to have IPv6. We are not there yet, but I believe the push will only increase with time.

Especially all new internal networks must be IPv6, and IPv4 is optional.

[0]: https://lovdata.no/dokument/SF/forskrift/2013-04-05-959?q=ip... (Sorry that it's in Norwegian.)

[+] AdrianB1|3 years ago|reply
Norway is not big enough on Internet scale to make a difference. Equipment and software companies will put in balance the cost vs the benefit and may decide to ignore that market.
[+] CGamesPlay|3 years ago|reply
To use GitHub on an IPv6-only Hetzner instance, you'll need to use a NAT64 gateway. There's a list of public ones here: https://nat64.xyz/

This can just go into your /etc/hosts:

    2a01:4f8:c2c:123f:64::140.82.121.3 github.com www.github.com
[+] gnfargbl|3 years ago|reply
Huh, so I can effectively use a NAT64 gateway as an unauthenticated open proxy? Let's try it. First look up the IPv4 for a site that reads back your IP address:

    $ dig +short a icanhazip.com
    104.18.115.97
    104.18.114.97
(Those are Cloudflare IP; icanhazip.com is hosted on CF.) Next, try connecting to the IP-readback site via a NAT64 gateway, but presenting the correct Host header so that Cloudflare knows what to do with the request:

    $ curl -6 -k -H 'Host: icanhazip.com' https://[2001:67c:2960:6464::104.18.115.97]/
    141.98.136.43
OK, it reads back an IPv4...

    $ dig +short -x 141.98.136.43
    de-fra2-nat641.level66.network.
...with a PTR record associated with level66.network, which was the NAT64 gateway we chose.

How does this not see more abuse by bad actors? I guess it probably does, which is why there are so few of these public gateways?

[+] 9dev|3 years ago|reply
That’s what I ended up doing, but it’s a major turn-off to learn this by trial and error, after setup scripts fail with seemingly unrelated messages.
[+] ArchOversight|3 years ago|reply
Yes, I use IPv6 in production, alongside IPv4 (dual-stack). I've got an IPv6 only EKS cluster that has IPv4 on the local instance that NAT's any IPv4 outgoing connections, but the entire cluster and all communications inside the cluster/between pods is IPv6 only.

IPv6 stand-alone without IPv4 dual stack is not yet an option, but it is getting closer. If you can mirror content and deploy from your mirrors it is entirely possible to do everything over IPv6 alone.

About 30% of my production traffic is IPv6, referring to outside traffic coming in to the systems. Internally almost 90% of the traffic is IPv6 due to the k8s cluster being IPv6 only, and preferring that over IPv4.

Interestingly enough at home on my home connection about 60% of my traffic is IPv6, which has been increasing steadily over the years as other companies have started bringing on-line IPv6 for their services.

[+] UI_at_80x24|3 years ago|reply
And this is where government regulation would have helped. It's a shame nobody listened to me back in the 90's: (it's a joke. I'm a nobody, there is nobody listening.)

All cell-phones should have been ipv6 from the get-go.

Now if we could just go back in time and warn everybody about IPv4 address space being exhausted. And maybe the climate too while we are at it.

Seriously though, cell-phone adoption would make that a game changer. Hey Apple, sell it as an exclusive feature!

[+] withinboredom|3 years ago|reply
Still no ipv6 support with WSL2 in windows. So, if you’re a Microsoft Windows dev in an ipv6 environment, you won’t make it too far.
[+] dboreham|3 years ago|reply
> The GitHub API and its code load endpoints are not reachable via IPv6

This is quite strange given the period around 2010 where there were government edicts that IPv6 must be supported. Perhaps GitHub wasn't mission critical back then.

[+] bombcar|3 years ago|reply
There are a number of companies (not naming names) that do NOT support IPv6 for commoners but if you are on a government account/contract you have different government endpoints that ... work over IPv6.

Infuriating, but perhaps understandable; they're willing to support the government over IPv6 because it's required and they pay more. But I wish there was a "I know what I'm doing and understand you'll make fun of me if I ask for support on this" option.

[+] jlokier|3 years ago|reply
It's 2022 and I've never yet seen an IPv6-capable internet connection at any home I've lived in, office I've worked at, coffee shop or library whose network I've used. And I've lived in a lot of places

Phones have it. But when I connect my computers via my phone's hotspot, the computers only get IPv4.

My data SIM at home enables IPv6 when used in a phone. But when installed in my 4G router, same contract, same provider, there is no IPv6 despite the router supporting it. That is, supporting in theory. Since I've never seen it enabled, I can't say if it works.

My bare metal servers have IPv6. But my cloud VMs and containers only get IPv4.

The last major network protocol I worked with, a P2P mesh protocol that needs to pass around IP addresses, has IPv6 on the todo list. Last I saw, earlier this year, there was a debate about whether it was worth adding or not. At the moment it's defaulting to no, even though I've seen a few IPv6 addresses attempt to join and be rejected

I can add IPv6 tunnels (VPNs and such) wherever I want, if I want to test software. But I could do that 20 years ago. It feels a bit artifical, not having the real thing.

My current work has a VPN we must use to access work dev servers. Obviously that doesn't use IPv6.

Forget about IPv6-"only", I'm still yet to see much IPv6 at all (except on phones).

[+] aidenn0|3 years ago|reply
My experience is quite different from yours:

- my ISP has offered IPv6 /64 delegations for over 5 years now.

- One cloud provider I use gives a discount for VMs with no IPv4 address, the other has ipv6 addresses.

[+] Vespasian|3 years ago|reply
I've had IPv6 at my home for at least 10 years I would say.

My parents do as well.

I was recently surprised that ssh to my dual stack private server didn't connect at work. It turned out I had set up the config file as V6 only.

Location: Southern Germany.

[+] api|3 years ago|reply
The biggest problem remains cloud and CDN companies with poor to nonexistent IPv6 support. Most ISPs, especially on mobile, have it now or are adding it very soon.

I've wondered whether some might be dragging their feet because they see an advantage in IP address scarcity to sell cloud gateways, CDNs, and other middle box type services. But the most likely explanation remains that not enough customers are asking for it so it's not the highest priority.

I happen to know that Google Cloud started moving on IPv6 seriously only when they lost some big telecom customers to AWS because they didn't have it.

[+] mort96|3 years ago|reply
I've never been on a non-mobile connection with IPv6. It's not a thing in Norway.
[+] dboreham|3 years ago|reply
> Most ISPs, especially on mobile, have it now or are adding it very soon.

Except Charter/Spectrum in the US.

[+] xlmnxp|3 years ago|reply
Hetzner should provide free CGNAT IPv4 Addresses (IPv4 Gateway) for IPv6-Only VMs