> We are working with the small number of networks with a higher network/ISP density than Cloudflare (e.g., Netflix, Facebook, Google/YouTube) to come up with an EDNS IP Subnet alternative that gets them the information they need for geolocation targeting without risking user privacy and security. Those conversations have been productive and are ongoing. If archive.is has suggestions along these lines, we’d be happy to consider them.
I actually forgot to consider this angle, as the discussion was centred at archive.is. I'd imagine 1.1.1.1 has a very negative consequences for Netflix local caches, for example. This is where your local homegrown provider gets to save significant $$$$ due to interconnect/bandwidth costs by hosting a local Netflix cache through their appliance, and you get to benefit by the latest shows being locally cached, delivered at maximum speeds, instead of being hailed through the whole internet each time for each device.
If you're using 1.1.1.1, you're basically not only making sure that your internet will be much slower due to suboptimal CDN performance by any CDN other than Cloudflare CDN, but that you're also going to needlessly be running extra simultaneous streams of your favourite shows instead of fetching a local copy from your own ISP, increasing the cost of bandwidth transit to your ISP.
And, remember, you don't actually get any extra privacy by bypassing ECS in the first place, because your exact full IP address will have to be used to establish any subsequent TCP or UDP connections to make those requests for actual content in any case. You're basically breaking the whole internet by using 1.1.1.1, all for no real benefit! It's worse than we initially thought!
This link and the two answers within demonstrate something important, broader than the DNS related issue at hand.
Both make implicit assumptions. One assumes the worst of Cloudflare and thinks “what’s the worst reason Cloudflare could have for doing this. How do they profit off this?” And the other assumes that Cloudflare has good intentions.
Neither answer is technically wrong. Both flow logically from their initial assumptions. But it shows how different our conclusions can be depending on where our initial biases lie. For the person who believes the first answer and says “prove to me that Cloudflare isn’t doing something nefarious”, it’s not possible. The analysis is correct and can’t be challenged unless the initial assumption is challenged. And for people who strongly believe that Cloudflare has bad intentions, nothing can be done to change their mind.
In this example it’s Cloudflare but it applies to any person or organisation that we feel strongly about.
> I consider EDNS-less requests from Cloudflare as invalid.
If your site depends on a DNS extension that's only 3.5 years old (and designed to be optional), I think it's fair to say your site is just offline for some users due to a config mistake.
You're free to set up your servers however you like, but there's wisdom in Postel's law.
Another interpretation of the Law by Mark Crispin, father of IMAP:
This statement is based upon a terrible misunderstand of Postel's
robustness principle. I knew Jon Postel. He was quite unhappy with
how his robustness principle was abused to cover up non-compliant
behavior, and to criticize compliant software.
Jon's principle could perhaps be more accurately stated as "in general,
only a subset of a protocol is actually used in real life. So, you should
be conservative and only generate that subset. However, you should also
be liberal and accept everything that the protocol permits, even if it
appears that nobody will ever use it."
Archive.is does not block all requests lacking EDNS. They specifically block requests coming from Cloudflare's datacenters. Cloudflare is not accidentally misconfiguring their EDNS, Cloudflare is intentionally not sending EDNS.
I really don't see this as a problem of Cloudflare.
End users switching to Cloudflare's DNS endpoint are doing so because they feel the DNS provider is both faster and more secure.
They rightly made the decision NOT to pass on the end user's IP information to the upstream DNS server. I agree with this decision and they are acting in my best interests in doing so. To draw some kind of nefarious intention from this is absurd.
Until Cloudflare are proven to be nefarious actors, I'll continue to use their service.
> They rightly made the decision NOT to pass on the end user's IP information to the upstream DNS server. I agree with this decision and they are acting in my best interests in doing so. To draw some kind of nefarious intention from this is absurd.
In this instance, the upstream DNS server and the resultant HTTP server are operated by the same organisation. Cloudflare have opted to not provide the /24 (or /56 if IPv6) network that the original DNS request came from, in the DNS request. Your computer will then provide the /32 (or /128 if IPv6) that your request is coming from when you connect to the HTTP server.
What privacy win have you gained by Cloudflare not providing that information in this instance?
> they feel the DNS provider is both faster and more secure.
'Feel' being the keyword. Faster, generally yes. More secure, not well defined and users are generally wrong.
> nefarious intention
I don't believe I've heard any complaints of nefarious intent.
But let's be clear, this advantages Cloudflare over other CDNs. That they treat the DNS data very well does not mean they won't have an incident. As well, they are more of a target due to the concentration.
> Until Cloudflare are proven to be nefarious actors,
Nefarious wrt whom? For end-users taken individually, I agree, I don't see and it's hard for me to imagine mal intent.
But IMHO they are bad for the Internet. I mean, more power to them and were I a leader there I'd press the same agenda, but as a 3rd party, the way I see it is that in 10 years they are going to be an anti-power much like Google is. Addiction to their services will allow them to trample over what's good for all.
What I dislike most about them is that they promote themselves as purely a force for good. Except for a few PMs and execs I'm 100% sure they believe it. But it's a disservice to never discuss the negative aspects of any of their services. And woe to anyone who does.
As for proven nefarious deeds, do you not consider "banning" sites from using CF nefarious? What if they take it to the next step now, and stop providing DNS for those sites? Given their stated reason for bans, yes it could happen. Why must you wait until they prove to be nefarious? The concentration of power per se is a bad thing.
That looks like uncompetitive behavior from Cloudflare, so it's their problem also. Cloudflare can send EDNS if nameserver and the actual server run by the same party, but they don't
I'm with some of the people on Twitter: It seems weird (to put it mildly) to just blackhole your own site with no explanation whatsoever to the end-user. For everyone on 1.1.1.1 archive.is will now be "down" and they're none the wiser.
Maybe there's a big backstory here, but without context that seems passive-aggressive and quite random?
What's especially weird is that they're returning "127.0.0.3" to Cloudflare's DNS, rather than a DNS SERVFAIL or REFUSED error. On most systems that will cause a connection refused error or a TCP timeout. I would assume that was a network issue on their end, not a DNS problem.
Indeed. This is the first I'd heard of this situation. I'd previously just assumed archive.is was a shonky service that didn't work properly. Hadn't connected it with my use of 1.1.1.1
I am no expert by any means. However, I strongly suspect EDNS is not actually needed to run a CDN. There’s a lot of approaches to balancing load and distributing traffic. An example of another approach would be using anycast IPs.
I’m also surprised that traffic from Cloudflare DNS users caused any significant problem. Was it really that much traffic?
> However, I strongly suspect EDNS is not actually needed to run a CDN.
It's not. The proof is that CDNs existed long before edns-client-subnet was introduced. All it does is allow the CDN's DNS servers to return the most optimal A/AAAA records for the client. But the worst that should happen without it is you get sent to a more distant CDN server, and the content loads more slowly.
The fact that archive.is somehow suffers without this feature (which, btw, wasn't standardized until 2016) suggests they're doing something really, really odd. If I were them, I'd focus on making my system more robust, rather than demanding the rest of the Internet adopt a relatively young, optional DNS extension.
> An example of another approach would be using anycast IPs.
Anycast IP is very expensive, unfortunately. Just getting a /22 has been expensive for years, and is now also getting difficult as well. It is beyond the reach of smaller companies.
GeoDNS is extremely cheap in comparison. You can run distributed services using GeoDNS for low latency on multiple continents on a hobby budget these days.
Anycast is technically better in many ways (the combination of anycast and geoDNS is better again), but anycast is so expensive that smaller operators just can't use it.
These days, smaller operators can use Cloudflare for their CDN, and the suspicious mind might think that suits Cloudfare just fine. But that doesn't really help for low-latency interactive services, or non-HTTP services.
> I’m also surprised that traffic from Cloudflare DNS users caused any significant problem.
Maybe the problem isn't amount of traffic, but rather that the site doesn't want to gain a reputation as slow (and therefore incompetently administered, and offputting to use) when everyone running Firefox switches over to 1.1.1.1 DoH automatically.
> Absence of EDNS and massive mismatch (not only on AS/Country, but even on the continent level) of where DNS and related HTTP requests come from causes so many troubles so I consider EDNS-less requests from Cloudflare as invalid.
> massive mismatch (...) of where DNS and related HTTP requests come from causes so many troubles
Does anyone know what they could mean here? I get that having more open connections and slow requests is not great, but there are popular attacks people will try against them in this case. They already have to handle pathologic cases of slow requests, so handling some small number of slower clients shouldn't be an issue.
A lot of folks here seem to be saying "if you're going to make a DNS query, you're only going to make a HTTP request," which is simply untrue. Hell, you can add a HTML tag to your page to prefetch DNS queries. Browsers prefetch DNS just for hovering your mouse over a link or typing something into your address bar (without actually navigating). Should some DNS server know your IP address just because you moved your mouse over a link? IMO, no.
Please can you rephrase your argument. 100% serious, I'd like to know what point you're making.
Pre-emptively: because whatever DNS server you are using already knows your IP address, regardless whether it's the first query for the site itself, or subsequent queries for site-related additional resources.
ECS is not equivalent to 'send the IP' but is revealing.
the fact that I subsequently connect to another place over HTTP or some other protocol is distinct from telling a DNS authority who is asking a question about a domain name: the article implies "its the same leakage" but it isn't: different people get told.
I don’t understand the privacy reason. If I am querying for domain x, why does it matter that domain x’s DNS servers know what IP I am querying them from? I am going to hit their web server directly with that very same IP in a few milliseconds anyway.
There are a few reasons. Here are three I can think of off the top of my head:
Many browsers prefetch DNS for links on webpages these days. So it’s entirely possible and even common that you may query DNS for sites you never visit, which would indeed be a privacy leak.
Secondly, many sites have their DNS hosted elsewhere so it may not be the same people you are leaking the information to.
Thirdly, if the DNS query is transmitted to the site’s DNS servers in plain text (which most DNS is), then despite eSNI etc anyone who has access to the wire traffic along the route from the DNS proxy to the site’s DNS servers (which is probably different from the route your own traffic takes to their servers) can see which site you wanted to access.
Does 1.1.1.1 send ECS info to Cloudflare’s own nameservers? More generally, does 1.1.1.1 in any way treat Cloudflare’s own nameservers in a special way and send it information that it doesn’t send to others?
If the answer to these questions is no, then Cloudflare’s reasons for blocking ECS (ie privacy) carry weight. Otherwise no.
I think decision of archive.is is very interesting.
1) They attracted a lot of attention;
2) They showed the way to struggle with Cloudflare business that abuse their service.
If several bigger CDNs like akamai or softlayer will consider requests from 1.1.1.1 without EDNS as invalid and block them, Clouldflare wouldn't be able just to say that it's their own problems
I think that's because Firefox falls back to system resolver if something funky happens. I'm using it strictly over DoH and it isn't working for me still.
A very silly one at that given their reasons for not resolving archive.is are quite rational and on the contrary makes me want to swap google's DNS servers for theirs.
[+] [-] amarshall|6 years ago|reply
[+] [-] cnst|6 years ago|reply
I actually forgot to consider this angle, as the discussion was centred at archive.is. I'd imagine 1.1.1.1 has a very negative consequences for Netflix local caches, for example. This is where your local homegrown provider gets to save significant $$$$ due to interconnect/bandwidth costs by hosting a local Netflix cache through their appliance, and you get to benefit by the latest shows being locally cached, delivered at maximum speeds, instead of being hailed through the whole internet each time for each device.
If you're using 1.1.1.1, you're basically not only making sure that your internet will be much slower due to suboptimal CDN performance by any CDN other than Cloudflare CDN, but that you're also going to needlessly be running extra simultaneous streams of your favourite shows instead of fetching a local copy from your own ISP, increasing the cost of bandwidth transit to your ISP.
And, remember, you don't actually get any extra privacy by bypassing ECS in the first place, because your exact full IP address will have to be used to establish any subsequent TCP or UDP connections to make those requests for actual content in any case. You're basically breaking the whole internet by using 1.1.1.1, all for no real benefit! It's worse than we initially thought!
[+] [-] dvfjsdhgfv|6 years ago|reply
[+] [-] nindalf|6 years ago|reply
Both make implicit assumptions. One assumes the worst of Cloudflare and thinks “what’s the worst reason Cloudflare could have for doing this. How do they profit off this?” And the other assumes that Cloudflare has good intentions.
Neither answer is technically wrong. Both flow logically from their initial assumptions. But it shows how different our conclusions can be depending on where our initial biases lie. For the person who believes the first answer and says “prove to me that Cloudflare isn’t doing something nefarious”, it’s not possible. The analysis is correct and can’t be challenged unless the initial assumption is challenged. And for people who strongly believe that Cloudflare has bad intentions, nothing can be done to change their mind.
In this example it’s Cloudflare but it applies to any person or organisation that we feel strongly about.
[+] [-] profmonocle|6 years ago|reply
If your site depends on a DNS extension that's only 3.5 years old (and designed to be optional), I think it's fair to say your site is just offline for some users due to a config mistake.
You're free to set up your servers however you like, but there's wisdom in Postel's law.
[+] [-] throw0101a|6 years ago|reply
Further discussion on the topic:
* https://news.ycombinator.com/item?id=9824638
[+] [-] rumanator|6 years ago|reply
For the lazy like me: robustness principle, aka Postel's law
https://en.wikipedia.org/wiki/Robustness_principle
Thank you for the reference. I learned something today!
[+] [-] Thorrez|6 years ago|reply
[+] [-] akvadrako|6 years ago|reply
https://tools.ietf.org/html/rfc2671
[+] [-] tedk-42|6 years ago|reply
End users switching to Cloudflare's DNS endpoint are doing so because they feel the DNS provider is both faster and more secure.
They rightly made the decision NOT to pass on the end user's IP information to the upstream DNS server. I agree with this decision and they are acting in my best interests in doing so. To draw some kind of nefarious intention from this is absurd.
Until Cloudflare are proven to be nefarious actors, I'll continue to use their service.
[+] [-] oarsinsync|6 years ago|reply
In this instance, the upstream DNS server and the resultant HTTP server are operated by the same organisation. Cloudflare have opted to not provide the /24 (or /56 if IPv6) network that the original DNS request came from, in the DNS request. Your computer will then provide the /32 (or /128 if IPv6) that your request is coming from when you connect to the HTTP server.
What privacy win have you gained by Cloudflare not providing that information in this instance?
[+] [-] jiveturkey|6 years ago|reply
'Feel' being the keyword. Faster, generally yes. More secure, not well defined and users are generally wrong.
> nefarious intention
I don't believe I've heard any complaints of nefarious intent.
But let's be clear, this advantages Cloudflare over other CDNs. That they treat the DNS data very well does not mean they won't have an incident. As well, they are more of a target due to the concentration.
> Until Cloudflare are proven to be nefarious actors,
Nefarious wrt whom? For end-users taken individually, I agree, I don't see and it's hard for me to imagine mal intent.
But IMHO they are bad for the Internet. I mean, more power to them and were I a leader there I'd press the same agenda, but as a 3rd party, the way I see it is that in 10 years they are going to be an anti-power much like Google is. Addiction to their services will allow them to trample over what's good for all.
What I dislike most about them is that they promote themselves as purely a force for good. Except for a few PMs and execs I'm 100% sure they believe it. But it's a disservice to never discuss the negative aspects of any of their services. And woe to anyone who does.
As for proven nefarious deeds, do you not consider "banning" sites from using CF nefarious? What if they take it to the next step now, and stop providing DNS for those sites? Given their stated reason for bans, yes it could happen. Why must you wait until they prove to be nefarious? The concentration of power per se is a bad thing.
[+] [-] varelaz|6 years ago|reply
[+] [-] darklajid|6 years ago|reply
Maybe there's a big backstory here, but without context that seems passive-aggressive and quite random?
[+] [-] profmonocle|6 years ago|reply
[+] [-] spzb|6 years ago|reply
[+] [-] jchw|6 years ago|reply
I’m also surprised that traffic from Cloudflare DNS users caused any significant problem. Was it really that much traffic?
[+] [-] profmonocle|6 years ago|reply
It's not. The proof is that CDNs existed long before edns-client-subnet was introduced. All it does is allow the CDN's DNS servers to return the most optimal A/AAAA records for the client. But the worst that should happen without it is you get sent to a more distant CDN server, and the content loads more slowly.
The fact that archive.is somehow suffers without this feature (which, btw, wasn't standardized until 2016) suggests they're doing something really, really odd. If I were them, I'd focus on making my system more robust, rather than demanding the rest of the Internet adopt a relatively young, optional DNS extension.
[+] [-] jlokier|6 years ago|reply
Anycast IP is very expensive, unfortunately. Just getting a /22 has been expensive for years, and is now also getting difficult as well. It is beyond the reach of smaller companies.
GeoDNS is extremely cheap in comparison. You can run distributed services using GeoDNS for low latency on multiple continents on a hobby budget these days.
Anycast is technically better in many ways (the combination of anycast and geoDNS is better again), but anycast is so expensive that smaller operators just can't use it.
These days, smaller operators can use Cloudflare for their CDN, and the suspicious mind might think that suits Cloudfare just fine. But that doesn't really help for low-latency interactive services, or non-HTTP services.
> I’m also surprised that traffic from Cloudflare DNS users caused any significant problem.
Maybe the problem isn't amount of traffic, but rather that the site doesn't want to gain a reputation as slow (and therefore incompetently administered, and offputting to use) when everyone running Firefox switches over to 1.1.1.1 DoH automatically.
[+] [-] tomp|6 years ago|reply
https://twitter.com/archiveis/status/1018691421182791680
> Absence of EDNS and massive mismatch (not only on AS/Country, but even on the continent level) of where DNS and related HTTP requests come from causes so many troubles so I consider EDNS-less requests from Cloudflare as invalid.
[+] [-] viraptor|6 years ago|reply
Does anyone know what they could mean here? I get that having more open connections and slow requests is not great, but there are popular attacks people will try against them in this case. They already have to handle pathologic cases of slow requests, so handling some small number of slower clients shouldn't be an issue.
Or are they talking about some other problem?
[+] [-] miyuru|6 years ago|reply
Just try one of the akamai endpoints to test it. (E.g media.steampowered.com)
For me 1.1.1.1 serves akamai singapore IPs, while 8.8.8.8 serves IPs of my ISPs akamai cache in Sri Lanka.
If your ISP has a bad route to 1.1.1.1, this just gets worse.
[1] https://en.wikipedia.org/wiki/GeoDNS
[+] [-] bastawhiz|6 years ago|reply
[+] [-] jiveturkey|6 years ago|reply
Please can you rephrase your argument. 100% serious, I'd like to know what point you're making.
Pre-emptively: because whatever DNS server you are using already knows your IP address, regardless whether it's the first query for the site itself, or subsequent queries for site-related additional resources.
[+] [-] ggm|6 years ago|reply
the fact that I subsequently connect to another place over HTTP or some other protocol is distinct from telling a DNS authority who is asking a question about a domain name: the article implies "its the same leakage" but it isn't: different people get told.
[+] [-] cnst|6 years ago|reply
[+] [-] cm2187|6 years ago|reply
[+] [-] tomatocracy|6 years ago|reply
Many browsers prefetch DNS for links on webpages these days. So it’s entirely possible and even common that you may query DNS for sites you never visit, which would indeed be a privacy leak.
Secondly, many sites have their DNS hosted elsewhere so it may not be the same people you are leaking the information to.
Thirdly, if the DNS query is transmitted to the site’s DNS servers in plain text (which most DNS is), then despite eSNI etc anyone who has access to the wire traffic along the route from the DNS proxy to the site’s DNS servers (which is probably different from the route your own traffic takes to their servers) can see which site you wanted to access.
[+] [-] unknown|6 years ago|reply
[deleted]
[+] [-] chaitanya|6 years ago|reply
If the answer to these questions is no, then Cloudflare’s reasons for blocking ECS (ie privacy) carry weight. Otherwise no.
[+] [-] jgrahamc|6 years ago|reply
No
> More generally, does 1.1.1.1 in any way treat Cloudflare’s own nameservers in a special way and send it information that it doesn’t send to others?
No
[+] [-] aeyes|6 years ago|reply
ECS doesn't even forward the IP, only the /24.
[+] [-] PeterStuer|6 years ago|reply
https://news.ycombinator.com/item?id=19828317
[+] [-] varelaz|6 years ago|reply
If several bigger CDNs like akamai or softlayer will consider requests from 1.1.1.1 without EDNS as invalid and block them, Clouldflare wouldn't be able just to say that it's their own problems
[+] [-] unknown|6 years ago|reply
[deleted]
[+] [-] Santosh83|6 years ago|reply
[+] [-] Operyl|6 years ago|reply
[+] [-] unknown|6 years ago|reply
[deleted]
[+] [-] unknown|6 years ago|reply
[deleted]
[+] [-] vesche|6 years ago|reply
[+] [-] known|6 years ago|reply
[+] [-] known|6 years ago|reply
It works;
[+] [-] LeoNatan25|6 years ago|reply
[+] [-] nobodyshere|6 years ago|reply
[+] [-] pnako|6 years ago|reply