Am I the only one who is getting fed up with everything under the sun having their own DNS resolver and DNS cache? It's border-lining madness right now.
Before you had a situation like this. ISP: Server and cache. Router: Client, server and cache. OS: Client and cache.
So if you ever encounter DNS failing, you only had 3 possible places to look for errors, or maybe not even errors, just stale data that may or may not correct itself over time. Flush the caches you can and hope for the best. Rinse and repeat.
But then came the browsers and decided that 3 levels of caching was good enough. Let's add more which can fail in the name of performance!
Combine this with mobile, mobility and data-roaming and now you not only have a DNS-cache (based on already tripple-cached data) which often is guaranteed to be wrong, your browser may actually have cached the wrong DNS-servers, so when you roam from Wifi to cell-data your precious LAN DNS servers are no longer available and your browser just looks stupid.
And now systemd adds to this. What the flying fuck?
All this nonsense because someone somewhere decided that a nice, shared, global OS level client and cache wasn't good enough. Handling this complexity once would just be too easy.
Can someone tell me why? Why wasn't it good enough? What was it that prompted someone to go reimplement all this wrongly, yet again?
Yes this is a rant, but if there are underlying technical reasons for this nonsense, I'm very, very interested to hear about it.
The short answer is: the glibc DNS interface sucks balls and systemd wanted extra features.
The long answer is:
Often, systems do not come feature-complete. There are many cases of a platform, or OS, or service, or framework, or some other piece of software being released without every possible imaginable feature being implemented. When this happens, some other software may need to re-implement some core piece of technology in order to add functionality to it. Usually care is taken to not interfere with other systems or cause additional burden to the user. In the case of DNS, our use of DNS and different networks simultaneously requires additional functionality in order to process requests independently and efficiently. As the DNS protocol itself has an inherent flaw allowing for cache poisoning, it's non-trivial for people to implement without running into this security hole. And as someone else mentioned, the kernel and glibc don't have their own DNS cache.
I didn't downvote you for the rant because you weren't being mean or insulting and you genuinely seem to want to know what's going on. But next time you feel yourself posting out of emotional reaction, I recommend typing it out in a draft and save it for a half hour, and see if you might want to reword it later to be less angry/ranty.
> And now systemd adds to this. What the flying fuck?
While I agree with you that these layers of caching can lead to headaches, Linux doesn't have a built-in cache (and distros often don't provide one by default). Systemd would be one way to provide this, instead of other daemons that have been used in the past, like nscd, dnsmasq or bind. So it's not adding yet another layer, just providing the layer at the local host.
As for technical reasons, I'm not sure what you want to hear, each layer implements a cache for performance and/or efficiency. No one throws in a DNS cache just because they feel like it.
Actually Unix out of the box (unlike for example windows) does not have system level dns cache, people often setup things like bind, unbound, dnsmasq, nscd etc.
Systemd here tries to be the official standard system level caching.
I agree though, about your extra caching sentiment to a degree. Actually as long the caches are RFC complaint (e.g. they obey TIL - looking at you nscd & java) or should be fine.
Every client operating system under the sun implements a DNS cache, its part of the intended design of DNS. This is just systemd's bikeshed of a function already available on Linux, via nscd.
Without caching, all DNS queries degrade to worst case behaviour, which is a recursive query involving 4-6 or more round trips, starting at the DNS roots (which, good luck finding them, since you don't have their addresses cached, which is exactly what the root hints file is).
As for invalidating the cache when network configuration changes arise, that sounds like a bug report for wherever you experienced it. Certainly I've never seen this kind of problem on OS X, nor any time in the past 10 years on Windows.
Yes, that's the full commit message for what I think is the commit that adds the feature. I was trying to dig up an authorative source for the "why" you're asking, but I guess that failed. Perhaps it was discussed/motivated/described/something somewhere else, like a mailing list, I didn't search further.
But the solution is not to complain, but to use /etc/hosts, run your own root and your own cache.
Any software that overrides /etc/hosts is not worth using, in my opinion. There are no "underlying technical reasons" for not letting the user control their own name resolution that benefit the user. Of course, there could be reasons that benefit someone else.
I find the design of systemd-resolved to be very strange. It uses dbus to talk to glibc, and it seems to be a new, from-scratch implementation of a DNS resolver. To be clear, I don't really think it matters whether systemd-resolved is under the systemd umbrella, but I do think that the design has a lot of unnecessary NIH syndrome.
It turns out that there's a very well-specified protocol by which clients can ask a local cache on their system to answer DNS queries. That protocol is called DNS :) I don't see why routing something DNS-like over dbus makes any sense in contrast to doing it using DNS itself on port 53.
Fedora is experimenting with running unbound as a local caching resolver [1]. This gives caching, DNSSEC validation, and all the benefits from the fact that unbound is probably much better hardened than the average libc or application-side DNS client implementation.
This is a perfect example of why the systemd approach of putting a bunch of disparate components under a single tightly-coupled umbrella is bad engineering. Generally, projects focused on doing one thing are able to do it well, while projects that attempt to do many things have a hard time doing any of them well. Unfortunately, as the steady stream of cache poisoning vulnerabilities in other DNS servers have demonstrated, implementing DNS securely is really hard. A less-than-perfect attempt at implementing a caching DNS resolver will be insecure. For this reason, it's a task best left to a mature project run by people with DNS-specific experience who can focus on doing that one thing well.
> This is a perfect example of why the systemd approach of reinventing a bunch of existing components under a single tightly-coupled umbrella is a bad approach.
Why? This wasn't caused by systemd's complexity or having too much components interconnect poorly. It was a bug in one of their implementations that could have equally happened had it been a stand-alone DNS cache resolver.
> Generally, projects focused on doing one thing are able to do it well, while projects that attempt to do many things have a hard time doing any of them well.
I'd totally buy that if BIND didn't have such a checkered history. I mean with BIND 9 they scrapped the entire project and started again because it was so insecure...
Yet they're only attempting to do "one thing" so according to you they should have done it well.
> Unfortunately, as the steady stream of cache poisoning vulnerabilities in other DNS servers have demonstrated, implementing DNS securely is really hard.
This line oddly contradicts the line that directly proceeds it.
> For this reason, it's a task best left to a mature project run by people with DNS-specific experience who can focus on doing that one thing well.
According to the previous line they aren't doing that one thing well...
Pretty weird for Systemd to have its own cache and resolver. If you need a local resolver and a cache simply run a local DNS server rather than to re-invent the wheel (in a broken way at that). I'm sure they have their reasons but this is not the unix way.
Systemd is just not the Unix Way. It's some shit program that requires to be run as PID 1, and takes over the system in the forms of an octopus. And, you need some gods-awful binary program to even read their logfiles. Yes, it's like the Windows registry.
The submitter doesn't state affected versions, but judging from systemd's NEWS file, the resolved component was first introduced in 213 and later dubbed as "a pretty complete caching DNS and LLMNR stub resolver" in 216: http://lwn.net/Articles/609740/
Considering most users run on a backported 208-stable IIRC (I know RHEL 7 settled on that), I'm not sure how widespread this is. Still concerning that they felt to roll their own and not follow RFC guidelines, though.
Considering how plagued with vulnerabilities BIND was, I'd assume DNS is a hard thing to do.
> "a pretty complete caching DNS and LLMNR stub resolver"
That quote is gold. Most of DNS isn't implementing the protocol, but implementing all the protections against cache poisoning, many of which are not intuitive.
But as the oss-sec email says: "systemd-resolved does not implement any of the hardening recommendations of rfc5452."
Of all the components of systemd that doesn't need to exist, resolved is definitely one of them. Why they're reinventing the wheel rather than allowing the user to set up something that works and is battle-tested, such as unbound, is beyond me.
I don't necessarily disagree with the heart of your statement, but it's worth mentioning that systemd does not require one to use resolved. The user is still "allowed" to run unbound, dnsmasq, etc., and the preferred solution will be a question for distribution maintainers and users to answer.
I understand concerns about systemd's scope, but most of the "questionable" functionality is optional.
Systemd... a very large mass of c code being written in a short amount of time with huge functionality creep. Yeah, this is going to end/begin really well. I swear all of the people commenting about it being a good thing are not seasoned developers.
[+] [-] josteink|11 years ago|reply
Before you had a situation like this. ISP: Server and cache. Router: Client, server and cache. OS: Client and cache.
So if you ever encounter DNS failing, you only had 3 possible places to look for errors, or maybe not even errors, just stale data that may or may not correct itself over time. Flush the caches you can and hope for the best. Rinse and repeat.
But then came the browsers and decided that 3 levels of caching was good enough. Let's add more which can fail in the name of performance!
Combine this with mobile, mobility and data-roaming and now you not only have a DNS-cache (based on already tripple-cached data) which often is guaranteed to be wrong, your browser may actually have cached the wrong DNS-servers, so when you roam from Wifi to cell-data your precious LAN DNS servers are no longer available and your browser just looks stupid.
And now systemd adds to this. What the flying fuck?
All this nonsense because someone somewhere decided that a nice, shared, global OS level client and cache wasn't good enough. Handling this complexity once would just be too easy.
Can someone tell me why? Why wasn't it good enough? What was it that prompted someone to go reimplement all this wrongly, yet again?
Yes this is a rant, but if there are underlying technical reasons for this nonsense, I'm very, very interested to hear about it.
[+] [-] peterwwillis|11 years ago|reply
The long answer is:
Often, systems do not come feature-complete. There are many cases of a platform, or OS, or service, or framework, or some other piece of software being released without every possible imaginable feature being implemented. When this happens, some other software may need to re-implement some core piece of technology in order to add functionality to it. Usually care is taken to not interfere with other systems or cause additional burden to the user. In the case of DNS, our use of DNS and different networks simultaneously requires additional functionality in order to process requests independently and efficiently. As the DNS protocol itself has an inherent flaw allowing for cache poisoning, it's non-trivial for people to implement without running into this security hole. And as someone else mentioned, the kernel and glibc don't have their own DNS cache.
I didn't downvote you for the rant because you weren't being mean or insulting and you genuinely seem to want to know what's going on. But next time you feel yourself posting out of emotional reaction, I recommend typing it out in a draft and save it for a half hour, and see if you might want to reword it later to be less angry/ranty.
[+] [-] ominous_prime|11 years ago|reply
While I agree with you that these layers of caching can lead to headaches, Linux doesn't have a built-in cache (and distros often don't provide one by default). Systemd would be one way to provide this, instead of other daemons that have been used in the past, like nscd, dnsmasq or bind. So it's not adding yet another layer, just providing the layer at the local host.
As for technical reasons, I'm not sure what you want to hear, each layer implements a cache for performance and/or efficiency. No one throws in a DNS cache just because they feel like it.
[+] [-] takeda|11 years ago|reply
Systemd here tries to be the official standard system level caching.
I agree though, about your extra caching sentiment to a degree. Actually as long the caches are RFC complaint (e.g. they obey TIL - looking at you nscd & java) or should be fine.
[+] [-] _wmd|11 years ago|reply
Without caching, all DNS queries degrade to worst case behaviour, which is a recursive query involving 4-6 or more round trips, starting at the DNS roots (which, good luck finding them, since you don't have their addresses cached, which is exactly what the root hints file is).
As for invalidating the cache when network configuration changes arise, that sounds like a bug report for wherever you experienced it. Certainly I've never seen this kind of problem on OS X, nor any time in the past 10 years on Windows.
[+] [-] unwind|11 years ago|reply
resolved: add DNS cache
Yes, that's the full commit message for what I think is the commit that adds the feature. I was trying to dig up an authorative source for the "why" you're asking, but I guess that failed. Perhaps it was discussed/motivated/described/something somewhere else, like a mailing list, I didn't search further.
[+] [-] 101914|11 years ago|reply
No.
But the solution is not to complain, but to use /etc/hosts, run your own root and your own cache.
Any software that overrides /etc/hosts is not worth using, in my opinion. There are no "underlying technical reasons" for not letting the user control their own name resolution that benefit the user. Of course, there could be reasons that benefit someone else.
[+] [-] jacquesm|11 years ago|reply
[+] [-] zurn|11 years ago|reply
[+] [-] amluto|11 years ago|reply
It turns out that there's a very well-specified protocol by which clients can ask a local cache on their system to answer DNS queries. That protocol is called DNS :) I don't see why routing something DNS-like over dbus makes any sense in contrast to doing it using DNS itself on port 53.
Fedora is experimenting with running unbound as a local caching resolver [1]. This gives caching, DNSSEC validation, and all the benefits from the fact that unbound is probably much better hardened than the average libc or application-side DNS client implementation.
[1] http://fedoraproject.org/wiki/Features/DNSSEC_on_workstation...
[+] [-] digi_owl|11 years ago|reply
Hell, i have recently seen a fedora bug regarding the use of su -. Where Poettering argued that people should not use su because it broke dbus.
Instead he seemed to advocate that people ssh into 127.0.0.1 to do their thing with a different account.
[+] [-] agwa|11 years ago|reply
[+] [-] Someone1234|11 years ago|reply
Why? This wasn't caused by systemd's complexity or having too much components interconnect poorly. It was a bug in one of their implementations that could have equally happened had it been a stand-alone DNS cache resolver.
> Generally, projects focused on doing one thing are able to do it well, while projects that attempt to do many things have a hard time doing any of them well.
I'd totally buy that if BIND didn't have such a checkered history. I mean with BIND 9 they scrapped the entire project and started again because it was so insecure...
Yet they're only attempting to do "one thing" so according to you they should have done it well.
> Unfortunately, as the steady stream of cache poisoning vulnerabilities in other DNS servers have demonstrated, implementing DNS securely is really hard.
This line oddly contradicts the line that directly proceeds it.
> For this reason, it's a task best left to a mature project run by people with DNS-specific experience who can focus on doing that one thing well.
According to the previous line they aren't doing that one thing well...
[+] [-] erhardm|11 years ago|reply
Do one thing and do it well.
[+] [-] unknown|11 years ago|reply
[deleted]
[+] [-] jacquesm|11 years ago|reply
[+] [-] kefka|11 years ago|reply
Even Debian has succumbed to Systemd.
[+] [-] pjc50|11 years ago|reply
[+] [-] userbinator|11 years ago|reply
[+] [-] marcosdumay|11 years ago|reply
[+] [-] digi_owl|11 years ago|reply
[+] [-] vezzy-fnord|11 years ago|reply
Considering most users run on a backported 208-stable IIRC (I know RHEL 7 settled on that), I'm not sure how widespread this is. Still concerning that they felt to roll their own and not follow RFC guidelines, though.
Considering how plagued with vulnerabilities BIND was, I'd assume DNS is a hard thing to do.
[+] [-] agwa|11 years ago|reply
That quote is gold. Most of DNS isn't implementing the protocol, but implementing all the protections against cache poisoning, many of which are not intuitive.
But as the oss-sec email says: "systemd-resolved does not implement any of the hardening recommendations of rfc5452."
[+] [-] talideon|11 years ago|reply
[+] [-] JeremyNT|11 years ago|reply
I understand concerns about systemd's scope, but most of the "questionable" functionality is optional.
[+] [-] SixSigma|11 years ago|reply
By the same author
"There's no such thing as a simple cache bug."
"Caches aren't architecture, they're just optimization."
[+] [-] tedunangst|11 years ago|reply
[+] [-] 4ydx|11 years ago|reply