In part one of this rant, I talked briefly of DNS, why we use it, its history, and proceeded to be an old man yelling at the cloud; likening the cloud services culture of today to the good old days of the 70’s and 80’s in which services were centrally located, and every action you performed was billable.
I also briefly discussed how this consolidation of computing power, resources, and network services is concerning in that there is this illusion that trusting these large companies whose business model is more or less selling your information and/or providing services to fuckin’ anyone who gives them money is somehow alloting you security and privacy in spite of the fact that all of the big players providing DNS service are recording your queries in some way, either for their own edification, or for finding malware. While its a net positive to have your DNS provider looking out for you and not allowing you to resolve malicious domains, privacy of your DNS resolutions, and the provider with less than pure intentions monitoring your DNS queries to “protect” you are, in my mind, mutually exclusive.
That brings me to one of the final points I made in the last post: DNS over HTTPS is more or less a message from these providers, and browser developers that ‘we more or less shape the internet now, whether or not you like what we provide, you have no choice but to accept it if we choose to foist it upon you.’
Not only that, they’re more or less letting you know that its okay for them to have your queries because they ‘ReSpEcT YoUr PrIvAcY’, until they don’t. Its okay for google and cloudflare to have your DNS queries, but god forbid your network admin and security team, those goddamn people who are responsible for making this pile of computers , network gear, and cat5 fucking work have access to a vital data source for network troubleshooting and identifying potential security threats.
Getting to Know You
So this is where we pick back up. Welcome to this brave, new world of DNS over HTTPS. I think its time that we got familiar with one another, and learned a little bit more about one another, don’t you agree?
The EFF has come out in support of DoH, championing it as a privacy solution and well, I think a good focus for this blogpost is tearing that decision apart as short-sighted. The majority of this post is going to be spent analyzing this article, providing couter-arguments against it, and talking about the DNS over HTTPS RFC itself.
Now, the first thing I’ll say is that the uptick in use of SSL/TLS thanks to Let’s Encrypt is pretty cool. 3/4ths of the internet loads content over SSL now, and that is a massive change that has happened in less than a decade. I’ll also say that Lets Encrypt has its problems (namely, the problem of “Oh, well this SSL certificate is being abused by this domain? Lol, thats not my fucking problem. I don’t care about revoking abused SSL certs. “), But that is a discussion for another time.
But DNS, the system that looks up a site’s IP address when you type the site’s name into your browser, remains unprotected by encryption.
Because of this, anyone along the path from your network to your DNS resolver (where domain names are converted to IP addresses) can collect information about which sites you visit. This means that certain eavesdroppers can still profile your online activity by making a list of sites you visited, or a list of who visits a particular site. Malicious DNS resolvers or on-path routers can also tamper with your DNS request, blocking you from accessing sites or even routing you to fake versions of the sites you requested.
https://www.eff.org/deeplinks/2019/09/encrypted-dns-could-help-close-biggest-privacy-gap-internet-why-are-some-groups
The first thing the EFF does is inform you that because DNS isn’t encrypted, that your DNS traffic can be intercepted, and you can be redirected to malicious sites without knowing about it. Yeah. That’s a problem for sure. However, DoH would do very little to resolve that problem other than make to where if your DNS is being hijacked, or actively being redirected to another IP address, the only party that would know about it would be the DoH provider.
Lets go back to 2012-2013, when we were first notified that SSL MITM was occuring at service providers all over the place.
Nobody had any fucking idea that this was happening. What makes you think that it wouldn’t be the same old song for your favorite conglomerate DoH provider? You’re placing an awful lot of trust into corporations and enterprises who, so far as you know, are willing to betray that trust for a couple of dollars and/or if the law demanded it. Remember: Nobody is going to jail for you, so if the FBI, DoJ, or your country’s equivalent rolls up and says “Do this thing for me, or we’re going to throw you in a hole.” You’re probably gonna do it.
You’re probably thinking to yourself that there’s something built into DoH that would prevent tampering of DNS records or sneaky shit like what I’m implying above, and well… you’d be incorrect, unfortunately.
Now, I know I just linked you to an RFC document, and I know that you’re eager to read dry, technical documentation, but you don’t have to read very far into the RFC to figure out that Integrity or Confidentiality wasn’t really built into the protocol, That they aren’t doing any sort of integrity-checking on the server-side or anything, that they’re just relying on TLS to provide Confidentiality and Integrity in transit. Once your queries reach the server, thats it. We don’t care if you’re logging, what you’re logging, or what gets modified.
sending DNS [RFC1035] queries and getting DNS responses over HTTP [RFC7540] using https [RFC2818] URIs (and therefore TLS [RFC8446] security for integrity and confidentiality). Each DNS query-response pair is mapped into an HTTP exchange.
The described approach is more than a tunnel over HTTP.
[…]
Two primary use cases were considered during this protocol’s development. These use cases are preventing on-path devices from interfering with DNS operations, and also allowing web applications to access DNS information via existing browser APIs in a safe way consistent with Cross Origin Resource Sharing (CORS)
https://tools.ietf.org/html/rfc8484#section-1
So they’re relying on a proper TLS configuration, and encrypted SNI to provide the Confidentiality and Integrity, and then only in transit. There is no magic here and this is one of the things that pisses me off the most about this protocol: Its piggybacking over HTTPS because you really don’t have the option of “blocking 443” out of your network without disabling about 2/3rds of the web. This is what we call a catch-22.
You’re providing what you perceive as a massive leap in consumer privacy, at the cost of rest of us having no playbook or method of blocking or defending against abuses of DNS over HTTPS. We’ve already seen at least two malware campaigns utilizing DoH for various purposes. Considering that we’re already seeing malware campaigns utilizing the protocol to hide their comms, how long until we see security researchers jerking themselves off on stage at $REDTEAMCONFERENCE that they’ve updated updated/forked, say, dnscat2 for DNS over HTTPS support? After all, if you aren’t emulating threats, then threats aren’t emulating you.
The same sort of shit happened with tor when it was announced: People jerked themselves off over it being a massive leap in consumer privacy on the internet, but as it turns out, running a hidden service is a bit harder than people think. Oh and of course, just like DoH it is also regularly abused for malware campaigns. The big difference here is that if I don’t want tor running on network and assets I’m responsible for, its easy to block and control because it isn’t a unique protocol trying to piggyback on the same port as an established protocol. Well, not yet anyway.
I’m starting to go off the rails a little bit. As some point in the future, I plan on talking about how enterprise defense playbooks for DNS over HTTPS are woefully inadequate, and that every solution or mitigation you can use is, at this point, a stopgap measure. But that probably won’t be until part 3. For now, lets continue focusing on the EFF’s endorsement of DoH. Lets take a look at another excerpt:
Members of civil society have also expressed concerns over plans for browsers to automatically use specific DNS resolvers, overriding the resolver configured by the operating system (which today is most often the one suggested by the ISP). This would contribute to the centralization of Internet infrastructure, as thousands of DNS resolvers used for web requests would be replaced by a small handful.
That centralization would increase the power of the DNS resolver operators chosen by the browser vendors, which would make it possible for those resolver operators to censor and monitor browser users’ online activity.
https://www.eff.org/deeplinks/2019/09/encrypted-dns-could-help-close-biggest-privacy-gap-internet-why-are-some-groups
This is a very valid concern and fits in remarkably well with the first blog I wrote regarding this subject: We’re consolidating computing resources and services into the hands of a few players. So whats the solution?
This capability prompted Mozilla to push for strong policies that forbid this kind of censorship and monitoring.
https://www.eff.org/deeplinks/2019/09/encrypted-dns-could-help-close-biggest-privacy-gap-internet-why-are-some-groups
The EFF is all like “Just promise not to fuck over people, and everything will be fine.” I don’t have words for how fucking naive this is. I mean, lets not forget that these are the same companies who have been invading your privacy for years, sold out to the Intelligence Community, and sold out to foreign powers in spite of terrible human rights violations because it means a bigger marketshare. Lets remember that these are the same companies that think silencing hate groups counts as silencing free speech. I don’t have any words for this “Just trust them, bro.” approach they got going on, but lets continue.
to avoid having this technology deployment produce such a powerful centralizing effect, EFF is calling for widespread deployment of DNS over HTTPS support by Internet service providers themselves.
https://www.eff.org/deeplinks/2019/09/encrypted-dns-could-help-close-biggest-privacy-gap-internet-why-are-some-groups
Aw dude, its in bold-faced font so you know this is important. This statement is more or less the EFF dictating to companies “Run this thing because we say so.”
While this solves the problem of only handful of large corporations having access to your DNS over HTTPS queries, there is absolutely no incentive for any of these ISPs to adopt the standard. Not only that, it doesn’t resolve the problem of your DoH provider logging your queries, and/or your ISPs allegedly selling your web browsing and DNS data to whoever they want to sell it to. It also doesn’t solve the issue of ISPs hijacking NXDOMAIN responses and pointing you to a search page/adware instead of letting your queries fail. This doesn’t solve the potential issue of a government or law enforcement entity requesting access to DoH queries or otherwise tampering with DNS results. I’m sorry, how the fuck is this service a privacy win again?
The next paragraph is then the EFF jerking themselves off because the CTO of an ISP wants to run a DoH resolver. The blog post ends with the EFF being super happy about DoH and issuing another warning/demand:
Browsers must be transparent about who will gain access to DNS request data and give users an opportunity to choose their own resolver.
https://www.eff.org/deeplinks/2019/09/encrypted-dns-could-help-close-biggest-privacy-gap-internet-why-are-some-groups
So uh. I have some unfounded thoughts about that. Remember above when I pointed out the two uses cases of DoH were to prevent query intercept/tampering and to expose DNS queries to the browser API? Why do you think they want to expose DNS to the browser API? I’m going to do something I’m pretty good at, and I’m going to speculate.
Ads and ad delivery networks are pretty lucrative. One of google’s primary business models is delivering ads. Not to mention their response to some of their platforms (e.g. youtube) having excessive ads is to just shovel more down your throat. After all, what choice do you have you plebeian? Surely you won’t block our ads, will you? How will Pewdiepie live without your ad money, you heartless bastard??
The thing is, a lot of people elect to block ads because yes, they believe that is a valid response to the excessive number of ads being shoveled down our throats, not to mention the very real threat of how a lot of ad exchanges don’t police their content and don’t actually give a shit about malvertising. There are those among us who see ads as a form of remote code execution that is not being safeguarded and policed very well.
So in direct response to this blase attitude towards their userbase, a variety of ad-blocking and anti-tracking solutions have been created. This impacts ad revenue, because if you’re not seeing or clicking on ads, nobody is making money. This gives people who buy ads, and companies that sell ads a massive sad. ublock origin, umatrix, noscript, privacy badger (an EFF product no less, showing that I can hate their opinion on shit and yet respect their overall mission), pi-hole, etc. People are sending a very clear message that ad delivery needs to change, and that they don’t want to be tracked or have “tailored” ads delivered to them, and its a message that companies like Google are choosing to ignore.
Now, I’m going to ask you again. Why do you think companies and organizations that support DNS over HTTPS support having DNS directly accessible via the browser APIs? If you answered “Probably because they want to enable more granular tracking, and counter anti-adware solutions” then congratulations, you get a cookie. I can see this being used as wall to preventing access to content. If you fail to resolve/display ads on the page, do not allow that client access until they enable DNS over HTTPs or some such nonsense.
Additionally, I want you to think about how easy it is for companies to fingerprint your web browser today, here and now. Now, think about this: When you do standard recursive DNS and your DNS query recurses through the DNS infrastructure, the source IP address making the DNS query you originally made changes. Its pretty hard for me to narrow down what client made a particular DNS request, which is why its usually a good idea to pair DNS logs with netflow for helping to narrow things down. While DNS over HTTPS allows you Confidentiality and Integrity for your DNS query in transport, consider the fact that your DoH is able to track you by your browser fingerprint. All of you are unique. All of you are potentially trackable. This is a potential ad delivery and analytics gold mine that is probably giving a lot of these companies a massive boner, because they can now track you like you’ve never been tracked before.
Also, congratulations, two of the most popular web browsers in the world will be shipping with this feature enabled by default. We more or less have a browser monoculture at this point, with Microsoft abandoning their own browser platform and opting to make something chromium based. Bringing us back to the point I made in the first post of this series in that a few organizations now decide what the internet is, and fuck you if you don’t like it. Remember when shit that changed your DNS and proxy settings was considered malware? da_667 remembers.
Personally, I can’t wait for the first piece of malware that covertly enables DNS over HTTPS and redirects it to toteslegit[.]ru and/or hijacks your DNS over HTTPS settings covertly. Its going to be a joy to find and remediate that. Remember that malware doesn’t care about your company’s stance or security policies. Even if you block or disable DoH usage in applications, there is probably nothing to stop malware from using it however it sees fit.
All of this talk about whether or not DoH is privacy enhancing/respecting or not, and we haven’t even gotten into the performance aspects and considerations of it.
Now, DoH uses HTTP as a transport. HTTP/HTTPS are TCP-based technologies. It is SSL/TLS wrapped. Most DoH providers encode their requests and responses as JSON. And of course, its all SSL-wrapped. Not only that, there aren’t any /local/ DoH providers. All of this means significant increases in the amount of time it takes for queries to be made, and responses to be returned, even assuming that caching DoH is possible, and done properly. The RFC makes specifications for caching, but the protocol is wrapped in SSL/TLS: Is your proxy able to process HTTPS requests, if so, the privacy advocates are probably screeching at you that certificate pinning and TLS 1.3 are an end to SSL inspection, but don’t get ahead of me, We’ll talk about that in the next post. STAY FOCUSED. We’re talking about PERFORMANCE here. Lets assume you /do/ have a proxy, but you can’t do HTTPS proxying.
So here are all of the areas in which you have to worry about overhead now:
-Establishing a TCP connection. Three way handshakes and session establishment are expensive. While traditional DNS can and does use TCP for large DNS requests, most queries/responses are UDP because they’re small, and its fire and forget.
-Formatting your DNS request to follow HTTP/HTTP 2.0 protocol specifications. True, DNS has its own protocol specifications, but HTTP requests are very finicky about where headers are and what types of requests you send. DNS doesn’t give a shit about that. Here are some bits and bytes that define what type of request it is, and what features you want enabled. Now you have to vomit that into an HTTP request, and not only that, you have to format it as a JSON request, adding more overhead to parse the request/response. Hell, the RFC itself recommends against using anything less than HTTP2 for DNS over HTTPS comms because of the poor performance using earlier versions of the HTTP protocol promise:
HTTP/2 [RFC7540] is the minimum RECOMMENDED version of HTTP for use with DoH. The messages in classic UDP-based DNS [RFC1035] are inherently unordered and have low overhead. A competitive HTTP transport needs to support reordering, parallelism, priority, and header compression to achieve similar performance. Those features were introduced to HTTP in HTTP/2 [RFC7540]. Earlier versions of HTTP are capable of conveying the semantic requirements of DoH but may result in very poor performance.
https://tools.ietf.org/html/rfc8484#section-5.2
Theres a lot of complexity and weird shit you’re asking HTTP/2 to do in order to approach the performance that just using UDP provided. (To be fair, UDP as a transport has a host of its own problems, mainly related to Denial of Service attacks) And as we all know, the more complexity you add to a software product the more likely you are to create places where developers interpret the spec differently and allow for programming errors. That means bugs and vulns.
-Its wrapped in SSL/TLS, meaning that those requests need to be decrypted, and responses need to be re-encrypted. Imagine being forced to do that for multiple small requests, repeatedly, over and over again, and all of the extra overhead that comes with it. It has all of the hallmarks of a performance nightmare.
Let’s come back to proxying really quick, because this is interesting. The RFC champions proxying/caching as a method to reduce the load and make DoH an easier pill to swallow. You can use a client’s web cache to reduce the number of requests they are making to a DNS over HTTPS provider, and the DoH provider can cache on their end to reduce their load, but what are network people supposed to do on their local networks? There is no option to set up a local DoH server to do recursive lookups and cache yet (maybe this will change at some point, but again we come back to the problems listed above with overhead, load, latency, performance, and now you have to throw SSL/TLS cert management into the mix.). So with no local DNS resolver options and a reliance on a handful of monolithic DoH providers, you are creating a massive DNS resolution point of failure. Again, remember when Dyn’s DNS servers went down in 2016 (because of the mirai botnet)? Remember what happened when cloudflare pushed a bad regular expression just this year? Most of the internet was down. If I sound like a broken record, its because I’m seriously concerned that a handful of organizations have such massive sway in how the internet works. Its pretty scary to think about.
I saw an article from 2018 when I googled for “dns over https performance” and well, I’d like you to read this:
However, even though GET requests are more cache-friendly than POST requests, there is still one problem. Usually, DNS packets have an ID field, that is used in order to correlate request and response packets. It is a unique, random identifier that results in different request URLs for what is essentially the same request. Therefore, clients should set this ID field to ‘0’.
https://dzone.com/articles/pros-and-cons-of-dns-over-https
So uh. the ID field is kind of important to DNS. Do you know why? Its to prevent spoofing and being able to easily MITM DNS requests. The ID field is randomized with each request to prevent hijacking the connection and replacing the DNS server’s response with a malicious one. This blog post is telling you in order to improve cache hits, just statically set it to zero.
The ID field is one of the few protections DNS is allotted, and you’re just telling people to ignore it so cached results can be returned more easily. However, you’re opening up DNS queries to being easily tampered with, depending on how this field is handled. What happens when the DoH provider has to recurse to a non DoH DNS server? Are you just going to set the ID field to 0 and just say fuck it? Are you just going to leave the ID field as 0 internally on the web server an make it easier for the DoH to modify the response just for shits and giggles? I’m just aghast by the security implications of this.
Sure you can probably scoff at this advice and say “its a developer blog, or blog website, or whatever.” And normally I’d be the one right next to you laughing it off, however, this was fifth google search result. Its a page 1 result. and DZone alleges that over 1 million developers have joined their network. Even if you and I know that this is a bad idea, perhaps a million others don’t. These people could be the ones who end up writing a DoH server/client implementation that does this. Thats fuckin’ bad news bears.
Oh, and if you want a second opinion on DoH performance, Go check this out. The long and short of it, is that DoH introduces delay. Might not be a lot, but consider DNS resolution is time-sensitive, every little bit counts.
Title of this post inspired by Alexisonfire – We Are The Sound