top | item 43229201

NIH.gov DNS servers down, making PubMed, BLAST, etc. unreachable [fixed]

494 points| raphman | 1 year ago |nslookup.io | reply

385 comments

order
[+] fweimer|1 year ago|reply
It's been pointed out that these servers still respond over TCP: https://mstdn.social/@rysiek/114089755401568345 https://lists.dns-oarc.net/pipermail/dns-operations/2025-Mar...

Given that the service is still partially operational (albeit not in a useful way), it's difficult to say from the outside what is going on.

[+] stwrzn|1 year ago|reply
This is for sure a firewall misconfiguration. If there would be malicious intent, the bad actor would for sure not just close UDP.
[+] raphman|1 year ago|reply
Can confirm.

Thanks for pointing this out. This makes it much more likely that someone messed up a server/firewall configuration than that Musk is ripping out network cables at NIH.

[+] thomasingalls|1 year ago|reply
Pubmed is one of the most reliable services in the history of the internet. Sure the exact issue is opaque, but the fish rots from the head.
[+] fabian2k|1 year ago|reply
Pubmed is essentially Google for scientists. Anytime you search for scientific publications you usually use Pubmed. Of course there are alternatives, but until now you didn't really have to know about those. Everyone just used Pubmed, I'd bet that even most European scientists didn't know the local alternatives until now.

And there's a lot more functionality made available to scientists by the NIH.

[+] rossant|1 year ago|reply
I almost never use pubmed, while I use Google Scholar everyday (for neuroscience/medical/computer science research). But I admit that all medical researchers I know only use pubmed.
[+] gary_0|1 year ago|reply
Apparently the FAA database that tracks accident investigations is down also, and probably a bunch of other systems that regular people aren't aware of that various organizations rely on.

If this was a Chinese cyberattack, it would be the scandal of the decade. But it's on purpose.

[+] belorn|1 year ago|reply
So looking at this from a technical point. NIH.gov has three name servers. Each host are still up, but only answering dns on TCP and not UDP. All three are located under the same AS, which implies that there is a single operator responsible. No ipv6. From the outside I can't see any sign that they have delegated operation of either the servers or the service to any external company.

Doing some more looking around, it seems like NIH has a department/group/structure called Center for Information Technology, which is the IT support side of NIH and are the operators for the DNS servers.

[+] gilleain|1 year ago|reply
For quick reference, BLAST refers to the 'Basic Local Alignment Search Tool' that's a commonly used part of the bioinformatics toolkit. You 'BLAST' sequences by sending a query sequence of interest against a database of other sequences to find similarity hits.
[+] bow_|1 year ago|reply
I have been out of the field for some time, so I am not sure how much BLAST is used these days.

Therer was a time when BLAST-ing a DNA and protein sequence you have is like doing a Google search on it: it simply tells you where the sequence might come from. This is useful especially when your research is to figure out what that specific sequence is doing. It won't give you the answer immediately (otherwise why bother doing the research at all), but it certainly gives context: sequence similarity often hints at similar / related functions.

As an analogy: imagine if StackOverflow is suddenly down and you don't know *if* it's going to be up again.

[+] 1vuio0pswjnm7|1 year ago|reply
DNS servers may be down temporarily, e.g., NIH is doing maintenance, but this is not making websites unreachable

Perhaps "unreachable" in the title is a figure of speech

I have no problem reaching these websites and can provide IPs to anyone who needs them

For example,

www.nih.gov 23.41.4.71 (Akamai)

www.nih.gov 2.22.31.155 (Akamai)

www.nih.gov 60.254.143.7 (Akamai)

www.nih.gov 95.101.74.96 (Akamai)

www.nih.gov 88.221.24.17 (Akamai)

www.nih.gov 184.51.148.226 (Akamai)

www.nih.gov 54.235.145.223 (Amazon)

pubmed.ncbi.nlm.nih.gov 34.107.134.59 (Google)

blast.ncbi.nlm.nih.gov 130.14.29.110

Usage example

echo 130.14.29.110 blast.ncbi.nlm.nih.gov|busybox sed -i -e 1r/dev/stdin -e1N /etc/hosts

echo 34.107.134.59 pubmed.ncbi.nlm.nih.gov|busybox sed -i -e 1/r/dev/stdin -e1N /etc/hosts

[+] jjallen|1 year ago|reply
FYI it has been like this for at least sixteen hours as last night I was trying to read something there too and it wouldn’t work. I hope there’s backups somewhere. I definitely wouldn’t feel bad about using non sanctioned sources at this point.
[+] raphman|1 year ago|reply
It seems someone shut down all NIH DNS servers. Right now the NIH website, PubMed and BLAST are up, but not resolvable.

Cloudflare's 1.0.0.1 DNS resolver seems to still have cached the records. Google and most others I tried did not. This probably explains why some people on social media could access the sites while others couldn't.

Workaround via /etc/hosts :

  156.40.212.210 nih.gov
  96.17.96.9     www.nih.gov
  34.107.134.59  pubmed.ncbi.nlm.nih.gov
  130.14.250.10  ftp.wip.ncbi.nlm.nih.gov
  130.14.250.10  ftp.ncbi.nlm.nih.gov
  130.14.29.110  blast.ncbi.nlm.nih.gov
EDIT:

While ns.nih.gov, ns2.nih.gov, and ns3.nih.gov do not respond, the nameserver at lhcns1.nlm.nih.gov (130.14.55.72) does.

Also see https://tldr.nettime.org/@ww/114089972404202687

[+] stwrzn|1 year ago|reply
They were not shut down.

Confirm yourself using:

  dig +tcp @$(dig +short ns.nih.gov @a.ns.gov) www.nih.gov
[+] anotherpaul|1 year ago|reply
Thank you for writing this down. Might become relevant if they don't solve that before Monday.

One can only hope this is not intentional.

[+] imhoguy|1 year ago|reply
Cool, thanks! While the chaos monkey is at work, the true ARPANET spirit is still alive!
[+] qgin|1 year ago|reply
At some point we have to consider the possibility that these aren’t just cuts to the budget, but an intentional plan to visit each government department and giving it a stab wound to let it bleed out and die over the course of the next year or two.
[+] ldoughty|1 year ago|reply
There's a proven track record with the Republican plan and USPS that they like to sabotage the agency and then point to the dysfunction they created as a reason to get rid of it. I fully suspect that we're going to hear that several agencies have been non-productive since January 20th and should thus be cut... Despite the fact that they BLINDLYfired not only the new employees, but all the employees that have had promotions over the last year as well (And were probationary in their new role).
[+] smallmancontrov|1 year ago|reply
If history is any indication, it won't even go towards fixing the debt. Capital Gains tax cut incoming in 3... 2... 1...
[+] someothherguyy|1 year ago|reply
It is well-known, but it seems like it is hip to be indifferent on this sinking ship.
[+] xg15|1 year ago|reply
I wonder how long until they pull the "who needs public libraries when you have Jeff Bezos?" shtick again.
[+] llamaimperative|1 year ago|reply
IMO we don’t need to attribute malice to it, which will drag everyone into a debate about what’s going on inside the clearly broken minds of a few rich men.

The sheer incompetence of it all is disqualifying by itself.

In my experiences arguing with MAGA, the incompetence argument lands a lot more convincingly than the evil argument.

[+] jordanpg|1 year ago|reply
** WAKE UP FOLKS! ** THIS IS HAPPENING RIGHT NOW. NO ONE IS COMING TO SAVE US.

If you're in the US, it's time to start asking yourself what you are going to want to be able to say to your kids or grandkids when they ask what you did about it when the constitutional republic was falling.

[+] UncleMeat|1 year ago|reply
Curtis Yarvin has been writing about this forever. JD Vance has spoken publicly about his agreement with ideas like RAGE (Retire All Government Employees). Musk has similarly spoken publicly about how CEOs are the best people to run the world. Although Thiel is not directly in power, he's also a Yarvin-ite and has been instrumental in both Vance's rise and also the pipeline for the various people working at DOGE to destroy the government.

Alongside these budget cuts, the GOP is proposing massive tax cuts and massive increases in funding for parts of the government related to law and immigration enforcement that will massively outweigh any savings from DOGE destroying entire programs overnight. DOGE is also demolishing organizations that are revenue positive and Trump has fired Inspector Generals, who are responsible for identifying waste and inefficiency in government operations.

It isn't a possibility. It is very very clear.

[+] MattGaiser|1 year ago|reply
This is an explicit Project 2025 goal.
[+] ck2|1 year ago|reply
and it's not like private industry is going to do this work, there's no profit in such services, at least not short term

AND if they are doing this to NIH, imagine what they are doing to food and drug inspection for safety

just assume all regulation is gone or being ignored without penalty, now scale that out for four years

Forget "great depression" it's a dark ages of sorts with smartphones for distraction

[+] MPSimmons|1 year ago|reply
It's really hard to have private companies charge for doing work that a functional government agency performs.
[+] yapyap|1 year ago|reply
Don’t wanna be that guy but if not really being about budget cuts seemed pretty obvious from the start, if Elon and Donald got a hand in it it’s pretty much always ego driven.
[+] akudha|1 year ago|reply
Eh? “At some point”? You must’ve been sleeping. This has nothing to do with efficiency or improving the budget. Bill Clinton laid off thousands of employees, but he did it in an orderly fashion, got congressional approval etc. This is the exact opposite. The goal here is to cause havoc, be cruel, cut regulations (by reducing budget and reducing personnel) etc. After the carnage, divvy up what’s left among the billionaires. So far, it seems going according to plan.

Did you hear what Russell Vought said? “We want the bureaucrats to be traumatically affected,”

Anyone genuinely interested in efficiency should start with why the Pentagon has failed audits and is unable to account for tens (hundreds?) of billions of dollars

[+] agumonkey|1 year ago|reply
And at some point it's time to take action to go around a rogue non government. We have the technology, we can rebuild it.
[+] sirolimus|1 year ago|reply
Really pubmed?!?! USA is a disgrace!
[+] koliber|1 year ago|reply
Move fast and break things.

We don't know the root cause, but the kinds of moves that Musk is pulling are straight from a sabotage manual.

[+] decasia|1 year ago|reply
It basically is like having politically motivated human chaos monkeys running around the data center randomly killing systems and seeing whether anyone notices.
[+] sneak|1 year ago|reply
…which seems like not the worst idea?

The very fact that it takes such a battle to even get read only access to analyze where the money is going is evidence enough that there is insanely baroque levels of waste and unnecessary services happening here.

If it’s that important it can be put back.

Why is it so scandalous that conservatives want a smaller government, got elected, and are now making the government smaller?

[+] greatgib|1 year ago|reply
Things like PubMed are probably some of best good to the world that did the US in recent years. Especially because of its great openness with widely open API and so...

If it becomes unavailable, my feeling is that the world will really enter in a dark age.

[+] sunils34|1 year ago|reply
The entire 18f org was let go. Many of these employees oversaw the building of digital services.

Login.gov is a most critical SSO resource for logging into services such as the IRS for tax payments. It’ll see peak usage next month for taxes. The teams maintaining these have been indiscriminately let go.

More of these online services will start to go down, rapidly.

If the point of DOGE is really to tackle the deficit, all these moves are incredibly shortsighted.

(Yesterday the 18F former employee put https://18f.org/ as a transparent warning of what’s to come)

[+] stellababe|1 year ago|reply
It is a DNS issue. Try and change which net you are using. If you are on wifi at home try your cell, however if your home network and cell provider are the same that wont help. A solution is to change which DNS you are using. Its a bit nerdy but actually very easy. Do a search on "change which DNS you are using for browsing" On your computer you can add following to your hosts file located in C:\Windows\System32\drivers\etc add this 156.40.212.210 nih.gov 96.17.96.9 www.nih.gov 34.107.134.59 pubmed.ncbi.nlm.nih.gov 130.14.250.10 ftp.wip.ncbi.nlm.nih.gov 130.14.250.10 ftp.ncbi.nlm.nih.gov 130.14.29.110 blast.ncbi.nlm.nih.gov
[+] wileydragonfly|1 year ago|reply
Another cute and recent development is that they’ve really restricted the hours on the federal payment system, making it difficult for everyone to draw down funds owed. It’s all electronic banking, why does it ever have to go offline?

https://pmsapp.psc.gov/pms/app/login

[+] slater|1 year ago|reply
nih.gov seems to be back up.

And if you search for "trans", you get search results, tho unrelated (e.g., "transnational", etc.).

If you search for "transgender", it just redirects you to the homepage.

The cruelty is the point.