top | item 42796383

(no title)

billyhoffman | 1 year ago

Beyond just IPs, there is a giant class of "DNS record pointing to X shared cloud resource that organization no longer controls" issues. The bigger the company, the more widespread the problem. These resource names get released back into a common pool that anyone can register.

Think:

* CNAME pointing to an S3 bucket, and the S3 bucket gets released

* CNAME pointing to Azure Website/WebApp Instance

* A record to an non-elastic IP, and the box gets rebooted

* DNS name using a Route53 name server that no longer part of the org's AWS account

* CNAME pointing to a Heroku/Shopify/GitHub pages account and the account gets deleted/deactivated freely up those names for registration

* MX record pointing to old transaction email provider start up that dies, and someone else registers that domain name...

Why does that happen?

* Decentralization of IT means people spinning up infrastructure not knowing what they are doing

* Great a spinning up infra, but when decomissioning they forget about DNS

* Lots of subsidiaries, lots of brands, different groups, operating in different geographies. All this makes it difficult to discover and enforce proper policies

* Geo-specific websites/apps (Think of all the country-specific websites Coke runs)

* Using some 3rd party vendor and never telling security about it (Marketing spinning up some landing pages on some fly-by-night martech provider or wordpress host, and never turning them off)

I am the Field CTO at a venture backed Israeli cyber security company in this space. I was literally talking to a major computer part company yesterday about the dozen or so Indonesian gambling websites that are "running" on their domain names using their pagerank and links. This is a weekly conversation

discuss

order

davchana|1 year ago

> CNAME pointing to a Heroku/Shopify/GitHub

At least Gitlab (similar to Github pages, I never used Github Pages, always Gitlab Pages) gives you a verification TXT record in your Gitlab Account, which needs to stay in DNS as TXT. So if I used to host hi.example.com on Gitlab (& my own TXT record was hosted, and publicly visible), now I don't own example com, or gitlab account got deleted (but still left DNS CNAME records intact) and scammer gets the domain, when he grabs domain and adds hi.example.com to his Gitlab Account to scam people, his Gitlab Account will have his own TXT record. (now) His hi.example.com can never point to "my" gitlab project or page.

https://docs.gitlab.com/ee/user/project/pages/custom_domains...

alwa|1 year ago

I’m not sure he’d want to, it would seem like he might want to point to his own scam. But if he did, I imagine he could add back your TXT record after looking it up in any of a large number of historical DNS databases. I can’t vouch for the quality, but a casual Google suggests there are still many, primarily paid but some free-ish, in the mix. Examples:

https://dnshistory.org/

https://whoisfreaks.com/tools/dns/history/lookup

I really don’t think a TXT record is a good place to keep a secret… although it is a good place to prove you control a domain.

ok_dad|1 year ago

What types of actions can you do to correct and prevent this class of errors? I think you could probably enforce deployment and shutdown checklists, perhaps, or have automated DNS checking software to see if any of the issues exist (I bet you guys have a solution for that) but there are so many human-error problems in manufacturing, and I kinda consider the large-scale deployment of apps to have similar issues and failure modes on the human side.

pastage|1 year ago

We have an inventory of everything running, and where they are supposed to be running. If service X does not respond on resource Y the team responsible get an ticket. Check is on IP and names, and some other services. There are no good ways to do this other than being meticulous IMHO. Getting dumps of what is running where from all services is rather hard but more or less doable.

It helps not using the cloud.

stackskipton|1 year ago

Azure has options when you use their DNS that they tie resource, Public IP, Azure WebApp and other to DNS. If resource is deleted, the record will be NXDomain. AWS probably has something for Route53.

Otherwise, good IaC can help but even in larger companies, I see more ClickOps then I should.

nonameiguess|1 year ago

The simplest things you can do are either:

- Stay within the cloud provider's ecosystem as much as possible, including for domain registration and DNS. All records then should be pointing to resources that include your account id in them and can't be taken over by others. If you delete the entire account, there'd be nothing to take over.

- Do everything with Infrastructure as Code, including DNS. If a single "terraform apply" creates everything, then a single "terraform destroy" deletes it all, leaving nothing dangling, provided of course that it is setup correctly and doesn't error out midway through a run.

Otherwise, it's a matter of being thorough. Automate what you can, including creating and deleting resources, if not through a single cloud provider API or some standard IaC product, then roll your own software to do it, but have software do it. Regularly roll out and tear down entire test installations of full systems, including valid DNS records. When you intend for them to be gone, ensure they are really, truly gone.

If you can't automate it, then yeah, checklists.

It's one of those things that is simple but not easy. It takes an organization that respects the tedious and time-consuming nature of ops, plans for it, and doesn't push people to cut corners for the sake of speed when the first time trying to do something takes much longer than someone's uninformed first guesstimate.

Really, automate. At a small enough scale, it doesn't matter, but if you're Mastercard doing this kind of thing thousands of times over the course of decades, humans will inevitably make mistakes. Software will make mistakes, too, but at least when you test software, it will do the same thing every time it is tested. Humans do not provide that guarantee, even if they have checklists.

Edit: Note the above is not true for LLMs, so when I say use software, I mean classical deterministic software. Don't have AI do it for you, because LLMs can and will produce different responses every time you make the same request. Don't devolve to making software that is just as flaky as humans.

abhgh|1 year ago

Do these Indonesian gambling sites somehow exploit the AMP cache? The apex domain of my blog was recently hijacked by such a site. I was only using the blog.* subdomain. One day I noticed on the Google search console that someone had verified themselves as a co-owner, and there was an entry for an AMP page (this was the gambling page). Putting a parking page at the apex url seemed to stop the redirects to the AMP page - and that's the solution I have for now.

I am still curious though - how does AMP make such exploits easier? Would you happen to know?

sakisv|1 year ago

> A record to an non-elastic IP, and the box gets rebooted

Oh mate, I've seen that happen with a company that was selling security-adjacent services, which were running on servers with just random IPs ffs