This type of issue can be incredibly annoying to deal with, because the legitimate answer to the abuse report ("someone is spoofing my IP, it isn't me, and the machine is not compromised") is the exact same excuse that a malicious actor would provide.
Then, as noted in the article, you're trying to prove a negative to someone who doesn't really care at all, which is borderline impossible.
Hertzner says in the email that no response is necessary.
Automated abuse reports of things that are easily spoofed don't justify a report, but might justify a quick check to make sure your box is still operating correctly and hasn't been taken over.
> the legitimate answer to the abuse report ("someone is spoofing my IP, it isn't me, and the machine is not compromised") is the exact same excuse that a malicious actor would provide.
The legitimate answer would include some sort of real-world attestation about you from a trusted third party. Probably the very least, some evidence of your identity and jurisdiction. Maybe including a video call or something. Not just you anonymously claiming you're a good guy over the internet and expecting to be believed.
> The internet was broken 25 years ago and is still broken 25 years later. Spoofed source IP addresses should not still be a problem in 2024, but the larger internet community seems completely unwilling to enforce any kind of rules or baseline security that would make the internet safer for everyone.
Same with spoofed MAC addresses, email addresses, ARP messages, Neighbor Discovery, MitM TLS certificates ... It's amazing anything works anymore :D
The thing is, obviously, that the Internet isn't broken, it has incredible utility and reliability. If it was designed and operated to be perfect, then it would likely be massively broken quite often. It is the tolerance for mild brokenness that has contributed significantly to its robustness and utility.
That isn't an argument for not improving things though, just a warning against perfection, if you chase it then you're liable to make really big mistakes that ruin everything.
It’s quite sad the only mail server out there which checks if you are allowed to use a email address is exchange. With all others you can set the from: header however you like.
Back in the day I would scan for DrDoS reflectors in a similar way, no hosting provider wants to get reports for port scanning so the source address of the scan would belong to an innocent cloud provider with a reputable IP that reflectors would happily send UDP replies to. The cloud provider would of course get a massive influx of complaints but you would just say that you aren't doing any scanning from your server (which they would verify) and they wouldn't shut your service off. The server sending out the spoofed scan packets is undetectable so you're able to scan the entire internet repeatedly without the typical abuse issues that come with it.
I'm not sure how often this happens in practice but tracing the source of a spoofed packet is possible if you can coordinate with transit providers to follow the hops back to the source. One time JPMorgan worked with Cogent to tell us to stop sending packets with their IP addresses (Cogent is one of the most spoofer friendly tier 1's on the internet btw).
This is the first time I've heard of this being used to target TOR specifically which seems counterintuitive, you would think people sending out spoofed packets would be advocates of TOR. Probably just a troll, luckily providers that host TOR won't care about this type of thing.
This is nothing new. A few years back, I implemented a very basic firewall rule: if I received a TCP packet with SYN=1 and ACK=0 to destination port 22, the source IP would get blacklisted for a day. But then I started getting complaints about certain sites and services not working. It turned out that every few days, I'd receive such packets from IPs like 8.8.8.8 or 1.1.1.1, as well as from Steam, Roblox, Microsoft, and all kinds of popular servers—Facebook, Instagram, and various chat services. Of course, these were all spoofed packets, which eventually led me to adjust my firewall rules to require a bit more validation.
So, I can assure you this is quite common. As a personal note, I know I’m a bit of an exception for operating multiple IP addresses, but I need the flexibility to send packets with any of my source addresses through any of my ISPs. That’s critical for me, and if an ISP filters based on source, it’s a deal-breaker—I’ll switch to a different ISP.
> As a personal note, I know I’m a bit of an exception for operating multiple IP addresses, but I need the flexibility to send packets with any of my source addresses through any of my ISPs. That’s critical for me, and if an ISP filters based on source, it’s a deal-breaker—I’ll switch to a different ISP.
If you actually have your own IP addresses this is normal and expected, but if you're able to use ISP A's IP addresses through ISP B or vice versa that has always been a bug that you are wrong to use.
If you are doing the latter this is firmly in the "reenable spacebar heating" category and I hope your ISPs fix their broken networks.
Okay, looks like I will reply to a few of the comments to clarify things.
I’ll give a concrete, real example.
I worked at a company that hosted some web assets on-prem in one of their branches. They had a 1Gbps connection there. However, at HQ, we had multiple 10G connections and a pretty good data center. So, we moved the web VM to HQ but kept the assigned IP address (a public static from ISP-A). We routed it through a VPN to HQ. The server used our default GW and sent responses with source IP (ISP-A) via ISP-B (10G).
That way, we utilized 10G outbound, even though the inbound was limited to 1G. It was only for GET requests anyway. I know this wasn’t the most optimal setup, and we eventually changed the IP, but it seems like a valid use case.
Scenario 2: We had two connections from two different ISPs (our own ASN, our own /23 addresses). We wanted to load balance some traffic and sent half of our IPs through ISP-A and the other half through ISP-B. It worked fine, but when we tried to mix the balance a bit, we found an interesting glitch. We announced the first /24 to ISP-A and the second /24 to ISP-B, but ISP-A had RP filtering. So, we had to announce all the IPs to them.
The way the RP filter works, as you may guess, means we cannot prepend or anything. All traffic must come through them. If they see a better route for that prefix, they will filter it. For a few months, they refused to fix this, citing security. There’s no shame in security best practices, so I might as well name the ISP—Virgin Media.
Note that the internet with rp_filter is not $20/month. It was more like 5K+/month!! And we did not change it due to lack of alternatives there. But otherwise guess who loses the contract :)
>As a personal note, I know I’m a bit of an exception ...That’s critical for me, and if an ISP filters based on source, it’s a deal-breaker—I’ll switch to a different ISP.
"...and obviously, Pennywise, I must spoof ingress and egress..."
The “someone hates Tor relays” theory doesn’t sound worth the effort. This could be an entity running malicious relays, while also trying to unethically take down legitimate relays to increase the percentage of the network that they control.
This is almost certainly it. There’s a lot of head-sand-burying around here about just how easily an attacker with access to logs of a not-even-that-large segment of the nodes can gain visibility into individuals’ service access patterns.
Yeah. If you hate the tor network an easier thing to do is just to overwhelm it with traffic and degrade the service. Running some bittorrent downloads might be enough.
> Which means, if you just find one transit provider which doesn’t do BCP38 filtering… you can send IP packets tagged with any source IP you want! And unfortunately, even though the origins of BCP38 date back to 1998… there are still network providers 25 years later that don’t implement it.
What would it take to get enough network providers to start rejecting traffic from all ASes that don't implement this, so that spoofing was no longer possible?
Cloudflare is probably enough. They already control enough ingress that their "checking the security of your connection" could actually mean something.
You'd have to find some way to make network providers care. Especially 'tier 1' transit providers and other networks of unusual size.
It's much easier to work on reducing reflection multipliers though, because you can scan (ipv4 anyway) for reflection vectors and yell at people that will respond with 10x the input bytes.
It seems like systems shouldn't report abuse (at least automatically) for single packet, no round trip, requests unless its reaching denial of service levels of traffic (and maybe these are). Like in particular for SSH there's no way thats even a valid connection attempt until some sort of handshake has occurred.
But since anyone can submit an abuse complaint, maybe server providers should actually check the abuse reports before triggering the "respond in 2 days or we suspend your server" or similar measure of their ToS.
I've had my main server thrown offline by a bogus abuse report claiming that they received an over 1Gbps DoS attack from my IP even though my server only has a 400 Mbps cap. Had a human actually read the report, they would've seen it was impossible and wouldn't have had to spend 2 days arguing with phone support on my holiday.
This is the IP version of SWATting, patent trolls, framing an innocent person, or using DMCA takedowns to remove the competition. It's basically weaponizing abuse-protection mechanisms to instead attack a target that is disliked. Interesting that the authorities can become a weak link here and be actively weaponized by unscrupulous actors to achieve their aims, but it's not really a new phenomena.
There's no in-band solution to this problem, but out-of-band solutions might exist! For example: (1) Notify the destination ISP that you're receiving backscatter. (2) That ISP checks where the packets are coming from, and notifies that ISP. (3) Repeat step 2 until source is found. (4) Quarantine that part of the network until it behaves better.
How difficult would it be to highjack this attack by sending these packages to everyone, so that providers like hetzner would get swamped with abuse emails? This way the attack would not work anymore. Either the honeypots would stop sending abuse emails, or the providers would filter those out.
This is likely a very naive question, but how did the spoofer know his IP was participating as an internal Tor node? From what vantage point can that be seen? I imagine internal Tor nodes must know to connect to each other, so it must propagate through Tor. Is the attacker also a Tor node? Is it trivial to map all Tor hosts?
Tor has something called a consensus that lists all relays and their flags. Clients need this to know which relays to make a circuit with. For most clients, they select a relay labelled as a guard from this file which is where their traffic first enters Tor. Some countries realized they could just block all of these IP addresses and stop people using Tor, so there are unlisted guard nodes called bridges designed for censorship circumvention that you have to get by filling out a captcha or say sending an email.
gah, I remember once when I was working at a company, and we got an email complaining "stop hacking my systems!"
in the end, we had a load-balancer at .1 balancing a bunch of backend servers.
the complainer would have traffic to .1 that the load balancer would receive. Thing is, old or stale connections would drop out of the load balancer mapping table, and eventually the backend server connection would not get mapped, and the guy would get traffic direct from the backend server real ip address.
the traffic was actually generated by the customer, but these "unrelated" backend servers looked like they were hacking him.
They have posted several screenshots of discussions among people affected on various channels, including Mastodon and the official #tor-relays channel on IRC.
I don't understand what you're advocating for. Are you against Tor because there was a vulnerability (that has since been mitigated) which led to the deanonymization of users or are you against Tor because there was an illegal service that used it? By killing existing good relays you're making it easier for someone malicious to come in, add a bunch of relays to the network, and have those relays be more likely to be used. And it won't be a saintly do-gooder either, it'll probably be some guy trying to mitmproxy crypto exchanges to steal money.
[+] [-] ziddoap|1 year ago|reply
Then, as noted in the article, you're trying to prove a negative to someone who doesn't really care at all, which is borderline impossible.
[+] [-] toast0|1 year ago|reply
Automated abuse reports of things that are easily spoofed don't justify a report, but might justify a quick check to make sure your box is still operating correctly and hasn't been taken over.
[+] [-] dataflow|1 year ago|reply
The legitimate answer would include some sort of real-world attestation about you from a trusted third party. Probably the very least, some evidence of your identity and jurisdiction. Maybe including a video call or something. Not just you anonymously claiming you're a good guy over the internet and expecting to be believed.
[+] [-] mrbluecoat|1 year ago|reply
Same with spoofed MAC addresses, email addresses, ARP messages, Neighbor Discovery, MitM TLS certificates ... It's amazing anything works anymore :D
[+] [-] colechristensen|1 year ago|reply
That isn't an argument for not improving things though, just a warning against perfection, if you chase it then you're liable to make really big mistakes that ruin everything.
[+] [-] alwayslikethis|1 year ago|reply
[+] [-] neop1x|1 year ago|reply
[+] [-] grotorea|1 year ago|reply
[+] [-] timokoesters|1 year ago|reply
[+] [-] Asmod4n|1 year ago|reply
[+] [-] Rasbora|1 year ago|reply
I'm not sure how often this happens in practice but tracing the source of a spoofed packet is possible if you can coordinate with transit providers to follow the hops back to the source. One time JPMorgan worked with Cogent to tell us to stop sending packets with their IP addresses (Cogent is one of the most spoofer friendly tier 1's on the internet btw).
This is the first time I've heard of this being used to target TOR specifically which seems counterintuitive, you would think people sending out spoofed packets would be advocates of TOR. Probably just a troll, luckily providers that host TOR won't care about this type of thing.
[+] [-] SSLy|1 year ago|reply
> Probably just a troll
Or someone wanting TOR to be treated like nuclear waste, because it offends their surveillance ops.
[+] [-] Habgdnv|1 year ago|reply
So, I can assure you this is quite common. As a personal note, I know I’m a bit of an exception for operating multiple IP addresses, but I need the flexibility to send packets with any of my source addresses through any of my ISPs. That’s critical for me, and if an ISP filters based on source, it’s a deal-breaker—I’ll switch to a different ISP.
[+] [-] wolrah|1 year ago|reply
If you actually have your own IP addresses this is normal and expected, but if you're able to use ISP A's IP addresses through ISP B or vice versa that has always been a bug that you are wrong to use.
If you are doing the latter this is firmly in the "reenable spacebar heating" category and I hope your ISPs fix their broken networks.
[+] [-] Habgdnv|1 year ago|reply
I worked at a company that hosted some web assets on-prem in one of their branches. They had a 1Gbps connection there. However, at HQ, we had multiple 10G connections and a pretty good data center. So, we moved the web VM to HQ but kept the assigned IP address (a public static from ISP-A). We routed it through a VPN to HQ. The server used our default GW and sent responses with source IP (ISP-A) via ISP-B (10G).
That way, we utilized 10G outbound, even though the inbound was limited to 1G. It was only for GET requests anyway. I know this wasn’t the most optimal setup, and we eventually changed the IP, but it seems like a valid use case.
Scenario 2: We had two connections from two different ISPs (our own ASN, our own /23 addresses). We wanted to load balance some traffic and sent half of our IPs through ISP-A and the other half through ISP-B. It worked fine, but when we tried to mix the balance a bit, we found an interesting glitch. We announced the first /24 to ISP-A and the second /24 to ISP-B, but ISP-A had RP filtering. So, we had to announce all the IPs to them.
The way the RP filter works, as you may guess, means we cannot prepend or anything. All traffic must come through them. If they see a better route for that prefix, they will filter it. For a few months, they refused to fix this, citing security. There’s no shame in security best practices, so I might as well name the ISP—Virgin Media.
Note that the internet with rp_filter is not $20/month. It was more like 5K+/month!! And we did not change it due to lack of alternatives there. But otherwise guess who loses the contract :)
[+] [-] Jerrrrrrry|1 year ago|reply
"Of course, Agent Bond."
[+] [-] jcalvinowens|1 year ago|reply
As someone who always enables rp_filter everywhere... I'm very curious why?
[+] [-] pixl97|1 year ago|reply
I mean, technically those ISPs would be in violation too. You need your own ASN.
[+] [-] cowboylowrez|1 year ago|reply
don't we want source based filtering tho? sounds like the problem is a LACK of source based filtering.
[+] [-] rvnx|1 year ago|reply
[+] [-] buildbuildbuild|1 year ago|reply
[+] [-] aphantastic|1 year ago|reply
[+] [-] alwayslikethis|1 year ago|reply
[+] [-] JoshTriplett|1 year ago|reply
What would it take to get enough network providers to start rejecting traffic from all ASes that don't implement this, so that spoofing was no longer possible?
[+] [-] benlivengood|1 year ago|reply
[+] [-] toast0|1 year ago|reply
It's much easier to work on reducing reflection multipliers though, because you can scan (ipv4 anyway) for reflection vectors and yell at people that will respond with 10x the input bytes.
[+] [-] cobbal|1 year ago|reply
I suppose a difference is that they use unaffiliated parties to send the complaint, instead of contacting the authority directly.
[+] [-] jmuguy|1 year ago|reply
[+] [-] franga2000|1 year ago|reply
I've had my main server thrown offline by a bogus abuse report claiming that they received an over 1Gbps DoS attack from my IP even though my server only has a 400 Mbps cap. Had a human actually read the report, they would've seen it was impossible and wouldn't have had to spend 2 days arguing with phone support on my holiday.
[+] [-] Avamander|1 year ago|reply
[+] [-] nostrademons|1 year ago|reply
[+] [-] wizzwizz4|1 year ago|reply
At the end of the day, the internet is people.
[+] [-] ahofmann|1 year ago|reply
[+] [-] preciousoo|1 year ago|reply
[+] [-] dataflow|1 year ago|reply
[+] [-] fullspectrumdev|1 year ago|reply
Just acquire a few boxes that don’t block spoofing outbound SYN packets and start spamming random IP’s from random IP’s with SYN packets.
It will generate a shitload of abuse emails and accomplish mostly nothing except fill up disk space with useless emails and such.
[+] [-] skygazer|1 year ago|reply
[+] [-] neckardt|1 year ago|reply
[+] [-] costco|1 year ago|reply
[+] [-] m463|1 year ago|reply
in the end, we had a load-balancer at .1 balancing a bunch of backend servers.
the complainer would have traffic to .1 that the load balancer would receive. Thing is, old or stale connections would drop out of the load balancer mapping table, and eventually the backend server connection would not get mapped, and the guy would get traffic direct from the backend server real ip address.
the traffic was actually generated by the customer, but these "unrelated" backend servers looked like they were hacking him.
[+] [-] unknown|1 year ago|reply
[deleted]
[+] [-] chaz6|1 year ago|reply
They have posted several screenshots of discussions among people affected on various channels, including Mastodon and the official #tor-relays channel on IRC.
[+] [-] stronglikedan|1 year ago|reply
https://archive.is/Eb7TI
[+] [-] 71bw|1 year ago|reply
[+] [-] encom|1 year ago|reply
[+] [-] gherard5555|1 year ago|reply
[+] [-] flemhans|1 year ago|reply
[+] [-] preciousoo|1 year ago|reply
[+] [-] 0x_null|1 year ago|reply
[deleted]
[+] [-] costco|1 year ago|reply
PS: The fake abuse report technique was invented like 5 years ago: https://www.theregister.com/2019/04/16/spamhaus_port_scans/