WingNews logo WingNews GitHub [2]
top | item 20205566

SACK Panic – Multiple TCP-based remote denial-of-service issues

416 points| cdingo | 6 years ago |access.redhat.com

131 comments

order
[+] LinuxBender|6 years ago|reply
This is the way I block such things on my own VM's (not at work) using iptables:

    iptables -t raw -I PREROUTING -i eth0 -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m tcpmss ! --mss 640:65535 -j DROP
Here it is in action:

    iptables -L -n -v -t raw | grep mss  
    84719 3392K DROP       tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0            tcp flags:0x17/0x02 tcpmss match !640:65535
My settings may be a little aggressive and may block some old pptp/ppoe users. Perhaps 520 would be a safer low end. As a funny side note, this also blocks hping3's default settings (ping floods) as it doesn't set mss. This also blocks a slew of really poorly coded scanners.

For everything else at work, we are behind a layer 7 load balancer that is not vulnerable.

You may also find it useful to block fragmented packets. I've done this for years and never had an issue:

    iptables -t raw -I PREROUTING -i eth0 -f -j DROP
If you have the PoC, then feel free to first verify you can browse to https://tinyvpn.org/ then send the small MSS packets to that domain, then see if you can still browse to it. I don't care if the server reboots or crashes. Just don't send DDoS please, as the provider will complain to me.

To see the counters increase, here is a quick and silly cron job that will show you the MSS DROPs in the last minute, that I will disable after a couple days: [1]

[1] - https://tinyvpn.org/up/mss/

[+] ilikepi|6 years ago|reply
The iptables commands listed in the Mitigation section of the RedHat article lists only SYN in the tcp-flags mask section:

    iptables -I INPUT -p tcp --tcp-flags SYN SYN -m tcpmss --mss 1:500 -j DROP
If I interpret the man page correctly, the above is more broad because it does not care about the presence or absence of other flags, whereas your rule explicitly requires the other listed flags to be unset. In fact it seems like the above might be broad enough to include incoming SYNACK response packets that are the result of outgoing connections.

Am I understanding this correctly, and if so, do you have a thought about why they suggest this?

[+] chupasaurus|6 years ago|reply
FYI Debian Security Team recommends setting new sysctl value net.ipv4.tcp_min_snd_mss to 536, even though they (Debian) are preserving the default kernel value of 48 for compatibility.
[+] hackersword|6 years ago|reply
How would you do that for the INPUT table (not a VM)? Just: > iptables -t raw -I INPUT -i eth0 -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m tcpmss ! --mss 640:65535 -j DROP

??

here [1] gives example of ... is your just inverting/negating the DROP rule ?

>iptables -A INPUT -p tcp -m tcpmss --mss 1:500 -j DROP

[1] https://github.com/Netflix/security-bulletins/blob/master/ad...

[+] LinuxBender|6 years ago|reply
In the event anyone needs this, you can also override outbound mss in case someone is telling your client to use a low mss.

This goes in the mangle table. DO NOT use this example unless you know for sure what you are doing.

    iptables -t mangle -I POSTROUTING -o eth0 -p tcp -m tcp --tcp-flags SYN,RST,ACK SYN -m tcpmss --mss 1:100 -j TCPMSS --set-mss 1360
This example should work for most use cases, but don't do this unless you for sure know the implications. Dropping bad inbound options is easy, but outbound can get more complicated. I am just showing this in case anyone asks and I am asleep. :-) Talk to your network admins and ask what is the highest mss/mtu your VPN's and 3rd party networks support.

This may not even help, as the packet has already been generated and we are too late in the process. I just figure someone might ask. There are probably use cases where this may help (for proxies, edge firewalls, hypervisors, docker hosts, maybe)

Or just log and drop the connections, or send yourself (your app) a tcp-reset.

[+] peterwwillis|6 years ago|reply
And ipv6? Fragmentation and mss are handled somewhat differently, iirc
[+] isodude|6 years ago|reply
How about stripping MSS instead? Like so:

iptables -t mangle -I PREROUTING -p tcp --tcp-flags SYN SYN -m tcpmss --mss 1:500 -j TCPOPTSTRIP --strip-options mss

I experimented with a host that sends out mss 216 and the communication was still ok with the above, but not while dropping the traffic.

[+] Avamander|6 years ago|reply
How does one do this with nftables?
[+] talawahtech|6 years ago|reply
AWS Bulletin: https://aws.amazon.com/security/security-bulletins/AWS-2019-...

FYI if your instances are behind an Application Load Balancer or Classic Load Balancer then they are protected, but NOT if they are behind a Network Load Balancer.

A patched kernel is available for Amazon Linux 1 and 2, so you won't have to disable SACK. You can run "sudo yum update kernel" to get it, but of course you have to reboot. Updated AMIs are also available.

Amazon Linux 1: https://alas.aws.amazon.com/ALAS-2019-1222.html Amazon Linux 2: https://alas.aws.amazon.com/AL2/ALAS-2019-1222.html

For Amazon Linux 2 the fixed kernel is kernel-4.14.123-111.109.amzn2. Looking at my instances, it look like I have been on that version since Friday.

[+] ones_and_zeros|6 years ago|reply
Even if your instances are behind ALBs or ELBs they may not be protected if they make outbound connections to the internet.
[+] usrname|6 years ago|reply
Some solution: echo 0 > /proc/sys/net/ipv4/tcp_sack

https://github.com/Netflix/security-bulletins/blob/master/ad...

[+] href|6 years ago|reply
Thanks for pointing this out. Applying this buys us time before we can properly patch all our systems. In our case this was easy to roll out in a jiffy.

I do wonder though, can anyone guess what kind of impact one might see with TCP SACK disabled? We don't have large amounts of traffic and serve mostly websites. Maybe mobile phone traffic might be a bit worse off if the connection is bad?

[+] topranks|6 years ago|reply
The MSS change is better, won’t affect performance.
[+] hoffie|6 years ago|reply
Red Hat's article on these issues also provides further explanations: https://access.redhat.com/security/vulnerabilities/tcpsack
[+] bauruine|6 years ago|reply
There is also an ansible playbook on the resolve tab to easily apply the net.ipv4.tpc_sack workaround on all your hosts.
[+] geggam|6 years ago|reply
Is this the same thing ?

https://www.cvedetails.com/cve/CVE-2005-0960/

Multiple vulnerabilities in the SACK functionality in (1) tcp_input.c and (2) tcp_usrreq.c OpenBSD 3.5 and 3.6 allow remote attackers to cause a denial of service (memory exhaustion or system crash).

[+] wademealing|6 years ago|reply
Similar concept, different operating system.
[+] jkern|6 years ago|reply
So you can remotely cause a kernel panic on basically any system with kernel newer than 2.6.28? The doesn't sound great
[+] syn0byte|6 years ago|reply
No, not "any system". Besides needing SACK enabled (which is by default) you also need segment offloading and non-shite networking hardware that will respect and preserve stupid MSS fields in packets.

pending a patch simply disable SACK: ~$ echo 0 > /proc/sys/net/ipv4/tcp_sack

and/or disable segmentation offloading: ~$ ethtool -K eth? tso off

TCP and Checksum offloading still aren't super standard on customer grade NICs or virtual machines. I'd assume less than half of the internet's linux hosts are actually at risk.

[+] bhauer|6 years ago|reply
Sorry, this is probably a bit simplistic, but I am curious: How likely is this to affect embedded devices? E.g., hardware firewalls, routers, IoT devices that all use a Linux kernel?
[+] LinuxBender|6 years ago|reply
If you are not exposing any tcp ports or reaching out directly from those devices to a malicious host, then very unlikely. Either way, it's best to check the vendors site or open a ticket with them, if that is an option.

An alternate option would be to put the device behind another firewall or load balancer or proxy that you know is not vulnerable.

[+] ravingraven|6 years ago|reply
They are equally as affected. Linux is Linux, it makes no difference in what box it runs. What is usually a mitigating factor is that embedded devices usually have a very different configuration compared to non-embedded devices (built with minimal options, not a lot of services running on them etc.).
[+] rphlx|6 years ago|reply
When was the last time Linux had a similar, reliably-remotely-exploitable kernel panic in the TCP/IPv4 stack? Pre-2000?
[+] foofc7c8|6 years ago|reply
Never, from what I can recall.
[+] fortran77|6 years ago|reply
Will there be a big performance hit if I just turn sack off (for now) until I finish running all our tests on the new kernel?
[+] ahoka|6 years ago|reply
Maybe this helps someone:

  [Unit]
  Description=Disable TCP SACK

  [Service]
  Type=simple
  ExecStart=/sbin/iptables -A INPUT -p tcp -m tcpmss --mss 1:500 -j DROP

  [Install]
  WantedBy=sysinit.target
[+] keyle|6 years ago|reply
What does OpenBSD/NetBSD do differently to not be affected by this?
[+] rphlx|6 years ago|reply
They use a different TCP/IP stack which implemented SACK without introducing this bug.

It's a Linux-specific implementation defect, not an intrinsic problem with the TCP SACK wire protocol or spec.

[+] sl-1|6 years ago|reply
Is there any way to quickly test if any of my machines are vulnerable?
[+] davoti|6 years ago|reply
Folks,

imo, TSO is intel NIC card function, does this affect others like from Cavium CPU?

thanks

[+] shereadsthenews|6 years ago|reply
Looks like the issue was fixed upstream a month ago. Might have been nice to know earlier? Is this how long it takes for the distros to lurch into action?