Related story time: At university, we ran a local quakeworld server (yes, this was in the stone age XD). And I wanted to write a tool to allow users to control to server. It would listen on a network port and pipe its input into the quakeworld server running as a child subprocess.
Sounds simple enough, right? Disregarding accept(2)ing connections,etc... it's just going to sit in an endless loop, select(2) on the sockets then read what comes in, and write that to the quakeworld server.
To debug this, everytime someone sent some data over the network, it would write a simple message ("blah") into a file on my home drive, called "~/blah" (very sophisticated debugging, I know).
Now, the semantics of select(2), cousin of poll(2), are interesting.
My understanding: select(2) blocks until you can read data from the socket.
Actual behaviour: select(2) blocks until a subsequent read on the socket does not block.
Fun fact: if there's not data available on the socket, read does not block, but returns with an error immediatly.
Result: Over the weekend, the process was spinning in an endless loop, writing 'blah' into a file on the nfs-home drive as quickly as the filesystem allowed. This being a university scale solaris NFS server, that's pretty fast.
End Result: A test file called 'blah', filled with about 800 GB of 'blah'. Good thing the admins where nice people and didn't shoot me when I went to explain the next monday why the home drive was filled with "blah" XD
EDIT: read(2) doesn't actually error on EOF, just returns 0, effect is the same though.
Read will return 0 when the peer has closed the connection, that's how you detect it. When there is no data, the call will block instead until some data arrives (unless the socket is set to non-block mode).
This is why open source (and controlling your whole stack) matters in big business.
For example, Microsoft may be changing their image, but their core software is closed source. <cloud provider> may be great, but can you pull off something like this when you've got an issue? The importance of being able to debug and patch your mission critical systems is hard to overstate.
Please encourage your employer to fiscally support the open source code that they rely upon!
> For example, Microsoft may be changing their image, but their core software is closed source. <cloud provider> may be great, but can you pull off something like this when you've got an issue? The importance of being able to debug and patch your mission critical systems is hard to overstate.
Not OSS != no source access. A cloud provider on top of the MS stack would most likely have Shared Source Initiative licenses.
There's a different model in this case. You pay for support, and then the original developers go away and fix it for you.
When this works well, it's great. You've got the experts on the problem. Not all support is good though, obviously, and if you have a mission critical problem that you need to fix overnight, nothing beats the 'self-support' that you can do with open source software.
They just rediscovered silly window syndrome, but at the application level rather than the TCP level.
In early TCP implementations, if you read from a TCP connection one byte at a time, and the sender was writing faster than the receiver, the sender TCP would get an ACK opening one byte of window. The sender TCP would then transmit a packet with only one byte. This often came up if a TCP connection was driving a slow device. The fix was to not send a window update until there was at least one full packet worth of window available.
This is the same problem, but at the OS buffering level. Linux has a similar solution. The problem is 1) servers today have to protect themselves against readers that are "too slow", because that can be used as a denial of service attack, and 2) the approach to doing this was not well designed.
This is a tough one for Cloudflare, because they're in the DDOS-prevention business. They have to decide whether a slow reader is an attacker or just someone on a slow connection. Still, insisting that a reader consume at least 5MB/minute is a bit much. Some people are still on dialup.
I really wish people would stop misusing TCP resets. It's not just a fast way to close a connection. That's not what it's for at all. If you're ever thinking about sending a reset, please read RFC 793 and RFC 3360 first.
Well the goal here seems to be aborting a connection, not just closing it. They specifically don't want to empty the buffer, because time is up. Is that a misuse of reset?
Great that they solved it! My company (degoo.com) had to use CloudFront instead of CloudFlare for our binaries because of this bug. Our conversion rate went down about 10% whenever we used CloudFlare instead. Perhaps time to switch back.
You're correct, but the title is way less helpful without the "nginx". Unless the reader knows that cloudflare uses nginx, he won't know the article pertains to nginx
[+] [-] BugsBunnySan|10 years ago|reply
Sounds simple enough, right? Disregarding accept(2)ing connections,etc... it's just going to sit in an endless loop, select(2) on the sockets then read what comes in, and write that to the quakeworld server.
To debug this, everytime someone sent some data over the network, it would write a simple message ("blah") into a file on my home drive, called "~/blah" (very sophisticated debugging, I know).
Now, the semantics of select(2), cousin of poll(2), are interesting.
My understanding: select(2) blocks until you can read data from the socket.
Actual behaviour: select(2) blocks until a subsequent read on the socket does not block.
Fun fact: if there's not data available on the socket, read does not block, but returns with an error immediatly.
Result: Over the weekend, the process was spinning in an endless loop, writing 'blah' into a file on the nfs-home drive as quickly as the filesystem allowed. This being a university scale solaris NFS server, that's pretty fast.
End Result: A test file called 'blah', filled with about 800 GB of 'blah'. Good thing the admins where nice people and didn't shoot me when I went to explain the next monday why the home drive was filled with "blah" XD
EDIT: read(2) doesn't actually error on EOF, just returns 0, effect is the same though.
[+] [-] brazzy|10 years ago|reply
They should be the ones explaining why they don't have disk quotas set up on a shared system.
[+] [-] ajnin|10 years ago|reply
[+] [-] colejohnson66|10 years ago|reply
[+] [-] bigethan|10 years ago|reply
For example, Microsoft may be changing their image, but their core software is closed source. <cloud provider> may be great, but can you pull off something like this when you've got an issue? The importance of being able to debug and patch your mission critical systems is hard to overstate.
Please encourage your employer to fiscally support the open source code that they rely upon!
[+] [-] masklinn|10 years ago|reply
Not OSS != no source access. A cloud provider on top of the MS stack would most likely have Shared Source Initiative licenses.
[+] [-] rkangel|10 years ago|reply
When this works well, it's great. You've got the experts on the problem. Not all support is good though, obviously, and if you have a mission critical problem that you need to fix overnight, nothing beats the 'self-support' that you can do with open source software.
[+] [-] Animats|10 years ago|reply
In early TCP implementations, if you read from a TCP connection one byte at a time, and the sender was writing faster than the receiver, the sender TCP would get an ACK opening one byte of window. The sender TCP would then transmit a packet with only one byte. This often came up if a TCP connection was driving a slow device. The fix was to not send a window update until there was at least one full packet worth of window available.
This is the same problem, but at the OS buffering level. Linux has a similar solution. The problem is 1) servers today have to protect themselves against readers that are "too slow", because that can be used as a denial of service attack, and 2) the approach to doing this was not well designed.
This is a tough one for Cloudflare, because they're in the DDOS-prevention business. They have to decide whether a slow reader is an attacker or just someone on a slow connection. Still, insisting that a reader consume at least 5MB/minute is a bit much. Some people are still on dialup.
[+] [-] scurvy|10 years ago|reply
I'm looking at you Arbor.
[+] [-] Dylan16807|10 years ago|reply
[+] [-] Yrlec|10 years ago|reply
[+] [-] LoSboccacc|10 years ago|reply
[+] [-] speps|10 years ago|reply
[+] [-] gruez|10 years ago|reply
[+] [-] Dylan16807|10 years ago|reply
[+] [-] bluesign|10 years ago|reply
[+] [-] scurvy|10 years ago|reply
Slow requests (like slowloris) were much more effective.
[+] [-] dredmorbius|10 years ago|reply
[deleted]