This is a story of caching

[pilif]

... and then the customer called and asked why the graphs on the front page were wrong even though they clearly just edited hugetable.

You explain them that it'll just take a little while to be updated, but the customer didn't like that answer. The data needs to always be current.

Apparently, you need to flush parts of the cache as new data arrives. Unfortunately though, you can't as memcache is a strict key/value store. So you change how you name the cache keys and make them dependent of, say the max(timestamp) of your hugetable.

Load goes back up to 2 because all requests now still have to check the table.

But it's still not as bad.

Until the next phone call...

[fliph]

Or you could just update the cache when the data changes.

[jemfinch]

> Apparently, you need to flush parts of the cache as new data arrives. Unfortunately though, you can't as memcache is a strict key/value store.

Huh? You can delete keys in memcached just fine.

> So you change how you name the cache keys and make them dependent of, say the max(timestamp) of your hugetable.

Or you could use memcached's existing expiration support.

[tomjen3]

So you make the app update the cache when the customer visits - now the newest version is available and it works.

[fliph]

For some reason, I started reading the story with the assumption that it was a "don't do it this way" tutorial, and I got very nervous towards the end. ("But that's exactly how I use memcache!")

[fizzfur]

hehe, all software documentation should come in 3 forms: Reference, Tutorial and Pop-up Book

[steveklabnik]

_why actually proposed (on a few separate occasions) that there should be more computer books in the 80 page range. More like the Poignant Guide or Nobody Knows Shoes than a dead-tree, slow version of Google.

[sloak]

The real story is in how they push untested code into production just to see what happens. ;)

[bigiain]

Heh - 'cause none of _us_ have ever done that, right? <looks around nervously for any colleagues reading>

[viraptor]

It depends on what they meant by load average of 20. Or specifically, what kind of workload is it. In some cases LA 20 is pretty much standard, in some other you cannot even login to that box anymore. If it was the second case... well - since nothing works anyways, someone might just as well push untested code into production ;)

Although "One day the Sysadmin realizes" is pretty bad. They should have daily trends showing them it's coming days before it actually happened (unless it was some big release / marketing day).

[adamtj]

Programmer and Sysadmin were either very lucky, or not working on anything important, or else they would have been fired or gone out of business. You can't just add caching and magically expect things to work. You have to think hard about expiration policies and test to make sure you aren't going to get wrong answers, or else you need to prove that wrong answers are ok.

[mikeklaas]

"All programming is an exercise in caching."

-Terje Mathisen