top | item 17838631

(no title)

minor3rd | 7 years ago

My team deploys probably 5 times in a given day, including Friday. They are all small deploys, and none can happen at the end of the day. If shit hits the fan, we rollback and maybe people figure out the root cause over the weekend but they aren't sweating bullets.

discuss

order

rjzzleep|7 years ago

I guess all the people commenting are much more competent than the people I've worked with.

Even with tests, I've seen startups do double charges on accounts and whatnot on deployments which didn't show up until the next day. I've also seen ops people updating OES where the storage service would segfault a day later. How does DevOps and OES go together in one sentence, you ask? it doesn't, but it just means not all ops people are pure wisdom either. The guy caused others to waste 72 hours of compute resources, because of this. So it's not limited to Dev. And yes, the first company did learn from the double spending bug, but why learn on a saturday?

So even if your DevOps practices are amazing and you have 70% test coverage on all your components, that doesn't mean you can't deploy faulty components where the deployment itself appears successful. Now what? Things aren't failing, they appear just fine. Someone has to go in and debug the problem, it may affect multiple components, it may be critical, and a simple rollback may not cut it.

Friday deployments are fine for certain components, but surely not as a general rule for everything? Friday deployments are like Monday morning or Friday meetings. You can do them, and most of the times they'll be fine, but maybe out of respect for your colleagues you shouldn't anyway.

protomyth|7 years ago

I've worked in places that required a checklist worthy of NASA to deploy software, and also had blackout periods where no changes could be made to the system without an executive team sign-off[1]. The thought of any of them deploying 5 times a day is just beyond anything they would be allowed to do. I would expect most enterprises are closer to the event deploys then rapid, multiple deployments.

In that type of "event" style deployments, week night deploys probably are safe, but anything scheduled for Friday is trouble.

1) common in agriculture during harvest season