> AWS purportedly puts design documents forward in the form of six-pagers. They start meetings with a 20 minute silent reading session. It's like the book club from hell.
Not just AWS- that's an Amazon-wide technique. And it's freaking amazing. You should try it.
Look, there's two other options. 1- let someone bullshit you with PowerPoint for an hour and skim over critical details. 2- send out the doc ahead of time for everyone to read before the meeting and have every single attendee say "ah yeah I only had time to skim it, but looks good to me". Wasted time.
The Amazon design review meeting involves no bullshitting, no homework before the meeting that gets skipped. We all arrive. We read the entire doc, on paper, red pens in hand. Then we dive into questions on the important bits and anything circled by those red pens.
Generally takes a lot less time as far as meetings go.
[Bias note: I've been at Amazon, but not AWS, for 7 years. These are my own opinions and not company statements]
Perhaps I’m just a particularly slow thinker, but 20 minutes is not enough time for me to read 6 pages of highly technical documentation and consider all of the implications and consequences of the proposed technical design. Even a full one hour meeting is insufficient. I need at least a day. I don’t want to come out of the meeting and two hours later have a “L'esprit de l'escalier” about a flaw in the design I just approved come up.
If I’m being abusively frank, if you can’t be trusted to read a 6 page document the day before a meeting and put some serious thought into it, you’re not doing your job.
> Despite no fewer than 6 attempts to patch the Open S3 Bucket problem, it remains. You can't patch people--legally, anyway.
Heh, I use S3 for hosting static sites only.
2 weeks ago they sent an email saying "[...] your AWS account xxxxxxxx has one or more S3 buckets that allow read or write access from any user on the Internet."
And I go "oh shit, what has write access set up?".
But nothing did. When they said "read or write", they meant 1, the other, or both. They just sent the same ambiguous email to everyone.
"Surely this AWS service can't be as poorly integrated with that other AWS service as it seems, because if it were that poorly integrated, it would be almost completely useless."
I got the same email about S3 and some lambdas, with a link to fix them, that went to some generic page. You'd think Amazon could do better. Why not link me right to the bucket's configuration page? Why not list all the problem buckets in the email?
BTW, the email was sent at 3:45AM EST (my time zone), or a bit after midnight Seattle time.
Dunno if there can be a wrong time to send an email, but I usually like to send important things like resumes around 9AM so it's at the top of the morning inbox.
I had so many clients and people in my company ask me about this. The wording was horrible. It would have been better to identify things properly, or at the least, call it out clearer.
Most businesses can probably run on a single server from OVH (see Paul Tyma's Mailinator architecture for inspiration: 1.2 billion emails on a single server, http://highscalability.com/mailinator-architecture), but I suspect a lot of folks in tech want to pad their resume with cloud buzzwords, so they recommend overcomplicated architectures to business that they absolutely wouldn't spend their own money on. So much of tech design, nay, tech thinking, is based on following fashion trends, it's shocking.
It's really good machine sympathy for keeping the unfunded service cheap to run, but it's one bad fan bearing away from losing a lot of messages, so I wouldn't say its design assures four nines. And the optimization work to get all the way down to n=1 wouldn't have paid off if Tyma needed to hire someone at market rate to build that and maintain it, because an engineer-month costs hundreds of instance-months.
What you describe is both a problem and an opportunity. We live in strange times when people pay through the nose for something that could be easily implemented on dedicated server in much simpler way and with way less risk, all for the fraction of the original amount. But tech comes in circles, and I bet we're going to see some "miraculous discoveries" of how companies saved millions by migrating from AWS to dedicated servers.
We run servers in colos in a bunch of POPs. We've tried to do the "how much would this cost in AWS" a few times now over the years, and it's hilarious how much staggeringly more expensive it is. Even if you do stupid things like assume hardware will depreciate over 2 years (in real life, try 5-10) AWS always ends up being an order of magnitude more expensive. And it's not really business-model specific: we've analyzed compute-heavy/bandwidth-heavy/HA-esque projects and it's always insane how much they get away with charging for services.
Maybe if you're in the "under $1k/mo" range it makes sense to microservice in AWS, but even then VPS hosts are so much cheaper and easier to use.
The difference between the needs of Mailinator ("Email can reside in RAM because it is temporary (3-4 hours)") and most corporate e-mail services are... large. If you treat e-mail as ephemeral, sure, you can run on a single box somewhere, and in the case of a failure, a stateless recovery will have the service up and running again.
Meanwhile, if you're a business, you have actual compliance requirements on how long you have to keep an e-mail around.[1] SarbOx says you have to keep documents related to insider dealings _indefinitely_. Do you, the sysadmin, know which e-mails those are?
So you need backups. If you aren't testing your backups, you might as well not have backups. So you need a second server sometimes to test backup restores on. Do you deal with HIPAA protected information? If so, now you have a bunch of compliance requirements about the security of those e-mails and those backups you just made.[2]
It turns out that "you can save hundreds of dollars a month on e-mail by introducing an existential risk to your business in the event of a lawsuit" is not a great value proposition for most businesses, and most businesses are better off just going with Office 365 or Google Apps for Business, which have dedicated compliance officers and certifications for all of these issues and a lot that I haven't named.[3][4]
Yes, it is possible to overarchitecture things. Yes, there are CIOs and CTOs who want to cargo cult their way to success by imitating successful cloud migrations done by others. But there are a lot of real business problems out there that can be solved by doing things differently than they were done 10 or 20 years ago.
I think you are right. And I started out doing just that with my business. I could easily run the core of the business on a single server. But I have found that what really starts requiring scalability are all the add-ons. Then we needed caching with redis, then elastic stack for logs, then prometheus for monitoring and on and on. Also, OVH doesn't have much presence in the U.S.
> Amazon's managed ElasticSearch offering is awful because it's ElasticSearch.
Had I never used ES or ELK I wouldnt even bat an eyelash. But man... This one hurt me in the tech feels. I already dont like ES when its on premise, I cant imagine on the cloud where you have even less control of it.
I've hit basically every limit on CloudSearch when I was at a previous role. We increased the cluster count and index sizes to their 'absolute limits'. And we were still projected to blow through them by 2020.
I believe there's a project underway now to move it to ElasticSearch, but still, CloudSearch is and should always be considered a prototyping tool for proper ES implementations.
> If you've got an old account charging you 22¢ a month, don't get mad. Start a snarky Twitter account and make sure you cost them orders of magnitude more than that in doing damage control each month instead.
I use S3 for hosting static sites only, and only in North American zones.
Some time ago, I see some billing lines stating:
"US West (Oregon) data transfer to Asia Pacific (Tokyo) 0.001 GB $0.01"
I had no idea why I'd be paying for transfer to a zone outside my own. Obviously I don't care about the 1cent, but my small problem may be someone else's big problem.
Instead of looking into it, they refunded me a month of service (a few dollars).
I guess that's the opposite of @QuinnPig's thought, but seriously, what was the charge for? Someone running their own crawler on EC2 so I paid for internal DC-DC charges?
>I guess that's the opposite of @QuinnPig's thought, but seriously, what was the charge for? Someone running their own crawler on EC2 so I paid for internal DC-DC charges?
Yes. Other customers accessing your publicly available resources pay the internal AWS fees.
Which is nice in that it saves you money, not nice in that it's not super intuitive so if you see it you think you've got some resource sitting around somewhere that shouldn't be.
> If you're a GM of a service at
@awscloud, and you price it at a simple fixed fee of only $X per month, you can expect to be walked out that day.
As a micro-customer, I like the "Only pay for what you use" model.
But AWS charges fixed fees for Route53. You have to pay 50 cents/month/domain when hosting static sites. My volume is small enough that I pay 4x$0.50/month for Route53 DNS, and like 29 cents total for the actual storage/transfer.
The profit margin on that 50cents/month must be 99%.
- Nobody has figured out how to make money from AI/ML other than by selling you a pile of compute and storage for your AI/ML misadventures.
- "AWS stole our open source project and turned it into a service!" is the rallying cry of people who suck at business models.
- Amazon's managed ElasticSearch offering is awful because it's ElasticSearch.
- A major reason to go public cloud that @awscloud can't say outright is "you people freaking suck at running datacenters."
- Route 53 isn't really a database, but then again, neither is Redis.
- MultiCloud is a good idea if you're tetched in the head; it treats cloud solely as "a place to run a bunch of VMs." If that's all you're doing, go you I guess. Bring money!
- Reserved Instances are the best way to take the on-demand promise of the cloud, and eviscerate it completely by forcing customers to think of it like it's an ancient datacenter. "Enjoy your three year planning cycles, schmucks!"
- Baby seals get more hits than the [AWS] forums do.
- "You should deploy everything to be HA across multiple regions" is the rallying cry of armchair architects who don't pay their own AWS bills by a long shot.
- "What does AWS have that GCP doesn't?" "A meaningful customer base"
- There's only one place to see every resource in your AWS organization, in every region: the AWS bill.
- DocumentDB isn't a perfect MongoDB clone yet, and can't be until it's just as good at trashing your production data.
- Netflix has assembled many of the most brilliant engineers on the planet so they can... use @awscloud to stream movies. Draw your own conclusions.
As this thread will draw people that know 'the cloud' I'm wondering if I could get experience learning 'the cloud'?
It it as simple as taking each individual service listed in the AWS console individually and learning exactly how they work, or is there something more in-depth that matters?
Integration between the various services. Any individual service is pretty straightforward, but bringing all the pieces together can be a challenge (security, networking, etc.) Billing is also a big deal - your solution can end up being very expensive.
A better way to learn it, IMO, is to come up with some project, and implement it. For example, a typical webpage with a backend db, some storage, DNS, maybe load balancing. Avoid EC2 options, only because that's too easy (it's just a VM).
It won't be incredibly difficult, but it isn't as easy as spinning up a VM on your local machine.
This is really funny, but I'm curious for people in the know on this: if you were starting from scratch (so no legacy), would GCP or Azure be better? As far as I can tell AWS's main advantage is cost -- is that fair?
>Everyone likes to make fun of outages in us-east-1 that break the internet, but Azure takes outages and everyone's websites all stay up. One wonders why.
huehuehue, doesn't the have i been pwned service use Azure? I can't recall that ever being down due to Azure outages...
[+] [-] mabbo|6 years ago|reply
Not just AWS- that's an Amazon-wide technique. And it's freaking amazing. You should try it.
Look, there's two other options. 1- let someone bullshit you with PowerPoint for an hour and skim over critical details. 2- send out the doc ahead of time for everyone to read before the meeting and have every single attendee say "ah yeah I only had time to skim it, but looks good to me". Wasted time.
The Amazon design review meeting involves no bullshitting, no homework before the meeting that gets skipped. We all arrive. We read the entire doc, on paper, red pens in hand. Then we dive into questions on the important bits and anything circled by those red pens.
Generally takes a lot less time as far as meetings go.
[Bias note: I've been at Amazon, but not AWS, for 7 years. These are my own opinions and not company statements]
[+] [-] falcolas|6 years ago|reply
If I’m being abusively frank, if you can’t be trusted to read a 6 page document the day before a meeting and put some serious thought into it, you’re not doing your job.
[+] [-] chid|6 years ago|reply
[+] [-] Scoundreller|6 years ago|reply
Heh, I use S3 for hosting static sites only.
2 weeks ago they sent an email saying "[...] your AWS account xxxxxxxx has one or more S3 buckets that allow read or write access from any user on the Internet."
And I go "oh shit, what has write access set up?".
But nothing did. When they said "read or write", they meant 1, the other, or both. They just sent the same ambiguous email to everyone.
[+] [-] jjoonathan|6 years ago|reply
"Surely this AWS service can't be as poorly integrated with that other AWS service as it seems, because if it were that poorly integrated, it would be almost completely useless."
"Oh. It is. FML."
[+] [-] anon1m0us|6 years ago|reply
[+] [-] Scoundreller|6 years ago|reply
Dunno if there can be a wrong time to send an email, but I usually like to send important things like resumes around 9AM so it's at the top of the morning inbox.
[+] [-] duxup|6 years ago|reply
[+] [-] mike503|6 years ago|reply
[+] [-] rapind|6 years ago|reply
[+] [-] cheez|6 years ago|reply
[+] [-] vijucat|6 years ago|reply
[+] [-] erik_seaberg|6 years ago|reply
[+] [-] dvfjsdhgfv|6 years ago|reply
[+] [-] parliament32|6 years ago|reply
Maybe if you're in the "under $1k/mo" range it makes sense to microservice in AWS, but even then VPS hosts are so much cheaper and easier to use.
[+] [-] cwyers|6 years ago|reply
Meanwhile, if you're a business, you have actual compliance requirements on how long you have to keep an e-mail around.[1] SarbOx says you have to keep documents related to insider dealings _indefinitely_. Do you, the sysadmin, know which e-mails those are?
So you need backups. If you aren't testing your backups, you might as well not have backups. So you need a second server sometimes to test backup restores on. Do you deal with HIPAA protected information? If so, now you have a bunch of compliance requirements about the security of those e-mails and those backups you just made.[2]
It turns out that "you can save hundreds of dollars a month on e-mail by introducing an existential risk to your business in the event of a lawsuit" is not a great value proposition for most businesses, and most businesses are better off just going with Office 365 or Google Apps for Business, which have dedicated compliance officers and certifications for all of these issues and a lot that I haven't named.[3][4]
Yes, it is possible to overarchitecture things. Yes, there are CIOs and CTOs who want to cargo cult their way to success by imitating successful cloud migrations done by others. But there are a lot of real business problems out there that can be solved by doing things differently than they were done 10 or 20 years ago.
1) https://www.intradyn.com/email-retention-laws/ 2) https://www.hipaaguide.net/hipaa-email-compliance-requiremen... 3) https://docs.microsoft.com/en-us/office365/servicedescriptio... 4) https://support.google.com/googlecloud/answer/6056694?hl=en
[+] [-] jjeaff|6 years ago|reply
[+] [-] giancarlostoro|6 years ago|reply
Had I never used ES or ELK I wouldnt even bat an eyelash. But man... This one hurt me in the tech feels. I already dont like ES when its on premise, I cant imagine on the cloud where you have even less control of it.
[+] [-] c-|6 years ago|reply
I believe there's a project underway now to move it to ElasticSearch, but still, CloudSearch is and should always be considered a prototyping tool for proper ES implementations.
[+] [-] idnefju|6 years ago|reply
[+] [-] Scoundreller|6 years ago|reply
I use S3 for hosting static sites only, and only in North American zones.
Some time ago, I see some billing lines stating:
"US West (Oregon) data transfer to Asia Pacific (Tokyo) 0.001 GB $0.01"
I had no idea why I'd be paying for transfer to a zone outside my own. Obviously I don't care about the 1cent, but my small problem may be someone else's big problem.
Instead of looking into it, they refunded me a month of service (a few dollars).
I guess that's the opposite of @QuinnPig's thought, but seriously, what was the charge for? Someone running their own crawler on EC2 so I paid for internal DC-DC charges?
[+] [-] cthalupa|6 years ago|reply
Yes. Other customers accessing your publicly available resources pay the internal AWS fees.
Which is nice in that it saves you money, not nice in that it's not super intuitive so if you see it you think you've got some resource sitting around somewhere that shouldn't be.
[+] [-] QuinnyPig|6 years ago|reply
[+] [-] wj|6 years ago|reply
[+] [-] ISL|6 years ago|reply
Thanks, @QuinnyPig. I needed that laugh today.
[+] [-] Scoundreller|6 years ago|reply
As a micro-customer, I like the "Only pay for what you use" model.
But AWS charges fixed fees for Route53. You have to pay 50 cents/month/domain when hosting static sites. My volume is small enough that I pay 4x$0.50/month for Route53 DNS, and like 29 cents total for the actual storage/transfer.
The profit margin on that 50cents/month must be 99%.
[+] [-] revicon|6 years ago|reply
[+] [-] donmcronald|6 years ago|reply
[+] [-] etaioinshrdlu|6 years ago|reply
[+] [-] rgoldste|6 years ago|reply
Is or was he ever affiliated with AWS?
[+] [-] jdonald|6 years ago|reply
Does not appear to be affiliated, but launched "Last week in AWS" more than 2 years ago so may know a thing or two.
https://www.linkedin.com/in/coquinn/
https://www.linkedin.com/pulse/last-week-aws-corey-quinn/
https://www.lastweekinaws.com/
[+] [-] peterwwillis|6 years ago|reply
[+] [-] cwyers|6 years ago|reply
"Despite all of the attention Serverless, AI/ML, etc. get on stage, the majority of AWS's income comes from EC2."
[+] [-] erichurkman|6 years ago|reply
[+] [-] akhilcacharya|6 years ago|reply
[+] [-] SomaticPirate|6 years ago|reply
[+] [-] mooreds|6 years ago|reply
[+] [-] eropple|6 years ago|reply
And it still ain't a real cloud.
[+] [-] yodon|6 years ago|reply
It's like the Git Man Page Generator [0] but trained on the AWS docs. Each time you click the "ASW" home page logo, it regenerates the docs.
[0] https://git-man-page-generator.lokaltog.net/
[+] [-] uikoeixueo|6 years ago|reply
It it as simple as taking each individual service listed in the AWS console individually and learning exactly how they work, or is there something more in-depth that matters?
[+] [-] itgoon|6 years ago|reply
A better way to learn it, IMO, is to come up with some project, and implement it. For example, a typical webpage with a backend db, some storage, DNS, maybe load balancing. Avoid EC2 options, only because that's too easy (it's just a VM).
It won't be incredibly difficult, but it isn't as easy as spinning up a VM on your local machine.
Hope this helps.
[+] [-] StavrosK|6 years ago|reply
[+] [-] sdan|6 years ago|reply
Thinking more about it, have companies made much with ML other than in analytics/self driving companies/Google?
[+] [-] overgard|6 years ago|reply
[+] [-] Jedd|6 years ago|reply
Recently from the same person this story[1] on the minefield that is AWS costs for data transit.
So, whether the 'Main advantage is cost' is accurate for you is very much an 'It depends' proposition.
[1] https://news.ycombinator.com/item?id=20972687
[+] [-] dexterdog|6 years ago|reply
[+] [-] eternalny1|6 years ago|reply
[+] [-] timeimp|6 years ago|reply
huehuehue, doesn't the have i been pwned service use Azure? I can't recall that ever being down due to Azure outages...