Hi, small SaaS founder here (tardis.dev) - I've been heavy Cloudflare Workers user (currently 4 billions requests & 1PB of data per month) for about 4 years already and today at 00:00 UTC without any warning my account was restricted, both website and APIs are down or very very slow to respond/time out, customers are angry obviously. I confirmed with support that "hmm, I see that your zone seems like being restricted due to 2.8 Limitation on Serving Non-HTML Content, see that there's high JSON data transfer". - which is bit strange as I'm using workers which have different terms - https://news.ycombinator.com/item?id=20791660 (confirmed by their CTO)...anyways I get it, perhaps I pay too little and should be on enterprise plan already, but when I got approached by Cloudflare sales team I explicitly asked if I can still be on pay as you go/self server model and reply was: "Enterprise wise, that's up to you and you could likely get away with utilising self-serve as you go, but if you did choose to go enterprise (without R2) I might be able to have something approved in the xx/month range."I would fully understand that I am required to upgrade, but why not sending me an email before shutting down my business completely? I even asked about such scenario on zoom meeting I had with their Sales and they said it will never happen - few weeks forward and here we are...anyways going back to replying to my customers emails regarding service outage.
[+] [-] plesiv|3 years ago|reply
There are so many questions this leaves unanswered:
- Was this a one-off error in Cloudflare's processes? (These things happen on a big enough scale.)
- Were you violating a specific clause of Cloudflare's T&C? How clear was the clause? What did you do to fix this?
- Was the issue that Cloudflare estimated that you're not paying enough given the bandwidth you're consuming? Did you end up signing up for the Enterprise plan?
Transparency would benefit both Cloudflare (in not making people unnecessarily apprehensive about becoming/remaining a customer) and you (in demonstrating that you're handling this issue in a professional and responsible manner).
[+] [-] tardis_thad|3 years ago|reply
Sales over the phone (was fastest) told me that it's good I contacted as otherwise in 24hours my account would be fully banned(whatever it means) and that they will prepare me an offer in 15 minutes, but it was taking longer (no response after an hour or so) and in the meanwhile I wrote Twitter and HN post which CTO of Cloudflare noticed and then after a while I've got another phone call from sales that I should update my ticket to ask unbanning my account as it was approved now by CTO which I did and that solved the issue at least for now - and that's it - no further info what the issue was, still waiting on Enterprise plan quote for me.
[+] [-] runnerup|3 years ago|reply
That would be the right way to give back for customers using us as an amplifier and for corporations relying on us to be a shibboleth (a prefilter so providers know that this is a true issue unsolvable through existing support channels they have established for customers).
Sharing these learnings with other potential founders would also be in line with the raison d’etre of HN. It would provide other founders with lessons they can takeaway and apply to their future startups to maybe do a few of these things right the first time around.
For both sides, HN is “picking up slack” in the system and it would be right to support the community with candid postmortems.
[+] [-] nullcaution|3 years ago|reply
That is rather aggressive?? Maybe thry live in another time zone and are asleep, or have other obligations like school pickup. Given them at least 24 hours to respond. sheesh...
[+] [-] dezb|3 years ago|reply
[+] [-] tiew9Vii|3 years ago|reply
[+] [-] tardis_thad|3 years ago|reply
[+] [-] NicoJuicy|3 years ago|reply
Reality is that Cloudflare serves 60% of the internet and this issue popped up. They are checking it internally what happened, as I understand from jgrahamc.
[+] [-] byteofbits|3 years ago|reply
Cloudflare's sales team and Enterprise pricing model are one of the least effective sales organisations I have encountered in this space. Given the technical nature of their product, it's extremely hard to explain even basic uses of the tool and things like Workers are near impossible to discuss with them. I was really unsurprised to see that OP had a failed Enterprise negotiation with them as I have had the exact same conversation at three different companies now and can imagine perfectly what you were told.
The current offerings of Enterprise and Enterprise Lite simply do not map to the reality of how people use the tool and scale businesses on top of it. I think in part due to Cloudflare's history essentially selling bandwidth and caching, the model is fixated on high binary traffic workloads and simply cannot comprehend the SaaS service model that runs on it and tools like Workers.
This is mostly a rant and hopefully a small +1 signal that this area needs major improvement - but I would also love to hear if anyone else has had interactions with Cloudflare Enterprise and how they found that process?
(Disclaimer: I'm a massive fan of Cloudflare, a user of their products and hold their stock)
[+] [-] tardis_thad|3 years ago|reply
[+] [-] bob1029|3 years ago|reply
I have seen this everywhere. Any large software company seems to operate with 2 completely different heads when it comes to technical sales support.
The "best" experience I've had was with GitHub Enterprise sales, but mostly because they just gave me access to the docs/binaries without much frustration. If I had a bunch of questions about the technology vs cost vs how we actually want use their product, it would have been a substantial nightmare.
[+] [-] asmor|3 years ago|reply
It was pretty novel and refreshing.
[+] [-] IanCal|3 years ago|reply
https://developers.cloudflare.com/workers/examples/return-js...
From the terms
> 2.8 Limitation on Serving Non-HTML Content
> The Services are offered primarily as a platform to cache and serve web pages and websites. Unless explicitly included as part of a Paid Service purchased by you, you agree to use the Services solely for the purpose of (i) serving web pages as viewed through a web browser or other functionally equivalent applications, including rendering Hypertext Markup Language (HTML) *or other functional equivalents, and (ii) serving web APIs subject to the restrictions set forth in this Section 2.8*. Use of the Services for serving video or a disproportionate percentage of pictures, audio files, or other non-HTML content is prohibited, unless purchased separately as part of a Paid Service *or expressly allowed under our Supplemental Terms for a specific Service*. If we determine you have breached this Section 2.8, we may immediately suspend or restrict your use of the Services, or limit End User access to certain of your resources through the Services.
Supplemental terms
> The Cloudflare Developer Platform consists of the following Services: (i) *Cloudflare Workers*, a Service that permits developers to deploy and run encapsulated versions of their proprietary software source code (each a “Workers Script”) on Cloudflare’s edge servers; (ii) Cloudflare Pages, a JAMstack platform for frontend developers to collaborate and deploy websites; (iii) Cloudflare Queues, a managed message queuing service; and (iv) Workers KV, Durable Objects, and R2, storage offerings *used to serve HTML and non-HTML content.*
I can't quite figure out how to parse this such that workers would be deemed unusable to just run an API.
I'd absolutely have gone ahead with using it for an API.
[+] [-] tardis_thad|3 years ago|reply
2.Cloudflare may, with or without notice to you and without liability of any kind, temporarily limit your storage and/or the number of requests you can make or receive using the Developer Platform for any reason (in its sole reasonable discretion), including without limitation, if processing such requests would put an undue burden on the Cloudflare network, adversely impact the Service, or otherwise threaten the integrity of Cloudflare’s networks.
To be fair I'm using lots of requests and bandwidth so could be reason, just if only I got an email about that before shutting everything down.
[+] [-] vishalchandra|3 years ago|reply
When ever there is non-transparent pricing, it's scary to try and use an infrastructure related service.
The sales teams can't go around saying that you are not a profitable customer, and they can't argue with the marketing team to be more honest about pricing on the pricing page.
So, end result, let's bump of these small free loaders. Large enterprise deals is what gets us the bonus anyways.
I like fly.io pricing in that sense. And I am sure there might be others offering a more transparent pricing, otherwise like me still stuck on AWS.
[+] [-] vb-8448|3 years ago|reply
I just repost the same comment I put in the above thread
> The thing that scary me most is that his business get shut down without any notice period (at least the author not mentioning any previous communications from Cloudflare team about the issue).
> This is really a shitty thing from Cloudflare, you cannot shut down an already running business without any notice/grace period.
[+] [-] mynameisvlad|3 years ago|reply
I don’t really feel any sympathy for that poster. They knowingly broke the rules, they had to have known that CF could come and shut them down at any time, and they still went ahead and threw the pity party knowing that they are pretty much entirely in the wrong. It’s very much a “play dumb games, win dumb prizes”.
Would it be nice for CF to give a heads up? Sure. But I don’t think it’s required, and especially not in an egregious case like that one.
[+] [-] tardis_thad|3 years ago|reply
[+] [-] cultofmetatron|3 years ago|reply
[+] [-] rexreed|3 years ago|reply
"The main issue is not that [COMPANY] is working hard to protect itself and its customers, but that customers feel very powerless in these situations. When it takes a massive effort to get attention, especially if you're small and powerless, you feel that you have no control, and that your issues will go unanswered. What can the average, powerless customer who doesn't have the weight of social media, HN, @dang, or others on their side do when their hard-earned money or business is being held, locked, or otherwise prevented, and when the cause is not fraudulent, or if the customer is unaware of that activity? The problem is that accounts are just shut down, moneys are held, and there's no quick or clear communication, with customer support simply saying it's not in their control. It's this feeling of powerlessness that's the issue, regardless of whether or not [COMPANY] is in its rights or doing what it feels is in its and its customers best interests.
What can you do to help empower the powerless customers when their livelihoods are at stake? Can you provide some way to not instantly assume fraud or malicious intent on behalf of the customer and provide some quick and direct way for the customer to feel empowered?"
Having to resort to HN to get major problems resolved that are major customer service and potential legal / liability issues causes me a lot of stress when I realize that I have don't have nearly the same sort of power or influence as some of the others here do on HN. I worry that my complaints would simply go ignored.
@jgrahamc would love you to comment on what we can do to avoid people having to resort to HN for a solution to these problems, which favors the well-connected and squeaky wheels and disfavors everyone else.
[0] https://news.ycombinator.com/item?id=34274456
[+] [-] Dowwie|3 years ago|reply
[+] [-] tiew9Vii|3 years ago|reply
> I would fully understand that I am required to upgrade, but why not sending me an email before shutting down my business completely? I even asked about such scenario on zoom meeting I had with their Sales and they said it will never happen
[+] [-] tardis_thad|3 years ago|reply
[+] [-] twawaaay|3 years ago|reply
It is very easy (relatively) to build a SaaS platform that serves this amount of traffic and this can be done by even a one determined individual or a small startup team.
I don't think it is useful to measure the size of the company in the amount of requests they are serving. Revenue/number of employees are much better measurements saying more about the type of things that are/can be happening. They may have relatively low margins per request and need to get to 4B to get by to pay for couple salaries?
[+] [-] thejosh|3 years ago|reply
[+] [-] danuker|3 years ago|reply
https://framagit.org/dCF/deCloudflare/-/blob/master/readme/e...
[+] [-] wrldos|3 years ago|reply
[+] [-] jamespo|3 years ago|reply
[+] [-] genewitch|3 years ago|reply
And whatever happened to ngate?
[+] [-] andrewstuart|3 years ago|reply
[+] [-] grosswait|3 years ago|reply
[+] [-] jgrahamc|3 years ago|reply
[+] [-] throwaway6845|3 years ago|reply
I say this slightly nervously as a Cloudflare customer who serves some amount of binary data. One message is "it's ok if you're on a paid plan". Another is "it's not ok at any time". My suspicion is that "it's ok unless we notice you".
If you could come up with consistent understandable messaging that would help a lot. I don't mind paying (stay competitive against AWS and Hetzner and that's all I need) but the uncertainty is not good.
[+] [-] rafaelturk|3 years ago|reply
Why Cloudflare cancel paying Workers customers? Makes no sense to me.
[+] [-] nickcw|3 years ago|reply
On the R2 page https://www.cloudflare.com/products/r2/ we see:
> No more egress charges. You shouldn’t have to pay to access your data. Pay no egress charges for data accessed from R2. Our affordable and consistent pricing means no more surprise bills.
Whereas I think the non-HTML traffic terms still apply to R2. Or do they?
[+] [-] tardis_thad|3 years ago|reply
[+] [-] archon810|3 years ago|reply
[+] [-] tlonny|3 years ago|reply
[+] [-] asmor|3 years ago|reply
It's not a company I trust to not randomly screw me over out of the blue anymore.
[+] [-] sgarg26|3 years ago|reply
[+] [-] tardis_thad|3 years ago|reply
I was able to contact via support chat to confirm it's indeed Cloudflare related issue as wasn't sure as it's not displayed in any form on Cloudflare dashboard that indeed account is restricted. That was around 8AM UTC.
Since then I also contacted with sales team (got the details already as they approached me in last few weeks as mentioned before) in order to upgrade to Enterprise plan as it seems like the only solution, but did not get the quote yet and account is still restricted.
[+] [-] is_true|3 years ago|reply
[+] [-] majestic5762|3 years ago|reply
[+] [-] genewitch|3 years ago|reply
[+] [-] andrewstuart|3 years ago|reply
None of Cloudflare's marketing or technical documentation makes any explicit reference to "permitted usages" for Cloudflare services such as R2 and Workers.
This page for example means one thing without any reference to permitted usages and would mean something entirely different if the permitted usages were promoted with the same level of visibility as the benefits.
https://www.cloudflare.com/products/r2/
Nothing here tells me I cannot write my own video serving code with Workers:
https://workers.cloudflare.com/
You might even believe "whatever you need" from this paragraph from the above link:
"Static assets with dynamic power. Say goodbye to build steps which pre-generate thousands of assets in advance. Harness the unrivaled raw power of the edge to generate images, SVGs, PDFs, whatever you need, on the fly, and deliver them to users as quickly as a static asset."
This developer documentation would takes on an entirely new meaning if a link to "acceptable uses" was prominent at the top of each page (not fine print).
https://developers.cloudflare.com/r2/get-started/
https://developers.cloudflare.com/r2/data-access/workers-api...
https://developers.cloudflare.com/r2/examples/demo-worker/
Have built an entire application around assuming there were no such limitations I now need to rebuild elsewhere.
Humph.
I now no longer even understand what "no egress fees" means - in a way that's worse than the big cloud providers where at least you know they are charging you 9 cents per gigabyte.
[+] [-] Roark66|3 years ago|reply
[+] [-] KomoD|3 years ago|reply
[+] [-] matvp|3 years ago|reply
[+] [-] mathattack|3 years ago|reply
In general you can’t trust salespeople and need to get everything in writing. Cloudflare is a prime example of why.
And I’d add in my case because we were keeping track of their promises, we caught them before the sales process completed. It cost them seven figures a year. But maybe it doesn’t matter - their sales approach still has them worth $20 billion.
[+] [-] iinnPP|3 years ago|reply
I asked them to delete my data or provide the Yubi offer and they did neither. So they sit in an email folder known as bad companies. Because my data has value and they lied to obtain it for their own gain (aka fraud).
In Canada we have private prosecution/rules about falsely acquired data. Every bad story on HN puts me closer to opening that folder up and ensuring my data costs at least 100k.
Enough is enough.