Joyent: 54+ hours of downtime

[enduser]

I am a customer of Joyent.

For the past two days, the server on which I host some customer websites and mail servers has been down. I did not receive any warning or explanation by email or telephone. When I filed a support ticket, I was given a link to their help site with some status updates.

Status updates (paraphrased):

2011-02-26 09:22:19 GMT: server offline

2011-02-26 14:38:38 GMT: no ETA

2011-02-26 17:09:42 GMT: estimated downtime 12+ hours

2011-02-27 22:23:08 GMT: estimated downtime 20+ hours

That final estimate of 20+ hours is in addition to what is now 37 hours of downtime. Joyent has offered no reparation or backup service (e.g. backup MX service to keep mail from bouncing).

For your consideration when choosing hosting providers.

[jasonhoffman]

I get the frustration. I really do and I apologize for any bump like this. I'm the founder of those old textdrive "share hosting" servers (they really have nothing to do with Joyent's current product line). These servers are ~6 years old and they're largely populated by "lifetime" customers who have tricked out various bits of their accounts enough that an automated migration has been basically impossible. We've been telling everyone for 3 years to really get off of them, we've been offering free upgrades for 4 years, including you btw, you got a free upgrade back in Nov 2007, and there's even support tickets detailing it (which you've participated on). So now we're piecing together everything that was left but those good ole FreeBSD UFS file systems take awhile.

[enduser]

Offering a transition from FreeBSD to Solaris that requires the effort of the customer is not free. Asking a customer who knows FreeBSD to take on administration of a Solaris virtual server for your convenience is not reasonable.

You are offering an unacceptable level of service to customers who have supported and recommended you from your early days. You were paid in advance for a lifetime of service. I took you on your word and offered lifetime hosting to friends with small websites for small businesses. If you can't make good on your agreements, I will eat the cost of moving these sites to Rackspace Cloud. I can't risk another 54 hours of downtime.

With proper backups and a fresh server it should not take 54 hours to get back online.

[jedsmith]

Is the personal account information for the submitting individual really necessary for your point? Did the submitter give you permission to tell us about his ticket history and account actions?

Although relatively harmless, it seems like planting the seed to a disregard for privacy.

[patrickgzill]

What I don't understand is that if the servers are 6 years old, why not migrate the old bad hardware to exactly the same OS, as virtual machines running under the virtualization platform of your choice?

A 6 year old server would have a max RAM capacity of say 32GB, you can get a newish 64GB RAM system for what, $3k or so?

In such a case, migration is basically a few shell scripts plus rsync from old hardware to new VM. You rsync a couple of times until the rsync takes very little time, then turn off services; do a final rsync; turn on services on the new VM, swap IP addresses and test.

You can even run FreeBSD/Xen on top of Solaris XVM, right?

[fingerprinter]

"lifetime" textdrive customer here. I don't ever remember getting a "free" upgrade. I was just offered a $45/month upgrade for a joyent smartmachine, but while a large discount, is not 'free'.

I would be interested in getting my account into a virtual env but I can't really justify paying $45/month at this moment...what are my options?

[pmb]

Free upgrade sounds fine - when were these offered, and how would a person take advantage? I have one of the aforementioned permanent accounts (on jervis, even) and if migrating to a new server means that my site is less likely to fall over, I certainly would do it. However, I just dug through all the emails I ever received from Joyent or Textdrive, and none of them mentioned migrating to another server for free. Lots of stuff about how I could pay more to have more, but since things are basically fine for me (running http://imprompt.us and http://tracyfood.com both of which are low-bandwidth vanilla wordpress sites), I ignored all of that.

tl;dr: how can I go about moving my stuff? To where? And why am I in this handbasket?

[sayrer]

oh dude, I have one of those lifetime accounts. Last summer, I looked into how to migrate, but I got lost in outdated pages. Can you drop some links in this thread?

Maybe you can shut down jervis.textdrive.com if I migrate my account :)

[payingcust]

I am not a lifetime customer, I pay $15/month. I have never received a migration offer and I don't have a "tricked out" account. I only really care about my email and I don't have any now for four days. Fool that I am my registrar has the email of record being the one hosted by joyent so I cant even move until you get my email up and running. I don't think you can even begin to feel my frustration. Despite the new status last night, saying everything is back up for most people, I am not. The Timeliness of support responses is also a joke, my companies email has been down for four days and you appear to be working 9 to 5 on the issue.

[dasil003]

Frankly, I'm surprised you are still on there. I too was an original VC and I left years ago due to the support failings.

Look, we were investors in the truest sense. TextDrive was this revolutionary new shared hosting platform with a lot of promise (flexibility, no overselling, etc). Problem was that shared hosting is a commodity with no profit margins. Since TextDrive was started by people who want to build cool shit, rather than people who knew anything about operations in a customer service industry, the result is a massive series of pivots that left us out in the cold.

I'm not making excuses for anyone here, personally I think Jason Hoffman spent way too much time in the forum sucking up the love, and not enough time figuring out how to run a support operation. So he burnt his bridge with me as a customer. I won't touch Joyent with a ten foot pole no matter how good it is.

But here's the thing: that was their bridge to burn.

It doesn't matter what 200 early customers think, because those 200 one-time payments were not going to build a successful company. The bottom line is that the market for a "premium" shared hosting service got squeezed by commodity hosting on one side, and VPSes on the other. Even if they had managed reasonable uptime (which they didn't because they totally underestimated the technical cost of giving people the freedom they did), it still would have been a doomed market, because VPSes got so cheap anyway. We can complain about what a bad deal we got, but it doesn't change the fact that the last salary we paid was in 2005, and the hosting world has moved on. They did what they had to.

[kylecordes]

I generally like Joyent, and use them. So I offer this tip with a tinge of sadness:

It looks like these message are from a problem with their Howe server. That is one of their old servers, which came over from the Textdrive merger years ago. I used to have some sites hosted on that same server, and was likewise not very happy with things. It appears to me that Joyent as a company doesn't care very much about that legacy equipment and line of business.

The way to make the pain stop, is to stop using Joyent's old TextDrive servers. You can do that by leaving Joyent entirely, or by moving to their newer stuff. Either way it's a fair amount of work, depending on the complexity of what you are hosting.

I took the path of moving to their newer stuff, and have been very happy with Joyent ever since.

[iskander]

I had a lifetime account with TextDrive. I don't know if it was official policy at Joyent to neglect the old customers, but I certainly felt like I was being pressured to repurchase services I had already paid for. After several outages, data loss and extremely slow responses from tech support I switched to linode. No problems since.

[zackola]

Holy crap! Fucking Howe! I was on that server three years ago for about two weeks before I could no longer take it. It was a misconfigured piece of junk when it was brand new. I've had a lot of success with Web Faction since then.

[tastybites]

The way to make the pain stop

No, the way to make the pain stop (and when I say pain, I mean hours and hours of downtime) is to work with a real managed hosting vendor that cares about your servers and your business, and to move your mission critical software off of commodity shared servers where it will get lost in a sea of customers, none of whom have an account executive, a sales representative, or assigned engineers who understands your stack from top to bottom.

Cloud services are great for some things like elastic computing power and rapid prototyping, but having my business' front door hosted on a service where I can't even call someone up in case of trouble seems like absolute craziness. I mean, how would you even implement a disaster recovery/business continuity solution when all the damn servers are in one building?

[jarin]

There seems to be some confusion in this thread about the difference between "free" and "paid for in advance".

Cutco will re-sharpen your knives for free, no matter how old they are, even if you bought them on eBay, because they made a 'lifetime resharpening' promise with the original purchase. And they're basically a multi-level marketing scam.

[frankwiles]

I have one single client who refuses to move off Joyent. My experience with them has been less than stellar. Frequent problems with no explanations, ask a support question like "How does one do X on your system?" and often get "oh you want to do Y I see, here is a link to the support forum for that." and the link isn't at all related to X or even the Y.

They aren't horrible, but I would never use or recommend them to anyone.

[jasonhoffman]

If you're on an old textdrive server, then you need to move, like we've been saying.

[itsnotvalid]

http://www.joyentcloud.com/about/policies/cloud-service-leve...

For clause 2:

  2. Service Level

  Goal: Joyent’s goal is to achieve 100% Availability for all customers.

  Remedy: Subject to Sections 3 and 4 below, if the Availability of customer’s Grid Container is less than 100%, Joyent will credit the customer 5% of the monthly fee for each 30 minutes of downtime (up to 100% of customer’s monthly fee for the affected server).

So probably you can get this month for very little (Takes only ten hours for getting full refund if SLA is signed)

But at the same time their exceptions list are pretty long, things like emergency maintenance and upgrades are considered as exceptions.

[swombat]

Sounds like the OP is getting the service for free anyway...

[fingerprinter]

FOR JASONHOFFMAN

I see you posting quite a bit here...you keep saying the same thing: "take the free upgrade that is being offered".

There also seems to be quite a few old textdrive lifetime members here.

Perhaps the best approach would be for you to outline the steps for those textdrive folks to migrate off of those servers and take the free upgrade you are offering. I personally have no idea how...so if you could explain it, give a link or some concrete information it would be helpful.

I for one would be happy to migrate if it means better servers, I just need to know how...I'm sure I'm not alone.

[tylerritchie]

From one of the very many forum posts on the exact topic:

"If you have not sent in a request and would like to get a jump on the migration, please send an email to support [at] joyent [dot] com with the following information:

- Subject line: migration request + plan type if known (startup, plus, premier) - Full Name - User Name(s) / Primary Domain(s) (please note new info if you want to change it at this time) - Existing Server(s)

If you wish your username or domain name to be changed, please include this in the email and we will process when we send your golden ticket. Waiting until after your account is set up will incur a $50 fee and you will receive a brand new account (old one will be blasted).

Please do not send follow up requests as it only slows the process down. If you asked questions when you sent in your migration request, we will answer those when we send your golden ticket out.

Once you receive your 'golden ticket', you will have 60 days to migrate to the OpenSolaris environment. If you need the second account for more than 60 days, regular monthly charges will apply."

[soulclap]

Haven't read all the comments but this really seems like a 'special case' being an exception and does definitely not apply to the experience and quality you'll usually get on Joyent.

I have been with them for a couple of years and the only reason why I left was because the project I used to host there was put on hiatus and I don't need that kind of 'enterprise' hosting for my other stuff.

I agree there shouldn't be downtimes on any of their machines but just going by the headline, you'd think Joyent is unreliable when indeed their own server plans are rock-solid.

You make it sound like they are Dreamhost. That said, hope your problems will get solved soon.

[enduser]

I will edit my original post as soon as they solve the problem. No response yet, other than excuses and denigration from Jason Hoffman (Joyent founder) in this thread.

Edit: I can no longer edit the original post, but I will post a comment.

[asuth]

We're Joyent customers as well, and their support has always been great, so this is definitely surprising. You should get on the phone with them (they provide an emergency number). What kinds of machines are you running? Is it just one, or many? It shouldn't be necessary, but message me if you're still not getting through and I can ping some contacts.

[jasonhoffman]

Thank you, we actually take our support very seriously and we're pretty accessible. He's talking about a different product than what you have.

[kenmck]

I have a lifetime account as well. Joyent's tech support has always been excellent. I take this as a wakeup call to get my stuff off of the old severs beyond that I can see little fault with Joyent.

[kenmck]

btw. Elapsed time between opening the support request and an answer from someone at Joyent - 9 minutes... Elapsed time from opening support request to receiving account details for an account on a new machine about 26 minutes. That includes the time it took me to get back to them about some details of the account.

[nphase]

I'm sure I'll get down-modded for saying this, but on WHT people will tell you that you get what you pay for, which is in stark contrast with what most people here are saying (lack of gratitude for being a charter customer, etc).

Sure, you paid $200. But if you've been a customer since 2007 then you've basically paid the same price as any cheap shared hosting provider. Contrast this experience with Layered Technologies that ran three price hikes and then when I still didn't leave the server I had with them essentially told me to fuck off once their harddrive failed.

At least jasonhoffman had the courtesy to tell people their best option is to switch servers to avoid trouble.

But, like they say, you get what you pay for. If lifetime support and stability is what you'll come to expect, don't pay bottom dollar prices up front.

Disclaimer: I am not a Joyent customer in any way.

[Klinky]

The better option would be for Jason Hoffman to say that he stands by his offer & will ensure that those who took advantage of the TextDrive deal will continue to see the same level of service or better that they received back when they bought it. Advertise your new services fine, but don't try to upsell people anytime there is downtime & don't go around telling everyone that these people are whiners getting their service for "free".

[jsdalton]

I had some issues with a Textdrive server recently as well and migrated over to one of their smart machines (at a lower price than what I had been paying). I was a bit frustrated like you, but Joyent was extremely, extremely helpful in assisting me with the migration and now everything is running smooth.

I would try to touch base with them again and see if they can help you out.

[jasonhoffman]

And there's no reason that you needed to pay additional unless you bought a different product.

[andrewvc]

I remember trying to get a quote / get a few details about buying large quantities of bandwidth from Joyent a couple years back.

They consistently took days to answer simple questions via email. I generally take it that if a company can't even get its sales people responding quickly there's a snowballs chance in hell it's techs will.

[jasonhoffman]

If it was a couple of years ago and for large quantities of bandwidth, then you would have been talking to me. We didn't have "sales people" a couple of years back. And I would have said "No" because we weren't in the business of selling "large quantities of bandwidth"

[pygorex1]

You get what you pay for.

This isn't a glib statement. A business transaction is about more than coming to agreement about the price of a thing or service. It's also about creating and maintaining incentives for both sides.

I pay slicehost for a VPS. They get $50/month from me in return for disk space, CPU cycles, uptime and support. Our incentives are aligned.

With Joyent/Textdrive, every month you get hosting and every month they get nothing. The incentives are not aligned. In fact, from Joyent's perspective, the incentive to provide any meaningful service diminishes over time. No about of cajoling or complaining or threatening Joyent will change this imbalance. Sure, it's 54 hours of downtime today - but I can guarantee that a year from now the outages will be far longer and far more frequent.

"But we had a deal!" Yes, you did. And you were a fool to enter into it.

Here's the bottom line:

Your customers don't care what's causing the downtime. It's still your fault.

Website users don't care that the deal has soured. It's still your responsibility.

Joyent doesn't care that they're providing cr@ppy service to Textdrive users, despite the postings on HN by Jason Hoffman (hey Jason, instead of wasting time posting on some internet forum, why not spend that time fixing the actual problem?)

Seriously - Joyent doesn't give a sh?t about the lifetime accounts. If they did, then why are you experiencing 54 hours of downtime? If they do care, and are applying resources to solve the problem, then why is it taking so long? 54 HOURS TO FIX A SERVER. That, to me, is evidence of a shocking level of incompetence. In either case I would get the hell out of there.

[oziumjinx]

T minus 10 minutes before this hits TechCrunch

[grandalf]

I've had a very bad experience with Joyent. Abysmal performance, tech support in denial in spite of obvious tests. Just leave, your life will be better.

[sid_g]

I remember that chunkhost has really aggressive pricing.. And I remember there service being pretty good...

[jasonhoffman]

He's currently at "free", so get any more aggressive than that.

[desigooner]

Is there any time that Mr. Hoffman's going to call it quits on this thread and actually take it offline in an attempt to resolve the OP's issues?!

Frankly, the tone of most of his comments have been borderline denigrating and it's doing nothing but adding nuisance value to the thread.

[ericd]

He's probably getting exasperated by the virtual lynch mob which seems to be downvoting everything he says and upvoting every angry comment, which is ridiculous. He's actually saying quite a lot of reasonable and useful stuff which is sitting at 0 or below.

[netik]

joyent had this sort of failure regularly about 4 years ago and continues to do so. Avoid.

[jasonhoffman]

Also incorrect. "About 4 years ago" we always did >99.99% uptimes even on the old textdrive stuff. I'd be happy to show all the old stats and graphs, we still have them.

[payingcust]

I will be moving from joyent. I am paying 15usd per month and have not been told to migrate. now I read the founder saying it's my fault that I am suffering so much downtime because I should have searched the forums and migrated years ago. Well I am going to migrate now.