top | item 13607890

GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down

520 points| craigcannon | 9 years ago |blog.ycombinator.com

311 comments

order
[+] bryanh|9 years ago|reply
160 employees remote is impressive and commendable. Zapier is fully remote as well (but half the size in employee count). I'd say "write everything down" is a great shortcut to the sorts of practices you need to cultivate.

We've also noticed that over-communicating is critical but hard - it is surprising the things that are "yeah yeah, we know" to some but are "oh we're doing that?" to others. This is only natural - organizations become complex as they grow, and individuals are busy doing their thing. You often have to bring the important data to them.

On another note, working remote is awesome. I recommend everyone give it a spin once in their careers - but try to find a team that embraces it. I've heard mixed experiences from those who were the single remote person on a team.

[+] Perihelion|9 years ago|reply
Honestly, I haven't seen a company that does the hybrid remote/on-site thing well. I'm sure they exist, but every time I've worked at one the people who were remote were out of the loop on almost everything.

Hybrid remote/on-site requires some great discipline. If you have a watercooler chat with someone about a feature, it's easy to forget that a remote colleague wasn't there for the discussion/not document what was said.

+1 for working remotely on a team that makes an effort to make it work.

[+] dvcrn|9 years ago|reply
The problem with remote, at least for me, is the visa. Being remote-only is nice for traveling around but the sad reality is that you will always be a non-citizen no matter where you go. Even if you really like a place, you will be working from there mostly illegally and can never fully integrate into local things that require bank accounts, IDs and friends. That was maybe the thing I disliked the most when I went completely remote, but for a year or two it's for sure awesome.

In my particular case, I want a company that is remotely open, but sponsors a visa for city/country X which could hold a tiny cheap apartment and a space to call home. But in my case these countries I want to call home are Japan and Korea and now try to find a remote company there.

[+] connorshea|9 years ago|reply
I didn't know Zapier was a fully remote company, that's awesome :)

I pretty much agree with your thoughts about over-communicating, it's super important and we've run into exactly the same things, where people aren't always caught up on everything going on at the company. In my experience this happens in physical workplaces as well, just to a lesser extent.

[+] mwetzler|9 years ago|reply
We're around 50 and we have about half of our company working remotely at any given time - and we love it. We've introduced a lot of communication rituals to make this work. Starting a weekly all-hands with good AV setup was a huge improvement for us. In addition to saving the space for people to ask questions in a big group, it also got us in the habit of preparing data & updates to present to the entire company (e.g. write everything down). It can be monotonous at times, and we resisted big group meetings for 4 years, but it's pretty critical and works well at this size.

We also schedule quarterly travel to the office, and have a weekly meeting where remotes get together and share what they are currently feeling anxious & excited about. That's a remarkably effective way to learn a lot of things you might otherwise pick up on w/ body language, and more.

[+] compuguy|9 years ago|reply
I've been in that situation (the only remote person on a team). With poor use of internal communication tools, it can be almost downright depressing.
[+] rimantas|9 years ago|reply
MySQL AB was about 400 before being bought by Sun and it was said that about 70% were remote so that gives about 280 persons. I was impressed how well organized that work was.
[+] bravura|9 years ago|reply
What tools do you (Zapier, Gitlab, etc.) use for internal documentation?

We have found that many wikis don't have good "related pages" functionality, i.e. they are hard to browse.

[+] deepaksurti|9 years ago|reply
>> 2:41 – GitLab values boring solutions: our product should be exceptional

Exceptional products have exceptional UX. Gitlab IMHO has the worst UX of all git based products out there, I much rather take BitBucket over Gitlab. I tried using Gitlab, but no, I would much rather pay the 7$ to GH for my private repos.

I sincerely hope they make an exceptional product. And 'should' better be 'must'!

[+] kminehart|9 years ago|reply
This is pretty subjective obviously.

I think the GitLab interface is pretty good, especially compared to GitHubs. I can't speak for BitBucket since I've only used it once, but I do remember having a hard time finding my way around it.

[+] amyjess|9 years ago|reply
GitLab may not be as good as some others in the space, but I've never seen a repo UI that actively repulses me nearly as much as BitBucket's. Even SourceForge doesn't put me off like that.

Honestly, all Atlassian products have terrible UI. JIRA is just as bad.

[+] zenlikethat|9 years ago|reply
I'm curious what you dislike about the UX. Personally I like the "feel" of the website generally (though it's not as polished as GitHub) and just find it to be slow, which I think they're working on improving.
[+] Perihelion|9 years ago|reply
Hey Deepak, thanks for the feedback. We've built out a UX team and improving that aspect of the product is high on our list of priorities, but please feel free to reach out if you have any specific issues. Happy to document those for you.
[+] shasheene|9 years ago|reply
> Gitlab IMHO has the worst UX of all git based products out there

Remember that git-based products include Gerrit (a UX which I hear hate about, but frankly, I like a lot)

[+] dandersh|9 years ago|reply
The most frustrating thing about working remote was the complete lack of documentation around anything -- meetings, requirements, internal frameworks/libraries, etc. Good for them for emphasizing documentation.

I lost track of the amount of times I was told to "look at the source code" when asking about documentation for the internal framework.

Also fun was being assigned a feature and having it's functionality explained over the phone, with any subsequent follow-up being met with "Didn't we already talk about this"?

I never would have thought that a remote based team would have the worst documentation out of anywhere I've worked, but that's exactly what happened.

[+] bgaluszka|9 years ago|reply
> I lost track of the amount of times I was told to "look at the source code" when asking about documentation for the internal framework.

Can you elaborate on this one? This is my preferred way.

[+] purerandomness|9 years ago|reply
You seem to work at a horrible place, with abrasive characters.

It begins with the fact that the company's code needs documentation. I'm a big proponent of Robert C. Martin's "Clean Code" and the idea that code should read like prose - by reading the code, its purpose and ideas should come across naturally.

It ends with people asking "Didn't we already talk about this?".

Is this your first job? Don't let yourself get treated like this.

[+] manojlds|9 years ago|reply
After the mess up, I don't really like seeing these posts about Gitlab. Maybe this is their problem after all.
[+] cookiecaper|9 years ago|reply
I think the problem has more to do with their recruiting. To be a successful distributed company, you need a disciplined and highly experienced workforce (note that this does not mean a highly educated workforce, which may actually be a contraindication).

GitLab offers a middling base salary for a role and applies modifiers for experience and city-based cost-of-living (both of which may modify the base downward; my CoL adjustment is -40%, and I live in the United States!). They explicitly state on their hiring page that they prefer people from cheaper areas. [0] [1]

On top of that, just from their headcount, it sounds like they have way too many people. 160 for a technical company like this is hard to manage when the employees are local; it's even harder to manage when employees are remote. Again, for remote work, you need mature, highly-skilled staff with a strong history of successful self-direction. You're not going to get that when you're chopping their salaries in half based on their location. Bad local wages is a major reason to work remotely in the first place.

[0] https://about.gitlab.com/jobs/developer/ ; use the calculator at the bottom of the page

[1] https://about.gitlab.com/handbook/people-operations/global-c... ; gems include "you are required to notify us when you move and we may adjust your compensation up or down" and "[a]ll things being equal we will hire people in lower cost markets vs. higher cost markets."

[+] ridruejo|9 years ago|reply
Every company goes through rough patches, specially one that is growing as fast as they are doing. Very few companies are as open as transparent as they are, even in the fuckups. That is really hard to do and kudos to them
[+] cjbprime|9 years ago|reply
Yes, seems a little tone-deaf to publish a puff piece about GitLab at the moment.
[+] geofft|9 years ago|reply
"This" being remote work?

Also, I'm a little surprised by the anti-GitLab sentiment on this thread. I thought the consensus was that they had bad luck but didn't do anything particularly more wrong than anyone else? (I may have missed some more analysis of the cause of the failure.)

[+] snarf21|9 years ago|reply
Scale is freaking hard. It is absurd to say they figured out the secret after a 7 layer failure on disaster recovery, even more so when one of the main reasons for source control is to have disaster recovery and the ability to rebuild old source.
[+] aedocw|9 years ago|reply
I don't understand why hearing some effective management practices for a distributed team would not be welcome here at any time. Their recent mistake dealt with a specific technical process and some failures around it. That incident in no way invalidates the entire company.

As a remote worker myself, I'm always keen to hear about how teams make it work. I'm also a huge advocate for distributed teams (and nobody having a commute), so personally I'm really glad this link was shared today.

[+] Kiro|9 years ago|reply
Given the complete backlash I'm seeing on HN it seems like transparency actually hurt you. Very sad.
[+] JustSomeNobody|9 years ago|reply
Wow. You all in the comments must be a bunch of perfect people, huh?

Good grief.

Also, nice cameo by the office cat.

[+] sytse|9 years ago|reply
Thanks, but the critics are right. This is not a good time to be promoting ourselves. We should be working on our postmortem and figure out how to improve our service. We're actually doing that, this video was recorded before the outage. But impressions count.
[+] overcast|9 years ago|reply
Imperfect doesn't mean you can be completely negligent of basics.
[+] Cofike|9 years ago|reply
Judging by the number of joke and low hanging responses on this thread I think they should have held this article for a little longer.
[+] sytse|9 years ago|reply
I think we earned a reality check. Have such a promotional video go out after the outage was out of touch on our part. We were aware it would be posted and didn't ask to hold it.
[+] geoelectric|9 years ago|reply
While I was at Mozilla, John O'Duinn gave a pretty great presentation about this sort of stuff.

The core message (mixed with my own takeaway) was that you have to consider all of your offices, even HQ, more like just another field office or coworking site--no location is "central" compared to others. As long as you consider accommodating remote people to be a separate task, it's a task that can be deprioritized. Ideally, anyone in your office should be able to work remotely at moment's notice with little-to-no change in procedure.

Much of his presentation was outlining concrete techniques towards making this actually doable. He still has the slide deck online.

http://oduinn.com/blog/2014/11/09/we-are-all-remoties-nov201...

In practice, of course, real life was imperfect and there's no question that you take a productivity hit over people in the same room. But if you do want a heavily location-agnostic organization as a core value, his take is a nice start.

[+] nedsma|9 years ago|reply
Glad that Gitlab is pushing the remote working boundaries even further. Yes, document everything, write easy to understand code, offer your remote coworkers to call you whenever they need help, leave feedback, praise work as if they're local. It all matters and even more when remotely.
[+] smarx007|9 years ago|reply
It would be so much better if YC would made it a podcast instead (or in addition). Would be a perfect thing to listen on the go.
[+] peterwwillis|9 years ago|reply
I think the worst team I've ever worked on was the one that had a team lead that literally never wrote anything down. Everything was verbal, and what wasn't verbal was private messaged. We would have the same discussion three times because nobody wrote it down the first or second time. Nobody even knew what the changes to the formal design were, because they never had a documented change review.

I always tell people on my team that if it isn't written down, it doesn't exist. When you get into the habit (and learn how to manage all that information) it really saves your bacon.

The other thing that would have saved them from the backup drama was learning how to document and date your to-do list. Suddenly someone notices that you've had a test process for backups on the to-do for over a year and they bring it up at the next meeting.

And another thing: in team meetings, people can write their own meeting notes so everyone becomes responsible for documenting their responsibilities and what affects them. It's easier to do this remote than at a stand-up, because you're already at your keyboard.

[+] siliconc0w|9 years ago|reply
People seem to have some subtle insecurities around broadcasting their activities which contributes a lot to the problem of distributed teams. Ideally all communication goes to everyone and everyone can apply their own client-side filters for what they want to care about. The problem is people want to present a certain face up the org and a different one to their peers so they PM each other and create isolated channels of communication (distribution lists, slack channels, etc). Now have some cognitive overhead of whom to loop in when which creates some other complications because it means if you're receiving a message it's explicit rather than implicit. TLDR - broadcast everything to everyone by default and create a culture of transparency.
[+] wallstquant|9 years ago|reply
That's impressive. Do they use open channels to communicate everything? compensation? Financials?
[+] newsat13|9 years ago|reply
Last I checked a year ago, GitLab had just 80 employees. So, it's now doubled. That is quite some growth. More often than not growing companies just collect lots of employees and then a startup comes and beats them...
[+] gravypod|9 years ago|reply
I can't wait for a remote-work company that has a policy like this to get into some kind of legal troubles that warrents subpoenas. This is a prosecutors gold mine. I feel bad for the defense lawers already.
[+] yuhong|9 years ago|reply
I am taking issues like admission of wrongdoing in settlements seriously now.
[+] mahyarm|9 years ago|reply
Thats what retention policies are for.
[+] whoiskevin|9 years ago|reply
I have to watch a video that then tells me to write everything down? Sorry I couldn't resist. I believe and practice that as much as I can because it makes things easier regardless.
[+] ceejay|9 years ago|reply
I think hindsight will reveal that the things that make a "distributed company" successful are really the same things that make a "localized company" successful. I think it's just that having all employees on-site probably makes it easier for companies to "fake it" and stumble into success through sheer grit and determination. Probably a lot of times even when they have less-than-adequate (but highly motivated) human resources.
[+] ddebernardy|9 years ago|reply
When everyone is on site, there's enough implicit communication going on that you can get away with not documenting things in writing for a lot longer than in a remote team.
[+] Paul-ish|9 years ago|reply
This is interesting, as I interned at Mozilla, and they have a large number of remote employees. There was only one person (other than me) on my team in my location. Mozilla also records a lot of things, but the intent is more to publish that stuff publicly, in the spirit of openness towards the community. For example, all my team meetings had public notes. I wouldn't be surprised if this tendency to record everything for the sake of the community also has positive benefits inside Mozilla.
[+] MrFurious|9 years ago|reply
Obviously, Gitlab don't have the secret for best uptime.
[+] winteriscoming|9 years ago|reply
More than knowing how they manage remote employees, at this point, with more than a week since the data loss incident, I would be genuinely curious to know if their backups are now functional and what plan they have put in place or planning to, to verify backups work.

Furthermore, I won't be surprised if they have seen more concentrated spam attack, after the news of this data loss surfaced.

[+] jgalt212|9 years ago|reply
160 employees? That seems like a lot support a product that is a wrapper around another product. I know the lede is skeptical, but does anyone have any idea how those numbers break down? i.e. is there a large number of sales/support due to complex installs?