gopi_ar's comments

Nice post! Seeing that you integrating with 2 different services, I'm wondering if you looked at other, more integrated, options? Mapbox, HERE and LocationIQ (shameless plug) are options that offer geocoding and maps.

@dbatten, LocationIQ team member here. We have a new API backed by a new geocoding engine that uses additional datasets (OA, GNAF, etc) currently in final stages of BETA. Could you shoot us an email at [email protected] and you can try it out. This should give you rooftop accuracy in a number of countries (US for sure) and street level accuracy in most others.

We did a study on self-hosting vs saas for geocoding. Here it is: https://view.attach.io/ryiFAKzmb

It's a bit marketing-y though :-)

Disclosure: we host an OSM compatible geocoder at https://locationiq.org

We're putting up a mapzen / pelias compatible endpoint soon on our geocoding service.

https://locationiq.org

It's OSM compatible at the moment.

They refunded the 'rate-limit' charge. Phew! Thanks to @jgrahamc, their CTO, for weighing in.

It's just a matter of scale; DevOps is a luxury at our scale. That said, we're getting other devs trained in basic ops to prevent such things..

Interestingly, it's their 7th birthday this week and they announced 'unmetered mitigation'.

https://blog.cloudflare.com/unmetered-mitigation/

"So today, on the first day of our Birthday Week celebration, we make it official for all our customers: Cloudflare will no longer terminate customers, regardless of the size of the DDoS attacks they receive, regardless of the plan level they use. And, unlike the prevailing practice in the industry, we will never jack up your bill after the attack.

Doing so, frankly, is perverse.

We call this Unmetered Mitigation. It stems from a basic idea: you shouldn't have to pay more to be protected from bullies who try and silence you online. Regardless of what Cloudflare plan you use — Free, Pro, Business, or Enterprise — we will never tell you to go away or that you need to pay us more because of the size of an attack. Cloudflare's higher tier plans will continue to offer more sophisticated reporting, tools, and customer support to better tune our protections against whatever threats you face online. But volumetric DDoS mitigation is now officially unlimited and unmetered."

:-|

This makes sense, we can rest easy knowing they didn't break out of the container. Thanks!

Update: I sent an email to the email on that script. And the person at the other end replied and mentioned that he/she is doing it for extra pocket money and was only mining on the server. We aren't going to pursue any legal charges, might even pay the person a bounty for pointing out this vulnerability. I'd like to thank all of you, with special mention to some folks over at reddit for all your help!

Here you go: https://kevinchen.co/blog/postmortem-server-compromised/

We were able to get in touch with the hacker and he told us he was just mining and not stealing stuff. We're still cleaning the whole system; might even pay him/her a bounty for this though.

Yes, it's a Redis vulnerability (caused by bad config on our part) in one container where the firewall was down.

Strange thing if we run 'top' from the main host, all containers running redis say 'statd' as their user; inside the container the user showed 'redis'. We removed nfs and all related files, and now it shows a user ID number. Is this something we should worry about?

Thank you for responding.

We searched the whole system for authorized_keys files and found one created in a /var/lib/redis/ of a staging container (with no firewall) on this host. We then came across the redis vulnerability https://kevinchen.co/blog/postmortem-server-compromised/ . A junior dev had spawned this container without help from dev-ops and hence left ports open.

What doesn't make sense to us is how this daemon (yam) was running under a statd username when the container doesn't have such a user, but the host does? Are LXC containers able to run daemons on the host?

Funny thing is we don't do uploads anywhere and there's no CMS whatsoever.. Which leads us to believe it's an OS vulnerability.

Would you how we could hire professionals to investigate this for us? And report it to appropriate groups..?

PS: These are dedicated servers :-/

This is actually why we had to setup our own servers in the first place!

LocationIQ wasn't meant to compete with paid offerings. It was merely a way for the team at Unwired Labs to give back to the OSM community. We think the good folks at OSM deserves a great free tier.

If geocoding needs are enterprise-grade / you are OK with spending a bit, you should look at Mapzen, OpenCage, and now, Geocodio.

This actually started out as a free-offering for our customers once MapquestOpen started charging unreasonably. Our devs put up LocationIQ as a standalone project that we expect to support fully for a long time to come.

Thanks for your wishes!

Indeed, and we need SSDs on a RAID! I remember we tried importing this on HDDs a year ago and it took 2 weeks and the average response time was 2000ms!

Our current config allows an import in 8 hours and responds within 20ms (not including network latency). It's not cheap though.

We'll check that out right away, thanks!

Haha, sure. :-)