barsxl's comments

Automattic | Remote | Full-time | Senior Systems Engineer (Systems Wrangler) | $110-$210k+

We're looking for the world's best systems engineers to help build and maintain the infrastructure that supports over a billion people each month across our entire line of products and services such as WordPress.com, WooCommerce, WordPress VIP, Jetpack, Tumblr, Texts.com, Day One, Pocket Casts and many more.

Here are the goals of the team that you would join:

* Remove as much friction as possible between Automattic developers and their goal of shipping software. When there are questions, we can provide answers in a matter of minutes, sometimes seconds.

* Make Automattic services as fast as possible through optimization of server-side and client-side interactions.

* Maximize the availability and uptime of all services through proactive planning, sound architectural decisions, and rapid response to failures.

* Ensure our services are safe and secure for both our users and employees through a combination of proactive monitoring and enforcement, real-time response, and ensuring data integrity/backups to allow recovery from disasters.

* Share our code, experience, and knowledge, both internally and externally whenever possible. Internally, this is usually accomplished via Slack, IRC, internal blogs, and code reviews. Externally, we try to blog publicly, speak at conferences, contribute to open source software, and release our own software under an open source license.

* Control costs through careful selection of technology partners and prudent negotiations of pricing and contracts. We work with 3rd party services where it makes sense, but always look at the big picture and value control and performance over cost.

https://join.a8c.com/hn

Automattic | Remote | Full-time | Senior Systems Engineer (Systems Wrangler)

We're looking for the world's best systems engineers to help build and maintain the infrastructure that supports over a billion people each month across our entire line of products and services such as WordPress.com, WooCommerce, WordPress VIP, Jetpack, Tumblr, and many more.

Here are the goals of the team that you would join:

* Remove as much friction as possible between Automattic developers and their goal of shipping software. When there are questions, we can provide answers in a matter of minutes, sometimes seconds.

* Make Automattic services as fast as possible through optimization of server-side and client-side interactions.

* Maximize the availability and uptime of all services through proactive planning, sound architectural decisions, and rapid response to failures.

* Ensure our services are safe and secure for both our users and employees through a combination of proactive monitoring and enforcement, real-time response, and ensuring data integrity/backups to allow recovery from disasters.

* Share our code, experience, and knowledge, both internally and externally whenever possible. Internally, this is usually accomplished via Slack, IRC, internal blogs, and code reviews. Externally, we try to blog publicly, speak at conferences, contribute to open source software, and release our own software under an open source license.

* Control costs through careful selection of technology partners and prudent negotiations of pricing and contracts. We work with 3rd party services where it makes sense, but always look at the big picture and value control and performance over cost.

We're currently #4 on DNSPerf.com, can you help us get to #1?

https://join.a8c.com/hn

Automattic | Remote | Full-time | Senior Systems Engineer (Systems Wrangler)

We're looking for the world's best systems engineers to help build and maintain the infrastructure that powers tens of billions of page views and supports over a billion people each month across our entire line of products and services, including WordPress.com, WooCommerce, WordPress VIP, Jetpack, Tumblr, and many more.

As a member of a compact, vertically integrated team overseeing all of Automattic's core services, you'll seize the unique opportunity to foster innovation and enhance our platforms' reliability, scalability, and efficiency. Your skills will be vital in identifying bottlenecks, addressing risks, augmenting infrastructure resilience and performance, and simplifying intricate system design and architectural challenges into scalable solutions.

Automattic is globally distributed and has been since its founding in 2005. Embracing a remote-first work culture, we prioritize flexibility and communication in our collaborative environment.

If you're passionate about simplicity at scale, let us know.

https://join.a8c.com/hn

Thanks for the heads up. We have removed this file from the platform for now while we review.

Thanks for pointing that out. I fixed the post.

> Can you elaborate just a smidge? Is WordPress.com, for example, not encrypting content when it's requested by iTunes? (Thanks!)

Yes, we have some targeted exceptions for incompatible clients.

Many of the larger webhosts have free (but not mandatory) SSL support in production, beta, or on their near-term roadmap.

This happens with the https:// vary rarely. The http:// case happens more often. This is one of the complications in adding HSTS headers with includeSubDomains. Right now, the http:// version works as expected, but if we added HSTS with includeSubDomains for "wordpress.com" those URLs would stop working and generate a SSL error.

Somewhat related - if you have a mapped/custom domain on WordPress.com, even though we are strongly no-www[0] the "www" will work over HTTPS, assuming that DNS for the sub domain points to our servers[1]

0. http://no-www.org/ 1. https://www.1912.me/

Photon[0] does exactly this (and more!) and is free to use as part of Jetpack[1]

0. http://developer.wordpress.com/docs/photon/ 1. https://wordpress.org/plugins/jetpack/

Yes, we're working on it.

> I believe it's breaking podcast feeds being served with WordPress.com, because iTunes doesn't support Let's Encrypt certificates.

Do you have an example? We have already implemented workarounds for iTunes. If they aren't working I would love to know the specifics so we can fix it.

WordPress.com offers two-step authentication for all of our users. You can use any application which supports Time-Based One-Time Passwords (TOTP) such as Google Authenticator, Authy, etc. and you can also receive a one time password via SMS.

WordPress.com has pretty sophisticated brute force detection mechanisms and protections in place. I am not sure why you would say otherwise.

The majority of our transient bugs on WordPress.com are probably related to APC opcode issues. We try to catch them and handle it gracefully, but it isn't always so easy.

Of the 2000 servers, about 90% of them are running something related to WordPress.com. There are 36 "load balancer" servers in total. I added up the req/sec across those 36 machines and came up with the 70k/sec number in the article. The requests aren't evenly distributed across that subset of machines though, so you can't just divide evenly to figure out a req/sec/CPU rate. I left another comment in this thread that mentions 5k req/sec on a "normal" load balancer and that the limiting factor isn't Nginx CPU usage.

WordPress.com currently has 12 load balancers per data center. They are HA and used for different subsets of traffic.

We started using Percona's MySQL builds by default a couple of years ago. There are some nice performance and convenience features not included in the MySQL.com builds. (Un)fortunately we still have quite a few MySQL 4.1 instances happily running and are still using MyISAM across over 360 million tables. Most of that stuff is not running Percona.

Many more dynamic pages than you would expect. We do hundreds of thousands of database queries/sec. We do some caching with Batcache - http://wordpress.org/extend/plugins/batcache/ but it's more to handle large spikes in traffic for single pages/blogs. CNN during the US elections, for example.

As with any tiered web application, the requests aren't evenly distributed across all the servers :) Most of our main Nginx proxies serve about 5k req/sec and have about 50k established connections. They are usually 8 cores + HT for a total of 16 "threads". They aren't at 100% utilization and the limiting factor isn't Nginx CPU usage - it's software interrupts generated by the NICs or bandwidth or something else... We have seen single machines serve upwards of 20k req/sec under "real world" conditions before.

WordPress.com hasn't run on FreeBSD since the TextDrive days which was way before Layered Tech, before me and before WordPress.com was open to the public. We are 100% Debian today, but have used Ubuntu in the past. We never really used HAProxy either. I posted about out load balancer choices back in 2008 - http://barry.wordpress.com/2008/04/28/load-balancer-update/