liyanchang's comments

See also https://99percentinvisible.org/episode/invisible-women/

"Car crash test dummies are also generally male, based on an average man... and ignores anatomical differences, plus specific individual circumstances like a person being pregnant"

There's lot of places where technology is tilted in unintentional ways

Yeah. I’d love to see the academic paper with formalizations that help me understand the true scope of this problem. Your example is a great one that prompts many questions. Is it possible to travel directly to the commit o(1) or will the code have to calculate the diff of each commit and apply them one at a time o(n) and how much definition and dependency mapping humans need to do to have it work correctly?

I’ll openly admit that we don’t have everything figured out. You’re absolutely right that currently, we constrain ourselves on what we migrate to admittedly simple migrations.

I think there’s a whole set of problems to be solved in this space and frankly, I’m as surprised as anyone that given how SQL is declarative, we use procedural code to do migrations so part of my post was hoping people would tell me what tool I should be using or how this approach fails over time. So your examples are really helpful for me as I think through if it’s possible to do automatically, workaround, or get by without.

It seems to me that we just lack the ability to express these transitions mathematically that can help us do them. And of those, there’s probably only a subset which are possible to do without taking downtime.

In particular, the class of migrations that you outlines are a combination of DDL and DML changes and also have quite a bit of code complexity to do without downtime. It’s definitely a current weakness.

I’ll openly admit that we don’t have everything ironed out. In fact my next big project is to tackle derived columns (rename is a column where the transformation is the identity function).

It requires a bit more finesse and integration into our code base as it requires multiple deploys with code that knows how to handle both columns.

That's correct. The DML changes as part of CI are somewhat new so we haven't ironed it all out yet.

Here's the scenario that I think you're laying out: 1. Commit A creates column foo 2. Commit B has DML that reference column foo 3. Commit C removes column foo

This works fine if our CI deployer does each commit individually. First roll out any schema changes, then run any DML SQL.

However, our deployer might pick up all those changes and since we roll out the schema migrations first (in this case a create + drop -> NOP) and then runs the DML (which will error), this is an issue because of the rollup.

In practice, we have yet to see this case (most of the time, the dev who write the DML is close enough to the code to know if it's going to be dropped soon and we don't drop that many columns - in part because we know that there be dragons) but truthfully, I haven't thought about it much and need to think through what the impact is beyond this example. Thanks for helping me refine my thinking and I'll have something to ponder on this weekend!

This is good to know. As someone who didn't do much with databases before, I was frankly worried given how it didn't seem like many others were taking this approach when it made so much sense (we did have the advantage of having a defined schema which I know isn't always available). Seems like I just didn't know what to search for.

Git would never have worked it required devs to write the up/down patches - why should we have to write the up/down migrations for my schema?

Excited to see more tooling around declarative schema!

I've been really happy with how my current company[0] has been doing migrations and I've seen a couple others do it but it seems like it should be more widespread.

Database Schema as Code

Instead of writing up and down migrations, you define what the end state should look like. Then the computer will figure out how to get here. This is just how the industry started managing server configurations (Puppet) and infrastructure (Terraform).

We use protocol buffers so it was pretty straight forward to have a definition of what our tables should look like. We have a script that figures out what the delta is between two states (either proto files or a db) and can calculate the schema migration SQL (e.g. CREATE TABLE, etc).

From there, we run it through a safety check. Any unsafe migration (either for data loss or performance issues e.g. DROP TABLE) requires an extra approval file.

There's no real difference between an up migration and a down migration (except that one tends to result in an unsafe migrations). It's calculable at CI time so we can give devs a chance to look at what it's going to do and approve any unsafe migrations. API compatability checks enforce that you need to deprecate before you can drop.

DML, that is data changes, are handled via standard check in a sql file and CI will run it before the code deploy and after the schema migration.

Alembic is the one other place I've seen this concept (a couple others mentioned this) so it's not new, but surprised I haven't seen it more places.

[0] Shameless plug: We're hiring if you're interested in changing how healthcare is paid for, delivered, and experienced. https://www.devoted.com/about/tech-jobs/

Devoted Health | Boston, MA | Software Engineers, SREs, Designers, Data Scientists, Head of Corp IT | Full Time | ONSITE | https://www.devoted.com/

We're building a better and more human health insurance company to improve the lives of seniors in America

We are guided by a deep belief that every senior should be treated like we would treat a member of our own family: with loving care and a profound commitment to their health and well-being through world-class technology and customer service.

Stack: Go, Python, Postgres, GraphQL, gRPC, React, TypeScript, Docker/Kubernetes, Terraform, AWS

Disclosure: I'm an engineer at USDS and these are my own opinions.

I'm personally appreciative of the work that you've done and your continued work, in spite of the many challenges and frustrations. There are many ways to serve. It's always incredibly rewarding to be working along side dedicated and talented public servants and contractors. Thank you.

Thanks. I appreciate that. I think you actually have a keen eye for the issues.

Since my time in government, I've become more sympathetic to agency employees, contractors, management - pretty much everyone. The entire environment is really challenging to do the right thing.

As an executive, the typical tools you have at your disposal, budget and HR, are circumscribed by congressional appropriations and federal hiring guidelines. Your ability to promote and demote and reorganize are limited by laws and unions and you definitely can't grant stock options. The metrics that you typically use to measure success are much fuzzier because there isn't a PNL but it's really easy for the press to find a single case where your org does a poor job.

I still maintain hope that good things can be done - I have seen it - and good management makes a huge difference.

Disclosure: I'm an engineer at USDS and these are my own opinions.

Disclosure: I'm an engineer at USDS and these are my own opinions.

We are very proud to be working along side the existing agency and contracting staff. You are right that many times, they do know how to fix it and need some help from us getting management to let them do it. Some times they are skeptical, sometimes there is friction - that's all completely understandable.

Ultimately, my time in government has opened my eyes to the talented folks inside government right now and I'm proud to be working along side them. I hope you didn't get a poor impression of us and the work that is happening.

Disclosure: I'm an engineer at USDS and these are my own opinions.

I absolutely agree that the interface that most citizens use to government can be better. President Obama agrees with you (from his sxsw remarks):

"I could change the politics of America faster than just about anything if I could just take control of all the DMVs in the country."

"If their primary interaction with government is the IRS, you just don’t have a good association with government when you’re writing that check."

USDS has been working with the IRS to help them get their Transcript service back online with better identity proofing though admittedly just a small step in a better online experience. (irs.gov/transcript)

A number of state and local governments have been making improvements as well (much harder to see if your state or your local government has not been). I'd encourage you to search them out or even start it.

Disclosure: I'm an engineer at USDS and these are my own opinions.

We hire for experience, which comes in all sorts of shapes and sizes. Based on my personal observations, the team is actually the most diverse team I've ever worked on by a number of dimensions. We've pulled folks from retirement, others who are chronologically younger but have been contributing to Python and Debian for years, those who have uprooted their family. The job is an incredible mix - it requires being able to debug a slow performing database index one moment, briefing a deputy secretary the next, all while empowering the other federal employees and contractors to do great things.

I would be remiss not to take this opportunity to link to our hiring page. https://www.usds.gov/join

I do recognize that there are lots of talented folks that aren't able to move to DC or can't make it work for a variety of reasons. That's okay. There are many other ways to contribute to our country [0] - working for state/local government, volunteering in your community, being a good parent, inventing the next breakthrough, actually using your turn signals, and many more.

[0] Yes I recognize that not everyone here is US based. Though the point actually is probably true for all countries.

Disclosure: I'm an engineer at USDS and these are my own opinions.

That's right. The US Government is probably not going to win on compensation - salary limits, no stock options, no lunch. However, while it's not a money making enterprise, it's enough to do just fine. I recognize the sacrifices that many of my colleagues have made to be here, not to mention the sacrifices that federal employees and contractors have made (some who are very good and could be making more in the private sector) and that makes this even more worthwhile.

The thing that government can offer is impact. I've always known that government has a big impact on people's lives but not sure if I could personally make an impact. That's what USDS, 18F, a number of other opportunities are offering.

This is not a job for everyone. But if you're a certain type, there's nothing quite like it.

Disclosure: I'm an engineer at USDS and these are my own opinions.

You make a great point - USDS is just one part of the solution. In fact, in almost all of the USDS projects, we work very closely with agency employees and contractors (many of which are just as talented and have chosen to serve their country). Most of the times, I spend very little time hands-on-keyboard and helping empower the existing team.

As for the longer term solution, there is a less publicized version of what folks are doing. USDS has a number of contracting officers who are helping teach others in government how to be savvy customers of technology. 18F and GSA have been doing a lot on this front as well to help bring in really good contractors and writing agile purchasing agreements. The Office of Federal CIO is rewriting and simplifying tech policy and the Office of Federal Procurement Policy has been working on the procurement side. These are the long term changes that I'm personally excited about.

Disclosure: I'm an engineer at USDS and these are my own opinions.

So in my admittedly short time in the government [0], I've witnessed how all of these problems are due to good intentions. That's what makes this all really tough because everything you think is bonkers actually has a reason.

The 1400 page travel regulations is a result of trying to prevent fraud - every single issue that comes up results in a new rule.

The fact that it takes some projects years to deploy is that we would like to plan and make sure that every resource is well-spent, that it's in a number of languages and accessible to the blind.

It makes it hard for everyone - I've met lots of smart talented civil servants and government contractors who want to do things differently but have their hands tied behind their back.

[0] 2 years feels like forever to me but flash in the pan to many of the dedicated civil servants I've met.

Hi Nick,

Agreed! There are lots of places where this tool could be improved! (Disclaimer: I work with people at 18F, Dept. Of Ed, USDS who built this tool, but I wasn't involved and don't have any privileged information and so here are some of my personal thoughts)

If I can add a bit of color:

1. This was developed open-source on GitHub. That means you're welcome to file an issue or comment at https://github.com/18F/college-choice/issues (Given the interest and number of issues filed so far, I'll probably take this opportunity to say that it might be a while before the team gets back to you and appreciate your help and patience!). This shouldn't impress you, but it is, because it's a complete paradigm shift from how government typically works.

2. Average salary is not a great metric and unfortunately, as far as I can tell, the data isn't sliced by major. You're right in your analysis of how it skews the data. I actually think loan repayment is probably the more interesting metric - have others in your income class been able to repay their debts? You're also right that this tool doesn't help people see that choosing college A over B because of a $X,000 delta in average salary isn't a good idea as it's subject to all sorts of personal decisions.

3. This is really cool not because the tool is perfect, but because the government is releasing an API to go along with this - a number of private companies and non-profits have already integrated the data into their tools. This data, which hasn't been possible to get before, can now be integrated into the tools that were helpful to you (the ones that calculate in-state tuition, and ask you for sat/act scores). The underlying API also has many more data fields than the tool exposes. If fact, you can build one yourself! (https://api.data.gov/signup and https://collegescorecard.ed.gov/data/documentation/)

I am not an accountant - I was looking most of these up so do let me know if I got something wrong and I'll edit as needed. Just trying to compile things in one place.

Gross Merchandise Value is how much money flows through your system while Revenue is how much lands in your bank account. For instance, a payments processor like Stripe might have a GMV of $100 million while their revenue would only be the 3% commission (in this case $3 million).

A contract is a legally binding and enforceable document. A letter of intent is when one party outlines what they are likely or would like to do - with some bits of it being enforceable like non-disclosure agreements. A memorandum of understanding is a letter of intent signed by all parties involved - it is still non-binding. A term sheet from a VC is like an LOI - however, it doesn't actually happen until after due diligence, negotiation, etc and only official when signed.

Burn rate is the delta in your bank account. Expenses is how much money left your bank account and revenue is how much entered. Thus burn rate is expenses - revenue and is -1 * profit.

Users are people on your site. Customers are paying users.

Signups are how many people created an account. Active users are how many people logged in over a certain period of time.

Cash flow positive means you have more in your bank account than you did before. However, a kickstarter which raised 1 million would be cash flow positive but not be profitable as it has many outstanding obligations.

Churn rate is the percentage of your users/customers who left over a certain duration. Retention is 1-churn.

Involuntary churn is when the customer leaves because they go out of business or in the case of dating apps, no longer need your services. Voluntary churn is all other churn.

Gross refers revenue - expenses of the product. Net is revenue - expense of the product - administrative costs - depreciation - payroll taxes etc.

Top line is referring to gross while bottom line refers to net. Top line growth means more revenue and bottom line growth means cost cutting.

Oh! Thanks for letting me know. Had no idea about that particular team. Will happily give you both credit next time!

Pensions have a 3 year cliff - but you're only allowed to work for 2 as a schedule A and you don't accrue years as an IC so one can conclude with very high confidence that I will not have a pension :D.