top | item 28408399

US Air Force chief software officer quits

170 points| Ziggy_Zaggy | 4 years ago |theregister.com

107 comments

order
[+] AndrewKemendo|4 years ago|reply
I'm honestly surprised this is on HN, but it's good that it is.

I worked with Nic on and off for almost his entire tenure while I was CTO for Kessel Run and I can state with full confidence that this is at best him mis-representing his importance and the problems with the DoD IT; and at worst this is his attempt to spin his being fired (or being asked to resign ala Nixon) by the incoming Secretary (timing here is not just a coincidence).

A couple of core points, that are important to keep in mind that have nothing to do with Nic's character, integrity, communication style or technical capabilities (which is a separate and important topic but not suitable for this public forum IMO):

- The CSO position was made up by him, it's not related to any GSA Schedule and it had about the kind of charter you would expect for the position: Namely ill-defined and loosely empowered.

- There was no office of the CSO in the sense that it was not congressionally funded, had no budget, no personnel and no real authority for writing, implementing policy or actually doing engineering.

- Nic never held a clearance, and as a result was never actually involved or aware of most of the programs that he intended to impact

- His primary mission seemed to be to push any organization that was developing software for the USAF to immediately adopt microservices architectures, containers/kubernetes and a couple of very specific "DevSecOps" practices - and specifically to the specifications that he approved/suggested. Make of that what you will.

That said, a lot of what he says is true and IT/network infrastructure, development and test etc... in the DoD is far from modern and in some places completely broken. Other places, where it matters a lot it's like nothing you've ever seen or will likely see in the commercial sector for decades.

Bottom line, I suggest taking this tirade with an EXTREME amount of salt.

[+] _bnmd|4 years ago|reply
Yeah. I aint saying who I am, but I'll anonymously corroborate this. Nic had a lot of good ideas, but he micromanaged the hell out of low-level product decisions, forced us to use specific broken products that made me swear the vendors must have naked photos of him somewhere. He threw new requirements out of left field in the middle of demos that were not real requirements written down anywhere, but for whatever reason, we had to change course in the middle of development and do it anyway.

I'm sitting here right now on a Friday afternoon while the Air Force is on a four-day weekend ahead of labor day, trying to deploy to a broken ass application his DevSecOps reference architecture forces everyone to use, but it doesn't work because it uses a checksum algorithm disabled by FIPS-compliance hardening in this environment, which we have absolutely no control over. The biggest impediment to even getting this far was another vendor enterprise service he forced us to use that was broken until July. We were stuck just waiting on a bug fix to be provided. And, of course, we have to use Iron Bank container images for everything, but Iron Bank container images are themselves perpetually broken. They do security hardening, but no functionality testing, and their process of pushing breaking changes to the same tags can break you in production unexpectedly. And you can't pin to the actual sha because Harbor only holds onto five orphaned shas at a time that don't correspond to a tag.

He's touting a lot of accomplishments here that are accomplishments because pushing broken functionality and calling it done is a very easy way to say you delivered faster than a normal DoD program that has to actually prove it does what it says it does.

[+] phkahler|4 years ago|reply
>> Other places, where it matters a lot it's like nothing you've ever seen or will likely see in the commercial sector for decades.

That's something I'd really like to see. How does that kind of difference come about? My guess is that it requires a certain degree of funding and commitment that may be impossible in wallstreet companies. But what else does it take for an organization to get there?

[+] poulsbohemian|4 years ago|reply
> Other places, where it matters a lot it's like nothing you've ever seen or will likely see in the commercial sector for decades.

Would love to more about this as it sounds like you are implying that their practices and technology are significantly ahead of the commercial sector.

[+] hiram112|4 years ago|reply
> - Nic never held a clearance, and as a result was never actually involved or aware of most of the programs that he intended to impact

I honestly don't know how he'd ever expect to succeed in a job like this without a clearance. Without it, he'd have little practical knowledge of how the actual air gapped networks (SIPR, JWICS, etc.) are designed and implemented.

Even if he didn't know the classified technical details, there's a lot of differences between deploying software in these environments vs the wide open public internet.

From experience, the type of exploits that would be devastating to a big company just aren't a big risk for most apps on these networks. OTOH, classified apps have all sorts of bureaucratic rules that the unclassified world doesn't have to consider.

For example, if this guy is coming from Silicon Valley, he might spend 80% of his time worrying about exploits that cost corporate American billions / year and which are due to unpatched systems which become targets for script-kiddies pounding every IP on the internet again and again, looking to take advantage of dozens of known vulnerabilities.

On the classified networks, these types of exploits just aren't as likely. There aren't fleets of bots in China and Russia trying known root passwords on every machine with port 22 open or trying to access open Mongo or Elastic Search systems on whole IP ranges.

On the other hand, a private sector SecOps guy isn't even considering that they absolutely cannot store multiple big program's databases in the same data center, staffed by the same contracting company. Because there's a huge risk that some rogue IT contractor with general root access in some small signals intelligence "datacenter" in Hawaii or Germany will end up having access to way too much sensitive data that is shared by multiple intel agencies and military branches, and it'd be disastrous if he decided to start taking raw disks home with him each night a la Snowden. The Air Force and other DOD agencies certainly must ensure that a single contractor doesn't have access to too many data sources.

And with everyone moving as fast as they can to AWS, Azure, etc, I'm guessing these type of procedure are far more important to the Air Force than worrying about some development team deploying an app on an older docker image based on some old version of Java or Python with a few known exploits, which could cost a private sector company a lot of pain and money the minute it was deployed on the open internet.

[+] nchaillan|4 years ago|reply
To be clear, I am not gone nor was I asked to leave. In fact, I was asked to stay. I do have a clearance as well and yes sorry I didn't let us get locked into Pivotal like you did and cost taxpayer millions.
[+] enkid|4 years ago|reply
The idea that he could fix Air Force IT in 6 months if empowered seems absolutely ridiculous given the size of the organization. What do you think the US gov needs to change to get better at it?
[+] Ziggy_Zaggy|4 years ago|reply
This is a very insightful and contrasting response.

Do you have any other articles/materials that we can reference for additional information related to this topic?

[+] ElijahLynn|4 years ago|reply
I encourage you to leave this comment on the article itself on The Register since you already made it public here.
[+] ryanmarsh|4 years ago|reply
Other places, where it matters a lot it's like nothing you've ever seen or will likely see in the commercial sector for decades.

It’s weird how the federal govt is like this across the board. Most things are “fine” being held together with bubblegum and duct tape. Some things matter a lot though, and when they do you get to see really smart people apply themselves in ways that are cooler than the movies.

[+] gfosco|4 years ago|reply
Likewise I will take this comment with an equal amount of salt.
[+] spfzero|4 years ago|reply
Thanks for posting this, the lack of a security clearance seems telling. Even contractors often have them. Also, a person in a real position of responsibility would never behave this way, and to be fair to him, they would never be treated this way.
[+] hamburgerwah|4 years ago|reply
^^ lol, political hit jobs representing in these comments
[+] nchaillan|4 years ago|reply
Of course Andrew did his best to get KR locked in to PCF which is now taking us 2 years to walk away from. Well done friend.
[+] _bnmd|4 years ago|reply
I feel Nic's pain. Here is the original article about the talk he gave before leaving: https://www.airforcemag.com/air-force-leadership-chief-softw...

> One of Chaillan’s main concerns is incorporating security into software development, a practice known among IT professionals as DevSecOps. With a lack of basic IT infrastructure, implementing DevSecOps has proven difficult, he said. What’s more, there has been some resistance among those used to the more traditional approach of considering security after development and operations.

We're standing up basically everything ourselves from scratch. The mandate was basically "we have a critical need for a new capability. Here is an AWS account and five developers, so make it happen." That's it. So everything from standing up CI/CD pipelines, to building out a cluster, to configuring storage and networking, to writing and testing the application code, to maintaining environments and deployments, is falling on us, with no support.

I'm not going to say what the product is for reasons of OPSEC, but it is inherently a product that has extremely high security needs. Yet in the rush to be able to tell some high-ranking people we have put an "MVP" in production, we've skimped in every which way it is possible to skimp. I am aware of so many holes in the system, but Air Force pen testers didn't find them, so our product manager is being pushed to go forward and we'll worry about security later.

To my mind, this is absolutely unacceptable for a critical defense system, but nobody is asking my opinion. Supposedly, we keep being told we'll lose funding and get the plug pulled if we don't hit some important milestone at some exact date. By being "agile," we can deliver a broken, insecure "MVP" and "iterate" on it until we have a real product that actually meets its requirements.

You can't do this crap with defense systems. This isn't Etsy. Deploying broken shit has far different implications than when all the exemplars from the DevOps Handbook do it in order to find all their bugs in prod and turn their users into beta testers.

[+] stult|4 years ago|reply
I can echo these concerns having been a contractor working on applications in more than one of the DoD cloud platforms, including CloudOne (a subset of PlatformOne, for readers not familiar with the flurry of DoD cloud offerings that have sprung up over the last few years). I recently changed jobs in no small part because of the massive incompetence on the USAF side. It’s really quite stunning. My entire schedule was eaten up by unnecessary meetings where wholly unqualified USAF officers (current and retired) in PM or similar roles would pontificate endlessly about just absolute nonsense concerns. Like hours of arguing about how to label a button on a form. Constant bike shedding. No users involved meaningfully in the feedback cycle. And I swear all these old USAF guys just straight up hate their users. They will suggest the most user-abusing possible design because they think their users are stupid and need 10000 confirmation dialogs to avoid making mistakes.

And on the legacy non-cloud side of things… it’s a horror show. No CI/CD. No testing (a lot of my job was bolting awkward test harnesses on to existing legacy software to compensate). Inconsistent and ever changing project management systems (they switched from TFS to Jama to Azure DevOps to Jama again and then when I left were talking about moving to JIRA. Our cocontractors were insanely unqualified. They were really proud of how cutting edge they were for adopting git for VCS. In 2019. It’s crazy how bad all of this software is, but at least it wasn’t on some internet connected server before.

[+] eitally|4 years ago|reply
I would wager that Etsy (and most big cloud-native unicorns) probably has far, far, far superior infra, SW & ops in place than just about any gov agency... and the ones that don't (Zoom) get called out and are forced to fix it.
[+] wolverine876|4 years ago|reply
That sounds disturbing. However, that is how the military has done things in other domains for generations, and probably forever.

Remember that the term SNAFU came from the military; watch some WWII through Vietnam depictions of it: Before the modern era of its glorification, the US military was synonymous with absurd, screwed-up systems and policies that the soldiers overcame with chewing gum, duct tape, initiative and a sense of humor. (Some say the reputational change is due to the shift from the draft, which caused a wide segment of the population to be familiar with the military, to volunteer professional personnel, which results in most people having no clue about it.)

I'm not saying it's a good thing or that it shouldn't be improved, but the military (and every large institution) have always had a lot of that crap. I remember a Marine officer telling me that to never fly in one of their tilt-rotor aircraft unless I see a lot of hydraulic fluid on the ground - because if I don't, then it's out of hydraulic fluid. As they explained, they go to war with - their lives depend on - tools made by the lowest bidder.

[+] neilv|4 years ago|reply
> has extremely high security needs ... I am aware of so many holes in the system,

It sounds like you need to have a very serious talk with your management.

I really hope that leads to a resolution of the problem.

If not, lives and nations sometimes depend on these systems working, so you might be obligated to ensure that the right people understand the problem.

[+] mbrodersen|4 years ago|reply
Security is clearly not incentivised in the organisation you are working for. But delivering on schedule and making your superiors look good is. So that’s the rules of the game you have to play with.
[+] nchaillan|4 years ago|reply
You should tell me what those issues are.
[+] evilos|4 years ago|reply
Sidenote, he lists "Push over-the-air software updates to weapon systems (U-2) while flying the jet" in his list of accomplishments. Is this what it sounds like? It sounds like a terrible idea.
[+] _bnmd|4 years ago|reply
Two notes on this:

1) If the military gets it right with anything, it's encryption. This isn't connecting to the aircraft over the Internet using Verisign PKI. You're not gonna man-in-the-middle inject your own code into the update. The only attack vector is the software supply chain itself, but that is already an attack vector regardless of how the software gets loaded.

2) Part of the purpose of being able to do something like this is to push new software capabilities to platforms that can't be brought back to manually do it at all, like satellites in orbit. A software update that doesn't require you to launch a new rocket into space can save billions.

[+] chrisseaton|4 years ago|reply
If you're watching a target, and it's going to be gone in a few hours, and the plane is already in the air, and you want to run a program to run the sensors in a certain way to get what you need, makes sense to me.
[+] panzagl|4 years ago|reply
'Weapon System' is acquisition speak for a project of a certain size that has to go through certain processes involving design, funding, acceptance, etc. Whether it is actually intended to harm someone is somewhat orthogonal to the designation.
[+] chrisseaton|4 years ago|reply
> IT is a highly skilled and trained job; staff it as such

I don't think it's highly trained at all!

What kind of training do major tech companies do? I've never done any in my career, outside my degrees, and not everyone does that even! Is that unusual?

Contrast that with the military, which is obsessive about training and invests a huge amount of time and effort into it throughout your entire career.

So who are we taking lessons from here?

[+] dragontamer|4 years ago|reply
https://www.linkedin.com/pulse/time-say-goodbye-nicolas-m-ch...

This linkedin post seems way more... balanced... than TheRegister.com implied.

[+] 2OEH8eoCRo0|4 years ago|reply
> I realize more clearly than ever before that, in 20 years from now, our children, both in the United States’ and our Allies’, will have no chance competing in a world where China has the drastic advantage of population over the US. If the US can’t match the booming, hardworking population in China, then we have to win by being smarter, more efficient, and forward-leaning through agility, rapid prototyping and innovation. We have to be ahead and lead. We can’t afford to be behind.

> While we wasted time in bureaucracy, our adversaries moved further ahead.

Zoinks! This matches my experience working in defense and is one of my biggest fears.

> I am becoming “technology stale”.

> The DoD is still using outdated water-agile-fall acquisition principles to procure services and talent

So glad that I left the industry. It's infuriating too because it's not a matter of if, but when. When the US faces a determined and modern adversary, the ones paying the price will be the men and women who serve in the military. It won't be the Pentagon brass or defense CEOs paying. This shit keeps me up at night. Worst of all the government has known it's a problem for decades if you read the Defense Innovation Board reports.

https://media.defense.gov/2019/May/01/2002126691/-1/-1/0/SWA...

> Nothing is changing: most of this has been said before and the 1987 DSB report on military software pretty much says it all. What is it going to take to actually do something?

[+] mikewarot|4 years ago|reply
>My office still has no billet and no funding, this year and the next.

From his LinkedIn post... this really is the crux of the matter... they want to whitewash security, not actually implement it.

[+] jrochkind1|4 years ago|reply
> Among the USAF's sins-according-to-Chaillan? The service is still using "outdated water-agile-fall acquisition principles to procure services and talent",

Wait, what?

[+] _bnmd|4 years ago|reply
It's not explained, but I know exactly what he means. We mandate that development teams and integrated product teams have to use agile methods, but the procurement process itself is inherently not agile. Contracts come with fixed dollar amounts, milestone delivery dates, and requirements that need at least signoff from senior agency officials to change and possibly acts of Congress. Further, the way your "customer" is always an acquisition office rather than actual users of your system, developers can't receive, solicit, or respond to direct feedback from users, which is a pretty basic core tenet of agile development, without which it's hard to see how it can ever work.
[+] bink|4 years ago|reply
I presume he means they're claiming to be agile while really continuing to use a waterfall process for deploying software. They create specs, hire a vendor, do regular reviews before testing, approvals, deployment, provide specs for changes, pay for those changes, test, approvals, deploy, and then repeat.
[+] markdjacobsen|4 years ago|reply
HN has two threads on this now. I just replied on the other but will copy here:

I can't speak for Chaillan, but as a military member who led an agile software development team similar to his during the same timeframe, I think he's referring to DoD's fondness for buzzwords.

Because "agile" is the new hotness, every DoD office and vendor tries to slap the language of agile onto a waterfall model. See this wonderful report from the Defense Innovation Board on "Detecting Agile BS": https://media.defense.gov/2018/Oct/09/2002049591/-1/-1/0/DIB...

[+] dragontamer|4 years ago|reply
The full paragraph reads:

> The DoD is still using outdated water-agile-fall acquisition principles to procure services and talent instead of leveraging “Capacity of work” agile contracts to staff teams. Improving acquisition ensures teams have the ability to groom their backlog and move at the pace of relevance. Only Platform One, and teams like Kessel Run, are truly end-to-end agile, from what I have seen to-date.

I don't know what "water-agile-fall" is exactly, but he's probably talking about some terms in the Air Force. Maybe he means that the waterfall model still exists, and a bunch of people are trying (unsuccessfully) to convert to agile. But he's only seen Agile properly happen in a minority of projects.

[+] pbpuckett3|4 years ago|reply
For those of you either contributing to PlatformOne or customers of PlatformOne struggling to get to prod, shoot me a message on LI and we will see if we can support you in cARMY/CReATE. I’m easy to find on LI or 365.
[+] arwhatever|4 years ago|reply
> We would not put a pilot in the cockpit without extensive flight training; why would we expect someone with no IT experience to be close to successful? They do not know what to execute on or what to prioritize which leads to endless risk reduction efforts and diluted focus. IT is a highly skilled and trained job; staff it as such.

Isn't this (general pattern) what led to the creation of the USAF as a separate military branch from the Army?

Perhaps we need a use military branch - The U.S. Software Force!

[+] RobRivera|4 years ago|reply
probably to make more money
[+] nchaillan|4 years ago|reply
I made enough money selling 12 companies.
[+] GartzenDeHaes|4 years ago|reply
Fun fact about the USAF: pilots are selected based on personnel's assessment of a cadet's probability of making general officer. Aptitude for flying and piloting ability have nothing to do with the assessment, which occurs before pilot training. As a result, many Air Force pilots are awful pilots, but they are world class ass kisssers and social climbers.