IMO this is a low quality article that sets up many straw man arguments to push its own agenda to coin a term.
It suggest IaS is better than IaC and dismisses AWS CDK because “it’s not software”.
Meanwhile I’m treating it entirely like software by writing unit tests, snapshot tests, integration tests. It’s composeable. It’s packageable. It’s open source. It’s shareable. You can enforce policies and coding styles. You can put it thru CI.
You can mix using Terraform modules written in Terraform CDK and regular Terraform HCL. Really there's no good reason to use Pulumi (or AWS CDK...) anymore IMHO.
I'm a fan of CF managing state versus TF's way of doing things. I also thing AWS CDK libs have much higher quality compared to the ones I generally find for TF.
I'm looking for opportunities to implement GitOps at $dayjob, but there are some challenges:
- Even developers don't know how to use Git properly. When it's just "code" with no actions associated with it, a mistake can be undone without side effects. With GitOps, a mistake can break infrastructure automatically. In a sense this is no worse than CI/CD, but I've found that many teams are allergic to even that level of automation, let alone CI/ plus automatic infrastructure changes.
- You're going to say something about repo security and policies to prevent uncontrolled merges into 'main', or something like that, right? Okay, now you've reinvented cloud IAM/RBAC and policies, but with none of the tooling around the cloud infrastructure roles and policies that the native platform has. Good luck establishing a new security policy that doesn't hamper productivity by getting in everyone's way, but also enforces reviews and tests!
- Most developers know nothing about infrastructure, and don't want to. With IaS their code has "infrastructure effects" that they can't avoid. It takes an immense amount of training to get people skilled up enough to understand the consequences of their actions. I've had an uphill battle explaining to developers that they need to regularly update their container base images for security!
- SecOps would have a fit, because with GitOps there are virtually no guardrails. Either you allow anyone with commit access to deploy load balancers with public ingress, or you're not really doing GitOps. Either SecOps needs to be involved[1] in Git merges ("DevSecOps"), or they're forced to put roadblocks in place that break the elegance and efficiency of the whole thing.
I like GitOps in principle, but at larger enterprises with random developers and minimal cross-team cooperation it seems like wishful thinking...
[1] For reference, I'm yet to meet anyone in a SecOps role that has ever installed Git, used Git, or knows what a commit even is.
I find it best to start small with this kind of thing.
Have a single team start practicing GitOps, with an eye to eventually accepting contributions from other dev teams. Establish good patterns in IaC repos, write documentation, add linters and other static checks over time. Tools like Atlantis and ArgoCD can provide deployment automation as well as rollbacks, with opportunities for manually diffing and reviewing changes before applying them.
Edited to add: if your "GitOps" implementation involves anything more complicated than the standard "short-lived PR to mainline" pattern, you are doing it wrong. Avoid environment branches at all costs and follow continuous delivery -- deploy incremental changes frequently.
Your workplace is not a good fit for GitOps (right now?). Anything *Ops needs political willpower to make them happen, you as a lowly IC have no leverage to implement something like this successfully. Sorry to be a downer.
Maybe I don't understand what's meant by GitOps - is it the concept of Infrastructure as Code, deployed typically automatically after merge? (Or whatever triggering event.)
I don't necessarily view GitOps as requiring Product Devs to do Infra Stuff. I mean, you're right, people specialize, and you will have some who are deep in the Infra weeds and some who aren't (they're deep in other weeds instead, hopefully).
At my last job (disclaimer: tiny company), we did lots of GitOps. Infra stuff was cordoned off into its own repos generally, which made security controls and such easier for sure. Not sure I get your point there, though, because if you care about insider risk, you need the same style of controls for the product code anyway; it's equally important.
We had some enterprising product devs who contributed to the infra repos, but as you observed, most didn't. Most of the infra code was maintained by the 'Platform Team' (essentially SRE-adjacent).
100% I’ve never seen Gitops implemented in an organization that has been successful. Most devs just do get it and shoot themselves in the foot with it.
This gets into gatekeeping in software. “I’ve built this perfect system and if you don’t get it you’re dumb” or “we just need to create a culture where people do this stuff”
These are unhelpful ideas in an organization. We should be meeting devs where they are and empowering them. Most devs don’t have time to fully understand infra and for 90% of use cases that should be totally fine.
Learn from Golang and Python. You don’t need a perfect system, what’s most helpful is to have a system people understand and can use to produce value.
GitOps not necessarily means developers should define every aspect of it. In my current work, SRE uses gitops and they are the main contributors to most repositories. But things like creating an AWS ECR repo for a new project or creating k8s resources is happily delegated to developers.
The authors (which I know) are failing to recognize all of the issues with Gitops.
Gitops, TF and Kubernetes are all mostly tools for infra people, who then try to push them on the rest of the organization they are a part of, usually causing lots of attrition.
Infrastructure people like Go and need to learn from its principles a bit more. Most people in software organizations are just mediocre devs. They don’t geek out on this stuff like you do, and we need to provide better interfaces that they feel comfortable using.
This creates the pattern where devs need to grab an infra person to do anything or pretend they know how to use a complex language/system that they don’t.
> Writing an application with typescript and infrastructure code in typescript means if you want to upgrade your version of typescript you need to upgrade both at the same time.
No, it really doesn't. I've been writing all my CDK projects in TypeScript for years and in the process I had to upgrade CDK multiple times,and not once did I had to upgrade TypeScript. Ever.
Also, CDK supports half a dozen languages, including JavaScript. Why did the author singles out one of the six as if it was such a critical step?
The article is a terribly uninformed and disingenuous read. I completely wasted a click on it.
And the reason for that is because it uses JSII to convert TypeScript to other languages. And JSII doesn’t understand any complex types. So CDK authors are forced to use very basic primitives. Which doesn’t take away anything from its quality in the end.
CDK project uses types that were probably supported in TypeScript v1.0.
IaC got people to understand at the idea that your infra code should be treated like your app code. Checked in to source control, deployed via COCD etc. Not defined by clicking buttons and stored in opaque databases somewhere.
"IaS" gets at the idea that it's a continuously running process. It's not a provisioning script you run once, there's a live reconciliation loop running in prod alongside your application code.
You lost me at `With terraform, my HCL modules worked no matter what language the service is written in`. With terraform your HCL modules only work if they are written in an obscure language called HCL.
[+] [-] moltar|3 years ago|reply
It suggest IaS is better than IaC and dismisses AWS CDK because “it’s not software”.
Meanwhile I’m treating it entirely like software by writing unit tests, snapshot tests, integration tests. It’s composeable. It’s packageable. It’s open source. It’s shareable. You can enforce policies and coding styles. You can put it thru CI.
How’s that not software?
[+] [-] candiddevmike|3 years ago|reply
[+] [-] pid-1|3 years ago|reply
[+] [-] mountainriver|3 years ago|reply
[+] [-] klysm|3 years ago|reply
[+] [-] jiggawatts|3 years ago|reply
- Even developers don't know how to use Git properly. When it's just "code" with no actions associated with it, a mistake can be undone without side effects. With GitOps, a mistake can break infrastructure automatically. In a sense this is no worse than CI/CD, but I've found that many teams are allergic to even that level of automation, let alone CI/ plus automatic infrastructure changes.
- You're going to say something about repo security and policies to prevent uncontrolled merges into 'main', or something like that, right? Okay, now you've reinvented cloud IAM/RBAC and policies, but with none of the tooling around the cloud infrastructure roles and policies that the native platform has. Good luck establishing a new security policy that doesn't hamper productivity by getting in everyone's way, but also enforces reviews and tests!
- Most developers know nothing about infrastructure, and don't want to. With IaS their code has "infrastructure effects" that they can't avoid. It takes an immense amount of training to get people skilled up enough to understand the consequences of their actions. I've had an uphill battle explaining to developers that they need to regularly update their container base images for security!
- SecOps would have a fit, because with GitOps there are virtually no guardrails. Either you allow anyone with commit access to deploy load balancers with public ingress, or you're not really doing GitOps. Either SecOps needs to be involved[1] in Git merges ("DevSecOps"), or they're forced to put roadblocks in place that break the elegance and efficiency of the whole thing.
I like GitOps in principle, but at larger enterprises with random developers and minimal cross-team cooperation it seems like wishful thinking...
[1] For reference, I'm yet to meet anyone in a SecOps role that has ever installed Git, used Git, or knows what a commit even is.
[+] [-] anchochilis|3 years ago|reply
Have a single team start practicing GitOps, with an eye to eventually accepting contributions from other dev teams. Establish good patterns in IaC repos, write documentation, add linters and other static checks over time. Tools like Atlantis and ArgoCD can provide deployment automation as well as rollbacks, with opportunities for manually diffing and reviewing changes before applying them.
Edited to add: if your "GitOps" implementation involves anything more complicated than the standard "short-lived PR to mainline" pattern, you are doing it wrong. Avoid environment branches at all costs and follow continuous delivery -- deploy incremental changes frequently.
[+] [-] candiddevmike|3 years ago|reply
[+] [-] rwiggins|3 years ago|reply
I don't necessarily view GitOps as requiring Product Devs to do Infra Stuff. I mean, you're right, people specialize, and you will have some who are deep in the Infra weeds and some who aren't (they're deep in other weeds instead, hopefully).
At my last job (disclaimer: tiny company), we did lots of GitOps. Infra stuff was cordoned off into its own repos generally, which made security controls and such easier for sure. Not sure I get your point there, though, because if you care about insider risk, you need the same style of controls for the product code anyway; it's equally important.
We had some enterprising product devs who contributed to the infra repos, but as you observed, most didn't. Most of the infra code was maintained by the 'Platform Team' (essentially SRE-adjacent).
[+] [-] mountainriver|3 years ago|reply
This gets into gatekeeping in software. “I’ve built this perfect system and if you don’t get it you’re dumb” or “we just need to create a culture where people do this stuff”
These are unhelpful ideas in an organization. We should be meeting devs where they are and empowering them. Most devs don’t have time to fully understand infra and for 90% of use cases that should be totally fine.
Learn from Golang and Python. You don’t need a perfect system, what’s most helpful is to have a system people understand and can use to produce value.
[+] [-] mvaliente2001|3 years ago|reply
[+] [-] mountainriver|3 years ago|reply
Infrastructure people like Go and need to learn from its principles a bit more. Most people in software organizations are just mediocre devs. They don’t geek out on this stuff like you do, and we need to provide better interfaces that they feel comfortable using.
This creates the pattern where devs need to grab an infra person to do anything or pretend they know how to use a complex language/system that they don’t.
Ideas like this harm organizations
[+] [-] dilyevsky|3 years ago|reply
> Ideas like this harm organizations
Organizations where this is true have already been harmed irreparably
[+] [-] arinlen|3 years ago|reply
> Writing an application with typescript and infrastructure code in typescript means if you want to upgrade your version of typescript you need to upgrade both at the same time.
No, it really doesn't. I've been writing all my CDK projects in TypeScript for years and in the process I had to upgrade CDK multiple times,and not once did I had to upgrade TypeScript. Ever.
Also, CDK supports half a dozen languages, including JavaScript. Why did the author singles out one of the six as if it was such a critical step?
The article is a terribly uninformed and disingenuous read. I completely wasted a click on it.
[+] [-] moltar|3 years ago|reply
And the reason for that is because it uses JSII to convert TypeScript to other languages. And JSII doesn’t understand any complex types. So CDK authors are forced to use very basic primitives. Which doesn’t take away anything from its quality in the end.
CDK project uses types that were probably supported in TypeScript v1.0.
[+] [-] wmf|3 years ago|reply
[+] [-] mds|3 years ago|reply
IaC got people to understand at the idea that your infra code should be treated like your app code. Checked in to source control, deployed via COCD etc. Not defined by clicking buttons and stored in opaque databases somewhere.
"IaS" gets at the idea that it's a continuously running process. It's not a provisioning script you run once, there's a live reconciliation loop running in prod alongside your application code.
[+] [-] furstenheim|3 years ago|reply
CDK has completely changed my understanding of infrastructure. So cool to be able to abstract and reuse components
[+] [-] mountainriver|3 years ago|reply
[+] [-] ameasure|3 years ago|reply