My belief is that we've been slowly building up to using general purpose languages, one small step at a time, throughout the infrastructure as code, DevOps, and SRE journeys this past 10 years. INI files, XML, JSON, and YAML aren't sufficiently expressive -- lacking for loops, conditionals, variable references, and any sort of abstraction -- so, of course, we add templates to it. But as the author (IMHO rightfully) points out, we just end up with a funky, poor approximation of a language.
I think this approach is a byproduct of thinking about infrastructure and configuration -- and the cloud generally -- as an "afterthought," not a core part of an application's infrastructure. Containers, Kubernetes, serverless, and more hosted services all change this, and Chef, Puppet, and others laid the groundwork to think differently about what the future looks like. More developers today than ever before need to think about how to build and configure cloud software.
We started the Pulumi project to solve this very problem, so I'm admittedly biased, and I hope you forgive the plug -- I only mention it here because I think it contributes to the discussion. Our approach is to simply use general purpose languages like TypeScript, Python, and Go, while still having infrastructure as code. An important thing to realize is that infrastructure as code is based on the idea of a goal state. Using a full blown language to generate that goal state generally doesn't threaten the repeatability, determinism, or robustness of the solution, provided you've got an engine handling state management, diffing, resource CRUD, and so on. We've been able to apply this universally across AWS, Azure, GCP, and Kubernetes, often mixing their configuration in the same program.
Again, I'm biased and want to admit that, however if you're sick of YAML, it's definitely worth checking out. We'd love your feedback:
This is a great analysis, but it's missing a fundamental point: why do we have a problem with these approximations of a programming language or just using a programming language to template stuff?
Because your build then becomes an actual program (i.e. Turing complete) and you have to refactor and maintain it! This is the common problem of using a "programming language as configuration" (e.g. gulp?)
It has the same premises of Pulumi, but without the Turing completeness (I don't know if/how Pulumi avoids that, but if it does it should be part of the pitch), so you cannot shoot yourself in the foot by building an abstraction castle in your build system/infrastructure config.
We use it at work to generate all the Infra-as-Code configurations from a single Dhall config: Terraform, Kubernetes, SQL, etc.
> My belief is that we've been slowly building up to using general purpose languages, one small step at a time, throughout the infrastructure as code, DevOps, and SRE journeys this past 10 years.
I think that you’re right, and I think it’s great, because we have a programming model in which code is data and data is code: Lisp & S-expressions.
It’d be downright awesome to have a Lisp-based system which used dynamic scoping to meld geographical & environmental (e.g. production/development) configuration items. But then, it’d be downright awesome if the world had seriously picked up Lisp in the 80s & 90s, and had spent the last twenty years innovating, rather than reïnventing the wheel, only this time square-shaped. But then, the same thing could be said about Plan 9 …
I’ve not yet had the time to take a look at Pulumi, but I hope to have time soon.
I think what you're doing with pulumi is the right answer and it's only a matter of time before this becomes the norm. The author's examples could easily be done with plain ol' JS/ES/TS with more far more extensibility and customization when the need arises.
I also feel this is where JSX got it right. Instead of creating yet-another-templating-language (looking at you Angular!), they used JavaScript and did a great job of outlining how interpolation works. Any new templating language is always going to be missing some key feature you expect out of a general programming language and your customers will continue to ask for more features.
In ROS we have these XML launch files that are just awful. They have enough features to be a really bad programming language for configuring and launching (often conditionally) numerous robot software nodes.
In ROS2 the launchfile can now just be a Python script. Very much learned all this the hard way and the solution was to just support Python. I think it's brilliant.
- the django like situation: the configuration is pure code, and it's a mistake. It was not necessary, it brought plenty of problems. I wish they went with a templated toml file.
- the ansible like situation: the configuration is templated static text. But with something as complex as deployment, they ended up adding more and more constructs, until they created a monstrous DSL on top of a their implementation language, with zero benefits compared to it and plenty of pitfalls. In that case, they should have made a library, with an API and documentation making an emphasis on best practices.
- and of course a big spectrum between those
The thing is, we see configuration as one big problem, but it's not. Not every configuration scenario has the same constraints and goals. Maybe you need to accept several sources of data. Maybe you need validation. Maybe you need generation. Maybe you to be able to change settings live. Maybe you need to enforce immutable settings. Maybe you need to pub sub your settings. Maybe you need to share them in a central place. Maybe they are just for you. Maybe you want them to be distributed. Maybe you need logic. Maybe you want to be protected from logic. Maybe the user can input settings. Maybe you just read conf. Maybe you generate it.
So many possibilities. And that's why there is not a single configuration tool.
What you would need, is a configuration framework, dealing with things like marging conf, parsing file, getting conf from the network, expressing constraints, etc.
But if you recreate a DSL for your config, it's probably wrong.
I owned the majority of the configuration system and ecosystem for Borg, Google's internal cluster management and application platform.
Unfortunately, what described here is good in many level, but not excellent in any.
If you are OK to describe the complexity of your infrastructure in a programming language similar to the general purpose language, then a well abstracted API built on original APIs from cloud providers are more familiar to devs. And it will be more reliable performance and flexible.
If you want a config experience, something like kustomize is leaner and more compatible with the text config model.
I also cannot see how this interoperate with other tools, which will seriously limit it's appeals to people using other tools.
The problem with code as configuration is that the config file is indeterministic and it takes longer to extract information from the file.
This has long been a problem in the python/pip community, as its basically impossible for the build tools to determine the dependencies of a package without fully downloading and running the setup.py file.
As a happy pulumi user, I have to say I am very impressed with the experience. An order of magnitude improvement on maintainability over our old terraform code base. Highly recommended.
This is my experience and it's clearly biased from maybe one bad example but ... Scons is an example of code over configuration and from what I could tell I never met someone that truly understood it. Because it was code over configuration every programmer added their own interpretation of what was supposed to happen, no programmer truly understood what was really going one and it turned into one giant mess of trying to understand different programmers hacks and code to get the build to work. I'm sure some Scons expert will tell me how I'm full of crap but I'm just saying, that's my experience.
So, what's my point? My point is configuration languages help in that they push "the one true way" and help to enforce it. Sure there are times you end up having to work around the one true way but given very powerful tools of a full language for configuration leads to chaos or at least that's my experience. Instead of being able to glance at the configuration and understand what's happening because it follows the one true way you instead end up with configuration language per programmer since every programmer will code up stuff a different way.
For what it's worth--I've been using Pulumi on a couple of different projects and, today, I couldn't imagine starting a cloud-based project on anything else. The Pulumi team has spent more time than almost anybody I know on understanding how to attack these problems; I guess I have a bit of an understanding of just how much work that is, as I've tried to do the same thing and their solution is better.
I appreciate that their revenue model doesn't require making the open-source version frustrating or stupid and I appreciate that they're incredibly responsive. And some of the stuff you'll see around cloud functions/Lambdas and the deployment thereof will fucking blow your mind.
I have been using ksonnet but that is now officially dead. Working with jsonnet seemed unnecessarily painful when coming from coding typescript. This information is quite timely and welcome, I'll look further at the ts example.
Build files (e.g. makefiles are their various descendants like SCons, rake, etc) seem to be in the same general boat except very early on mixing "real languages" (or at least shell scripting) was obviously allowed so they've always leaned far more towards the "yes, it is a general purpose language" end of the spectrum.
> My belief is that we've been slowly building up to using general purpose languages, one small step at a time, throughout the infrastructure as code, DevOps, and SRE journeys this past 10 years. INI files, XML, JSON, and YAML aren't sufficiently expressive -- lacking for loops, conditionals, variable references, and any sort of abstraction -- so, of course, we add templates to it. But as the author (IMHO rightfully) points out, we just end up with a funky, poor approximation of a language.
This is the why I prefer to use a JS file for configuration instead of native JSON or YAML file if those options are available.
Having dealt with puppet, cloudformation, ansible and other solutions that have gone in and out of fashion and also dealing regularly with Kotlin, Java, Javascript, and recently typescript, my view is that configuration files are essentially DSLs.
DSLs ought to be type safe and type checked since getting things wrong means all kinds of trouble. E.g. with cloudformation I've wasted countless hours googling for all sort of arcane weirdness that amazon people managed to come up with in terms of property names and their values. Getting that wrong means having to dig through tons of obscure errors and output. Debugging broken cloudformation templates is a great argument against how that particular system was designed. It basically requires you know everything listed ever in the vastness of its documentation hell and somehow be able to produce thousands of lines of json/yaml without making a single mistake, which is about as likely as it sounds. Don't get me started on puppet. Very pleased to not have that in my life anymore.
On a positive note, kotlin recently became a supported language for defining gradle build files in. Awesome stuff. Those used to be written in Groovy. The main difference: kotlin is statically compiled and tools like intellij can now tell you when your build file is obviously wrong and autocomplete both the standard stuff as well as any custom things you hooked up. Makes the whole thing much easier to customise and it just removes a whole lot of uncertainty around the "why doesn't this work" kind of stuff that I regularly experience with groovy based gradle files.
Not that I'm arguing using Kotlin in place of Json/yaml. But typescript seems like a sane choice. Json is actually valid javascript, which in turn is valid typescript. Add some interfaces and boom you suddenly have type safety. Now using a number instead of a boolean or string is obviously wrong. Also typescript can do multi line strings, comments, etc. and it supports embedding expressions in strings. No need to reinvent all of that and template JSON when you could just be writing type script.
I recently moved a yaml based localization file to typescript. Only took a few minutes. This resulted in zero extra verbosity (all the types are inferred) but I gained type safety. Any missing language strings are now errors that vs code will tell me about and I can now autocomplete language strings all over the code base which saves me from having to look them up and copy paste them around. So no pain, plenty of gain.
And yes, people are ahead of me and there are actually several projects out there offering typescript support for cloudformation as well.
To go with your general line of thought, see how many JS-based projects are increasingly moving towards a JS file with a default export as a config file.
I know I'm in a minority, but I really dislike YAML... I recently did a lot of Ansible and boy, at the beginning, I was just struggling a lot. Syntactic whitespace kills me.
I don't like it in Python either, but for some reason, when I write Python, it's a lot easier. Maybe YAML is just a bit more complex (and Python has better IDE support..?)
Okay, I'm gonna be the asshole in the room, but how hard is it to just use consistent indentation? I can't count how many times I've heard people complain about significant whitespace in languages.
Not only is it not difficult to begin with, but every code editor and IDE will show you where there's a syntax error in your YAML. People are free to dislike YAML, even for its significant whitespace, but how does it "kill you"?
Look at this example from the article:
```
something: nothing
hello: goodbye
```
This is pure sloppiness, and anyone who has trouble carelessly adding pointless bytes to code, no matter the language, is sloppy. I don't understand why people criticize YAML and Python because "whitespace is hard".
P.S.: There's a similar configuration language called ArchieML, which is similar to YAML but doesn't have significant whitespace.
YAML is a bit bonkers in that it's a superset of JSON (all valid JSON is valid YAML), so if you don't like the whitespace sensitivity, you can write your YAML like this:
{
a: 42,
# But you can have comments!
b: "hello world",
c: "and
multi-line
strings!", # and trailing commas!
}
I find YAML to be almost unusable. IMO it's just not intuitive. If I get to choose a format for my config files I would only use TOML, it's just better (again IMO).
Other than that, no major complaints. My editor understands YAML and shows the indentation level in the background (highlight-indentation-mode) and auto-formats files so they all have consistent indentation (prettier-mode). As a result, it is not much of a nightmare to edit, despite the fact that semantic whitespace COULD cause you a lot of problems.
I like looking at YAML when it doesn't use any of the insane YAML features. Even so I'm not convinced it should be used (at least not as widely as it is) for one big reason: it can be truncated almost anywhere and still be valid. This causes way more issues than you might think. JSON has no such issue - the only case I can think of where you can truncate JSON into valid JSON is if your JSON is just a number.
I generally dislike languages like YAML or Python where whitespace matters, and can break your code, however, YAML is way more easily human readable than JSON, so I started to appreciate it for readability purposes.
I guess YMMV, but after you've used both YAML and JSON for a while, you might appreciate YAML a little bit more.
Writing YAML is easily the part I hate most about writing/deploying software. It's unstructured, feedback cycles tend to be slow (e.g. when deploying k8s configs into prod), and you can't possibly write something useful without documentation pulled up. It's definitely easier to implement than purpose built DSLs, but it's not a good experience.
Python has better IDE support and, maybe more importantly, python does not completely disallow any indentation style so it handles unsupported IDEs better. But it's essentially an IDE support problem.
My life improved a lot since I got an YAML mode for emacs. Now things would be just perfect if haskell's cabal migrated to dhall...
If you have been around long enough you still remember the world that was excited about XML and templating it using XSLT. As a hindsight it was a horrible world.
Even though YAML is not optimal, it is a human friendly compromise between too verbose XML and machine only JSON. It lacks native templating, leading to funny constructs e.g. with Ansible files. However human kind has made progress and will make progress further, so it is just a matter of time until someone comes up with sane "native templated YAML" and all projects will adopt it.
I saw this title and immediately knew the article would be about Helm. I don't think anyone wants to use Helm. People use it for a set-and-forget thing that they don't care about (who cares that it's called impressive-leopard-kubernetes-dashboard, after all.)
It is actually a little bit too magical for my taste, but I continue to use it because it hasn't done anything stupid. I have one file that maps logical names to images in a container repository. If I create a service called "foo" pointing to selector.app.label="foo" in the base, then in production it's called foo-prd and the label magically updates to foo-prd for the selector. It actually understands what it's generating, and while they might have taken it a little bit too far, it's far better than just dumb text replacement.
As others in this thread have said: I ask this question all the time, except s/templating/using/.
YAML is insanely over complicated; it's as bad or worse than XML for config files, and it doesn't even have the nice streaming mode.
Not to mention that it's a bit of a security nightmare (seriously, who put pointers into the YAML spec?).
And, on a more subjective note, YAML is just confusing: between all the significant whitespace and the random single character symbols that no one ever remembers what they do, I never get a YAML document right on the first try.
Templating it really does add a whole new level of headache too.
File-based configs are a troublesome abstraction: they package unrelated concerns into a rigid document whose form must take a particular, application-dependent shape, and the assembly and disassembly of that document essentially becomes an API where key-value pairs are mixed with complex glue code. The application has to do this internally, but anyone who's generating their configs are also doing parts of this externally.
Templates try to bandage over that by drilling down the abstraction to key-value pairs themselves. And imperative constructs that sneak into templating languages are an artifact of wanting to gain expressiveness without losing the benefits of declarative form -- but really, the two are at odds.
YAML is a red herring -- we had the same headaches with XML a decade prior. The problem is always that there's relationships among the data (or even multiple instances of the config) that we care about, but that the structure of a single config file at rest cannot model.
Databases -- let's say, an SQL one -- are actually among the better solutions, because they allow the universe of config items to live in structured places without overspecifying the exact form the data must take when serialized into a file. Then, data can be normalized where it makes sense to avoid repetition and introduce propagation. An SQL database gives all the tools needed to accomplish this, using mostly declarative code.
Databases in a KV sense are often used for configuration, and SQLite's rise has increased richly structured configs that are specified at a higher level than what's typically done with other serialization formats, but the full approach has not caught on outside big enterprise systems and complex applications. Which is a shame, because it's hardly more complex than the current awkward pairing of a full serializer and a templating engine.
I feel this article is missing the bigger problem - one that for some reason just cannot die.
The problem is that of gluing strings together. YAML is not an unstructured text file, it's a tree notation. Whatever "templating" or "generation" mechanism you want to use, it needs to respect the tree nature of the language it operates on. It needs to respect semantics.
Gluing strings together is literally what causes SQL Injection to exist. It caused countless of defacements on the web, and countless of broken websites. I would think we've learned our lessons, but for some reason, I see these template languages still alive and kicking.
There have been times I've wanted to templatize my configuration but I don't want to do it with text-based templates but templates within the configuration files syntax (be it yaml, toml, or something else). Not sure what this is called, I've been calling it "structural templating".
Dhall-lang ( https://dhall-lang.org ) is another, somewhat interesting, attempt to solve this program: it comes with a non-Turing complete programming language, so you can bring some abstraction to your configuration files without having to worry about things like infinite loops.
LOL we are doing the same with k8s. We deploy to any environment, where each environment is a different k8s namespace. We even have a namespace for each developer. The variables are things like the image name (the tag is effectively a git commit id, or sometimes different for not-committed yet stuff). Beyond that just about nothing really needs to be a variable, but we do have different RAM amounts.
We have a couple ways that we template this out, but mostly we literally just do this in bash:
(Where CI_COMMIT_SHA comes from gitlab)
ENV comes from our gitlab CI file.
That all being said, the extent of our k8s integration is lots of stuff like that. We could write a JS file that creates a JSON k8s template, but honestly, that would be more work and more learning than we had to for what we're doing. Why would we do more just because we want to avoid templating in a YAML file?
I think they are missing the real selling point of JSON. It's basically interoperable with JavaScript objects.
That means you write it, send it, store it, operate on it, etc. with little or no modification.
The author says "converting between the two is trivial" which may be true, but the developer overhead is less trivial. And it will always be JSON in the client - JS doesn't support YAML objects.
Off-topic: The whole discussion is about deploy-time configuration management, but our problem is more about run-time configuration management.
We have done the classical memcached+database custom solution, but I was wondering if there is any accepted library/tool to change application run-time behavior. We have tried consul KV store [1], but does not quite fit in our environment.
My ideal solution would be a webapp with some text editor (think codemirror). Changes in this text file would push the configuration data to a running application.
It makes me throw up a little in my mouth every time I see hundreds of lines of YAML to configure something like Traefik with Kubernetes. The worst is when people say they prefer that because "I don't have to write a config file for my backend". That's true but instead now you have extremely verbose configuration mixed in with other verbose configuration.
But in YAML's defense I think it's more of a problem with the tools that use it more so than YAML itself. Ansible is a great example of how amazing YAML can be to manage complex configuration in a concise way.
YAML is a data stream, not a program. Please do not shove programs into data.
Your data does not need to be "expressive", it just needs to provide input to a program. If your data files need to be complex, you need a program to generate them for you.
I've danced the dance of ini -> json -> yaml -> weird hybrid -> embedded logic, and it ends with "program that asks for what the thing you want looks like and generates data files". Industrial software design figured this out ages ago.
* all configuration may be generated from any other format imaginable, but it's sure as fuck going into the Big Main Godlike Application as XML.
Separation, interfaces, etc. Disclaimer: I work in .NET almost exclusively. The .NET configuration APIs generally work, as long as you only ever use them for reading; treating config as something the application itself can fiddle with is a fast route to madness.
I feel like Ansible generating YAML with Jinja templates is really a sweet-spot, with idempotency and reusability.
I find it pushes me to write plain YAML files for variables and defaults (Ansible), while allowing strong templating of generated files (Jinja) and letting the result be readable (YAML). By readable I mean minimum programming bloat (code spread on many lines just to write a for loop) and minimum extra syntax that clutters the screen (brackets and quotes). It also lets me write very little custom code (aside from variables, obviously).
If I had to use a more "powerful" YAML-like replacement, it would mix all these into files, written differently by people with different styles, and it would have bloat all over the place.
The main issue I have with helm is that values.yml is not templatable by default so you have to generate it if you want reusability.
YAML does one thing and does it well, it's readable and bloat-free. Maybe we need more tools like "kubectl explain" to know the syntax though.
[+] [-] joeduffy|7 years ago|reply
I think this approach is a byproduct of thinking about infrastructure and configuration -- and the cloud generally -- as an "afterthought," not a core part of an application's infrastructure. Containers, Kubernetes, serverless, and more hosted services all change this, and Chef, Puppet, and others laid the groundwork to think differently about what the future looks like. More developers today than ever before need to think about how to build and configure cloud software.
We started the Pulumi project to solve this very problem, so I'm admittedly biased, and I hope you forgive the plug -- I only mention it here because I think it contributes to the discussion. Our approach is to simply use general purpose languages like TypeScript, Python, and Go, while still having infrastructure as code. An important thing to realize is that infrastructure as code is based on the idea of a goal state. Using a full blown language to generate that goal state generally doesn't threaten the repeatability, determinism, or robustness of the solution, provided you've got an engine handling state management, diffing, resource CRUD, and so on. We've been able to apply this universally across AWS, Azure, GCP, and Kubernetes, often mixing their configuration in the same program.
Again, I'm biased and want to admit that, however if you're sick of YAML, it's definitely worth checking out. We'd love your feedback:
- Project website: https://pulumi.io/
- All open source on GitHub: https://github.com/pulumi/pulumi
- Example of abstractions: https://blog.pulumi.com/the-fastest-path-to-deploying-kubern...
- Example of serverless as event handlers: https://blog.pulumi.com/lambdas-as-lambdas-the-magic-of-simp...
Pulumi may not be the solution for everyone, but I'm fairly optimistic that this is where we're all heading.
Joe
[+] [-] ff_|7 years ago|reply
Because your build then becomes an actual program (i.e. Turing complete) and you have to refactor and maintain it! This is the common problem of using a "programming language as configuration" (e.g. gulp?)
Dhall solves exactly this problem: https://dhall-lang.org
It has the same premises of Pulumi, but without the Turing completeness (I don't know if/how Pulumi avoids that, but if it does it should be part of the pitch), so you cannot shoot yourself in the foot by building an abstraction castle in your build system/infrastructure config.
We use it at work to generate all the Infra-as-Code configurations from a single Dhall config: Terraform, Kubernetes, SQL, etc.
And there is already an integration with Kubernetes: https://github.com/dhall-lang/dhall-kubernetes
[+] [-] rauhl|7 years ago|reply
I think that you’re right, and I think it’s great, because we have a programming model in which code is data and data is code: Lisp & S-expressions.
It’d be downright awesome to have a Lisp-based system which used dynamic scoping to meld geographical & environmental (e.g. production/development) configuration items. But then, it’d be downright awesome if the world had seriously picked up Lisp in the 80s & 90s, and had spent the last twenty years innovating, rather than reïnventing the wheel, only this time square-shaped. But then, the same thing could be said about Plan 9 …
I’ve not yet had the time to take a look at Pulumi, but I hope to have time soon.
[+] [-] okhudeira|7 years ago|reply
I also feel this is where JSX got it right. Instead of creating yet-another-templating-language (looking at you Angular!), they used JavaScript and did a great job of outlining how interpolation works. Any new templating language is always going to be missing some key feature you expect out of a general programming language and your customers will continue to ask for more features.
Take for example Terraform and HCL, they're continually adding more and more [templating features](https://github.com/hashicorp/terraform/blob/master/website/d...) and [functions](https://www.terraform.io/docs/configuration/interpolation.ht...) because there's so many different ways to skin configuration/infrastructure as code. What if TF just expect a "computed" JSON object and it's left up to the developer to figure out how to put it together?
I'm gonna keep an eye on Pulumi and hope to be able to use in a real project soon.
[+] [-] Waterluvian|7 years ago|reply
In ROS2 the launchfile can now just be a Python script. Very much learned all this the hard way and the solution was to just support Python. I think it's brilliant.
[+] [-] sametmax|7 years ago|reply
- the django like situation: the configuration is pure code, and it's a mistake. It was not necessary, it brought plenty of problems. I wish they went with a templated toml file.
- the ansible like situation: the configuration is templated static text. But with something as complex as deployment, they ended up adding more and more constructs, until they created a monstrous DSL on top of a their implementation language, with zero benefits compared to it and plenty of pitfalls. In that case, they should have made a library, with an API and documentation making an emphasis on best practices.
- and of course a big spectrum between those
The thing is, we see configuration as one big problem, but it's not. Not every configuration scenario has the same constraints and goals. Maybe you need to accept several sources of data. Maybe you need validation. Maybe you need generation. Maybe you to be able to change settings live. Maybe you need to enforce immutable settings. Maybe you need to pub sub your settings. Maybe you need to share them in a central place. Maybe they are just for you. Maybe you want them to be distributed. Maybe you need logic. Maybe you want to be protected from logic. Maybe the user can input settings. Maybe you just read conf. Maybe you generate it.
So many possibilities. And that's why there is not a single configuration tool.
What you would need, is a configuration framework, dealing with things like marging conf, parsing file, getting conf from the network, expressing constraints, etc.
But if you recreate a DSL for your config, it's probably wrong.
[+] [-] justicezyx|7 years ago|reply
Unfortunately, what described here is good in many level, but not excellent in any.
If you are OK to describe the complexity of your infrastructure in a programming language similar to the general purpose language, then a well abstracted API built on original APIs from cloud providers are more familiar to devs. And it will be more reliable performance and flexible.
If you want a config experience, something like kustomize is leaner and more compatible with the text config model.
I also cannot see how this interoperate with other tools, which will seriously limit it's appeals to people using other tools.
[+] [-] devxpy|7 years ago|reply
This has long been a problem in the python/pip community, as its basically impossible for the build tools to determine the dependencies of a package without fully downloading and running the setup.py file.
Static config files are static for a reason!
[+] [-] houqp|7 years ago|reply
[+] [-] tokyodude|7 years ago|reply
So, what's my point? My point is configuration languages help in that they push "the one true way" and help to enforce it. Sure there are times you end up having to work around the one true way but given very powerful tools of a full language for configuration leads to chaos or at least that's my experience. Instead of being able to glance at the configuration and understand what's happening because it follows the one true way you instead end up with configuration language per programmer since every programmer will code up stuff a different way.
[+] [-] eropple|7 years ago|reply
I appreciate that their revenue model doesn't require making the open-source version frustrating or stupid and I appreciate that they're incredibly responsive. And some of the stuff you'll see around cloud functions/Lambdas and the deployment thereof will fucking blow your mind.
It's good. You should strongly consider it.
[+] [-] kross|7 years ago|reply
[+] [-] freddie_mercury|7 years ago|reply
[+] [-] arvinsim|7 years ago|reply
This is the why I prefer to use a JS file for configuration instead of native JSON or YAML file if those options are available.
[+] [-] jonreem|7 years ago|reply
[+] [-] W1ndRunner|7 years ago|reply
[+] [-] jillesvangurp|7 years ago|reply
DSLs ought to be type safe and type checked since getting things wrong means all kinds of trouble. E.g. with cloudformation I've wasted countless hours googling for all sort of arcane weirdness that amazon people managed to come up with in terms of property names and their values. Getting that wrong means having to dig through tons of obscure errors and output. Debugging broken cloudformation templates is a great argument against how that particular system was designed. It basically requires you know everything listed ever in the vastness of its documentation hell and somehow be able to produce thousands of lines of json/yaml without making a single mistake, which is about as likely as it sounds. Don't get me started on puppet. Very pleased to not have that in my life anymore.
On a positive note, kotlin recently became a supported language for defining gradle build files in. Awesome stuff. Those used to be written in Groovy. The main difference: kotlin is statically compiled and tools like intellij can now tell you when your build file is obviously wrong and autocomplete both the standard stuff as well as any custom things you hooked up. Makes the whole thing much easier to customise and it just removes a whole lot of uncertainty around the "why doesn't this work" kind of stuff that I regularly experience with groovy based gradle files.
Not that I'm arguing using Kotlin in place of Json/yaml. But typescript seems like a sane choice. Json is actually valid javascript, which in turn is valid typescript. Add some interfaces and boom you suddenly have type safety. Now using a number instead of a boolean or string is obviously wrong. Also typescript can do multi line strings, comments, etc. and it supports embedding expressions in strings. No need to reinvent all of that and template JSON when you could just be writing type script.
I recently moved a yaml based localization file to typescript. Only took a few minutes. This resulted in zero extra verbosity (all the types are inferred) but I gained type safety. Any missing language strings are now errors that vs code will tell me about and I can now autocomplete language strings all over the code base which saves me from having to look them up and copy paste them around. So no pain, plenty of gain.
And yes, people are ahead of me and there are actually several projects out there offering typescript support for cloudformation as well.
[+] [-] crooked-v|7 years ago|reply
[+] [-] Aeolun|7 years ago|reply
Thanks for plugging!
[+] [-] saberience|7 years ago|reply
[+] [-] BossingAround|7 years ago|reply
I don't like it in Python either, but for some reason, when I write Python, it's a lot easier. Maybe YAML is just a bit more complex (and Python has better IDE support..?)
[+] [-] ravenstine|7 years ago|reply
Okay, I'm gonna be the asshole in the room, but how hard is it to just use consistent indentation? I can't count how many times I've heard people complain about significant whitespace in languages.
Not only is it not difficult to begin with, but every code editor and IDE will show you where there's a syntax error in your YAML. People are free to dislike YAML, even for its significant whitespace, but how does it "kill you"?
Look at this example from the article:
```
something: nothing
```This is pure sloppiness, and anyone who has trouble carelessly adding pointless bytes to code, no matter the language, is sloppy. I don't understand why people criticize YAML and Python because "whitespace is hard".
P.S.: There's a similar configuration language called ArchieML, which is similar to YAML but doesn't have significant whitespace.
http://archieml.org
[+] [-] atombender|7 years ago|reply
[+] [-] ilovetux|7 years ago|reply
[+] [-] jrockway|7 years ago|reply
[+] [-] Slippery_John|7 years ago|reply
[+] [-] nerdponx|7 years ago|reply
[+] [-] bluedino|7 years ago|reply
I wasn’t aware anyone liked it
[+] [-] illumin8|7 years ago|reply
I guess YMMV, but after you've used both YAML and JSON for a while, you might appreciate YAML a little bit more.
[+] [-] bpicolo|7 years ago|reply
[+] [-] marcosdumay|7 years ago|reply
My life improved a lot since I got an YAML mode for emacs. Now things would be just perfect if haskell's cabal migrated to dhall...
[+] [-] andrewflnr|7 years ago|reply
[+] [-] aetherlord|7 years ago|reply
[+] [-] miohtama|7 years ago|reply
Even though YAML is not optimal, it is a human friendly compromise between too verbose XML and machine only JSON. It lacks native templating, leading to funny constructs e.g. with Ansible files. However human kind has made progress and will make progress further, so it is just a matter of time until someone comes up with sane "native templated YAML" and all projects will adopt it.
[+] [-] jrockway|7 years ago|reply
kustomize is much more sane for your own stuff: https://github.com/kubernetes-sigs/kustomize
It is actually a little bit too magical for my taste, but I continue to use it because it hasn't done anything stupid. I have one file that maps logical names to images in a container repository. If I create a service called "foo" pointing to selector.app.label="foo" in the base, then in production it's called foo-prd and the label magically updates to foo-prd for the selector. It actually understands what it's generating, and while they might have taken it a little bit too far, it's far better than just dumb text replacement.
[+] [-] SamWhited|7 years ago|reply
YAML is insanely over complicated; it's as bad or worse than XML for config files, and it doesn't even have the nice streaming mode. Not to mention that it's a bit of a security nightmare (seriously, who put pointers into the YAML spec?).
And, on a more subjective note, YAML is just confusing: between all the significant whitespace and the random single character symbols that no one ever remembers what they do, I never get a YAML document right on the first try.
Templating it really does add a whole new level of headache too.
[+] [-] niftich|7 years ago|reply
Templates try to bandage over that by drilling down the abstraction to key-value pairs themselves. And imperative constructs that sneak into templating languages are an artifact of wanting to gain expressiveness without losing the benefits of declarative form -- but really, the two are at odds.
YAML is a red herring -- we had the same headaches with XML a decade prior. The problem is always that there's relationships among the data (or even multiple instances of the config) that we care about, but that the structure of a single config file at rest cannot model.
Databases -- let's say, an SQL one -- are actually among the better solutions, because they allow the universe of config items to live in structured places without overspecifying the exact form the data must take when serialized into a file. Then, data can be normalized where it makes sense to avoid repetition and introduce propagation. An SQL database gives all the tools needed to accomplish this, using mostly declarative code.
Databases in a KV sense are often used for configuration, and SQLite's rise has increased richly structured configs that are specified at a higher level than what's typically done with other serialization formats, but the full approach has not caught on outside big enterprise systems and complex applications. Which is a shame, because it's hardly more complex than the current awkward pairing of a full serializer and a templating engine.
[+] [-] TeMPOraL|7 years ago|reply
I feel this article is missing the bigger problem - one that for some reason just cannot die.
The problem is that of gluing strings together. YAML is not an unstructured text file, it's a tree notation. Whatever "templating" or "generation" mechanism you want to use, it needs to respect the tree nature of the language it operates on. It needs to respect semantics.
Gluing strings together is literally what causes SQL Injection to exist. It caused countless of defacements on the web, and countless of broken websites. I would think we've learned our lessons, but for some reason, I see these template languages still alive and kicking.
[+] [-] epage|7 years ago|reply
So far the only things close to this are
- Azure pipeline's syntax: https://docs.microsoft.com/en-us/azure/devops/pipelines/proc... - Something called Jasonette: https://docs.jasonette.com/templates/ - Something called Jsonnet: https://jsonnet.org/
Azure Pipeline's approach I think is closest to what I've been looking for.
Anything else in this space?
[+] [-] fiddlerwoaroof|7 years ago|reply
[+] [-] MrStonedOne|7 years ago|reply
[+] [-] equalunique|7 years ago|reply
>One of the clearest signals I’ve gotten from users is that Dhall is “the YAML killer”, for the following reasons:
>Dhall solves many of the problems that pervade enterprise YAML configuration, including excessive repetition and templating errors
>Dhall still provides many of the good parts of YAML, such as multi-line strings and comments, except with a sane standard
>Dhall can be converted to YAML using a tiny statically linked executable, which provides a smooth migration path for “brownfield” deployment.
Source: http://www.haskellforall.com/2019/01/dhall-year-in-review-20...
[+] [-] rhacker|7 years ago|reply
We have a couple ways that we template this out, but mostly we literally just do this in bash:
(Where CI_COMMIT_SHA comes from gitlab) ENV comes from our gitlab CI file.That all being said, the extent of our k8s integration is lots of stuff like that. We could write a JS file that creates a JSON k8s template, but honestly, that would be more work and more learning than we had to for what we're doing. Why would we do more just because we want to avoid templating in a YAML file?
[+] [-] CryoLogic|7 years ago|reply
That means you write it, send it, store it, operate on it, etc. with little or no modification.
The author says "converting between the two is trivial" which may be true, but the developer overhead is less trivial. And it will always be JSON in the client - JS doesn't support YAML objects.
[+] [-] the8472|7 years ago|reply
[+] [-] fasteo|7 years ago|reply
We have done the classical memcached+database custom solution, but I was wondering if there is any accepted library/tool to change application run-time behavior. We have tried consul KV store [1], but does not quite fit in our environment.
My ideal solution would be a webapp with some text editor (think codemirror). Changes in this text file would push the configuration data to a running application.
[1] https://learn.hashicorp.com/consul/getting-started/kv
[+] [-] nickjj|7 years ago|reply
It makes me throw up a little in my mouth every time I see hundreds of lines of YAML to configure something like Traefik with Kubernetes. The worst is when people say they prefer that because "I don't have to write a config file for my backend". That's true but instead now you have extremely verbose configuration mixed in with other verbose configuration.
But in YAML's defense I think it's more of a problem with the tools that use it more so than YAML itself. Ansible is a great example of how amazing YAML can be to manage complex configuration in a concise way.
[+] [-] peterwwillis|7 years ago|reply
Your data does not need to be "expressive", it just needs to provide input to a program. If your data files need to be complex, you need a program to generate them for you.
I've danced the dance of ini -> json -> yaml -> weird hybrid -> embedded logic, and it ends with "program that asks for what the thing you want looks like and generates data files". Industrial software design figured this out ages ago.
[+] [-] desc|7 years ago|reply
* all configuration, without exception, is XML.
* all configuration may be generated from any other format imaginable, but it's sure as fuck going into the Big Main Godlike Application as XML.
Separation, interfaces, etc. Disclaimer: I work in .NET almost exclusively. The .NET configuration APIs generally work, as long as you only ever use them for reading; treating config as something the application itself can fiddle with is a fast route to madness.
[+] [-] louiskottmann|7 years ago|reply
I find it pushes me to write plain YAML files for variables and defaults (Ansible), while allowing strong templating of generated files (Jinja) and letting the result be readable (YAML). By readable I mean minimum programming bloat (code spread on many lines just to write a for loop) and minimum extra syntax that clutters the screen (brackets and quotes). It also lets me write very little custom code (aside from variables, obviously).
If I had to use a more "powerful" YAML-like replacement, it would mix all these into files, written differently by people with different styles, and it would have bloat all over the place.
The main issue I have with helm is that values.yml is not templatable by default so you have to generate it if you want reusability.
YAML does one thing and does it well, it's readable and bloat-free. Maybe we need more tools like "kubectl explain" to know the syntax though.