Using the sqlite db as the canonical configuration is a nice idea, but reminds me of the fun of bind, editing one file to re-generate another. It is as if he is ignoring the advice in his older post about making software usable.
Instead of having a simple text format that is the configuration, he encourages everyone to write their own. I can imagine the fun now on a mailing list - debugging n+1 languages to start a webserver.
When you deploy, do you deploy the sqlite database file or the script that generates it? Does the daemon check if the db is out of date on startup? Does the script clear the existing db when it runs? Does it have merge logic? How do you keep them in sync?
'But it's powerful' I hear you cry, but my retort is simply 'it is effort to maintain, deploy and configure'. It smacks of: here is a bus port, go build yourself a keyboard. This sort of toolsmithery is what brings us delights like autoconf.
A simple change would improve things a lot, whilst maintaining the flexibility he craves:
instead of 'generate a config file' with your tool and hand it off to mongrel, the tool you write is part of mongrel.
You write a script called /etc/mongrel/makeconfig, that takes one argument, the name of the mongrel settings database. When mongrel starts, it invokes makeconfig to build the database.
You can provide a bunch of trivial makeconfig scripts as defaults, including a shell script that just runs a bunch of sqlite statements.
> Instead of having a simple text format that is the configuration, he encourages everyone to write their own.
Uh no, you obviously didn't read the article where at the end I show you the default format we provide to you in m2sh. I think before you go waxing poetical about something you should learn more about it.
And it's not powerful, it's simple. Take a look at the code that loads the configuration in mongrel2 from sqlite, and compare that to similar C code (also written by me) in the m2sh source to just parse a config file. Code doesn't lie and in this case, parsing a config is a hell of a lot harder than just loading it out of a sqlite file.
Agreed, the potential issue of having different people having different configuration sounds a bit dangerous. OTOH, most issues you mention are true whether you use a db or a plain configuration file - they are just more explicit with sqlite3 (which acts as the model in a MVC organization of your app settings).
Incidentally, I am thinking quite hard about those issues (configuration representation, live change/querying, versioing) for internal apps, it is not an easy issue once your app has more than a few parameters.
Bad call. The canonical version of the server configuration doesn't play nice with version control, and so staff has to work with a facsimile and hope that nobody makes out-of-process changes.
Because programmers are expensive, but sysadmins are free.
If you're a programmer working for a large organization, despite sitting in an open plan office surrounded by dozens of other programmers, and knowing that there are hundreds in the building, you probably nevertheless believe that yours is the only code running in production, and the entire operations staff exists in order to support it...
All I see are things about religion, dressing like the status quo, and other things that say "old way of doing things". I think you mean to say "official" or something like that. Let's use official.
In this case, the config file you use is the "official" configuration. If you use the m2sh format than that's the default official format. If you put those into version control then you have solved your problem.
But then, I'm sure you have some witty reply to this reply to your canonical one-liner:
"Conforming to orthodox or well-established rules or patterns, as of procedure."
I agree with you - I prefer a human readable/git friendly config file. That's why I invested 10 minutes coming up with this solution and putting it into run_mongrel2.sh.
[edit: earlier version of this post was unnecessarily snarky.]
I don't follow: surely, a sql dump is possible with sqlite, and you could use just that in your VCS ?
As for ini file, the format is supported in the python stdlib, which makes it a natural default if you don't need too much. If .ini is not enough, I think in general you need something with a bit some logic, at which step using the database itself makes sense in mongrel2 case.
Cisco configuration files don't play nice with version control either. Anyone (with access) can log into any Cisco router, make changes and forget to write the running config to the startup config. Even if they do that, they might forget to copy the new startup config to the TFTP server. And even if they do THAT, they might not even bother doing the "svn commit" or "git commit" or "whatevervcweareusing commit". I view the sqlite database as the running configuration of Mongrel2. And from where I sit, it seems vastly simpler to do a diff of a configuration file stored in sqlite than it is to do a diff of a configuration from a Cisco router.
I did opposite in large Windows-based real-time algorithmic system with tens of thousands of parameters. The parameters were stored in MS-SQL database, under complex schema.
For each branch and version, each developer were adding new parameters, so after merges or upgrades it quickly become parameters hell.
I changed the parameters storage implementation to use CSV files, instead of MS-SQL, while external API did't changed.
Now we were able to add .CSV files to the source control and parameters were versioned and automatically merged.
Additional bonus - you can edit .CSV files in Excel ;)
No opinion one way or another about how Mongrel2 handles configuration, but are INI files really "the fad today"? Just strikes me as an odd insult for something so old. (I don't work with Python, so maybe that's the faddish part of all this?)
A technology doesn't have to be recent to still be applicable. INI files have been around a long time but they are a well understood mechanism for storing confuguration settings and have mature, well debugged, components for handling them. Not that I don't think that sqlite isn't mature but it seems like overkill for storing simple configuration settings.
SQLite seems like more of a PITA for some cases, but when using something like chef to configure an instance, SQLite makes more sense than using something like sed to mangle a config file.
Now, yes, you can define a whole config file in a template, but sometimes you need to have a separate recipe that makes a change to an existing config (I'm looking at you engineyard), and for that, we must unfortunately resort to sed.
Exactly. With m2sh the regular small usage case is very easy, and you get tons of capabilities other servers don't have. It's not preventing you from working how you work now, just gives one more step.
But, for everyone else, and the future of operations, this kind of configuration storage is insanely useful. The idea that we could point mongrel2 at a redis store in the future and have the 1000s of servers some companies put out automatically update their configs in realtime is just sexy as hell.
I hate config files, having written a software tool GUI to configure lots of daemons that traditionally use config files.
The steps, in general, were parse-modify-overwrite, a real hassle considering the diversity of grammars each daemon employs - following the make your own parser mantra.
I welcome Zed's initiative of using sqlite, making a configuration GUI tool for it would be a breeze.
I've not found answer to question why mongrel2 doesn't use ini files..
It's not so hard to generate sqlite file but it is even easier to generate ini, yaml, json or xml if you don't like particular ini format. And you can manage these files by git/svn and config loader could show you line with error if loading fails.
A couple people have mentioned this, and really I'll tell you a little secret:
The config loading in Mongrel2 is all abstracted away and could support anything. I haven't done anything explicitly to let you write a "config load module" but it'd be possible. In theory, since the design kept this MVC idea, you could configure out of a config file, redis, couchdb, really anything you need to get your ops mojo working.
I could have easily just went with a config file as the default setup instead of sqlite3, but then where would all the FUD slingers and bikeshedders go to waste their energy? :-)
I haven't seen the Python community complaining about the config methodology. It usually just seems to be the people who can't understand new things that complain about change. As someone who would be managing this in production, I welcome it.
Already get those. There's folks who want to use redis, couchdb, just about anything that can store configurations.
I think the thing most existing sysadmins don't get is that this is written for people who have to manage mongrel2 servers in a modern way. It doesn't prevent you from doing your usual manual ssh work, but it's aimed at the future where people won't be doing that so much.
It's kind of sad, because I want sysadmins to learn to code so bad, and I even wrote a book so they could learn it, and have helped tons of them get better jobs, yet they still resist awesome features like this. They're really just holding themselves back by not realizing automation and code are their future.
[+] [-] tef_|15 years ago|reply
Instead of having a simple text format that is the configuration, he encourages everyone to write their own. I can imagine the fun now on a mailing list - debugging n+1 languages to start a webserver.
When you deploy, do you deploy the sqlite database file or the script that generates it? Does the daemon check if the db is out of date on startup? Does the script clear the existing db when it runs? Does it have merge logic? How do you keep them in sync?
'But it's powerful' I hear you cry, but my retort is simply 'it is effort to maintain, deploy and configure'. It smacks of: here is a bus port, go build yourself a keyboard. This sort of toolsmithery is what brings us delights like autoconf.
A simple change would improve things a lot, whilst maintaining the flexibility he craves:
instead of 'generate a config file' with your tool and hand it off to mongrel, the tool you write is part of mongrel.
You write a script called /etc/mongrel/makeconfig, that takes one argument, the name of the mongrel settings database. When mongrel starts, it invokes makeconfig to build the database.
You can provide a bunch of trivial makeconfig scripts as defaults, including a shell script that just runs a bunch of sqlite statements.
[+] [-] zedshaw|15 years ago|reply
Uh no, you obviously didn't read the article where at the end I show you the default format we provide to you in m2sh. I think before you go waxing poetical about something you should learn more about it.
And it's not powerful, it's simple. Take a look at the code that loads the configuration in mongrel2 from sqlite, and compare that to similar C code (also written by me) in the m2sh source to just parse a config file. Code doesn't lie and in this case, parsing a config is a hell of a lot harder than just loading it out of a sqlite file.
[+] [-] cdavid|15 years ago|reply
Incidentally, I am thinking quite hard about those issues (configuration representation, live change/querying, versioing) for internal apps, it is not an easy issue once your app has more than a few parameters.
[+] [-] tptacek|15 years ago|reply
(INI files though? What year is it?)
[+] [-] gaius|15 years ago|reply
If you're a programmer working for a large organization, despite sitting in an open plan office surrounded by dozens of other programmers, and knowing that there are hundreds in the building, you probably nevertheless believe that yours is the only code running in production, and the entire operations staff exists in order to support it...
[+] [-] zedshaw|15 years ago|reply
http://www.wordnik.com/words/canonical
All I see are things about religion, dressing like the status quo, and other things that say "old way of doing things". I think you mean to say "official" or something like that. Let's use official.
In this case, the config file you use is the "official" configuration. If you use the m2sh format than that's the default official format. If you put those into version control then you have solved your problem.
But then, I'm sure you have some witty reply to this reply to your canonical one-liner:
"Conforming to orthodox or well-established rules or patterns, as of procedure."
[+] [-] yummyfajitas|15 years ago|reply
[edit: earlier version of this post was unnecessarily snarky.]
[+] [-] cdavid|15 years ago|reply
As for ini file, the format is supported in the python stdlib, which makes it a natural default if you don't need too much. If .ini is not enough, I think in general you need something with a bit some logic, at which step using the database itself makes sense in mongrel2 case.
[+] [-] spc476|15 years ago|reply
[+] [-] nivertech|15 years ago|reply
I changed the parameters storage implementation to use CSV files, instead of MS-SQL, while external API did't changed. Now we were able to add .CSV files to the source control and parameters were versioned and automatically merged. Additional bonus - you can edit .CSV files in Excel ;)
[+] [-] zedshaw|15 years ago|reply
[+] [-] telemachos|15 years ago|reply
[+] [-] maximilianburke|15 years ago|reply
[+] [-] tvon|15 years ago|reply
[+] [-] andrewvc|15 years ago|reply
SQLite seems like more of a PITA for some cases, but when using something like chef to configure an instance, SQLite makes more sense than using something like sed to mangle a config file.
Now, yes, you can define a whole config file in a template, but sometimes you need to have a separate recipe that makes a change to an existing config (I'm looking at you engineyard), and for that, we must unfortunately resort to sed.
[+] [-] zedshaw|15 years ago|reply
But, for everyone else, and the future of operations, this kind of configuration storage is insanely useful. The idea that we could point mongrel2 at a redis store in the future and have the 1000s of servers some companies put out automatically update their configs in realtime is just sexy as hell.
[+] [-] clyfe|15 years ago|reply
The steps, in general, were parse-modify-overwrite, a real hassle considering the diversity of grammars each daemon employs - following the make your own parser mantra.
I welcome Zed's initiative of using sqlite, making a configuration GUI tool for it would be a breeze.
[+] [-] thefreshteapot|15 years ago|reply
Yet running a development setup on a VM and having live setups. ( A little bit like that the deviant guys wrote about
http://news.ycombinator.com/item?id=1769637 )
I can see sqlite data being very handy.
You could give each their own respective id and then just pull out all the records for them.
[+] [-] zedshaw|15 years ago|reply
[+] [-] yatsyk|15 years ago|reply
It's not so hard to generate sqlite file but it is even easier to generate ini, yaml, json or xml if you don't like particular ini format. And you can manage these files by git/svn and config loader could show you line with error if loading fails.
[+] [-] zedshaw|15 years ago|reply
[+] [-] nivertech|15 years ago|reply
Why not use some NoSQL distributed key/value store? [1]
With simple map/reduce you will be able to generate INI files or any other format you want ;)
[1] Full disclosure: this is what I doing in my new project
[+] [-] zedshaw|15 years ago|reply
The config loading in Mongrel2 is all abstracted away and could support anything. I haven't done anything explicitly to let you write a "config load module" but it'd be possible. In theory, since the design kept this MVC idea, you could configure out of a config file, redis, couchdb, really anything you need to get your ops mojo working.
I could have easily just went with a config file as the default setup instead of sqlite3, but then where would all the FUD slingers and bikeshedders go to waste their energy? :-)
[+] [-] pablohoffman|15 years ago|reply
[+] [-] danudey|15 years ago|reply
[+] [-] mfukar|15 years ago|reply
Good luck. :-)
[+] [-] zedshaw|15 years ago|reply
I think the thing most existing sysadmins don't get is that this is written for people who have to manage mongrel2 servers in a modern way. It doesn't prevent you from doing your usual manual ssh work, but it's aimed at the future where people won't be doing that so much.
It's kind of sad, because I want sysadmins to learn to code so bad, and I even wrote a book so they could learn it, and have helped tons of them get better jobs, yet they still resist awesome features like this. They're really just holding themselves back by not realizing automation and code are their future.